Operation monitoring support system

Abstract

To efficiently perform site management.SOLUTION: Provided are: a virtual production monitoring model processing part which generates a virtual production monitoring model M1 with a configuration of a site as a virtual three-dimensional image, acquires time-series information of a state of the site, and reflects the information on the virtual production monitoring model M1; and a display processing part which displays the virtual production monitoring model M1, on which the time-series information is reflected, on a VR terminal used by a user. Further, the virtual production monitoring model processing part generates, for the virtual production monitoring model M1 on which the current status of the site is reflected, diving three-dimensional information as a three-dimensional image that makes it appear as if the user were actually present at the site, and the display processing part displays the diving three-dimensional information on the VR terminal used by the user.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates to a technique for a work monitoring support system. [Background technology]

[0002] In industrial production factories, products are manufactured using production methods such as the line system, cell system, and job shop system. One product is produced through multiple processes, and one process consists of one or more tasks. On-site production is based on a production plan that was drawn up in advance, but there is often a discrepancy between the production plan and the actual work results. In particular, for manual work by on-site workers, which has a high degree of discrepancy, it is necessary to accurately collect the actual work hours, which are the actual work results, and take measures such as changing the worker allocation plan according to the progress of production.

[0003] In response to such technology, Patent Document 1 describes a "progress and operation monitoring system for monitoring the progress and operation status at a production site including a plurality of work processes, comprising a production performance management unit for managing production performance data acquired from each of the work processes, a production plan management unit for managing a process-by-process production plan at each of the work processes, and a production monitoring screen for displaying a problem in the work capacity at each of the work processes analyzed based on each of the production performance data managed by the production performance management unit and each of the process-by-process production plans managed by the production plan management unit, and for receiving an instruction to change the allocation of workers to each of the work processes, the production monitoring screen being provided to a user. a production monitoring unit that, when a destination process to which a worker is to be added and an allocation source process that provides a worker to the destination process are selected on the production monitoring screen, extracts candidates for workers that can be assigned to the destination process from the allocation source process and displays them on the setting screen (see claim 1).

[0004] Meanwhile, in recent years, the concept of the "metaverse" has emerged, which is a three-dimensional virtual space constructed inside a computer where many people can participate and communicate or where various content and services are provided. Due to the recent impact of COVID-19, the "metaverse" is expected to be used not only for entertainment such as games, but also in the business field. (For example, Non-Patent Document 1) [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6945369 [Non-patent literature]

[0006] [Non-Patent Document 1] "Overview and Trends of the Metaverse: Towards Use in Business Scenarios", [online], July 1, 2022, Advanced Technology Lab, Japan Research Institute, Ltd., [Retrieved November 3, 2022], Internet<URL: https: / / www.jri.co.jp / MediaLibrary / file / column / opinion / pdf / 13531.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 has the main purpose of managing the progress of production. Therefore, for example, when an inspection defect occurs during production, the relevant parties need to use a dedicated application or service to individually check and analyze various data related to the work process where the inspection defect occurred, and improvements are required. Similarly, in the technology described in Patent Document 1, when a delay occurs in the progress of work, it is necessary to use another application or service to perform a simulation to predict and avoid delays that are likely to increase in the future, and therefore improvements are required in terms of time and effort. In order to avoid this, it is necessary to improve the environment in which progress management, problem occurrence detection, analysis, and resolution can be performed seamlessly.

[0008] Furthermore, in the case of the technology described in Non-Patent Document 1, it is necessary to monitor not only the equipment and robots, but also the workers who perform manual work on-site. One issue that is raised is that the mental burden placed on on-site workers is heavy when their own behavior is constantly or irregularly monitored by cameras currently installed in many factories. Furthermore, when production managers monitor workers in a virtual space, the areas they want to check differ depending on the purpose of monitoring (e.g., work efficiency, quality, safety, etc.), so an appropriate way of looking is required.

[0009] The present invention has been made in view of the above background, and an object of the present invention is to efficiently perform on-site management. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the present invention is characterized in having a first virtual reality generation unit that generates a first virtual reality image in which the configuration of a specified site is a virtual three-dimensional image, a reflection unit that acquires time-series information of the state of the site and reflects it in the first virtual reality image, and a display processing unit that displays the first virtual reality image in which the time-series information is reflected on a terminal used by a user. Other solutions will be described in the embodiments as appropriate. Effect of the Invention

[0011] According to the present invention, on-site management can be carried out efficiently. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram illustrating the configuration of a work monitoring support system according to a first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a system configuration of a work monitoring support system. [Figure 3A] FIG. 2 is a diagram illustrating an example of the configuration of factory information. [Figure 3B] FIG. 13 is a diagram illustrating an example of the configuration of line information. [Figure 4A] FIG. 13 is a diagram showing an example of the configuration of tool and equipment information. [Figure 4B] FIG. 4 is a diagram illustrating an example of the configuration of robot information. [Figure 4C] FIG. 2 is a diagram illustrating an example of the configuration of worker information. [Figure 5A] FIG. 2 is a diagram illustrating an example of a configuration of product information. [Figure 5B] FIG. 4 is a diagram illustrating an example of the configuration of part information. [Figure 5C] FIG. 2 is a diagram showing an example of the configuration of customer information. [Figure 6A] FIG. 4 is a diagram showing an example of the configuration of work information. [Figure 6B] FIG. 4 is a diagram illustrating an example of the configuration of work plan information. [Figure 7A] FIG. 2 is a diagram showing an example of the configuration of on-site information. [Figure 7B] FIG. 4 is a diagram illustrating an example of the configuration of location information. [Figure 7C] FIG. 4 is a diagram illustrating an example of a configuration of sensor information. [Figure 8] FIG. 11 is a diagram illustrating an example of the configuration of work performance information. [Figure 9] FIG. 2 is a diagram illustrating a detailed configuration example of a work plan management unit. [Figure 10] 13 is a flowchart showing a processing procedure for a workflow generating unit to generate a workflow. [Figure 11] FIG. 2 is a diagram illustrating a detailed configuration example of a work monitoring unit. [Figure 12]13 is a flowchart showing a procedure for generating three-dimensional basic information. [Figure 13] FIG. 1 is an image diagram showing three-dimensional basic information. [Figure 14] 13 is a flowchart showing a processing procedure for generating a virtual production monitoring model. [Figure 15] This is an image (part 1) showing a virtual production monitoring model. [Figure 16] 13 is a flowchart showing a procedure for a status information superimposition process for superimposing work status information on a virtual production monitoring model. [Figure 17] This is an image (part 2) showing a virtual production monitoring model. [Figure 18] 13 is a flowchart showing a process procedure for superimposing progress information. [Figure 19] This is an image (part 3) showing a virtual production monitoring model. [Figure 20A] FIG. 13 is a diagram showing an example of plot position management information data. [Figure 20B] FIG. 13 is a diagram (part 1) showing an example of a lot position management setting screen. [Figure 20C] FIG. 13 is a diagram showing an example of a plot position management setting screen (part 2). [Figure 21] FIG. 1 is an image diagram of diving into a virtual production monitoring model. [Figure 22A] FIG. 13 is a diagram showing an example of a diving confirmation screen. [Figure 22B] FIG. 1 is a diagram showing an example of a three-dimensional screen for diving (part 1). [Diagram 23] 13 is a flowchart showing a procedure of a diving process. [Figure 24] FIG. 13 is a diagram illustrating an example of a virtual production monitoring model when a malfunction occurs. [Figure 25A] 13 is a diagram showing an example of a selection screen displayed on the VR terminal when a two-dimensional barcode is read by the VR terminal. FIG. [Figure 25B] FIG. 2 is a diagram showing an example of a three-dimensional screen for diving (part 2). [Figure 26]13 is a flowchart showing a procedure for a progress support measure process. [Figure 27] FIG. 13 is a diagram illustrating an example of a virtual production monitoring model in which countermeasure proposal information is displayed. [Figure 28] FIG. 1 shows an example of a virtual production monitoring model in which specific countermeasure proposals are displayed. [Figure 29] FIG. 2 shows an example of a virtual production monitoring model in which specific countermeasure proposals are displayed. [Diagram 30] FIG. 13 is a diagram illustrating an example of a virtual production monitoring model during the execution of countermeasures. [Diagram 31] FIG. 13 is a diagram illustrating an example of an updated virtual production monitoring model after countermeasures have been completed. [Figure 32A] FIG. 1 is a diagram showing an example of the installation of a movable three-dimensional camera. [Figure 32B] This is a diagram showing an example of an AGV or a robot equipped with a 3D camera. [Figure 32C] FIG. 1 is a diagram showing an example of a drone equipped with a three-dimensional camera at its bottom. [Diagram 33] FIG. 11 is a diagram illustrating an example of display level setting information. [Figure 34A] This is a diagram (part 1) showing the actual image. [Figure 34B] FIG. 3 is a diagram showing an example of a three-dimensional screen for diving (part 3). [Figure 35A] This is a diagram (part 2) showing the actual image. [Figure 35B] FIG. 4 is a diagram showing an example of a three-dimensional screen for diving (part 4). [Diagram 36] FIG. 2 is a diagram showing the hardware configuration of a computer used in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Next, a mode for carrying out the present invention (referred to as an "embodiment") will be described in detail with reference to the drawings as appropriate.

[0014] First Embodiment [Schematic diagram] FIG. 1 is a schematic diagram of the configuration of a work monitoring support system Z according to the first embodiment. The first factory F1, the second factory F2, and the third factory F3 are sites (production sites; predetermined sites) where products are manufactured through stages, and are managed by the work monitoring support system Z shown in FIG. 2. The first factory F1, the second factory F2, and the third factory F3 are, for example, an assembly factory, an inspection factory where an inspection process is performed, and a parts factory where work such as production, preparation, and inspection is performed, respectively. Hereinafter, the first factory F1, the second factory F2, and the third factory F3 are collectively referred to as factories F as appropriate. A field worker P who works in these factories F can access the work monitoring support system Z as a user U via the Internet N. In addition, each factory F is not limited to an assembly factory, an inspection factory, or a parts factory. Furthermore, each factory F may be a different building in the same base or site, or may be located at a different base in Japan or overseas. In addition, in the example shown in this embodiment, three factories F are targeted, but the number is not limited to three factories F.

[0015] In addition, users U at locations other than the factory F can also access the work monitoring support system Z via the Internet N. Possible users U at locations other than the factory F include a user U1 in the business management department who is in an office, and a user U2 in the design department who works from home. Furthermore, users U involved in product manufacturing can also access the work monitoring support system Z via the Internet N. Possible users U involved in product manufacturing include, for example, a user U3 who is a customer who purchases products, and a user U4 who is a supplier who provides tools, equipment, and parts.

[0016] Users U1 to U4 who are remote from the factory F can view the work progress status of the work monitoring support system Z via a PC (Personal Computer) or a tablet terminal (for example, the user terminal UT shown in FIG. 11), and can also view a virtual reality related to the inside of the factory F via a VR terminal CR and the Internet N. The VR terminal CR is a terminal worn (used) by the user U, such as a head-mounted display or a smartphone attached to VR goggles.

[0017] [system configuration diagram] Fig. 2 is a diagram showing the system configuration of the work monitoring support system Z. Fig. 1 will be referred to as appropriate. The work monitoring support system Z constructed in the first factory F1 (factory F performing an assembly process in the example shown in FIG. 2) has a common database 1, a work monitoring unit 2, a work plan management unit 3, a work performance management unit 4, and a production analysis and simulator unit 5. The work monitoring support system Z also has a site management unit 6, a site control unit 7, and a site data acquisition unit 8. Furthermore, the work monitoring support system Z has an AGV (Automatic Guided Vehicle) 91, tools and equipment 92, a robot 93, and a sensor 94. The common database 1 is connected to the work plan management unit 3, the work monitoring unit 2, etc., as shown in FIGS. 9 and 11 described later.

[0018] The work monitoring unit 2 has a feature of this embodiment, and collectively manages the progress and operation status of each site in one or more factories F. The work monitoring unit 2 then provides an environment in which the user U can check these statuses in a virtual space. The work monitoring unit 2 is a first virtual reality generation unit, a reflection unit, a display processing unit, and a second virtual reality generation unit.

[0019] The work plan management unit 3 manages production plans for the site. That is, it manages the production plan for the entire site, the production plan for each task, and the production plan for each product.

[0020] The work performance management unit 4 manages the production performance against the production plan of the site. That is, based on the information acquired from the site management unit 6, the work performance management unit 4 manages the production performance for each work and each product.

[0021] The production analysis and simulator part 5, which is the countermeasure proposal generation part, analyzes the progress and operation status using the production plan and production results of the site, and analyzes whether there are any problems such as delays or quality defects. It also has a function to generate countermeasure proposals for problems and to simulate how much the efficiency will change depending on the countermeasure proposals.

[0022] The site management unit 6, which also serves as a storage unit, manages the site.

[0023] The site control unit 7 displays and instructs the site AGV 91, tools and equipment 92, robot 93, and site worker P on the details of the work to be performed according to the production plan. If an omission or mistake in a procedure occurs during execution, the site control unit 7 notifies the fact and performs control such as stopping the operation → rechecking the procedure → re-executing.

[0024] The on-site data acquisition unit 8 acquires on-site data (not shown) relating to the work performed by the AGV 91, tools and equipment 92, robot 93, and on-site worker P at the site, and organizes and stores the data as time-series data. The on-site data includes equipment data and sensor data. The equipment data is data acquired by the on-site data acquisition unit 8 by directly communicating with the AGV 91, tools and equipment 92, and robot 93. The sensor data is data acquired by the on-site data acquisition unit 8 from the sensors 94 mounted on the AGV 91, tools and equipment 92, and robot 93, the sensors 94 worn by the on-site worker P, the environmental sensors 94e installed at the site, and the like.

[0025] AGV91 is a mobile robot that loads materials, workpieces, parts, tools, etc. required on-site and transports them to a designated location. AGV91 includes an automated guided forklift (AGF) and an AI (Artificial Intelligence) transport robot.

[0026] The tools and equipment 92 are tools and equipment required at the site. The tools include digital tools that can digitally set and acquire measurement values ​​and settings. In addition, the installation locations of the tools and equipment 92 can be managed using RFID (Radio Frequency Identification), barcodes, beacons, etc.

[0027] The robot 93 is an industrial robot or a human-collaboration robot used on-site, and does not include the AGV 91 described above. The site worker P is a worker who performs manual work such as assembly work and inspection work on site.

[0028] The sensors 94 are installed at the site to sense the site conditions. The sensors 94 include on-board sensors 94a that are built into or externally mounted on the AGV 91, the tools and equipment 92, and the robot 93. The on-board sensors 94a include pressure sensors, vibration sensors, current and voltage sensors, illuminance sensors, GPS (Global Positioning System), fingerprint sensors, and the like.

[0029] The sensor 94 may include a clothing-type sensor 94b or a glove-type sensor 94c worn by the field worker P. The sensor 94 may include a three-dimensional camera 94d and an environmental sensor 94e installed at the site of the factory F. The three-dimensional camera 94d is a three-dimensional information acquisition unit. The sensor 94 may also include a 2D camera, an infrared camera, an IMU (Inertial Measurement Unit), a laser distance measuring device, and the like, which are not shown in FIG. 2. The sensor 94 may also include a wearable camera worn by the field worker P, a blood pressure sensor, and the like, which are not shown in FIG. 2. The sensor 94 may also include a temperature and humidity sensor, an illuminance sensor, and the like, which are installed at the site and are mainly used to measure the site environment, which are not shown in FIG. 2. These sensors 94 are intended to detect changes in the site environment and the work content performed by the tools / equipment 92, the robot 93, and the field worker P.

[0030] In addition, each of the sensors 94 includes an existing sensor 94 and a newly installed sensor 94 for the work monitoring support system Z.

[0031] The common database 1 stores data that is used in common in each configuration of the work monitoring support system Z.

[0032] The work monitoring support system Z constructed in the second factory F2 (in the example of FIG. 2, an inspection factory that performs the inspection process) and the third factory F3 (in the example of FIG. 2, a parts factory that produces, prepares, and inspects parts) has a site management unit 6, a site control unit 7, a site data acquisition unit 8, an AGV 91, tools and equipment 92, a robot 93, and a sensor 94. A site worker P is also placed in the second factory F2 and the third factory F3. The second factory F2 and the third factory F3 are also equipped with a site management unit 6, a site control unit 7, a site data acquisition unit 8, an AGV 91, tools and equipment 92, a robot 93, a site worker P, and a sensor 94. These have the same functions as those installed in the first factory F1, so their explanations are omitted.

[0033] In addition, the site management units 6 of the second factory F2 and the third factory F3 can communicate with the work plan management unit 3 and the work performance management unit 4 installed in the first factory F1 using the Internet N shown in FIG. 1 or a dedicated network not shown.

[0034] In FIG. 2, the common database 1, the work monitoring unit 2, the work plan management unit 3, the work performance management unit 4, etc. installed in the first factory F1 may be constructed in a cloud on the Internet N (see FIG. 1). In addition, the common database 1, the work monitoring unit 2, the work plan management unit 3, and the work performance management unit 4 installed in the first factory F1 do not have to be centralized and installed in one place. Furthermore, the site management unit 6 installed in any of the first factory F1, the second factory F2, and the third factory F3 may be a master and manage all site information 610 (see FIG. 7A). The site information 610 will be described later, and stores information related to the site, such as the factory F and the line.

[0035] 2, the work monitoring unit 2, the work plan management unit 3, the work record management unit 4, the production analysis and simulator unit 5, and the common database 1 are installed in an assembly plant, but this is not limiting. For example, the work monitoring unit 2, the work plan management unit 3, the work record management unit 4, the production analysis and simulator unit 5, and the common database 1 may be installed in a plant other than an assembly plant, or may be installed in a head office or the like (not shown).

[0036] 2, it is assumed that the work monitoring unit 2, the work plan management unit 3, the work performance management unit 4, the production analysis and simulator unit 5, and the common database 1 are each an independent computer. However, this is not limiting, and any two or more of these units (the common database 1 to the production analysis and simulator unit 5) may be collectively installed as one server.

[0037] Next, information used in this embodiment will be described with reference to Fig. 3A to Fig. 8. Fig. 2 will be referred to as appropriate in Fig. 3A to Fig. 8. Note that the user can refer to and edit the information shown in Fig. 3A to Fig. 8 as necessary.

[0038] [Information managed in common database 1] 3A to 5C, the factory information 110, line information 120, tool and equipment information 130, robot information 140, worker information 150, product information 160, part information 170, and customer information 180 managed in the common database 1 will be described. FIG. 2 will also be referenced as appropriate.

[0039] (Factory Information 110) FIG. 3A is a diagram showing an example of the configuration of the factory information 110. As shown in FIG. The factory information 110 is composed of a factory registration number 111, factory identification information 112, factory name 113, location 114, three-dimensional information 115, and line identification information 116. The factory registration number 111 is a number uniquely assigned to each piece of information (record) registered in the factory information 110. The factory identification information 112 is identification information that uniquely identifies a factory F (i.e., a first factory F1, a second factory F2, and a third factory F3) that has a site that is the object of management of the work monitoring support system Z. In the example shown in Fig. 3A, "F1" indicates the first factory F1, "F2" indicates the second factory F2, and "F3" indicates the third factory F3.

[0040] The factory name 113 is the name of the factory F indicated by the factory identification information 112 . The location 114 is the address of the factory F indicated by the factory identification information 112 . The three-dimensional information 115 indicates a data file related to three-dimensional information of the factory F indicated by the factory identification information 302. The information stored in the three-dimensional information 115 is information created using 3D-CAD (for example, BIM (Building Information Modeling) information), but may also be information created based on images of the inside of the factory F actually captured by a three-dimensional camera 94d or the like.

[0041] The line identification information 116 is information indicating a line associated with the factory F indicated by the factory identification information 112. The identification information stored in the line identification information 116 is information that links with the line identification information 122 stored in the line information 120 shown in Fig. 3B. A plurality of pieces of identification information may be stored in the field of the line identification information 116. Note that, instead of the line identification information 116, pointer information for linking with the line information 120 may be stored.

[0042] (Line info 120) FIG. 3B is a diagram showing an example of the configuration of the line information 120. As shown in FIG. The line information 120 is made up of a line registration number 121 , line identification information 122 , a line name 123 , a valid / invalid flag 124 , detailed information 125 , and three-dimensional information 126 . The line registration number 121 is a number uniquely assigned to information (record) registered in the line information 120. The line identification information 122 is identification information that uniquely identifies a line at the site. The line name 123 is the name of the line indicated by the line identification information 122 . The valid / invalid flag 124 is information indicating whether the line indicated by the line identification information 122 is currently valid or invalid. A line being valid means that the corresponding line is in operation, and a line being invalid means that the corresponding line is not in operation.

[0043] The detailed information 125 is information related to an explanation of the line indicated by the line identification information 122 (for example, the name of the work, the name of the product to be produced, etc.) and an update history of the line. The three-dimensional information 126 indicates a data file related to three-dimensional information of the line indicated by the line identification information 122. The information stored in the three-dimensional information 126 is information created by 3D-CAD, but may be information created based on an image captured by the three-dimensional camera 94d or the like.

[0044] As described above, the line information 120 is associated (linked) with the factory information 110 by the line identification information 122 via the column of the line identification information 116 of the factory information 110 shown in FIG. 3A.

[0045] (Tools and equipment information 130) FIG. 4A is a diagram showing an example of the configuration of the tool and equipment information 130. As shown in FIG. The tool / equipment information 130 is composed of a tool / equipment registration number 131 , tool / equipment identification information 132 , factory identification information 133 , tool / equipment name 134 , detailed information 135 , and three-dimensional information 136 . The tool / equipment registration number 131 is a number uniquely assigned to information (record) registered in the tool / equipment information 130 . The tool / equipment identification information 132 is identification information that uniquely identifies the tool / equipment 92 used at the site. The factory identification information 133 is the factory identification information of the factory F that has the tool / equipment 92 indicated by the tool / equipment identification information 132. The identification information stored in the factory identification information 133 is linked to the factory identification information 112 of the factory information 110 shown in Fig. 3A. Furthermore, a plurality of pieces of identification information may be stored in the field of the factory identification information 133. The tool / equipment name 134 is the model name or name of the tool / equipment 92 . The detailed information 135 includes an explanation of the tool / equipment 92 indicated by the tool / equipment identification information 132. The explanation of the tool / equipment 92 includes, for example, the name of the work to be performed, specifications, maintenance information, manual information, and the like. The three-dimensional information 136 is a data file related to three-dimensional information of the tool / equipment 92 indicated by the tool / equipment identification information 132. The information stored in the three-dimensional information 136 is information created by 3D-CAD, but may also be information created based on an image captured by a three-dimensional camera 94d or the like.

[0046] (Robot Information 140) FIG. 4B is a diagram showing an example of the configuration of the robot information 140. As shown in FIG. The robot information 140 is made up of a robot registration number 141 , robot identification information 142 , factory identification information 143 , robot name 144 , detailed information 145 , and three-dimensional information 146 . The robot registration number 141 is a number uniquely assigned to information (record) registered in the robot information 140. The robot identification information 142 is identification information that uniquely identifies the robot 93 used at the site. The factory identification information 143 is identification information of the factory F in which the robot 93 identified by the robot identification information 142 is located. The factory identification information 143 is linked to the factory identification information 112 of the factory information 110 illustrated in Fig. 3A. Note that the factory identification information 143 may store multiple pieces of identification information. The robot name 144 is the model name or name of the robot 93 indicated by the robot identification information 142 . The detailed information 145 is information including a description of the robot 93 indicated by the robot identification information 142. The description of the robot 93 is, for example, the name of the work in which the robot 93 is used, specifications, maintenance information, manual information, and the like. The three-dimensional information 146 is a data file related to three-dimensional information of the robot 93 indicated by the robot identification information 142. The information stored in the three-dimensional information 146 is information created by 3D-CAD, but may be information created based on an image captured by the three-dimensional camera 94d or the like.

[0047] (Worker information 150) FIG. 4C is a diagram showing an example of the configuration of the worker information 150. The worker information 150 includes a worker registration number 151 , worker identification information 152 , a name 153 , an affiliation 154 , years of experience 155 , height 156 , age 157 , and past performance 158 . The worker registration number 151 is a number uniquely assigned to information (record) registered in the worker information 150. The worker identification information 152 is identification information that uniquely identifies a field worker P who is involved in work at the site. Name 153, department 154, years of experience 155, height 156, and age 157 store the name, department, years of experience engaged in the work, height, and age of the field worker P indicated by the worker identification information 152. Past performance 158 stores information on products, work, skills, etc. that the worker has been in charge of in the past. The number of times each work has been performed may also be stored in the worker information 150.

[0048] (Product Information 160) FIG. 5A is a diagram showing an example of the configuration of the product information 160. As shown in FIG. In the product information 160, information on the products produced in the factory F is stored. The product information 160 is made up of a product registration number 161 , product identification information 162 , product name 163 , three-dimensional information 164 , part identification information 165 , customer identification information 166 , and work identification information 167 . The product registration number 161 is a number uniquely assigned to information (record) registered in the product information 160. The product identification information 162 is identification information that uniquely identifies a product produced at the site. The product name 163 is the name of the product indicated by the product identification information 162 . The three-dimensional information 164 indicates a data file related to three-dimensional information of the product indicated by the product identification information 162. The information stored in the three-dimensional information 164 is information created by 3D-CAD, but may also be information created based on an image captured by the three-dimensional camera 94d or the like. Part identification information 165 indicates information on parts necessary to produce the product indicated by product identification information 162. Furthermore, the identification information stored in part identification information 165 is information that links with identification information stored in part identification information 172 of part information 170, which will be described later in FIG. 5B. A plurality of pieces of identification information may be stored in the field of part identification information 165. Furthermore, instead of part identification information 165, a pointer for linking to part information 170 may be stored.

[0049] The customer identification information 166 indicates information about a customer who provides the product indicated by the product identification information 162. The identification information stored in the customer identification information 166 is linked to the identification information stored in the customer identification information 182 of the customer information 180 described later in Fig. 5C. Also, instead of the customer identification information 166, a pointer for linking to the customer information 180 may be set. Work identification information 167 indicates information on work required to produce the product indicated by product identification information 162. The identification information stored in work identification information 167 is information that links to work identification information 312 of work information 310, which will be described later in Fig. 6A. Also, instead of work identification information 167, a pointer for linking to work information 310 may be set.

[0050] (Parts information 170) FIG. 5B is a diagram showing an example of the configuration of the part information 170. As shown in FIG. The part information 170 is made up of a part registration number 171 , part identification information 172 , supplier identification information 173 , delivery conditions 174 , part quantity 175 , and three-dimensional information 176 . The part registration number 171 is a number uniquely assigned to information (record) registered in the part information 170. The part identification information 172 is identification information that uniquely identifies a part required to manufacture a product. The supplier identification information 173 stores identification information related to a manufacturer (supplier) that supplies the part indicated by the part identification information 172. The identification information stored in the supplier identification information 173 is linked to supplier information (not shown). The supplier information stores, for example, the manufacturer's name, the manufacturer's location, contact information, contract information, and the like.

[0051] The delivery conditions 174 store conditions required for receiving parts from the supplier identified by the supplier identification information 173. Conditions stored in the delivery conditions 174 include, for example, information on the accuracy and delivery deadline of the parts, packaging and transportation methods, and legal regulations. The number of parts required to produce a product is stored in the part quantity 175. The quantity stored in the part quantity 175 may be the quantity per product, or may be the quantity in a specific unit. The three-dimensional information 176 is a data file related to three-dimensional information of a part. The information stored in the three-dimensional information 176 is information created by 3D-CAD, but is not limited to information created by CAD, and may be information created based on an image captured by the three-dimensional camera 94d or the like.

[0052] (Customer information 180) FIG. 5C is a diagram showing an example of the configuration of customer information 180. The customer information 180 is composed of a customer registration number 181 , customer identification information 182 , customer name 183 , and company information 184 . The customer registration number 181 is a number uniquely assigned to information (record) registered in the customer information 180. The customer identification information 182 is identification information that uniquely identifies the customer who provides the manufactured product. The customer name 183 is the personal name or company name of the customer indicated by the customer identification information 182 . When the customer is a company, information about the company is stored in the company information 184. Information stored in the company information 184 is, for example, information about the size of the company, contract information, location, and the like.

[0053] Next, examples of data of the work information 310 and the work plan information 320 that store information related to the production plan managed by the work plan management unit 3 will be described with reference to FIGS. 6A and 6B.

[0054] (Work information 310) FIG. 6A is a diagram showing an example of the configuration of the work information 310. As shown in FIG. The work information 310 is made up of a work registration number 311 , work identification information 312 , work name 313 , line identification information 314 , previous work identification information 315 , next work identification information 316 , and work plan identification information 317 . The work registration number 311 is a number uniquely assigned to information (record) registered in the work information 310. The work identification information 312 is identification information that uniquely identifies a work in the production of a product. The task name 313 is the name of the task indicated by the task identification information 312 . The line identification information 314 is identification information of the line on which the production work indicated by the work identification information 312 is carried out, and is information that links with the line identification information 122 of the line information 120 shown in FIG. 3B. The previous task identification information 315 stores identification information of a task to be performed immediately before the task indicated by the task identification information 312. If there is no task to be performed immediately before the task indicated by the task identification information 312, for example, "-" is set in the previous task identification information 315 field.

[0055] The subsequent task identification information 316 stores identification information of a task to be performed after the task indicated by the task identification information 312. If there is no task to be performed after the task indicated by the task identification information 312, for example, "-" is set in the subsequent task identification information 316 field. Work plan identification information 317 indicates production plan information related to the work indicated by work identification information 312. The identification information stored in work plan identification information 317 is information that links to identification information stored in work plan identification information 322 of work plan information 320, which will be described later in Fig. 6B. Note that a plurality of pieces of identification information may be stored in the field of work plan identification information 317. Also, instead of work plan identification information 317, a pointer for linking to work plan information 320, which will be described later in Fig. 6B, may be set.

[0056] (Work plan information 320) FIG. 6B is a diagram showing an example of the configuration of the work plan information 320. As shown in FIG. The work plan information 320 is made up of a work plan registration number 321, work plan identification information 322, work identification information 323, work name 324, site identification information 325, and worker identification information 326. Furthermore, the work plan information 320 is made up of part identification information 327, tool / equipment identification information 328, robot identification information 329, sensor identification information 330, product quantity 331, scheduled start date / time 332, scheduled end date / time 333, and three-dimensional link information 334. The work plan registration number 321 is a number uniquely assigned to information (record) registered in the work plan information 320. The work plan identification information 322 is identification information that uniquely identifies a work to be performed. Work identification information 323 is identification information that uniquely identifies the type / procedure of work, and is information that links to work identification information 312 of work information 310 shown in Fig. 6A. For example, information such as "screw up part A with serial number XXX" may be linked to work identification information 323. Note that instead of storing identification information in the work identification information 323 field, "screw up part A", which is a simplification of "screw up part A with serial number XXX", may be stored. Furthermore, multiple pieces of identification information may be stored in the work identification information 323 field.

[0057] The task name 324 stores the name of the task. The site identification information 325 is identification information of a site where the work indicated by the work plan identification information 322 is to be performed. The identification information stored in the site identification information 325 is linked to the identification information stored in the site identification information 612 included in the site information 610 described later in FIG. 7A. The worker identification information 326 stores information about the field worker P who performs the work indicated by the work plan identification information 322. The worker identification information 326 is information that links with the identification information stored in the worker identification information 152 of the worker information 150 shown in FIG. 4C.

[0058] Part identification information 327 stores information on parts used in the work indicated by work plan identification information 322. The identification information stored in part identification information 327 is information linked to the identification information stored in part identification information 172 of part information 170 shown in FIG. The tool / equipment identification information 328 stores information about the tool / equipment 92 used in the work indicated by the work plan identification information 322. The identification information stored in the tool / equipment identification information 328 is information linked to the identification information stored in the tool / equipment identification information 132 of the tool / equipment information 130 shown in FIG. The robot identification information 329 stores information about the robot 93 used in the work indicated by the work plan identification information 322. The identification information stored in the robot identification information 329 is information that links to the identification information stored in the robot identification information 142 of the robot information 140 shown in FIG.

[0059] The sensor identification information 330 stores information about the sensor 94 used in the work indicated by the work plan identification information 322. The identification information stored in the sensor identification information 330 is information that links to the identification information stored in sensor identification information 632 of the sensor information 630 described later in Fig. 7C.

[0060] In each of the fields of worker identification information 326, part identification information 327, tool / equipment identification information 328, robot identification information 329, and sensor identification information 330, a plurality of pieces of identification information may be stored.

[0061] The product quantity 331 is the quantity of a product (or intermediate product) produced in the work indicated by the work plan identification information 322 . The scheduled start date and time 332 and the scheduled end date and time 333 indicate the scheduled start date and time and the scheduled end date and time of the work indicated by the work plan identification information 322. Alternatively, the scheduled start date and time 332 and the scheduled end date and time 333 are set with an estimated time calculated using the years of experience and past performance of the field worker P who will perform the work. The years of experience and past performance of the field worker P are obtained from the years of experience 155 and past performance 158 columns of the worker information 150 shown in FIG. 4C using the identification information stored in the worker identification information 326 column as a key.

[0062] The three-dimensional link information 334 stores work flow information. The work flow information is information that links three-dimensional information 136, 146, 637 (see Figs. 4A, 4B, and 7C) of the tools / equipment 92, robots 93, and sensors 94 to be used in the work indicated by the work plan identification information 322 with three-dimensional information of the site to be described later, or information that combines these three-dimensional information. The work flow information will be described later.

[0063] Next, examples of data of site information 610, position information 620, and sensor information 630 that store information related to the site managed by the site management unit 6 will be described with reference to Figs. 7A to 7C.

[0064] (Scene information 610) FIG. 7A is a diagram showing an example of the configuration of the site information 610. As shown in FIG. The site information 610 is made up of a site registration number 611 , site identification information 612 , site name 613 , factory identification information 614 , line identification information 615 , work identification information 616 , and location identification information 617 . The site registration number 611 is a number uniquely assigned to information (record) registered in the site information 610. The site identification information 612 is identification information that uniquely identifies a site (block or area) where the work indicated by the work identification information 616 is performed. For example, identification information that identifies a specific line (or lines) in the case of a line method, a specific cell (or cells) in the case of a cell method, or a specific job shop unit in the case of a job shop method is stored in the field of the site identification information 612.

[0065] The site name 613 is the name of the site indicated by the site identification information 612 . The factory identification information 614 is identification information of the factory F in which the site indicated by the site identification information 612 is located. The identification information stored in the factory identification information 614 is information that links to the identification information stored in the field of the factory identification information 112 of the factory information 110 shown in FIG. 3A. The line identification information 615 is identification information of a line on which the site indicated by the site identification information 612 exists. The identification information stored in the line identification information 615 is information that links with the identification information stored in the line identification information 122 of the line information 120 shown in FIG. 3B.

[0066] The work identification information 616 indicates the type of work to be performed at the site indicated by the site identification information 612. Furthermore, the identification information stored in the work identification information 616 is information that links to the identification information stored in the work identification information 323 of the work plan information 320 shown in FIG. The location identification information 617 is information indicating where the site indicated by the site identification information 612 is located within the factory F. The identification information stored in the location identification information 617 is information that links to location identification information 622 of the location information 620 described later in Fig. 7B. Instead of the location identification information 617, a pointer for linking to the location information 620 may be set.

[0067] (location information 620) FIG. 7B is a diagram showing an example of the configuration of the location information 620. As shown in FIG. The location information 620 stores information about a location in the work site. The location information 620 is made up of a location registration number 621, location identification information 622, a building 623, a floor number 624, floor three-dimensional information 625, block information 626, tool / equipment identification information 627, robot identification information 628, and sensor identification information 629. The location registration number 621 is a number uniquely assigned to information (record) registered in the location information 620. The location identification information 622 is identification information for uniquely identifying the location information indicated by the location identification information. The building 623 is information for identifying a building in the case where the factory F having the position corresponding to the position identification information 622 has a plurality of buildings. The floor number 624 is information for identifying a floor in the case where the building 713 of the factory F having the position corresponding to the position identification information 622 has a plurality of floors. The floor three-dimensional information 625 indicates a data file related to three-dimensional information of the position indicated by the position identification information 622. The floor three-dimensional information 625 is information created by 3D-CAD, but may be information created based on an image of the floor actually captured by the three-dimensional camera 94d or the like.

[0068] The block information 626 stores information about the block in which the site is located. A block is a division into multiple blocks of the site space corresponding to the position identification information 622. For example, if the space of a floor including the position corresponding to the position identification information 622 is divided into a width of "10", a depth of "6", and a height of "5", the position corresponding to the position identification information 622 is stored in the form of being present in block (2,3,1) (shown as (X,Y,Z) in FIG. 7B). Instead of the unit of a block, information on the position in the actual site space may be used.

[0069] The tool / equipment identification information 627 is identification information of the tool / equipment 92 installed at the position indicated by the position identification information 622. The identification information stored in the tool / equipment identification information 627 is information that links to the identification information stored in the tool / equipment identification information 132 of the tool / equipment information 130 described above in Fig. 4A. Furthermore, a plurality of pieces of identification information may be stored in the field of the tool / equipment identification information 627.

[0070] The robot identification information 718 is the identification information of the robot 93 installed at the position indicated by the position identification information 622. The identification information stored in the robot identification information 718 is information that links to the identification information stored in the robot identification information 142 of the robot information 140 described above in Fig. 4B. Furthermore, multiple pieces of identification information may be stored in the field of the robot identification information 718.

[0071] The sensor identification information 629 is identification information of the sensor 94 installed at the position indicated by the position identification information 622. The identification information stored in the sensor identification information 629 is information that links to the identification information stored in the sensor identification information 632 of the sensor information 630 shown in Fig. 7C. A pointer that links to the sensor information 630 may be set instead of the sensor identification information 629. Furthermore, a plurality of pieces of identification information may be stored in the field of the sensor identification information 629.

[0072] (Sensor information 630) FIG. 7C is a diagram showing an example of the configuration of the sensor information 630. The sensor information 630 includes a sensor registration number 631 , sensor identification information 632 , a sensor name 633 , a model name 634 , a fixed / movable flag 635 , an installation position 636 , and three-dimensional information 637 . The sensor registration number 631 is a number uniquely assigned to information (record) registered in the sensor information 630. The sensor identification information 632 is identification information that uniquely identifies the sensor 94 by the record. The sensor name 633 and the model name 634 are the name, model name, etc. of the sensor 94 indicated by the sensor identification information 632 . The fixed / movable flag 635 is information indicating whether the sensor 94 indicated by the sensor identification information 732 is fixedly installed or movable (movable). A movable sensor 94 is a sensor 94 mounted on a movable robot 93 or AGV 91, or a movable three-dimensional camera 94d.

[0073] The installation position 636 stores information on the installation position of the sensor 94 indicated by the sensor identification information 632 (denoted as (x, y, z) in FIG. 7C). For example, the information stored in the installation position 636 may be information on the position in the block indicated by the block information 626 shown in FIG. 7B or information on the position in the actual site space, which is stored in the installation position 636 field. For the sensors 94 mounted internally or externally on the AGV 91, the tools / equipment 92, and the robot 93 described above, the current positions of these are stored in the installation position 636 field. Also, for the sensors 94 set to "movable" in the fixed / movable flag 635, the current position of the sensor 94 is stored in the sensor position field.

[0074] The three-dimensional information 637 stores a data file related to three-dimensional information of the sensor 94 indicated by the sensor identification information 632. The information stored in the three-dimensional information 637 is information created by 3D-CAD, but may be information created based on an image actually taken of the sensor 94 by a three-dimensional camera 94d or the like.

[0075] (Work performance information 410) FIG. 8 is a diagram showing an example of the configuration of the work performance information 410. As shown in FIG. The work performance information 410 is composed of a work performance registration number 411 , work performance identification information 412 , work plan identification information 413 , site identification information 414 , acquired data 415 , lot / serial number 416 , product quantity 417 , start date / time 418 , and end date / time 419 . The work record registration number 411 is a number uniquely assigned to information (record) registered in the work record information 410. The work result identification information 412 is identification information that uniquely identifies a work that has already been performed. The work plan identification information 413 is identification information for identifying a work plan corresponding to the work result identification information 412, and is information that links with the identification information stored in the work plan identification information 322 of the work plan information 320 described above in FIG. 6B.

[0076] The site identification information 414 is identification information of a site where the work indicated by the work result identification information 412 was performed. The identification information stored in the site identification information 414 is information that links to the site identification information 612 of the site information 610 described above in FIG. 7A. The acquired data 415 stores information acquired when the work indicated by the work result identification information 412 is performed. Specifically, information identifying the on-site worker P who performed the work, the materials used, the tools and equipment 92, the robot 93, and the sensor 94, etc. are stored in the column of the acquired data 415. Alternatively, the file names storing the raw data acquired by the on-site data acquisition unit 8, and the analysis results of the raw data are stored in the column of the acquired data 415. The analysis results include the presence or absence of an abnormality and the cause, the presence or absence of an inspection defect and the cause, the current status, etc.

[0077] The lot serial number 416 is a number that uniquely identifies a completed product (or an intermediate product) as a result of the work indicated by the work result identification information 412 being performed. The product quantity 417 is the quantity of a product (or intermediate product) produced as a result of the work indicated by the work result identification information 412 being performed. The start date and time 418 and the end date and time 419 are the date and time when the work indicated by the work result identification information 412 actually started and ended. If the actual work has not started, the start date and time 418 of the work result information 410 is left blank, and if the actual work has not been completed, the end date and time 419 of the work result information 410 is left blank. By comparing the scheduled start date and time 332 and the scheduled end date and time 333 in the work plan information 320 shown in FIG. 6B with the start date and time 418 and the end date and time 419 of the work result information 410, it is determined whether the work is delayed. The scheduled start date and time 332 and the scheduled end date and time 333 compared with the start date and time 418 and the end date and time 419 of the work result information 410 are the scheduled start date and time 332 and the scheduled end date and time 333 in the record of the work plan information 320 linked to the identification information stored in the work plan identification information 413.

[0078] The work plan information 320 shown in FIG. 6B and the work performance information 410 shown in FIG. 8 constitute time-series information on the site conditions.

[0079] (Work Planning Management Division 3) FIG. 9 is a diagram showing a detailed configuration example of the work plan management unit 3. As shown in FIG. The work plan management unit 3 is made up of a control unit 31, a communication unit 32, a work plan generation unit 33, and a work flow generation unit 34. The work plan management unit 3 also includes an input unit 35a such as a keyboard or a mouse, and a display unit 35b such as a display. The control unit 31 controls the overall operation of the work plan management unit 3 , and transmits and receives information to and from the work monitoring unit 2 and the site management unit 6 via the communication unit 32 . The work plan generating unit 33 stores information in the work information 310 shown in Fig. 6A and the work plan information 320 shown in Fig. 6B based on instructions input by a user U who creates a work plan. The work plan generating unit 33 stores information in the work information 310 and the work plan information 320 by, for example, the following steps (A1) to (A8).

[0080] First, it is assumed that the factory information 110, the line information 120, the tool and equipment information 130, the robot information 140, the worker information 150, and the product information 160 have been created by the user U. Furthermore, it is assumed that the part information 170, the customer information 180, the site information 610, the position information 620, and the sensor information 630 have been created by the user U. (A1) A user U determines the work required for the production of a product. For example, the user U inputs work identification information and a work name via the input unit 35a, and the work plan generation unit 33 acquires the input work identification information and the work name.

[0081] (A2) The line on which the work of (A1) will be performed is determined. This determination is made by the user U himself or by a computer running the work plan management unit 3. At this time, the work plan generation unit 33 generates line identification information 314 corresponding to the work name using the work identification information input in (A1). Also, when a work name is input in (A1), the work plan generation unit 33 generates work identification information using the input work name.

[0082] (A3) Next, the work plan generating unit 33 determines the order in which the work is to be performed. This sets the pre-work and post-work of the work to be processed. Specifically, the user U inputs the work names of the pre-work and post-work of the work to be set via the input unit 35a. The work plan generating unit 33 acquires the work identification information 312 corresponding to the pre-work and post-work input in (A3) from the work information 310. If the work identification information 312 corresponding to the pre-work and post-work does not exist in the work information 310, the work plan generating unit 33 assigns the work identification information to the work names of the pre-work and post-work. Then, the work plan generating unit 33 stores the work input in (A1) and the identification information of the pre-work and post-work input in (A3) in the pre-work identification information 315 and post-work identification information 316 columns of the work information 310.

[0083] (A4) Next, the work plan generating unit 33 determines the name of a specific task included in the task input in (A1). Then, the work plan generating unit 33 generates identification information for the task to be processed. Furthermore, the work plan generating unit 33 stores the determined task name and task identification information in the task identification information 323 and task name 324 columns of the work plan information 320 shown in Fig. 6B. The task name may be input by the user U via the input unit 35a.

[0084] (A5) Then, the work plan generating unit 33 determines the site, the site worker P, the parts, the tools and equipment 92, the robot 93, and the sensor 94 that will actually perform the work determined in (A4). At this time, the tool and equipment information 130 shown in FIG. 4A and the robot information 140 shown in FIG. 4B are displayed on the display unit 35b. Furthermore, the worker information 150 shown in FIG. 4C, the parts information 170 shown in FIG. 5B, the site information 610 shown in FIG. 7A, the position information 620 shown in FIG. 7B, and the sensor information 630 shown in FIG. 7C are displayed on the display unit 35b. Then, the user U selects the site identification information 612, the worker identification information 152, the parts identification information 172, the tools and equipment identification information 132, the robot identification information 142, and the sensor identification information 632 to be used via the input unit 35a. Then, the work plan generating unit 33 stores each piece of selected information in the corresponding column of the work plan information 320 shown in FIG. 6B.

[0085] (A6) Next, the work plan generating unit 33 determines the quantity of the product (or intermediate product) to be produced in the work determined in (A1). Specifically, the user U inputs the quantity of the product (or intermediate product) via the input unit 35a. The work plan generating unit 33 stores the input quantity in the product quantity 331 field of the work plan information 320.

[0086] (A7) Furthermore, the work plan generating unit 33 determines the scheduled start date and time and the scheduled end date and time using a standard time (ST) calculated in advance. The standard time is the scheduled work time, which is known in advance. The user U inputs the scheduled start date and time and the scheduled end date and time based on the standard time via the input unit 35a or via a computer or the like which is the user terminal UT (see FIG. 11). The work plan generating unit 33 stores the input scheduled start date and time and scheduled end date and time in the scheduled start date and time 332 and scheduled end date and time 3333 of the work plan information 320.

[0087] (A8) Then, the work plan generating unit 33 generates identification information for the record of the work plan information 320 into which information has been input in (A3), (A5) to (A7). Then, the work plan generating unit 33 stores the generated identification information in the field of the work plan identification information 322 of the work plan information 320. That is, the work plan generating unit 33 sets the work plan identification information 322 for the record of the work plan information 320 into which information has been set in (A3), (A5) to (A7). Note that the three-dimensional link information 334 of the work plan information 320 is set by the work flow generating unit 34, which will be described later, and is therefore not set at the stage of (A8).

[0088] The processing procedure shown in (A1) to (A8) is an example, and the order does not matter as long as the information required for generating the work information 310 and the work plan information 320 can be set.

[0089] When the work information 310 and work plan information 320 are generated in the above-described procedure, the control unit 31 instructs the workflow generation unit 34 to generate a workflow.

[0090] The workflow generation unit 34 generates workflow information that links the three-dimensional information of the site where each task is performed with the site information, based on the generated task information 310 and task plan information 320. As an example, the workflow generation unit 34 generates workflow information in the procedure shown in Fig. 10. 10 is a flowchart showing the procedure for generating a workflow by the workflow generating unit 34. Note that FIG. 9 will be referred to as appropriate.

[0091] The user U inputs information about the site to be created. For example, the user U inputs the site name and site identification information via the input unit 35a (S001). When the site name is input, the workflow generation unit 34 acquires the site identification information 612 from the site information 610 shown in FIG. 7A. The workflow generating unit 34 acquires the tool / equipment information 130 shown in FIG. 4A and the robot information 140 shown in FIG. 4B, and acquires three-dimensional information 136, 146 related to the tool / equipment 92 and robot 93 to be used (S002). The tool / equipment information 130 and the robot information 140 are generated and registered in advance. Specifically, the workflow generating unit 34 acquires the tool / equipment identification information 328 and the robot identification information 329 from the work plan information 320 shown in FIG. 6B, using the site identification information input or acquired in step S001 as a key. Then, the workflow generating unit 34 acquires the three-dimensional information 136 of the tool / equipment 92 from the tool / equipment information 130 shown in FIG. 4A, using the acquired tool / equipment identification information 328 as a key. Similarly, the workflow generating unit 34 acquires the three-dimensional information 146 of the robot 93 from the robot information 140 shown in FIG. 4B, using the acquired robot identification information 329 as a key.

[0092] Next, the workflow generation unit 34 refers to sensor information 630 shown in Figure 7C and acquires three-dimensional information 637 of the sensor 94 to be used (S003). The sensor information 630 has been generated and registered in advance. Specifically, the workflow generation unit 34 acquires sensor identification information 330 from the work plan information 320 shown in Figure 6B using the site identification information input or acquired in step S001 as a key. Then, the workflow generation unit 34 acquires three-dimensional information 637 of the sensor 94 from the sensor information 630 shown in Figure 7C using the acquired sensor identification information 330 as a key.

[0093] 7B, the workflow generator 34 identifies the space indicated by the block information 716 in the three-dimensional space indicated by the floor three-dimensional information 625 (S004). At this time, the workflow generator 34 acquires the block information 716 from the position information 620 using the site identification information input or acquired in step S001 as a key. The position information 620 has been generated and registered in advance.

[0094] Furthermore, the workflow generator 34 places the three-dimensional information 637 of the sensor 94 acquired in step S003 at the position indicated by the installation position 636 of the sensor information 630 within the space identified in step S004 (S005).

[0095] As in step S005, the workflow generation unit 34 places the three-dimensional information 146 of the robot 93 and the three-dimensional information 136 of the tools and equipment 92 acquired in step S002 within the space identified in step S004 (S006).

[0096] The workflow generating unit 34 stores the workflow information generated in steps S001 to S006 in the common database 1 (S007). The workflow generating unit 34 stores three-dimensional link information 334 for linking to the workflow information generated in steps S001 to S006 in the work plan information 320 shown in FIG. 6B (S008).

[0097] In this way, the workflow information includes three-dimensional information 136 (see FIG. 4A), 146 (see FIG. 4B), and 637 (see FIG. 7C) of the tools / equipment 92, robot 93, and sensor 94 linked to position information. Note that the processing procedure of steps S001 to S008 is an example, and any procedure may be performed as long as the workflow information can be generated. Also, the three-dimensional information 136, 146, and 637 of the tools / equipment 92, robot 93, and sensor 94 do not all have to be included in the workflow information.

[0098] In the process shown in steps S001 to S008, the three-dimensional information 637 of the sensor 94, the three-dimensional information 146 of the robot 93, and the three-dimensional information 136 of the tools / facility 92 are arranged in the space specified in step S004, but this is not limited to the above. For example, the three-dimensional information 637 of the sensor 94, the three-dimensional information 146 of the robot 93, and the three-dimensional information 136 of the tools / facility 92 may be linked to the information of the space specified in step S004.

[0099] (Work monitoring unit 2) FIG. 11 is a diagram showing a detailed configuration example of the work monitoring unit 2. As shown in FIG. The work monitoring unit 2 includes a control unit 21, a user communication unit 22, a system communication unit 23, a virtual production monitoring model processing unit 24, and a status information superimposition unit 25. The work monitoring unit 2 further includes a three-dimensional basic information generation unit 26, a display processing unit 28, and a setting information management unit 27.

[0100] The control unit 21 controls the overall operation of the work monitoring unit 2. The control unit 21 also transmits and receives information to and from the work plan management unit 3, the work performance management unit 4, and one or more site management units 6 provided in each factory F via the system communication unit 23. The control unit 21 also transmits and receives information to and from a user U1 (U) in the business management department who is in the office, a user U2 (U) in the design department who is working from home, and the like via the user communication unit 22. In the example shown in FIG. 11, the control unit 21 communicates with the business management department and the design department, but may also communicate with a user U3 (U) as a customer and a user U4 (U) as a supplier shown in FIG. 1. Each user U can access the work monitoring unit 2 via a user terminal UT. The user terminal UT is a terminal for inputting various information and displaying information generated by the work monitoring unit 2. The user terminal UT is a PC, a tablet terminal, a smartphone, or the like.

[0101] In addition, the user communication unit 22 and the system communication unit 23 can be integrated into one communication unit, but it is preferable to separate them in consideration of security and communication performance. The user communication unit 22 may use the Internet N (see FIG. 1), and the system communication unit 23 may use a dedicated line or an in-house network.

[0102] The three-dimensional basic information generating unit 26, which is a first virtual reality generating unit, generates three-dimensional basic information M0 (see FIG. 13) that is the base of the virtual site. The three-dimensional basic information M0 is three-dimensional information that virtually overlooks the site.

[0103] The virtual production monitoring model processing unit 24, which is the first virtual reality generating unit and the second virtual reality generating unit, arranges three-dimensional information of the tools / facility 92, robots 93, sensors 94, field workers P, etc. to be used in each task in the three-dimensional space indicated by the three-dimensional basic information M0 generated by the three-dimensional basic information generating unit 26. The three-dimensional information of the field workers P may be an avatar, etc., or may be three-dimensional information generated based on information photographed by a three-dimensional camera 94d. In this way, the virtual production monitoring model processing unit 24 generates a virtual production monitoring model M1 (FIG. 15) including the target monitored by the task monitoring support system Z.

[0104] The status information superimposition unit 25, which is a reflection unit, superimposes the progress status and operation status on the virtual production monitoring model M1 generated by the virtual production monitoring model processing unit 24 by using time-series information such as work plan information 320 (see FIG. 6B) and work performance information 410 (see FIG. 8).

[0105] The display processing unit 28 generates display data from the virtual production monitoring model M1 and the virtual production monitoring model M1a (see FIG. 17) on which the construction progress status and operation status are superimposed. The display data is data to be displayed on the VR terminal CR (see FIG. 1) used by the user U1 in the business management department who is in the office, the user U2 in the design department who is working from home, etc., via the user communication unit 22. Then, the display processing unit 28 sends the display data to the PC or tablet terminal (for example, the user terminal UT shown in FIG. 11) used by the user U, or the VR terminal CR. As a result, the virtual production monitoring model M1, etc., and a three-dimensional screen D3 for diving (see FIG. 22B) are displayed on the screen of the PC or tablet terminal (for example, the user terminal UT shown in FIG. 11) used by the user U, or on the VR terminal CR.

[0106] The setting information management unit 27 manages information such as the display method of the display data generated by the display processing unit 28 and setting information of the three-dimensional basic information M0.

[0107] (3D basic information generation processing) Fig. 12 is a flow chart showing the procedure by which the three-dimensional basic information generating unit 26 generates the three-dimensional basic information M0 shown in Fig. 13. Figs. 1 and 2 will be referred to as appropriate. First, the control unit 21 instructs the three-dimensional basic information generating unit 26 to generate three-dimensional basic information M0. Then, the three-dimensional basic information generating unit 26 refers to the factory information 110 shown in FIG. 3A and the line information 120 shown in FIG. 3B. Then, the three-dimensional basic information generating unit 26 acquires the three-dimensional information 115 of the factory information 110 and the three-dimensional information 115 of the line information 120 (S101). Note that before step S101, the user U specifies the factory F for which the three-dimensional basic information M0 is to be generated via the user terminal UT. Next, the three-dimensional basic information generating unit 26 acquires the three-dimensional information 126 of the line from the line information 120 using the line identification information 116 corresponding to the factory identification information 112 of the specified factory F as a key.

[0108] Next, the three-dimensional basic information generation unit 26 deletes three-dimensional information unnecessary for production monitoring from the three-dimensional information 115 of the factory F and the three-dimensional information 126 of the line, as necessary. Then, the three-dimensional basic information generation unit 26 interpolates the space that has become discontinuous due to the deletion (S102). For example, the three-dimensional basic information generation unit 26 deletes three-dimensional information of unused tools and equipment 92, rooms, warehouses, etc. The deletion may be performed by the user U via the user terminal UT, or may be performed by a computer. For example, the computer may determine whether or not deletion is necessary based on attribute information accompanying the three-dimensional information, etc.

[0109] Next, the three-dimensional basic information generating unit 26 judges whether the production of the product spans multiple factories F, multiple buildings, and multiple floors (multiple bases) (S103). Whether the production spans multiple buildings and multiple floors is judged by the user U or the computer running the work monitoring unit 2 based on the three-dimensional basic information generating unit 26, the site information 610 shown in Fig. 7A, the position information 620 shown in Fig. 7B, and the like. If the production of the product does not span multiple bases (S103→NO), the three-dimensional basic information generator 26 advances the process to step S105.

[0110] When the production of the product spans multiple bases (S103→YES), the three-dimensional basic information generation unit 26 generates three-dimensional information of a passageway (virtual passageway L1 (see FIG. 13)) that virtually connects the factory F, the building, and the floor. Then, the three-dimensional basic information generation unit 26 synthesizes the generated three-dimensional information of the virtual passageway L1 with the three-dimensional information generated in step S103 so as to prevent discontinuity in the three-dimensional information (S104).

[0111] Then, the three-dimensional basic information generator 26 stores the three-dimensional information generated in step S105 in the common database 1 as three-dimensional basic information M0 of the site (S105).

[0112] (Basic 3D site information M0) FIG. 13 is an image diagram showing three-dimensional basic information M0 of the site generated by the three-dimensional basic information generation procedure shown in FIG. In the example of three-dimensional basic information M0 shown in FIG. 13, it is assumed that products are produced across two factories F (see FIG. 1) located at different locations. Factory F (see FIG. 1) is shown in FIG. 13 as three-dimensional factory information MF1, MF2 of factory F. Each piece of three-dimensional factory information MF shows work line information L2, which is information about the line. Furthermore, each piece of three-dimensional factory information MF1, MF2 is connected by a virtual passage L1. This makes it possible to generate and display a three-dimensional site that provides an overhead view of a site that is actually located at a different location (factory F in the example of FIG. 13). Such three-dimensional basic information M0 makes it easier for user U to grasp the overall picture of production.

[0113] (Procedure for generating virtual production completion model) FIG. 14 is a flowchart showing a processing procedure in which the virtual production monitoring model processing unit 24 generates the virtual production monitoring model M1 (see FIG. 15) which is the first virtual reality image. First, the control unit 21 instructs the virtual production monitoring model processing unit 24 to generate a virtual production monitoring model M1. Upon receiving the instruction, the virtual production monitoring model processing unit 24 acquires three-dimensional basic information M0 (see FIG. 13) of the site corresponding to the generation target from the common database 1 (S201).

[0114] Moreover, the virtual production monitoring model processing unit 24 refers to the work information 310 shown in FIG. 6A and the work plan information 320 shown in FIG. 6B (S202). Next, the virtual production monitoring model processing unit 24 acquires three-dimensional information 136 (see FIG. 4A), 146 (see FIG. 4B), and 637 (see FIG. 7C) of the tools / equipment 92, robots 93, and sensors 94 used in the factory F indicated by the three-dimensional basic information M0 based on the information referred to in step S202 (S203). Specifically, the virtual production monitoring model processing unit 24 acquires flow information by tracing the three-dimensional link information 334 of the work plan information 320 shown in FIG. 6B.

[0115] Next, the virtual production monitoring model processing unit 24 arranges the three-dimensional information 136 (see FIG. 4A), 146 (see FIG. 4B), and 637 (see FIG. 7C) of the tools / equipment 92, robot 93, and sensor 94 acquired in step S202 in the three-dimensional basic information M0 acquired in step S201 (S204). By the process of FIG. 10 described above, the workflow information linked to the three-dimensional link information 334 of the work plan information 320 shown in FIG. 6B is acquired. In the workflow information, the three-dimensional information 136 (see FIG. 4A), 146 (see FIG. 4B), and 637 (see FIG. 7C) of the tools / equipment 92, robot 93, and sensor 94 are stored in a format linked to position information. Therefore, the virtual production monitoring model processing unit 24 arranges the three-dimensional information 136, 146, and 637 of the tools / equipment 92, robot 93, and sensor 94 according to the position information in the workflow information.

[0116] Next, the virtual production monitoring model processing unit 24 determines a plot position PR (see FIG. 15) where the progress status and operation status can be referred to (S205). The plot position PR is the location of a base in product production, such as a location where parts are carried in or a location where inspections are performed. As will be described later, it is also a location where three-dimensional information for diving, which is a second virtual reality image, can be displayed. The plot position PR is associated with a site managed by the site information 610 shown in FIG. 7A. In other words, the plot position PR is associated with the site identification information 612 in FIG. 7A.

[0117] Specifically, the virtual production monitoring model processing unit 24 refers to the site information 610 in the work plan information 320 acquired in step S202. Then, the virtual production monitoring model processing unit 24 selects all or a part of the site identification information 702 that corresponds to the site in the factory F that is the target of generation of the virtual production monitoring model M1 from the site information 610 as the plot positions PR required for production. The term "corresponding" means that the site identification information 702 corresponds to the site in the factory F that is the target of generation of the virtual production monitoring model M1. When a part of the site identification information 702 is selected, the user U may select the site identification information 702 via the user terminal UT.

[0118] Next, the virtual production monitoring model processing unit 24 specifies the position around the plot position PR determined in step S205 in the three-dimensional basic information M0 generated in step S204. Then, the virtual production monitoring model processing unit 24 cuts out three-dimensional information for the position around the specified plot position PR. Thereafter, the virtual production monitoring model processing unit 24 converts the three-dimensional information around the cut-out plot position PR into three-dimensional information (three-dimensional information for diving) in an MR (Mixed Reality) space (S206). A method for converting three-dimensional information into three-dimensional information for diving is used using a conventional technique or a commercially available product. The block information 626 in the position information 620 shown in FIG. 7B may be used as the area around the plot position PR to be cut out. In addition, not only the area around the plot position PR but also all of the three-dimensional basic information M0 generated in step S204 may be converted into three-dimensional information for diving.

[0119] Thereafter, the virtual production monitoring model processing unit 24 generates a dedicated two-dimensional barcode TB (see FIG. 15) as a marker associated with each plot position PR determined in step S205 (S207). In step S207, a dedicated marker, barcode, etc. may be generated in addition to the two-dimensional barcode TB. Finally, the virtual production monitoring model processing unit 24 stores the three-dimensional basic information M0, the diving three-dimensional information, and the two-dimensional barcode TB in the common database 1 as one virtual production monitoring model M1 (S208).

[0120] (Virtual Production Monitoring Model M1) FIG. 15 is a conceptual diagram showing a virtual production monitoring model M1 of the site generated by the virtual production monitoring model generation procedure. The virtual production monitoring model M1 is displayed by the display processing unit 28 shown in Fig. 11, and similarly the virtual production monitoring model M1 described below is displayed by the display processing unit 28 shown in Fig. 11 on the display screen of a PC or tablet terminal (for example, the user terminal UT shown in Fig. 11) used by the user U, or on a VR terminal CR worn by the user U. In this way, the user U can view the virtual production monitoring model M1 by the PC or tablet terminal (for example, the user terminal UT shown in Fig. 11) used by the user U, or on the VR terminal CR worn by the user U. In the following description, it is assumed that the virtual production monitoring model M1 is displayed on the VR terminal CR.

[0121] In the virtual production monitoring model M1 shown in FIG. 15, three-dimensional images TM of the tools and equipment 92, robot 93, and sensor 94 (refer to FIG. 2, respectively) used are arranged in the three-dimensional basic information M0 shown in FIG. 13, and further plot positions PR are displayed. In the virtual production monitoring model M1 shown in FIG. 15, an avatar AV of a site worker P is displayed. In the example shown in FIG. 15, a three-dimensional image TM of a three-dimensional camera 94d is displayed as the robot 93 and the sensor 94. Furthermore, a two-dimensional barcode TB is displayed so as to correspond to the plot positions PR. However, the two-dimensional barcode TB may be hidden. Note that the two-dimensional barcode TB is a marker that links the virtual production monitoring model M1 with the diving three-dimensional screen D3 (refer to FIG. 22B), which will be described later, and is displayed on the virtual production monitoring model M1 as shown in FIG. 15. Moreover, the avatar AV of the site worker P can be switched on and off. Furthermore, the avatars AV of all the site workers P may be displayed. Alternatively, several (eg, 10) field workers P may be displayed together as one avatar AV.

[0122] The plot positions PR may be color-coded according to the type of work. The virtual production monitoring model M1 displayed on the VR terminal CR or the like may be rotatable within the screen by line-of-sight manipulation or the like.

[0123] In this way, the virtual production monitoring model M1 is a virtual three-dimensional image of the site configuration. In the example shown in Fig. 15, a plot position PR capable of displaying three-dimensional information for diving, which is a second virtual reality image, is displayed in the first virtual reality image. Incidentally, as shown in Fig. 17 and other figures, the plot position PR is also a base for displaying time-series information.

[0124] (Status information overlay processing) Fig. 16 is a flowchart showing the procedure of a status information superimposition process for superimposing status information of an operation on the virtual production monitoring model M1 generated by the process shown in Fig. 14. Note that the process shown in Fig. 16 may be performed for each operation. First, the control unit 21 instructs the status information superimposition unit 25 to superimpose (map) information on the work plan and the progress and operation status onto the virtual production monitoring model M1 (FIG. 15). Upon receiving the instruction, the status information superimposition unit 25 acquires the virtual production monitoring model M1 of the site that is the target for superimposition from the common database 1 (S301).

[0125] Next, the status information superimposing unit 25 refers to the work information 310 shown in Fig. 6A, the work plan information 320 shown in Fig. 6B, and the work performance information 410 shown in Fig. 8 (S302). When the actual work is started, the status information superimposing unit 25 generates the work performance information 410 by the work performance management unit 4 collecting information from the sensor 94, the site information 610 shown in Fig. 7A, etc. based on the work plan information 320. Next, the status information superimposition unit 25 calculates the scheduled time and actual time for each task at the plot position PR set on the virtual production monitoring model M1, using the task performance information 410 (S303). The cumulative scheduled time and actual time are the accumulated scheduled time and actual time from the first task to a certain task.

[0126] Then, the production analysis and simulator unit 5 analyzes the planned time and actual time calculated in step S303. The production analysis and simulator unit 5 further analyzes the information of the acquired data 415 of the work result information 410. Then, the production analysis and simulator unit 5 judges whether or not an abnormality has occurred in the progress status or the operation status (S304). That is, the production analysis and simulator unit 5 compares the work plan information 320 shown in FIG. 6B with the work result information 410 shown in FIG. 8, and judges whether or not an abnormality has occurred in the progress status or the operation status. For example, the production analysis and simulator unit 5 judges that an abnormality has occurred in the progress status or the operation status in the following cases. (C1) The scheduled start date and time 332 of the work plan information 320 has passed, but the start date and time 418 of the work result information 410 is blank. (C2) Although the scheduled end date and time 333 in the work plan information 320 has passed, the end date and time 419 in the work result information 410 is blank.

[0127] Furthermore, the production analysis / simulator unit 5 judges whether or not an abnormality has occurred based on the contents of the acquired data 415 of the work performance information 410.

[0128] Finally, the state information superimposing unit 25 superimposes and displays all or part of the information of the results calculated or analyzed in steps S303 and S304 on the virtual production monitoring model M1 acquired in step S301. When part of the results calculated or analyzed in steps S303 and S304 is displayed, the user U may select information to be displayed on the PC or tablet terminal (for example, the user terminal UT shown in FIG. 11) or the VR terminal CR.

[0129] That is, the production analysis / simulator unit 5 superimposes the scheduled time and actual time calculated in step S303, and the judgment result in step S304 on the plot position PR (see FIG. 15) of the virtual production monitoring model M1 (S305). The superimposition method may be any display method that allows the scheduled time, actual time, and judgment result to be distinguished. For example, the superimposition may be performed by displaying a speech bubble at the plot position PR and describing information in the speech bubble. Alternatively, the superimposition may be performed by displaying a "status button" or a "menu button" at the plot position PR and displaying detailed information by selecting the button. The virtual production monitoring model M1a (see FIG. 17) on which the superimposition is performed in step S305 is stored in the common database 1.

[0130] (Display of plot position information PF) Fig. 17 is an image diagram showing a virtual production monitoring model M1a of the site generated by the processing procedure shown in Fig. 16. Note that in Fig. 17 and subsequent figures, the avatar AV (see Fig. 15) of the site worker P is assumed to be hidden. In the example of the virtual production monitoring model M1a shown in FIG. 17, plot position information PF of the time-series information of the site, such as the scheduled time, actual time, and judgment result, is superimposed in a balloon on the virtual production monitoring model M1 shown in FIG. 15. In order to avoid complication, in FIG. 17, the reference numerals are omitted for the elements already described in FIG. 15, and the two-dimensional barcode TB (see FIG. 15) is not displayed. In this manner, the virtual production monitoring model M1a shown in FIG. 17 is a model in which the time-series information of the state of the site is reflected in the first virtual reality image. That is, the virtual production monitoring model M1a shown in FIG. 17 is a model in which the first virtual reality image reflecting the time-series information is displayed on a terminal (PC, tablet terminal (e.g., user terminal UT shown in FIG. 11), VR terminal CR) used by a user U. In addition, as described above, the following description will be given assuming that the virtual production model M1a is displayed on the VR terminal CR. The same applies to the virtual production models M1b to M1h described later.

[0131] The virtual production monitoring model M1a shown in Fig. 17 allows the user U to visually grasp the overview and overall picture of the production of the product. Furthermore, the user U can check the progress and operation status of each task, analysis results, etc. together with the actual site.

[0132] (Progress information overlay processing) FIG. 18 is a flowchart showing the procedure of the process of superimposing progress information. The control unit 21 designates a specific product to the status information superimposing unit 25 in response to an instruction from the user U via the user terminal UT (S401). The process of step S401 is performed, for example, in a state in which the virtual production monitoring model M1a shown in Fig. 17 is displayed as a result of carrying out the process shown in Fig. 16 described above. If a specific product is not specified (S401→NO), the work monitoring support system Z ends the process.

[0133] If a specific product is specified (S401→NO), the status information superimposition unit 25 acquires workflow information of the specific product specified in step S401 (S402). Specifically, the status information superimposition unit 25 acquires work identification information 167 included in product information 160 shown in Fig. 5A using product identification information 162 of the specified product as a key. Then, the status information superimposition unit 25 acquires a record having work identification information 312 corresponding to the acquired work identification information 167 in work information 310 shown in Fig. 6A.

[0134] Next, the status information superimposing unit 25 acquires a record of the work plan information 320 (see FIG. 6B) and the work result information 410 (see FIG. 8) related to the product (S403). Specifically, the status information superimposing unit 25 acquires the work plan identification information 317 stored in the record acquired in step S402. Furthermore, the status information superimposing unit 25 acquires a record having the work plan identification information 413 corresponding to the work plan identification information 317 acquired from the work plan information 320 in the work result information 410.

[0135] Then, the status information superimposing unit 25 highlights the work line information L2 and plot position PR of the virtual production monitoring model M1a related to the work that has already been completed based on the acquired records of the work performance information 410 (S404). For example, the status information superimposing unit 25 highlights the work by coloring it in red. A work that has already been completed is a work whose date and time information is stored in the end date and time 419 in the work performance information 410.

[0136] Next, the status information superimposing unit 25 superimposes at least the planned completion time, actual completion time, and work result, as well as marks for displaying other detailed information, on the virtual production monitoring model M1a (S405). As a result, the virtual production monitoring model M1b shown in Fig. 19 is displayed on the VR terminal CR (see Fig. 1). The marks are speech bubbles MA1 and MA2 (see Fig. 19), buttons, menus, etc. Furthermore, the status information superimposing unit 25 performs the process of step S405 by referring to the work plan information 320 shown in Fig. 6B, the work performance information 410 shown in Fig. 8, etc.

[0137] Next, the status information superimposition unit 25 identifies the work currently being started (currently being performed), and highlights the work line information L2 and plot position PR of the virtual production monitoring model M1b in a different way from that in step S404 (S406). The different highlighting is, for example, coloring in yellow. The currently started work is work for which date and time information is stored in the start date and time 418 in the work performance information 410, but no date and time information is stored in the end date and time 419.

[0138] Then, the status information superimposing unit 25 superimposes at least a display of the scheduled completion time and a mark for displaying other detailed information on the virtual production monitoring model M1b (S407). The mark is a speech bubble MA3 (see FIG. 19), a button, a menu, etc. The scheduled completion time is the difference between the scheduled end date and time 333 of the work plan information 320 shown in FIG. 6B and the current date and time.

[0139] Next, the status information superimposition unit 25 calculates the scheduled start time and scheduled completion time for each plot position PR for the work to be started for the target product (S408). Then, the virtual production monitoring model processing unit 24 highlights the lines and plot positions PR to be started in the virtual production monitoring model M2b in a manner different from that in steps S404 and S406 (S407). In step S407, the virtual production monitoring model processing unit 24 colors the lines and plot positions PR in green, for example. Note that for the work to be started in the future, marks (such as speech bubbles, buttons, menus, etc.) that display the planned start time and planned completion time may be displayed.

[0140] (Displays work completed / scheduled to start) Fig. 19 is an image diagram showing a virtual production monitoring model M1b displayed as a result of performing the process shown in Fig. 18. In Fig. 19, to avoid complication, reference numerals are omitted for elements that have already been explained in Fig. 15 and Fig. 17. Also, the two-dimensional barcode TB shown in Fig. 15 is not displayed. In the virtual production monitoring model M1b shown in FIG. 19, the work line information L2 where the work has been completed and the virtual passage L1 show solid line work route information L31 which is information (line) of the work that has been performed. In the virtual production monitoring model M1b shown in FIG. 19, the work line information L2 which will be started (scheduled to be started) shows dashed line work route information L32 as information (line) of the work to be performed. In addition, the plot position PR1 shows information of the work currently being started. In the example shown in FIG. 19, the plot position PR1 which is currently being started is shown, and the work route information which is currently being started is not shown. In addition, the work route information which is currently being started may be displayed in a different highlighting from the solid line work route information L31 where the work has been completed and the dashed line work route information L32 which is to be started. Incidentally, currently being started means that the work is currently being performed.

[0141] Speech bubble MA1 displays at least the planned completion time, actual completion time, and work results for work that has already been completed, as well as other detailed information. Speech bubble MA1 is a type of mark displayed in step S405 of FIG. 18. Speech bubble MA2 displays the planned completion time for work currently being started, as well as other detailed information. Speech bubble MA2 is information about work currently being started (currently being carried out). Speech bubble MA3 is a type of mark displayed in step S407 of FIG. 18.

[0142] The work route information L31, L32 and the balloons MA1, MA2 are time-series information, and are information relating to the progress of work being carried out at the site. In this way, the virtual production monitoring model M1b shown in Fig. 19 displays information (time-series information) relating to the progress of work being carried out at the site.

[0143] In this way, in the example shown in Fig. 19, information on work that has already been performed (work route information L31), information on work to be performed (work route information L32), and information on work currently being started (balloon MA2) are displayed in different ways for the virtual production monitoring model M1b. By displaying the information as shown in Fig. 19, the user U (see Fig. 1) can visually check the overall view with on-site information and the progress and operation status of each product superimposed. This allows the user U to check at a glance the process that has been carried out in the past, the current production position, and how production will proceed in the future.

[0144] (Plot location management information 210) Fig. 20A is a diagram showing an example of data of the plot position management information 210 managed by the work monitoring unit 2. Fig. 11 will be referred to as appropriate. The work monitoring unit 2 can change the plot positions PR and the number of plot positions PR set in the virtual production monitoring model M1 shown in Fig. 15 etc. depending on the type of user U who checks the progress information and operation status. For this reason, plot position management information 210 according to the type of user U is managed by the setting information management unit 27. The type of user U refers to the position of the user U, whether the user is a person within the company or outside the company, etc. In other words, the information that can be viewed in the virtual production monitoring model M1 is divided according to the type of user U.

[0145] The plot position management information 210 is composed of a plot position registration number 211 , in-house / out-house information 212 , affiliation 213 , job title 214 , number of plot positions 215 , and plot positions 216 . The plot position registration number 211 is a number uniquely assigned to information (record) registered in the plot position management information 210. The in-company / out-of-company information 212, the department 213, and the position 2144 are information relating to the type of the user U. The number of plot positions 215 and the plot positions 216 may be determined in advance as default values ​​based on information input by the user U via the user terminal UT (see FIG. 11). Alternatively, they may be set by the user U every time the virtual production monitoring model M1 is displayed based on the plot position management setting screens D1A and D1B shown in FIG. 20B and FIG. 20C.

[0146] (Plot position management setting screen) 20B and 20C are diagrams showing examples of plot position management setting screens D1A and D1B. The plot position management setting screen D1A shown in Fig. 20B is a plot position management setting screen for a quality manager. The plot position management setting screen D1A shown in Fig. 20B allows the user U to select a location (plot position PR) that the quality manager wants to check. The selected information is stored in the plot position management information 210 shown in Fig. 20A.

[0147] In contrast, the plot position management setting screen D1B shown in Fig. 20C is a plot position setting screen for user U4 (see Fig. 1) who is a supplier. Since user U4 who is a supplier is an outsider, the plot positions PR (see Fig. 15) that can be viewed are fixed.

[0148] In this way, by dividing the plot positions PR that can be viewed depending on the type of user U, the user U can check only the plot positions PR that he or she needs in the virtual production monitoring model M1 (see FIG. 15). Furthermore, the user U who operates the work monitoring support system Z may set access rights for each user U in consideration of security. In this way, it becomes possible to adjust the scope of information disclosure.

[0149] 20A to 20C, the plot position PR to be displayed changes depending on the type of user U.

[0150] According to the first embodiment, by generating and displaying a virtual production monitoring model M1, which is a virtual realization that links the production progress and operation status with the site, those involved in production can easily check the production progress and operation status. This makes it easier to grasp not only the production status of each process, but also the overall picture and overview of production. And according to the first embodiment, it is possible to accurately check the points that need checking while reducing the burden on the site worker P.

[0151] <Second embodiment> Next, a second embodiment will be described with reference to FIGS. In the first embodiment, a virtual production monitoring model M1 (FIG. 15) is generated and displayed to provide an overview of the site, but in the second embodiment, a technique for diving into the virtual production monitoring model M1 will be described. Diving refers to generating a virtual space in which a user U appears as if he or she were inside the virtual production monitoring model M1.

[0152] (Diving into the virtual production monitoring model M1) FIG. 21 is an image diagram regarding diving into the virtual production monitoring model M1. In Fig. 21, a two-dimensional barcode TB is superimposed on a part of the plot position PR displayed on the virtual production monitoring model M1b shown in Fig. 19. In Fig. 21, in order to avoid complication, explanations of elements that are not necessary for explanation are omitted.

[0153] In the example shown in FIG. 21, each of the multiple users U dives into the virtual production monitoring model M1b from a different plot position PR using the VR terminal CR or a PC or tablet terminal used by the user U (for example, the user terminal UT shown in FIG. 11). In the example shown in FIG. 21, the multiple users U are a user U2 working from home, a user U1 in the business management department, a user U4 who is a supplier, and a user U3 who is a customer. For example, the user U2 working from home is diving at the plot position PR11. The user U1 in the business management department is diving at the plot position PR12. The user U4 who is a supplier is diving at the plot position PR13, and the user U3 who is a customer is diving at the plot position PR14. Note that the VR terminal CR, the PC or tablet terminal used by the user U, etc. can be used for the process of diving. The process for diving includes reading the two-dimensional barcode TB, displaying the diving three-dimensional screen D3 (see FIG. 22), and other processes shown in FIG. 21 to FIG. 31, FIG. 33 to FIG. 35B, etc. Also, the PC, tablet terminal, etc. used by the user U is, for example, the user terminal UT shown in FIG. 11. In the following description, it is assumed that the VR terminal CR is used when the process for diving is performed. Incidentally, when the two-dimensional barcode TB is read by the PC, the two-dimensional barcode TB displayed on the PC is clicked.

[0154] Also, as shown in Fig. 21, a two-dimensional barcode TB is superimposed on each of the plot positions PR11 to PR14. When each user U reads the two-dimensional barcode TB with a VR terminal CR (see Fig. 22A), the diving result (virtual space) is displayed on the screen of the VR terminal CR. Note that, as shown in Figs. 20A to 20C, the plot positions PR where the two-dimensional barcode TB is displayed (where diving is possible) differ depending on the type of user U.

[0155] Furthermore, the two-dimensional barcode TB may be displayed according to the restrictions of the plot position PR shown in FIG. 20A.

[0156] (Diving confirmation screen D2) Fig. 22A is a diagram showing an example of a diving confirmation screen D2, and Fig. 22B is a diagram showing an example of a diving three-dimensional screen D3. First, the user U reads the two-dimensional barcode TB of the virtual production monitoring model M1b as shown in FIG. 21 by the VR terminal CR. Then, options as shown in FIG. 22A are displayed on the VR terminal. If the user U selects "NO", the process ends, and if the user U selects "Go Dive", a diving three-dimensional screen D3, which is a second virtual reality image, as shown in FIG. 22B is displayed on the VR terminal CR. The diving three-dimensional screen D3 displayed is diving three-dimensional information at the plot position PR (see FIG. 21) linked to the read two-dimensional barcode TB (see FIG. 21). It is preferable that the selection on the diving confirmation screen D2 shown in FIG. 22A is performed using eye tracking. In addition, the diving three-dimensional information (second virtual reality image) displayed on the diving three-dimensional screen D3 in FIG. 22B is generated by the process shown in FIG. 23. However, the diving three-dimensional information generated in step S206 in FIG. 14 may be used as the diving three-dimensional information displayed on the diving three-dimensional screen D3 in FIG. 22B. Such a three-dimensional diving screen D3 is displayed on the screen of the VR terminal CR by the display processing unit 28 shown in FIG.

[0157] (Diving Processing) 23 is a flow chart showing the procedure of the diving process, with reference to FIGS. 2 and 21 to 22B as appropriate. First, the user U reads the two-dimensional barcode TB, which is a mark displayed on the virtual production monitoring model M1b, using the VR terminal CR (S501). Then, the diving confirmation screen D2 as shown in FIG. 22A is displayed on the screen of the VR terminal CR, and the user U selects either "Go Dive" or "Go Dive with" (see FIG. 25A), or "NO". This determines whether or not to dive around the plot position PR (S502). If the user U selects "NO" (S502→NO), the work monitoring support system Z ends the process.

[0158] When the user U selects "Go Dive" or "Go Dive with" (S502→YES), the virtual production monitoring model processing unit 24 refers to the work performance information 410 shown in FIG. 8. Then, the virtual production monitoring model processing unit 24 acquires the work currently being performed at the specified plot position PR from the work performance information 410 shown in FIG. 8 (S503). The specified plot position PR is the plot position PR linked to the read two-dimensional barcode TB. Moreover, the currently being performed work is a work for which date and time information is stored in the start date and time 418 in the work performance information 410, but the end date and time 419 is blank.

[0159] Next, the virtual production monitoring model processing unit 24 generates three-dimensional information for diving around the plot position PR (S504). The virtual production monitoring model processing unit 24 acquires corresponding position identification information 617 from the site information 610 shown in Fig. 7A using the site identification information 414 of the work result information 410 as a key. Furthermore, the virtual production monitoring model processing unit 24 acquires floor three-dimensional information 625 stored in the position information 620 shown in Fig. 7B using the acquired position identification information 617 as a key. Then, the virtual production monitoring model processing unit 24 generates three-dimensional information for diving around the plot position PR based on the acquired floor three-dimensional information 625.

[0160] Next, the virtual production monitoring model processing unit 24 superimposes the three-dimensional information acquired from the three-dimensional camera 94d etc. used in the work on the three-dimensional information for diving (S505). The three-dimensional information acquired from the three-dimensional camera 94d etc. is information on the current site condition acquired by the sensor 94.

[0161] Then, the virtual production monitoring model processing unit 24 judges whether or not there is a gap equal to or larger than a threshold value in the superimposed diving three-dimensional information (S506). In step S506, the virtual production monitoring model processing unit 24 judges whether or not there is a gap between the diving three-dimensional information generated from the floor three-dimensional information 625 and the three-dimensional information (actual three-dimensional information) acquired from the sensor 94 in step S505. The floor three-dimensional information 625 is information on the site condition stored in the storage unit.

[0162] If there is no gap (S506→NO), the virtual production monitoring model processing unit 24 advances the process to step S508. If there is a gap (S506→YES), the virtual production monitoring model processing unit 24 judges whether or not to delete the gap portion, and if so, performs spatial interpolation (S507). The virtual production monitoring model processing unit 24 judges whether or not to delete the gap portion based on attribute information of the floor three-dimensional information 625. For example, the virtual production monitoring model processing unit 24 judges to delete the gap portion if it is unnecessary for the production of the product. In addition, spatial interpolation is performed based on the floor three-dimensional information 625. Furthermore, if the gap portion is not deleted, the virtual production monitoring model processing unit 24 generates the diving three-dimensional information by prioritizing the three-dimensional information acquired from the sensor 94. In this process, the changes and the like are reflected. In this way, if there is a gap between the information on the current site state acquired by the three-dimensional camera 94d and the floor three-dimensional information 625, the virtual production monitoring model processing unit 24 reflects the current site information in the diving three-dimensional information.

[0163] Then, the virtual production monitoring model processing unit 24 superimposes the three-dimensional information of the tools and facilities 92, robot 93, field worker P, and sensor 94 used in the work on the diving three-dimensional information (S508). In step S508, the virtual production monitoring model processing unit 24 acquires the site identification information 325 corresponding to the plot position PR being processed in the work performance information 410 shown in FIG. 8. Then, the virtual production monitoring model processing unit 24 acquires the part identification information 327, the tool and facility identification information 328, the robot identification information 329, the worker identification information 326, and the sensor identification information 330 from the work plan information 320 shown in FIG. 6B using the acquired site identification information as a key. Furthermore, the virtual production monitoring model processing unit 24 acquires the three-dimensional information 176 of the part from the part information 170 shown in FIG. 5B using the acquired part identification information 327 as a key. Similarly, the virtual production monitoring model processing unit 24 acquires three-dimensional information 136 of the tool / equipment 92 from the tool / equipment information 130 shown in FIG. 4A using the acquired tool / equipment identification information 328 as a key. The virtual production monitoring model processing unit 24 also acquires three-dimensional information 146 from the robot information 140 shown in FIG. 4B using the acquired robot identification information 329 as a key. Then, the virtual production monitoring model processing unit 24 acquires a record from the worker information 150 shown in FIG. 4C using the acquired worker identification information 326 as a key. Furthermore, the virtual production monitoring model processing unit 24 acquires three-dimensional information 637 of the sensor 94 from the sensor information 630 shown in FIG. 7C using the acquired sensor identification information 330 as a key. The virtual production monitoring model processing unit 24 superimposes the acquired three-dimensional information 176, 136, 146, 637 on the diving three-dimensional information generated in step S504. In addition, the virtual production monitoring model processing unit 24 generates an avatar AV (see FIG. 25B) of the site worker P from each piece of information in the record acquired from the worker information 150, and superimposes the generated avatar AV on the diving three-dimensional information.

[0164] In this way, when the user U reads the two-dimensional barcode TB with the VR terminal CR, the three-dimensional screen D3 for diving linked to the read two-dimensional barcode TB is displayed on the VR terminal CR.

[0165] In short, for the virtual production monitoring model M1 reflecting the current situation of the site, three-dimensional information for diving, which is a three-dimensional image as if the user U were present in the site, is generated as a second virtual reality image. Then, the generated three-dimensional information for diving is displayed on the VR terminal CR used by the user U as shown in FIG. 22B via the display processing unit 28 of FIG. 11. In this way, the user U can grasp the situation of the site as if he or she were standing at the site.

[0166] (When a malfunction occurs) Fig. 24 is a diagram showing an example of the virtual production monitoring model M1c when a malfunction occurs. In Fig. 24, to avoid complication, reference numerals are not given to elements that have already been explained in Fig. 19. The defects include inspection defects, work delays, and the like. An inspection defect is detected when an abnormality is recorded in the acquired data 415 of the work performance information 410. A work delay is detected when the end date and time 419 of the work performance information 410 remains blank even after a predetermined time has passed since the scheduled end date and time 333 of the work plan information 320 shown in FIG. 6B. When such defects such as an inspection defect or work delay are detected, the location where the defect occurred is displayed on the virtual production monitoring model M1c as shown by the symbol MA in FIG. 24, and further, a two-dimensional barcode TB for diving is displayed. At this time, the virtual production monitoring model processing unit 24 identifies the plot position PR where the defect occurred in the work performance information 410 using the site identification information 414 of the record where the defect occurred as a key. Then, the virtual production monitoring model processing unit 24 displays defect information MB as shown in FIG. 24 for the plot position PR where the defect occurred.

[0167] Fig. 25A is a diagram showing an example of a selection screen D4 displayed on the VR terminal CR when the two-dimensional barcode TB shown in Fig. 24 is read by the VR terminal CR. Also, Fig. 25B is a diagram showing an example of a three-dimensional screen D5 for diving. When the user U reads the two-dimensional barcode TB shown in FIG. 24, a selection screen D4 as shown in FIG. 25A is displayed on the screen of the VR terminal CR.

[0168] When "Go Dive" or "Go Dive with" is selected, a three-dimensional screen D5 for diving as shown in FIG. 25B is displayed on the screen of the VR terminal CR. The three-dimensional screen D5 for diving is the three-dimensional information for diving generated by the process shown in FIG. 23 displayed on the screen of the VR terminal CR. Also, the three-dimensional screen D5 for diving displays an avatar AV of the field worker P generated based on the information of the worker information 150 shown in FIG. 4C. Incidentally, "Go Dive" is selected when the user U views the three-dimensional screen D5 for diving shown in FIG. 25B alone. Also, "Go Dive with" is selected when the three-dimensional screen D5 for diving is shared by multiple people.

[0169] In the three-dimensional diving screen D5 shown in FIG. 25B, in addition to the MR image, a record button D51, a play / stop button D52, a fast-forward button D53, and a rewind button D54 are displayed. By operating the play / stop button D52, the fast-forward button D53, and the rewind button D54, the user U can view the MR video at the location where the problem occurred. The MR video is made possible by the virtual production monitoring model processing unit 24 storing the three-dimensional information acquired by the sensor 94 in chronological order. In this way, the three-dimensional diving screen D5 displayed on the screen of the VR terminal CR can be fast-forwarded, rewound, and recorded.

[0170] This allows the user U to check the product flow at the plot position PR where the defect occurred using the MR video, which helps identify the cause of the defect.

[0171] In addition, the user U can record and save the MR video by selecting the record button D51. For example, by recording and saving the MR video that is the cause of the problem, the MR video of the problem can be shared with other users U.

[0172] In addition, by multiple users U reading the two-dimensional barcode TB shown in Fig. 24, the three-dimensional diving screen D5 shown in Fig. 25B can be shared among multiple users U. That is, a common three-dimensional diving screen D5 is displayed on the screen of the VR terminal CR used by each of the multiple users U. In this way, the multiple users U can discuss the cause of the malfunction.

[0173] At this time, the virtual production monitoring model processing unit 24 may control the screen of the VR terminal CR so that a plurality of users U with different positions (types of users U) can check the process leading up to the occurrence of the malfunction in the same space from a free viewpoint. In addition, the virtual production monitoring model processing unit 24 may control the screen of the VR terminal CR so that the playback speed is the same for all users U viewing the MR video, or can be set to any desired playback speed.

[0174] <Third embodiment> Next, a third embodiment of the present invention will be described with reference to FIGS. In the third embodiment, a process of presenting a countermeasure against a malfunction and adopting the countermeasure will be described. Note that, in the third embodiment, an example of a progress delay is shown as a malfunction, but similar processes are performed for other malfunctions.

[0175] (Progress Delay Countermeasures Processing) 26 is a flowchart showing the procedure of the progress support measures process, with reference to FIGS. First, when a problem is detected in an operation as shown in FIG. 24, a user U determines whether or not a countermeasure against the progress delay is necessary (S601). If no countermeasure is required (S601→NO), the work monitoring support system Z ends the process. If measures are necessary (S601→YES), the production analysis and simulator unit 5, which is a countermeasure plan generating unit, generates candidates for delay avoidance countermeasure plans (S602). Since the generation of candidates for delay avoidance countermeasure plans is a known technique, detailed description in this embodiment will be omitted, but the production analysis and simulator unit 5 generates countermeasure plans by simulating many work environments. For example, the production analysis and simulator unit 5 generates countermeasure plans for how many additional field workers P are required to avoid delays by simulating the case where the number of field workers P is increased. The number of countermeasure plan candidates generated may be one or more. Furthermore, the production analysis and simulator unit 5 may select the candidate with the highest delay countermeasure effect as a result of the simulation.

[0176] Next, the production analysis / simulator unit 5 generates information (measures plan information IF (see FIG. 27)) on the measures plan generated in step S602 (S603). In this way, the production analysis / simulator unit 5 generates at least one measures plan by performing a simulation. Next, the virtual production monitoring model processing unit 24 superimposes the countermeasure information IF (see FIG. 27) generated in step S603 onto the virtual production monitoring model M1 (FIG. 15) (S604).

[0177] In addition, the virtual production monitoring model processing unit 24 generates a two-dimensional barcode TB (see FIG. 21, etc.) associated with the countermeasure plan information IF (S605). Note that what is generated in step S605 is not limited to the two-dimensional barcode TB, but may be a one-dimensional barcode, a button, etc. Then, when the user U selects a countermeasure plan from the countermeasure plan information IF, the virtual production monitoring model processing unit 24 superimposes the two-dimensional barcode TB generated in step S605 on the virtual production monitoring model M1 (S606). The superimposed two-dimensional barcode TB corresponds to the countermeasure plan selected by the user U.

[0178] The virtual production monitoring model M1 on which the countermeasure plan information IF (countermeasure plan) is superimposed is displayed by the display processing unit 28 on the VR terminal CR worn by the user U. In other words, the display processing unit 28 displays the generated countermeasure plan together with the virtual production monitoring model M1 on the VR terminal CR. The user U can dive into the virtual production monitoring model M1 by reading the two-dimensional barcode TB displayed on the VR terminal CR.

[0179] The user U judges whether or not to adopt a countermeasure plan by viewing specific information of the countermeasure plan (S607). If there are multiple countermeasure plans, the user U may select one from the multiple countermeasure plans, or may select multiple countermeasure plans. If the user U decides not to adopt the currently displayed countermeasure plan (S607→NO), the virtual production monitoring model processing unit 24 returns to step S602 and generates countermeasure plan candidates again. At this time, it is preferable to change the conditions of the simulation performed by the production analysis / simulator unit 5.

[0180] When the user U adopts the countermeasure plan (S607→YES), the virtual production monitoring model processing unit 24 deletes countermeasure plan information other than the adopted countermeasure plan information, and superimposes information indicating that countermeasures are being taken on the virtual production monitoring model M1 (S608).

[0181] Then, at the site, the site workers P are secured and the line layout is changed in accordance with the proposed countermeasures.

[0182] Next, the virtual production monitoring model processing unit 24 judges whether or not the countermeasure has been completed (S609). Completion of the countermeasure in step S609 means completion of the countermeasure at the actual site. The completion of the countermeasure may be determined by inputting the completion of the countermeasure by the user U via the user terminal UT, or by recognizing an image captured by a camera installed at the site to judge whether or not the countermeasure has been completed.

[0183] If it is determined that the countermeasures have not been completed (S609→NO), the virtual production monitoring model processing unit 24 returns the process to step S609. If the countermeasures have been completed (S609→YES), the virtual production monitoring model processing unit 24 updates the virtual production monitoring model M1 (S610). The virtual production monitoring model M1 is updated using 3D-CAD information after the countermeasures have been completed, three-dimensional information obtained by a laser distance measuring device, etc. In addition, the virtual production monitoring model processing unit 24 updates information related to the countermeasures that have been taken, such as the 3D-CAD information of the factory information 110 shown in Fig. 3A, the line information 120 shown in Fig. 3B, and the work information 310 shown in Fig. 6B.

[0184] (Example of countermeasures screen) Next, a specific example of the delay countermeasure screen will be shown with reference to FIGS. Fig. 27 is a diagram showing an example of a virtual production monitoring model M1d on which countermeasure plan information is displayed, while Figs. 28 and 29 are diagrams showing examples of virtual production monitoring models M1e and M1f on which specific countermeasure plans are displayed. In Fig. 27, the work route (solid line) and countermeasure plan information IF are superimposed on the virtual production monitoring model M1 shown in Fig. 15, and the two-dimensional barcode TB shown in Fig. 15 is not displayed. Incidentally, the screen shown in Fig. 27 is the screen displayed in step S604 in Fig. 26. Also, the three-dimensional factory information MF1 and MF2 are the same as that shown in Fig. 13.

[0185] In the example shown in FIG. 27, multiple countermeasures such as "1. Increase the number of workers at site XX of the assembly plant" and "2. Increase the number of facilities at the inspection plant" are displayed. When a line designer U11 (see FIG. 28), which is a type of user U wearing a VR terminal CR, selects "1. Increase the number of on-site workers at the assembly plant site", the screen shown in FIG. 28 is displayed on the VR terminal CR (see FIG. 28) of the line designer U11. In FIG. 28, a virtual production monitoring model M1e is displayed. The screen shown in FIG. 28 is the screen displayed in step S606 in FIG. 26. Note that "site XX" corresponds to the plot position PR.

[0186] In Fig. 28, an increase in the number of site workers P is shown in plot position PR21 of the first factory F1 corresponding to the three-dimensional factory information MF1. Furthermore, countermeasure proposal information IF1 is displayed in the plot position PR21. The countermeasure proposal information IF1 displays specific countermeasure proposals, such as the number of site workers P to be increased and the reduction in work hours as a result of implementing the countermeasure. Furthermore, the plot position PR21 where the countermeasure is implemented is highlighted with a different display (diagonal lines in the example shown in Fig. 28) from the other plot positions PR.

[0187] 27, when the user U selects and inputs a countermeasure, information on the countermeasure is displayed as countermeasure information IF1. By displaying in this manner, the user U can easily grasp the countermeasure.

[0188] In addition, the two-dimensional barcode TB generated in step S605 in Fig. 26 is displayed in Fig. 28. When the line designer U11 or the like wearing the VR terminal CR reads the two-dimensional barcode TB displayed in Fig. 28, the line designer U11 or the like can dive into the plot position PR21 that is the target of measures in the virtual production monitoring model M1e. Note that the diving process is similar to the process shown in Figs. 21 to 22B, and therefore a description thereof will be omitted here.

[0189] Also, when the line designer U11 selects "2. Adding equipment to the inspection factory", a screen shown in Fig. 29 is displayed on the VR terminal CR of the line designer U11. In Fig. 24, the virtual production monitoring model M1f is displayed. Also, the screen shown in Fig. 29 is the screen displayed in step S606 in Fig. 26.

[0190] FIG. 29 shows that equipment is to be added to plot position PR22 of the second factory F2 corresponding to the three-dimensional factory information MF2. Moreover, countermeasure plan information IF2 is displayed at plot position PR22 corresponding to the equipment to be added. The countermeasure plan information IF2 displays specific countermeasure plans, such as the name of the equipment to be added and the reduction in work time as a result of the countermeasure. Moreover, the plot position PR22 where the countermeasure is implemented and a three-dimensional image TM1 of the equipment to be added are highlighted with a display, etc., different from the other plot positions PR and the work line information L21. In the example shown in FIG. 29, the three-dimensional image TM1 of the equipment and the plot position PR22 are indicated by diagonal lines.

[0191] In addition, the two-dimensional barcode TB generated in step S605 in Fig. 26 is displayed in Fig. 29. When the line designer U11 or the like wearing the VR terminal CR reads the two-dimensional barcode TB displayed in Fig. 29, the line designer U11 or the like can dive into the plot position PR22 that is the target of measures in the virtual production monitoring model M1f.

[0192] The line designer U11 or the like can dive into the plot position PR that is the target location of the countermeasure by using the two-dimensional barcode TB shown in Figures 28 and 29. This allows the line designer U11 or the like to consider countermeasure proposals as if they were standing at the target location of the countermeasure.

[0193] FIG. 30 is a diagram showing an example of the virtual production monitoring model M1g during the execution of a countermeasure. Fig. 30 is an example of a screen displayed in step S608 in Fig. 26. Fig. 30 also shows a case where two of the proposed countermeasures shown in Fig. 27, "1. Increase the number of on-site workers at the assembly plant site" and "2. Add equipment to the inspection plant", are adopted. In the virtual production monitoring model M1g shown in Fig. 30, balloons IF11 and IF12 in preparation are displayed at the plot positions PR21 and PR22 that are the targets of measures in Fig. 28 and Fig. 29. Moreover, in the three-dimensional image TM1 of the plot positions PR21 and PR22 and the equipment that are the targets of measures in Fig. 28 and Fig. 29, the same emphasis as in Fig. 28 and Fig. 29 is applied in Fig. 30.

[0194] FIG. 31 is a diagram showing an example of a virtual production monitoring model M1h after countermeasures have been completed and updated. The screen shown in FIG. 31 is the screen displayed in step S610 of FIG. In the example shown in FIG. 31, a virtual production monitoring model M1h in which the countermeasures are reflected is displayed.

[0195] When the countermeasures are completed, the virtual production monitoring model processing unit 24 updates the three-dimensional information 115 of the factory information 110 shown in FIG. 3A, the line information 120 shown in FIG. 3B, the work information 310 shown in FIG. 6B, and the like.

[0196] <Fourth embodiment> FIG. 32A is a diagram showing an example of installation of the movable three-dimensional camera 94d1. The movable 3D camera 94d1 is a type of the 3D camera 94d shown in FIG. 32A, movable rails RA are installed in a grid pattern on the ceiling of a factory F. A movable three-dimensional camera 94d1 moves along the movable rails RA. In this manner, three-dimensional information about the inside of the factory F is captured.

[0197] FIG. 32B is a diagram showing an example in which the AGV 91 and the robot 93 are equipped with a three-dimensional camera 94d. As the AGV 91 equipped with the three-dimensional camera 94d and the robot 93 move around the factory F, three-dimensional information within the factory F is captured.

[0198] FIG. 32C is a diagram showing an example in which a three-dimensional camera 94d is mounted on the bottom of the drone DR. A drone DR equipped with a three-dimensional camera 94d flies inside the factory F, thereby acquiring three-dimensional information about the interior of the factory F.

[0199] As shown in FIGS. 32A to 32C, three-dimensional camera 94d is movable, so that three-dimensional information about the inside of factory F can be acquired dynamically and easily.

[0200] <Fifth embodiment> Next, the setting of the display level 224 will be described with reference to FIGS. 33 to 35B. Fig. 33 is a diagram showing an example of the display level setting information 220, Fig. 34A and Fig. 35A are diagrams showing actual images D601, D611, and Fig. 34B and Fig. 35B are diagrams showing examples of diving three-dimensional screens D602, D612. The display level setting information 220 shown in Fig. 33 is data managed by the work monitoring unit 2 or the like. The display level setting information 220 shown in FIG. 33 stores various identification information such as product identification information 221, work identification information 222, work identification information 223, and the like, and a display level 224 in the MR space. The product identification information 221, work identification information 222, work identification information 223, etc. are the same as those described above, and therefore individual explanations will be omitted. Incidentally, the identification information stored in the product identification information 221 is information that links with the identification information stored in the product identification information 162 of the product information 160 shown in Fig. 5A. Moreover, the work identification information 222 is information that links with the identification information stored in the work identification information 312 of the work information 310 shown in Fig. 6A.

[0201] 33, there are a "common avatar" and a "common avatar + image". The "common avatar" means that the field worker P is displayed as an avatar, and the "common avatar + image" means that most of the field worker P is displayed as an avatar, but a part of the field worker P is shown as an actual image.

[0202] 34A and 34B are diagrams showing examples of a "common avatar." When the actual image is image D601 as shown in FIG. 34A, in the "common avatar," as shown in FIG. 34B, the field worker P is displayed as avatar AV11 in a diving three-dimensional screen D602 as a second virtual reality image. That is, in the "common avatar," the entire field worker P is displayed as avatar AV11. The "common avatar" is used when it is not necessary to display accurate actions such as "setting materials" and "operating equipment."

[0203] The "common avatar" does not have to be the avatar AV11 of the field worker P himself, and for example, an animation of the field worker P's movements that have been learned may be displayed. In addition, when it is desired to view the posture of the field worker P, the avatar AV11 may be displayed as a silhouette of a portion that is recognized as a person in an image captured by a 2D camera or a three-dimensional camera 94d. In this way, the field worker P working at a site corresponding to the diving three-dimensional screen D602 is displayed as the avatar AV11 in the diving three-dimensional screen D602. In this way, the field worker P can reduce the feeling of being watched, and the stress of the field worker P can be reduced.

[0204] In contrast to Fig. 34A and Fig. 34B, Fig. 35A and Fig. 35B are diagrams showing an example of "common avatar + image". When the actual image is the image D611 shown in Fig. 35A, the field worker P is displayed as shown in Fig. 35B on the diving three-dimensional screen D612 displaying the "common avatar + image". That is, in the "common avatar + image", the field worker P is displayed as the avatar AV12, and the hand movements of the field worker P and the field worker P's field of view object are displayed as the actual image D612a. The field worker P's field of view object is the place the field worker P is looking at. The "common avatar + image" is used when it is necessary to display accurate movements of the hands, etc., such as "inspection". In this way, in Fig. 35B, a part of the avatar AV12 of the field worker P is displayed as a real image (actual image D612a). In this way, the field worker P's sense of being monitored can be reduced, and the monitor can confirm the area required for monitoring as a real image. In other words, the user U checking the situation at the site can set the avatar display method for the site worker P, thereby reducing the psychological burden on the worker.

[0205] <Hardware configuration> FIG. 36 is a diagram showing the hardware configuration of a computer CA used in this embodiment. The computer CA includes the work monitoring unit 2, work plan management unit 3, work performance management unit 4, and production analysis / simulator unit 5 shown in FIG. 2, and the work monitoring support system Z includes the site management unit 6, site control unit 7, and site data acquisition unit 8. The computer CA has a memory CA1, an arithmetic unit CA2, a storage device CA3, and a communication device CA4. The memory CA1 is composed of a volatile memory such as a random access memory (RAM). The arithmetic unit CA3 is composed of a central processing unit (CPU), a graphic processing unit (GPU), etc. The storage device CA3 is composed of a non-volatile memory such as a hard disk (HD), a solid state drive (SSD), etc. The communication device CA4 is composed of a network interface card (NIC), etc. The communication device CA4 is the communication unit 32 in FIG. 9, the user communication unit 22, and the system communication unit 23 in FIG. 11.

[0206] A program is stored in the storage device CA3, and this program is loaded into the memory CA2. The loaded program is executed by the arithmetic device CA2, thereby realizing the units 31, 33 to 34 shown in Fig. 9 and 21, 24 to 28 shown in Fig. 11.

[0207] Furthermore, in this embodiment, the description is mainly directed to work, but similar processing can also be performed for a process that includes multiple works.

[0208] <About modified examples> The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0209] The work plan management section 3 and the work performance management section 4 shown in FIG. 2 may be integrated into one production management section.

[0210] The two-dimensional barcode TB displayed in Fig. 15 and the like may be readable by the VR terminal CR or a PC or tablet terminal (for example, the user terminal UT shown in Fig. 11) used by the user U. For example, a one-dimensional barcode or a specified mark may be used.

[0211] In the above embodiments, each functional unit can be configured by hardware such as a circuit device that implements that function, or can be configured by a calculation device CA2 such as a CPU executing software that implements that function.

[0212] In the above embodiment, an example of managing the work of a production process has been described, but this embodiment can also be applied to other fields, such as the management of facilities, robots 93, automobiles, and human actions (maintenance, monitoring, etc.) related to smart cities. The present invention can be applied to technology that maps each base on three-dimensional information and recognizes the work content of facilities, robots 93, and humans using various sensors.

[0213] Also, the above-mentioned configurations, functions, units 2-8, 21, 24-28, 31, 33-34, common database 1, etc. may be realized in hardware by designing some or all of them as integrated circuits, for example. Also, as shown in Fig. 36, the above-mentioned configurations, functions, etc. may be realized in software by a processor such as a CPU interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a HD, or in a memory, a recording device such as an SSD, or a recording medium such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, or a DVD (Digital Versatile Disc). In addition, in each embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0214] 1 Common Database 2 Work monitoring unit (first virtual reality generation unit, reflection unit, display processing unit, and second virtual reality generation unit) 3. Work Planning and Management Department 4. Work Performance Management Department 5. Production Analysis and Simulator Department (Countermeasures Proposal Generation Department) 6 Site management department (storage department) 7 On-site control section 8. Field Data Acquisition Department 24 Virtual production monitoring model processing unit (first virtual reality generation unit, second virtual reality generation unit) 25 Status information overlay unit 26 Three-dimensional basic information generation unit (first virtual reality generation unit) 91 AGV 92 Tools and equipment 93 Robot 94 Sensors 94d 3D camera (3D information acquisition section) 94d1 Movable 3D camera (movable 3D information acquisition unit) 94e Environmental Sensor 320 Work plan information (chronological information) 410 Work performance information (chronological information) AV Avatar AV11 Avatar AV12 Avatar CR VR terminal (terminal) D1A Plot position management setting screen D1B Plot position management setting screen D2 Dive confirmation screen D3 3D Screen for Diving (2nd Virtual Reality Image) D5 Diving 3D screen (second virtual reality image, common second virtual reality image) D51 Record button D52 Play / Stop button D53 Fast forward button D54 Rewind button D602 3D Screen for Diving (2nd Virtual Reality Image) D612 3D Screen for Diving (2nd Virtual Reality Image) DR Drone F factory F1 No.1 Factory F2 Second Factory F3 Third Factory IF Countermeasures Information (Countermeasures) IF1 Countermeasures Information IF2 Countermeasures Information IF11 Speech bubble IF12 Speech bubble L1 Virtual Passage L31 Work route information (information on completed work) L32 Work route information (information on the work to be carried out) M0 3D basic information M1 Virtual Production Monitoring Model (First Virtual Reality Image) M1a Virtual Production Monitoring Model (First Virtual Reality Image) M1b Virtual Production Monitoring Model (First Virtual Reality Image) M1c Virtual Production Monitoring Model (First Virtual Reality Image) M1d Virtual Production Monitoring Model (First Virtual Reality Image) M1e Virtual Production Monitoring Model (First Virtual Reality Image) M1f Virtual Production Monitoring Model (First Virtual Reality Image) M1g Virtual Production Monitoring Model (First Virtual Reality Image) M1h Virtual Production Monitoring Model (First Virtual Reality Image) MA1 Speech bubble MA2 Callout (Information about current work) MB defect information P Field worker PF Plot location information (on-site time series information) PR Plot Position PR1 Plot location (information on the current work) RA Movable rail TB 2D barcode (marker) U, U1~U4 users U11 Line Designer UT User Terminal U11 Line Designer Z Work monitoring support system

Claims

1. a first virtual reality generating unit configured to generate a first virtual reality image in which a configuration of a predetermined site is a virtual three-dimensional image; a reflection unit that acquires time-series information of the state of the site and reflects the time-series information in the first virtual reality image; a display processing unit that displays the first virtual reality image reflecting the time-series information on a terminal used by a user; a second virtual reality generating unit that generates a second virtual reality image, which is a three-dimensional image as if the user is present in the site, in response to the first virtual reality image in which the current site situation is reflected by the reflecting unit; and having The display processing unit is Displaying the second virtual reality image on a terminal used by the user A work monitoring support system comprising:

2. The display processing unit is Displaying information about the progress of work being carried out at the site together with the first virtual reality image.

2. The work monitoring support system according to claim 1.

3. The display processing unit is In the first virtual reality image, information on a task that has already been performed, information on a task to be performed, and information on a task currently being performed are displayed in different display methods.

3. The work monitoring support system according to claim 2.

4. A countermeasure plan generating unit generates at least one countermeasure plan by performing a simulation when a defect in the work is detected, The display processing unit is The generated countermeasure plan is displayed on the terminal together with the first virtual reality image.

2. The work monitoring support system according to claim 1.

5. When the user selects and inputs the countermeasure plan, Information about the proposed measures is displayed.

5. The work monitoring support system according to claim 4.

6. The three-dimensional information acquisition unit for acquiring the three-dimensional information of the site is movable.

2. The work monitoring support system according to claim 1.

7. The display processing unit is displaying a marker linking the first virtual reality image and the second virtual reality image on the second virtual reality image; When the user reads the marker with the terminal, the display processing unit displays the second virtual reality image associated with the read marker on the terminal.

2. The work monitoring support system according to claim 1.

8. a plot location where the second virtual reality image can be displayed is displayed in the first virtual reality image; The plot position displayed varies depending on the type of user.

2. The work monitoring support system according to claim 1.

9. The display processing unit is A common second virtual reality image is displayed on the screens of the terminals used by each of the plurality of users.

2. The work monitoring support system according to claim 1.

10. The second virtual reality image displayed on the screen of the terminal can be fast-forwarded, rewound, and recorded.

10. The work monitoring support system according to claim 9.

11. The display processing unit is In the first virtual reality image and the second virtual reality image, a site worker working at the site is displayed as an avatar.

2. The work monitoring support system according to claim 1.

12. The display processing unit is In the second virtual reality image, a portion of the avatar of the field worker is displayed as a real image.

12. The work monitoring support system according to claim 11.

13. The second virtual reality generator, When there is a gap between information on a current site state acquired by the sensor and information on a site state previously stored in a storage unit, the current site information is reflected in the second virtual reality image.

2. The work monitoring support system according to claim 1.

14. A sensor that is installed at a predetermined site and senses a state of the site; a first virtual reality generating unit configured to generate a first virtual reality image in which the configuration of the site is a virtual three-dimensional image; a reflection unit that acquires time-series information of the state of the site and reflects the time-series information in the first virtual reality image; a display processing unit that displays the first virtual reality image reflecting the time-series information on a terminal used by a user; a second virtual reality generating unit that generates a second virtual reality image, which is a three-dimensional image as if the user is present in the site, in response to the first virtual reality image in which the current site situation is reflected by the reflecting unit; and having The display processing unit is Displaying the second virtual reality image on a terminal used by the user A work monitoring support system comprising:

Images