Work support system, work object identification device and method

Abstract

To provide a work support system, a work object identifying device, and a method capable of improving usability.SOLUTION: A work support system 7 for supporting work of workers includes a work object identifying device 1 used by a worker U1 and a work support device 2 communicably connected to the work object identifying device and used by a work supporter U2 who supports the work of the worker. The work object identifying device acquires three-dimensional space data 52 of a work space including an object device, associates two-dimensional image data 51 obtained by photographing the object device with the object device in three-dimensional space data to create predetermined image data, and outputs the created predetermined image data to the work support device. The work support device displays the predetermined image data received from the work object identifying device, and transmits information input from the work supporter to the work object identifying device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a work support system, a work object identification device, and a method. [Background technology]

[0002] Factories, power plants, server rooms, and other facilities are equipped with a wide variety of devices, including distribution boards, control panels, and servers. When maintaining and inspecting these devices, workers follow the instructions in the work instructions. When a worker has little experience and is unable to perform the work alone, an experienced worker may provide on-site guidance while carrying out the maintenance and inspection work. However, when an experienced worker needs to instruct multiple less-skilled workers, it is difficult for them to accompany the less-skilled workers to the site due to time constraints.

[0003] Therefore, a technology is known in which an experienced worker supports an on-site worker from a remote location (Patent Document 1). In the conventional technology, for example, the on-site worker wears a wearable device and continuously transmits images captured by the wearable device to a remote expert. The equipment on the remote expert's side creates and displays an on-site image by stitching together the images received from the on-site wearable device. The expert selects the area to be worked on on the stitched together image. The worker's wearable device is notified to move to the vicinity of the image of the area to be worked on designated by the expert from the remote location.

[0004] In other conventional techniques, the characteristic points of a work site are acquired and registered in advance as a point cloud. When a worker performs work, the device used by the worker aligns the point cloud based on the coordinates at the time of registration and displays the work target location in three dimensions (Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-181751 A [Patent Document 2] JP 2019-159668 A Summary of the Invention [Problem to be solved by the invention]

[0006] The technology described in Patent Document 1 reproduces the work site only using two-dimensional images, making it difficult for experienced workers to intuitively grasp the position of the work target by looking at the two-dimensional images. With the technology described in Patent Document 1, it is particularly difficult to grasp the depth, making it difficult to identify the work target in a place with a complex network of shelves, piping, and equipment.

[0007] The technology of Patent Document 2 requires processing such as acquiring the 3D space of the work site as point cloud data and allocating it to a 3D map before starting work. Therefore, it is not possible to quickly respond to a new work site, and it is difficult to use. Furthermore, when the situation at the work site changes, the acquired point cloud data cannot be used as it is, so it is not possible to quickly respond to the change in the situation at the work site.

[0008] The present invention has been made in view of the above problems, and an object of the present invention is to provide a task support system, a task object specifying device, and a method thereof that can improve usability. [Means for solving the problem]

[0009] In order to solve the above problems, a work support system according to one aspect of the present invention is a work support system that supports a worker's work, and includes a work object identification device used by the worker, and a work support device used by a work assistant who supports the worker's work, and which is communicatively connected to the work object identification device, wherein the work object identification device acquires three-dimensional spatial data of a work space including the target device, creates specified image data by correlating two-dimensional image data obtained by photographing the target device with the target device in the three-dimensional spatial data, and outputs the created specified image data to the work support device, and the work support device displays the specified image data received from the work object identification device, and transmits information input by the work assistant to the work object identification device. Effect of the Invention

[0010] According to the present invention, it is possible to identify target devices at a work site. [Brief description of the drawings]

[0011] [Figure 1] Overall overview of the work support system. [Diagram 2] FIG. 2 is a block diagram of a work object identification device. [Diagram 3] FIG. 2 is a configuration diagram of a work object identification device. [Figure 4] FIG. [Diagram 5] FIG. 13 is an explanatory diagram showing a method for generating assistance content. [Figure 6] 13 is a flowchart of a work target identification process. [Figure 7] 13 is a flowchart of a work support process. [Figure 8] FIG. 11 is an explanatory diagram showing an example of support content. [Figure 9] FIG. 13 is an explanatory diagram showing a screen showing an object recognition result. [Figure 10] FIG. 13 is an explanatory diagram showing a screen when a plurality of candidates are detected. [Figure 11] FIG. 13 is an explanatory diagram showing an example of a screen in which two-dimensional image data of a target device is associated with the target device in three-dimensional space data. [Figure 12] FIG. 11 is an explanatory diagram showing a state in which predetermined image data acquired from a work object identifying device is operated on the screen of the work support device. [Figure 13] 4 is a flowchart showing a process of the work support device. [Figure 14] 10 is a flowchart of a process for evaluating an operation according to the second embodiment. [Figure 15] FIG. 11 is an overall schematic diagram of a work support system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a work support system that identifies a target device for work at a work site will be described. The work support system 4 according to this embodiment enables a work assistant at a remote location to intuitively understand the work status of a worker U1 on a target device for work. The work status that the worker assistant grasps remotely also includes confirmation of the position of the target device designated in advance to see if the worker is about to work on it.

[0013] As described with reference to Figs. 1 to 11, the work support system 7 according to this embodiment has a space grasping program (space grasping unit 111) for creating three-dimensional space data, a work instruction manual management database 122 for managing a work instruction manual, a work instruction manual analysis program (work instruction manual analysis unit 112) for analyzing a work instruction manual 140, an assistance content control program (assistance content control unit 114) for displaying assistance content 150 for work instructions based on the analyzed work instruction manual, an imaging unit (2D camera 131) for imaging the work situation, an object recognition program (object recognition unit 113) for recognizing objects included in the imaging results, and a part determination program (part determination unit 115) for displaying work instructions and object recognition processing results, creating space data of the work site from the created three-dimensional space data, displaying candidates for work target devices based on identification information of the work target device 4 included in the work instruction manual, and associating the location of the work target device selected from the displayed candidates with the three-dimensional space data. Hereinafter, the work target device 4 will also be referred to as the target device 4.

[0014] According to the work support system 7 of this embodiment, there is no need to collect three-dimensional spatial data at the work site WS in advance, and even if the work site WS changes dynamically, the remote supporter U2 (supervisor) can easily grasp the three-dimensional position of the target device 4 of the worker U1.

[0015] In this embodiment, as described below, when there are multiple identification information candidates for the target device 4 included in the work instruction manual 140, the object recognition program displays a message to select the target device. When there is only one identification information candidate, the object recognition program can prompt the worker U1 to only confirm it.

[0016] If the object recognition program is unable to identify the target device 4, it may request the operator U1 to input information for identifying the target device 4.

[0017] The object recognition program can display information about the target device 4 to be recognized on the output unit 134 of the work object identifying device 1 by analyzing the voice or captured image input by the worker U1. EXAMPLES

[0018] A first embodiment will be described with reference to Figures 1 to 13. In the first embodiment, an example will be described in which a worker U1 installs a target device 4, arrives at a work place WS, and specifies the part to be worked on, and then a remote work assistant U2 double-checks the specified part. Hereinafter, the work assistant U2 may be referred to as an expert U2.

[0019] The work in this embodiment means work including manual work on the device 4. For example, the work in this embodiment is maintenance work such as inspecting parts, replacing parts, and updating software. The work support system 7 in this embodiment is applicable to various devices 4 provided at various work sites WS such as factories, power plants, commercial facilities, hotels, and airports. The various devices 4 include, for example, air conditioners, copiers, servers, network devices, motors, control panels, switchboards, and machine tools. Below, the maintenance work of a server will be described as an example.

[0020] Fig. 1 is an overall schematic diagram of a work support system 7. Fig. 2 is a block diagram of a work object identification device 1.

[0021] The work support system 7 includes, for example, at least one work object identifying device 1 and at least one work support device 2. The work object identifying device 1 and the work support device 2 are connected to each other via communication networks CN1 and CN2 so as to be able to communicate bidirectionally with each other.

[0022] A work support management device 3 may be provided between the work object identification device 1 and the work support device 2. The work support management device 3 can acquire information such as three-dimensional spatial data and two-dimensional image data from the work object identification device 1, and transmit analysis results or processing results of the acquired information to the work support device 2. The work support management device 3 can have the functions of the work object identification device 1 described in Fig. 2. The work support management device 3 may be installed in the work support center SS as a so-called on-premise server. The work support management device 3 may be provided on a communication network as a so-called cloud service.

[0023] As in other embodiments described later, the work object identifying device 1 and the work support management device 3 can also be directly connected. In this case, at least a part of the functions of the work object identifying device 1 can be provided in the work support device 2, and only the two-dimensional image data and three-dimensional space data can be transmitted from the work object identifying device 1 to the work support device 2.

[0024] The work object identification device 1 and the work support device 2 may be connected one-to-one, or one-to-many, many-to-one, or many-to-many. One work support device 2 may be connected to one work object identification device 1, and one skilled person U2 may supervise one worker U1. One work support device 2 may be connected to multiple work object identification devices 1, and one skilled person U2 may supervise multiple workers U1. Multiple work support devices 2 may be connected to one work object identification device 1, and multiple skilled people U2 may supervise one worker U1. Multiple work support devices 2 may be connected to multiple work object identification devices 1, and multiple skilled people U2 may manage and supervise multiple workers U1 in a group. That is, a group consisting of multiple skilled people U2 can remotely supervise a group consisting of multiple workers U1.

[0025] When the worker U1 arrives at the work site WS, he / she finds the target device 4 among various devices provided at the work site WS and performs a predetermined task on the target device 4. The worker U1 uses the work object identification device 1 to find the target device 4 at the work site WS, generates predetermined image data by associating two-dimensional image data of the target device 4 with three-dimensional spatial data including the target device 4, and transmits the generated predetermined image data to the work support device 2.

[0026] The work object identification device 1 is configured, for example, as a tablet computer, a smartphone, a glasses-type wearable terminal, etc. In other words, the work object identification device 1 is a device that has a calculation function, a communication function, a memory function, a function for capturing two-dimensional images, and a function for acquiring three-dimensional spatial data, and can be named anything as long as it is portable.

[0027] The work support center SS is provided with a work support device 2. When the work support device 2 receives predetermined image data from the work object identification device 1 at the work site WS, the work support device 2 displays the predetermined image data on a monitor display. The work supporter U2 can confirm that the worker U1 has reached the location of the target device 4 by looking at the predetermined image 5 displayed on the work support device 2. The image 5 displayed on the work support device 2 is created from the predetermined image data and displayed. In the image 5, two-dimensional image data 51 corresponding to the target device 4, two-dimensional image data (not shown here) of a part of the device 40 existing in the vicinity of the target device 4, and three-dimensional space data 52 of the surroundings including the target device 4 are associated with each other. Since the two-dimensional image data 51 is associated with the three-dimensional space data 52, the work supporter U2 can confirm the two-dimensional image data of the target device 4 from multiple angles by, for example, rotating the three-dimensional space data 52.

[0028] The operation of the work object identification device 1 will be described with reference to Fig. 2. The work object identification device 1 shown in Fig. 2 is configured as a glasses-type wearable terminal. When a worker U1 arrives at the work site WS, the worker U1 scans the surrounding space with a three-dimensional sensor 132 serving as a "three-dimensional space data acquisition unit" to acquire scan data (point cloud data). The space recognition unit 111 recognizes the scanned space from the data acquired by the three-dimensional sensor 132. The space recognition unit 111 recognizes the scanned space means obtaining data indicating the structure of the scanned space.

[0029] When the worker U1 finds the target device 4, he / she photographs the target device 4 with the 2D camera 131 serving as a "2D image data acquisition unit." The object recognition unit 113 recognizes the photographed target device 4. The object recognition unit 113 determines whether the object shown in the 2D image data is the target device 4 by, for example, comparing device identification information such as a product name and / or model name printed or affixed to the target device 4 with device identification information described in a work instruction manual.

[0030] The part determination unit 115 has a function of determining the device or part of the device on which the worker U1 is going to work or is working. In other words, the part determination unit 115 identifies the work position, such as the place where the worker U1 works. The part determination unit 115 generates composite image data as "predetermined image data" by associating the two-dimensional image data of the recognized target device 4 with three-dimensional space data of the peripheral configuration 40 of the target device 4. The part determination unit 115 transmits the generated composite image data to the work support device 2.

[0031] When the work support device 2 receives the composite image data from the work object identifying device 1, it displays a composite image 5 on the monitor display. The composite image 5 includes a two-dimensional image 51 corresponding to the target device 4 and a three-dimensional image 52 corresponding to the configuration 40 surrounding the target device 4. For example, if the target device 4 is a server, the peripheral configuration 40 is a rack or the like. The work supporter U2 can rotate the three-dimensional image 52 (three-dimensional spatial data) by manipulating it, thereby rotating the two-dimensional image 51 as well, and confirming it.

[0032] The configuration of the work object identification device 1 will be described with reference to Fig. 3. The work object identification device 1 is a device used by a worker U1 when performing maintenance work, and has the functions of identifying the work target device 4, confirming the work procedure, and registering the work results. Note that the maintenance work referred to in this specification also includes inspection work and the like.

[0033] As described above, the work object identification device 1 is a computing device such as a personal computer, a smartphone, a tablet terminal, or a head-mounted display, and is equipped with a display device such as an LCD display, a keyboard, a mouse, a touch panel, a touch pen, a touch sensor, voice recognition, gaze recognition, and hand recognition, among other input devices.

[0034] Work object identifying device 1 includes, for example, a control unit 100, a main memory unit 110, an auxiliary memory unit 120, a 2D camera 131, a 3D sensor 132, an input unit 133, an output unit 134, other sensors 135, a power supply unit 136, and a communication unit 137. These electronic circuits 100, 110, 120, 131-137 are connected by a bus 101.

[0035] The control unit 100 controls the operation of the work object identifying device 1. The control unit 100 includes, for example, a microprocessor, a cache memory, a graphics processing unit (GPU), etc. The control unit 100 reads out and executes computer programs 111-115 stored in the main memory unit 110.

[0036] The main memory unit 110 is composed of, for example, a dynamic random access memory (DRAM) and stores computer programs such as a space grasping unit 111, a work instruction manual analyzing unit 112, an object recognizing unit 113, an assistance content control unit 114, and a part determining unit 115. These computer programs 111-115 may be read from the auxiliary memory unit 120 to the main memory unit 110 and deployed therein.

[0037] The auxiliary storage unit 120 is composed of a recording medium built into the work object identification device 1, a removable external recording medium, an optical disk, etc. The auxiliary storage unit 120 stores various data such as a spatial information management database 121, a work instruction manual management database 122, a support content management database 123, and an object recognition management database 124. These databases 121-124 are used and updated as appropriate by the computer programs 111-115.

[0038] The 2D camera 131 includes, for example, an optical lens, an imaging element, a signal processing circuit, etc., and photographs the target device 4 or its parts by operation of the worker U1. The photographed two-dimensional image data is stored in the object recognition management database 124.

[0039] The 3D sensor 132 is, for example, a ToF (Time of Flight) sensor or a stereo camera, and acquires three-dimensional spatial data. The three-dimensional spatial data acquired by the 3D sensor 132 is stored in the spatial information management database 121.

[0040] The input unit 133 is a device through which the worker U1 inputs information to the work object identification device 1. For example, the input unit 133 may be a keyboard, a touch panel, a mouse, a touch pen, a touch sensor, a microphone, a voice recognition device, a gaze recognition device, a device for recognizing hand movements, or a combination of these.

[0041] The output unit 134 is a device that outputs information to the worker U1 from the work object identification device 1. For example, the output unit 134 may be a display device such as a liquid crystal display or an organic EL display (organic electroluminescence display), a speaker, a printer, or a combination of these.

[0042] For example, the work instruction contents, the location of the work target device 4, etc. are displayed on the display serving as the output unit 134. Instructions from the work supporter U2 may be displayed as characters on the display, or may be output as voice from a speaker. The work instruction contents, the location of the work target device 4, advice from the work supporter U2, etc. may be printed out by a printer. The printer may be built into the work object identifying device 1, or may be connected via a wireless or wired communication network.

[0043] The other sensor 135 is, for example, an illuminance sensor, an acceleration sensor, a geomagnetic sensor, a pressure sensor, a temperature sensor, or a combination of these. The other sensor 135 senses the results according to the movement of the worker U1 who uses the work object identification device 1. The other sensor 135 does not need to be built in the work object identification device 1, and may be connected to the work object identification device 1 via a wireless or wired communication network. For example, the other sensor 135 may be provided in a wristwatch-type terminal worn by the worker U1, and the detected sensing data may be transmitted to the work object identification device 1 configured as a glasses-type terminal. The other sensor 135 may include, for example, a sensor that detects vital signs such as a pulse sensor and a blood pressure sensor. By transmitting the vital signs and work content of the worker U1 to the work support device 2, the work supporter U2 can more appropriately grasp the situation of the worker U1.

[0044] The power supply unit 136 is a device that supplies power to parts that require power among the components of the work object identification device 1. The power supply unit 136 is composed of, for example, a battery, an AC / DC conversion adapter, a charging circuit, etc. The power supply unit 136 may be configured to obtain power wirelessly from a wireless power supply device (not shown).

[0045] The communication unit 137 is a device that communicates with the work support device 2 via the communication networks CN1 and CN2.

[0046] The schematic configuration of the work support device 2 will be described with reference to Fig. 4. As shown in Fig. 4, the work support device 2 includes, for example, a control unit 200, a main memory unit 210, an auxiliary memory unit 220, an input unit 231, an output unit 232, a power supply unit 233, and a communication unit 234. These electronic circuits 200, 210, 220, 231-234 are connected by a bus 201.

[0047] The control unit 200 controls the operation of the work support device 2. The control unit 200 reads out and executes computer programs 211 and 212 stored in the main storage unit 210.

[0048] The main memory unit 210 stores computer programs such as a space grasping unit 211 and a work object identifying unit 212. These computer programs 212, 212 may be read from the auxiliary memory unit 220 to the main memory unit 210 and deployed. The auxiliary memory unit 220 stores a space information management database 221. The power supply unit 233 supplies power to each unit that requires power. The communication unit 234 communicates with the work object identifying device 1 via the communication networks CN1, CN2.

[0049] An example of a method for generating assistance content 150 from a work instruction manual 140 will be described with reference to Fig. 5. A work instruction manual 140 is prepared for each type of target device 4, and describes the procedure for maintenance work of the target device 4. The work instruction manual 140 may be composed of text data and still image data, or at least a part of it may be composed of moving image data or audio data.

[0050] When the information identifying the target device 4 is input, the work instruction manual analysis unit 112 acquires the work instruction manual 140 for the identified target device 4 from the work instruction manual management database 122. The information identifying the target device 4 may be input manually by either the worker U1 or the work assistant U2, or may be input automatically from a maintenance plan management system (not shown). Examples of the information identifying the target device 4 include the model, the model name, and the serial number. When the worker U1 manually inputs the information identifying the target device 4, the work assistant U2 who supervises the worker U1 may check whether the input information identifying the target device 4 is correct.

[0051] The work instruction manual analysis unit 112 acquires the support content 150 corresponding to the target device 4 from the support content management database 123 based on the information identifying the target device 4. The work instruction manual analysis unit 112 may generate the support content 150 based on the information identifying the target device 4.

[0052] The generated support content 150 is output to the output unit 134. The support content control unit 114 controls the display content of the support content 150. An example of the support content 150 will be described later with reference to FIG.

[0053] The process of identifying a work target will be described using the flows in Figures 6 and 7. The flow in Figure 6 describes a method in which a worker U1 identifies a work target device 4 while checking work instructions at the work site WS. Figure 6 is a flow explaining the operation of the work target identifying device 1, and Figure 7 is an overall flow of the work support process. In Figure 7, the operations of worker U1 are indicated by dotted line blocks.

[0054] When the work object identification process is started (S10), first, the space grasping unit 111 acquires point cloud data around the worker U1 using the 3D sensor 132, and creates surrounding spatial data (S11). This process starts when the worker U1 issues an instruction to start this process from the input unit 133. If the work object identification device 1 can use a position sensor such as a GPS, this process may be started automatically when the work object identification device 1 reaches a predetermined location.

[0055] The work instruction manual analysis unit 112 acquires the work instruction manual 140 stored in the work instruction manual management database 122 (S12-1 in FIG. 7), and the support content control unit 114 causes the output unit 134 to display the work instruction content 150 (S12-2). Step S12 in FIG. 6 can be divided into steps S12-1 and S12-2 shown in FIG.

[0056] Fig. 8 shows an example of the content 150 for supporting the work. Here, the procedure for identifying the target device 4 for performing the part replacement work is shown. Although not shown in Fig. 8, once the identification of the target device 4 is completed, instructions for a series of work such as removing the target device 4, disassembling the target device 4, and replacing a specified part are displayed in the support content 150.

[0057] The work instruction manual analysis unit 112 instructs the worker U1 to check the model name in front of the target device 4 (step 1). In this example, the model name of the target device 4 is "TMA-5000". The model name is not necessarily displayed on the front of the target device 4. It may be displayed on the side, back, bottom or top of the target device 4. Therefore, there is a possibility that the target device 4 cannot be found by simply taking a picture of the entire work site WS with the 2D camera 131. Therefore, in this embodiment, the behavior of the worker U1 is guided so that the worker U1 moves in front of a device that is thought to be the target device 4. The model name of the target device 4 is displayed in the support content 150 (1501).

[0058] When the worker U1 arrives in front of a device that is thought to be the target device 4, he / she reads out the model name (step 2). The work instruction manual analysis unit 112 acquires the model name read out by the worker U1 via a microphone included in the input unit 133, converts the acquired voice data into text, and stores it (S13).

[0059] In this embodiment, the worker U1 uses the microphone of the input unit 133 to read out specific information (such as model name, type name, and device number) of the device in front of the worker U1, and it is determined that the worker U1 has found the target device 4. Alternatively, the 2D camera 131 may capture an image of the target device 4 in front of the worker U1, and the specific information included in the captured image may be character-recognized, thereby determining that the worker U1 has found the target device 4 and is in front of it. This can prevent the worker U1 from making a reading error, and therefore the discovery of the target device 4 can be detected more smoothly and accurately.

[0060] The rest of the flow will be explained. The work instruction manual analysis unit 112 activates the object recognition unit 113 and performs object recognition of the device corresponding to "TMA-5000" (S14). In detail, the subject (think of the target device 4) in front of the worker U1 is photographed by the 2D camera 131, and the photographed two-dimensional image data is recognized and processed by the object recognition unit 113.

[0061] The object recognition unit 113 superimposes and displays the object recognition result on the output unit 134 (S15). When the work object identification device 1 is a tablet, the worker U1 points the tablet at the target device 4 using AR (Augmented Reality), and device candidates are displayed on the target device visible on the tablet screen.

[0062] If the work object identification device 1 is a head mounted display (HMD), candidates for the target device 4 are displayed superimposed on the view seen by the worker U1 by AR or MR (Mixed Reality) (S15). If there is one object recognition result (S16: YES), proceed to S17.

[0063] 9 is a confirmation screen 151 showing the results of recognizing candidates for the target device 4. The screen 151 also displays the peripheral configuration 1511 of the target device 4. When the operator U1 confirms the object recognition results, the "Yes" button 1512 is selected (S17: YES). On the other hand, when the operator U1 selects the "No" button 1513 (S17: NO), the process returns to step S16.

[0064] If there are multiple object recognition results (S16: NO), the work instruction manual analyzer 112 prompts the worker U1 to select which is the correct target device 4 (S18). Figure 10 shows a screen for confirming one device from multiple candidate target devices 4(1)-4(3).

[0065] 10 shows candidate target devices 4(1)-4(3) recognized from an image captured by the 2D camera 131. The worker U1 selects one candidate that is considered appropriate from among the candidates 4(1)-4(3) via the input unit 133 (S18).

[0066] When the work object identification device 1 is a tablet, the worker U1 touches a candidate that he / she considers appropriate on the touch panel. When the work object identification device 1 is an HMD, the worker U1 selects a candidate that he / she considers appropriate by moving his / her line of sight or moving his / her hands or fingers.

[0067] The part determination unit 115 associates the three-dimensional spatial data acquired in step S11 with the information of the target device 4 selected in either step S17 or step S18, photographs the surrounding area including the target device 4 with the 2D camera 131, and attaches the photographed two-dimensional image data to the three-dimensional spatial data acquired in step S11 (S19).

[0068] FIG. 11 is an example in which two-dimensional image data 51 of the periphery of a selected target device 4 is pasted onto three-dimensional space data 52. FIG. 11 is used in the flow of FIG. 13. Here, pasting two-dimensional image data onto three-dimensional space data means associating the two-dimensional image data so that it is displayed at a predetermined position in the three-dimensional space data 52. FIG. 12 is an explanatory diagram showing a state in which a work supporter U2 operates predetermined image data (image data in which two-dimensional image data and three-dimensional space data are combined) in the work support device 2. The operation means rotating an image to change its orientation. FIG. 11 and FIG. 12 will be described in further detail later.

[0069] When obtaining the three-dimensional space data 52 using ToF sensor data or a 3D sensor 132 such as a stereo camera, the three-dimensional space data 52 is acquired by connecting a plurality of image data captured by the 3D sensor 132. For this reason, the three-dimensional space data 52 is generally unclear and distorted or missing due to, for example, hand shake during measurement or fluctuation in the value of the ToF sensor, as shown in Fig. 11. Therefore, it may be difficult for the skilled work assistant U2 to accurately determine where the target device 4 is located based on only the three-dimensional space data 52.

[0070] Therefore, in this embodiment, the two-dimensional image data 51 captured by the 2D camera 131 is displayed superimposed on the three-dimensional spatial data 52. This allows the work supporter U2 to clearly confirm where the target device 4 is located in the work site WS.

[0071] The work object identifying device 1 transmits predetermined image data (also called composite image data) in which the two-dimensional image data 51 is superimposed on the three-dimensional spatial data 52 to the work supporting device 2 (S20). When the work supporting device 2 acquires the composite image data, it displays it on the output unit 232. By looking at the composite image data, the work supporter U2 can visually confirm that the worker U1 has reached the correct target device 4 and is about to start work. Then, the work supporter U2 can give advice to the worker U1 at the work site WS to support the work via the microphone included in the input unit 231.

[0072] 13, a process will be described in which the work supporter U2 checks the target device 4 identified by the worker U1 based on the composite image data. This process starts, for example, when the work supporter U2 starts up the work support device 2. Alternatively, this process may start automatically when data is received from the work object identifying device 1.

[0073] The space grasping unit 211 of the work support device 2 acquires the composite image data received from the work object identifying device 1 via the communication unit 234 (S31). The composite image data received from the work object identifying device 1 by the communication unit 234 is stored in the space information management database 221.

[0074] The work object identification unit 212 displays the screen 5 for confirming the work location based on the acquired composite image data (S32). FIG. 11 and FIG. 12 are examples of the screen displayed on the output unit 232 of the work support device 2.

[0075] The screen 5 can be rotated at any angle in a plurality of directions, for example, yaw, pitch, and roll, through operation input from the input unit 231. Furthermore, the image 5 can also be displayed enlarged or reduced forward, backward, left, right, etc. Operation buttons for rotation, enlargement, reduction, etc. may be displayed on the screen 5 so that the operation can be performed using a touch panel, keyboard, mouse, etc.

[0076] In Fig. 11, high-resolution two-dimensional image data is displayed superimposed around the work target device 4 specified by the worker U1. Of the high-resolution two-dimensional image data 51, the portion corresponding to the work target device 4 is displayed in a different color so as to be distinguishable from other devices. The work assistant U2 can easily check the positional relationship, including the depth, by displaying the image 5 at any position and angle.

[0077] 12 shows an example in which the device etc. shown in FIG. 11 is rotated 90 degrees with respect to the direction perpendicular to the ground surface. The work assistant U2 can confirm the positional relationship of the target device 4 in the surrounding configuration 40 from an angle other than the front captured by the 2D camera 131.

[0078] The work support system 7 of this embodiment configured as described above does not require the user to obtain 3D spatial data of the work site WS in advance to create a 3D spatial map before work begins, and obtains 3D spatial data to the extent required each time work is performed, making it easy to use. Furthermore, the work support system 7 of this embodiment can be adapted to cases where work is performed at a new work site WS or where a part of an existing work site WS is changed.

[0079] That is, the work support system 7 of this embodiment can three-dimensionally map two-dimensional image data including the target device 4 to three-dimensional spatial data even if the arrangement of devices and objects placed at the work site WS changes dynamically. Therefore, by the worker U1 and the work assistant U2 sharing the three-dimensional mapping result, the work assistant U2 can intuitively understand which device the worker U1 is about to start working on or is working on. Therefore, the work assistant U2 can provide the worker U1 with confirmation that the device selected by the worker U1 is the correct device and advice regarding the work on the target device 4 through the work object identification device 1. EXAMPLES

[0080] The second embodiment will be described with reference to Fig. 14. The following embodiments, including this embodiment, are modifications of the first embodiment, so the differences from the first embodiment will be mainly described. The work support system of this embodiment evaluates and saves the work performed by a worker U1.

[0081] 14 shows a flow of the scoring process. This process can be executed by either the work object identifying device 1, the work support device 2, or the work support management device 3. Alternatively, the scoring results by the work object identifying device 1 and the scoring results by the work support device 2 may be managed by the work support management device 3. Here, the scoring process will be described as being executed by the work object identifying device 1.

[0082] The work object identification device 1 judges whether the work by the worker U1 has been completed (S40). If the work has not been completed (S40: NO), the work supporter U2 obtains and stores the score for each task unit of the worker U1 (S41). That is, for each task unit such as the worker U1 removing the target device 4 from the rack, removing the top plate of the target device 4, and replacing parts in the target device 4, the work supporter U2 scores the work of the worker U1 and inputs it to the input unit 231. The input score result is transmitted to the work object identification device 1 and stored (S41). The completion of the work can be detected by the worker U1 inputting information to the input unit 133 that the work has been completed. The worker U1 may operate a work completion button (not shown) on the screen output by the output unit 134, or may input the completion of the work by voice or text, such as "finished replacing the fan of product A."

[0083] When the work by the worker U1 is completed (S40: YES), the work object identification device 1 acquires the self-assessment by the worker U1 (S42). The self-assessment is performed for the entire work by the worker U1. Alternatively, the worker U1 may self-assess for each task unit during or after the work is completed. For example, the worker U1 may input the result of his / her self-assessment for each task unit to the input unit 133, such as "Removal of device 4 completed. 100 points" or "Fan replacement work completed. 90 points."

[0084] The work object identifying device 1 centralizes the scoring by the work supporter U2 and the self-scoring by the worker U1 (S43). The work object identifying device 1 evaluates the work of the worker U1 based on the score aggregation and stores it in the auxiliary storage unit 120 (S44). The work object identifying device 1 stores image data captured by the 2D camera 131 during the work in the auxiliary storage unit 120 (S45), and also stores composite image data in the auxiliary storage unit 120 (S46).

[0085] The work object identification device 1 can evaluate the work of the worker U1, for example, by ranking or scoring the work, based on the tallying up of the scores by the work assistant U2 and the self-scoring by the worker U1. The evaluation result is associated with two-dimensional image data and composite image data of at least a part of the work at the work site WS, and recorded in a work history management database (not shown). The work history management database can also be provided in the work instruction manual management database 122.

[0086] This embodiment thus configured also has the same effect as the first embodiment. Furthermore, according to this embodiment, the work contents of the worker U1 can be scored and saved as a history, which can be used for the performance evaluation of the worker U1 and can also be used to improve the work instruction manual 140. For example, if there is a work item that is poorly rated by many workers U1, it can be determined that there is room for improvement in the design of the target device 4 or in the description of the work instruction manual 140. EXAMPLES

[0087] A third embodiment will be described with reference to Fig. 15. In a work support system 7A of this embodiment, a work object identifying device 1 and a work support device 2 are directly connected, and a work support management device 3 is not provided. This embodiment configured in this way also achieves the same effects as the first embodiment.

[0088] The present invention is not limited to the above-described embodiment. A person skilled in the art can make various additions and modifications within the scope of the present invention. The above-described embodiment is not limited to the configuration example shown in the attached drawings. The configuration and processing method of the embodiment can be appropriately changed within the scope of achieving the object of the present invention.

[0089] In addition, the components of the present invention may be selected arbitrarily, and the invention having the selected components is also included in the present invention. Furthermore, the components described in the claims may be combined in combinations other than those explicitly stated in the claims. [Explanation of symbols]

[0090] 1: work object identification device, 2: work support device, 3: work support management device, 4: work object device, 5: image displayed on work support device, 7: work support system, 111: space recognition unit, 112: work instruction manual analysis unit, 113 object recognition unit, 114: support content control unit, 115: part determination unit, 131: 2D camera, 132: 3D sensor, 133: input unit, 133: output unit, 211: space recognition unit, 212: work object identification unit

Claims

1. A work support system for supporting a worker's work, comprising: A work object identification device used by a worker; A work assistance device used by a work assistant who assists a worker in a task, the work assistance device being communicatively connected to the work object identification device, the work object identification device acquires three-dimensional spatial data of a work space including a target device, creates predetermined image data by associating two-dimensional image data of the target device with the target device in the three-dimensional spatial data, and outputs the created predetermined image data to the work support device; The work support device displays the predetermined image data received from the work object identifying device, and transmits information input by the work supporter to the work object identifying device. Work support system.

2. The work object identifying device creates assistance content data including guidance information for guiding the worker to the target device in the work space and provides the assistance content data to the worker. The work support system according to claim 1 .

3. The guidance information is information that requests the worker to input, in front of the target device, identification information that identifies the target device into the work target identification device. The work support system according to claim 2 .

4. The support content data is created based on work instruction manual data describing work instructions for the target device. The work support system according to claim 2 .

5. The work object identification device analyzes two-dimensional image data obtained by photographing the work space, extracts candidates for the target device, and presents the candidates to the worker. The work support system according to claim 1 .

6. When there are a plurality of candidates for the target device, the worker is prompted to select the target device from among the plurality of candidates for the target device presented to the worker. The work support system according to claim 5 .

7. The work object identification device acquires an evaluation of the work of the target device and stores the evaluation together with the work history. The work support system according to claim 1 .

8. The work object identifying device and the work support device are directly connected to each other. The work support system according to claim 1 .

9. A work object identification device that identifies a work object device, Acquire three-dimensional spatial data of a working space including the target device from a three-dimensional spatial data acquisition unit; Acquire two-dimensional image data of the target device from a two-dimensional image data acquisition unit; creating predetermined image data by associating the two-dimensional image data of the target device with the target device in the three-dimensional spatial data; Transmitting the created predetermined image data to a work support device; Assistance information based on the predetermined image data is received from the task assistance device. Work object identification device.

10. A work object identification method for identifying a work object device by a work object identification device, comprising: The work object identification device is Acquire three-dimensional spatial data of a working space including the target device from a three-dimensional spatial data acquisition unit; Acquire two-dimensional image data of the target device from a two-dimensional image data acquisition unit; creating predetermined image data by associating the two-dimensional image data of the target device with the target device in the three-dimensional spatial data; Transmitting the created predetermined image data to a work support device; Assistance information based on the predetermined image data is received from the task assistance device. How to identify the work target.

Images