Work support apparatus and work support method

Abstract

To provide work support information with appropriate content for a worker.SOLUTION: An estimation unit 313 estimates a difficult-to-identify portion that is less visually identifiable to a worker S, out of objects that are viewed by the worker S. A determination unit 314 determines a display position of a warning image PZ which is to be superimposed on the object, based on the difficult-to-identify portion RZ estimated by the estimation unit 313 and history information indicating past histories related to work of the worker S. An output unit 315 outputs control information for displaying the warning image PZ in the position determined by the determination unit 314.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a work assistance device and a work assistance method. [Background technology]

[0002] Conventionally, technologies have been developed to provide support to workers using MR (Mixed Reality) devices or AR (Augmented Reality) devices. For example, an assembly support system described in Patent Document 1 below includes an AR device (worker terminal), an instructor terminal, and an assembly support device. The assembly support device supports aircraft engine assembly work. An identifier acquisition unit acquires identifiers of aircraft engine parts. A work manual selection unit selects, from a work manual database, a work manual related to the part identified by the identifier acquired by the identifier acquisition unit. A display control unit displays at least a part of the work manual selected by the work manual selection unit on the worker terminal. The display control unit has a screen sharing mode that displays, on the instructor terminal, the same screen as the display screen of the worker terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-168870 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described conventional technology, the worker terminal assists the worker in the work by displaying a manual for a part specified by the worker or by displaying a pointer at a location specified by the assistant. However, in the above-described conventional technology, the content of the assistance provided to the worker is determined by the worker or the assistant, and is not actively determined by the system. However, for example, when the assistant determines the assistance content, the assistance content determined by the assistant may be inappropriate due to differences in the worker's and assistant's levels of expertise in the work. Furthermore, for example, when there are a wide variety of work types, the assistant's level of expertise in each individual work may not necessarily be high. If the assistant's level of expertise in the work to be assisted is not high, it may not be possible to provide appropriate assistance to the worker.

[0005] An object of the present disclosure is to provide workers with work support information with appropriate content. [Means for solving the problem]

[0006] A work assistance device according to a preferred aspect of the present disclosure includes an estimation unit that estimates difficult-to-identify parts of an object within a worker's field of view that have low visual identification accuracy by the worker, a determination unit that determines a display position of a warning image that is displayed superimposed on the object based on the difficult-to-identify parts estimated by the estimation unit and history information that indicates a past history of work performed by the worker, and an output unit that outputs control information for displaying the warning image at the position determined by the determination unit.

[0007] A work assistance method according to a preferred aspect of the present disclosure estimates difficult-to-identify parts of an object within a worker's field of vision that are difficult to identify visually by the worker with low accuracy, determines a display position of a warning image to be displayed superimposed on the object based on the estimated difficult-to-identify parts and historical information indicating the past history of work performed by the worker, and outputs control information for displaying the warning image at the determined position. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide appropriate work support information to a worker. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of a work support system 1 according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating a typical state in which the wearable terminal 10 is used. [Figure 3] 2 is a diagram schematically illustrating a state in which the assistant terminal 20 is used. FIG. [Figure 4] 1 is a diagram schematically illustrating a visual field VS1 of a worker S wearing a wearable terminal 10. FIG. [Figure 5] FIG. 2 is a block diagram showing the configuration of the wearable terminal 10. [Figure 6] FIG. 2 is a block diagram showing the configuration of an assistant terminal 20. [Figure 7] FIG. 2 is a block diagram showing the configuration of a work support server 30. [Figure 8] FIG. 10 is a diagram showing a schematic diagram of a difficult-to-distinguish portion RZ. [Figure 9] FIG. 10 is a diagram illustrating an example of history information. [Figure 10] FIG. 10 is a diagram schematically showing an example of an attention-calling image PZ. [Figure 11] 10 is a diagram showing another example of history information and an attention-calling image PZ. FIG. [Figure 12] 10 is a flowchart showing the operation of the processing device 303. DETAILED DESCRIPTION OF THE INVENTION

[0010] A. Embodiment A-1. System Configuration 1 is a block diagram showing the configuration of a work support system 1 according to an embodiment. The work support system 1 includes a wearable device 10, an assistant terminal 20, and a work support server 30. The work support server 30 is an example of a work support device. The wearable device 10, the assistant terminal 20, and the work support server 30 are connected via a communication network N.

[0011] 2 is a diagram schematically illustrating a state in which the wearable terminal 10 is used. As shown in FIG. 2, the wearable terminal 10 is a see-through head-mounted display that is worn on the head of a worker S. In this embodiment, the worker S performs work on a device DV (devices DV1 to DV3 are shown in FIG. 2) while wearing the wearable terminal 10.

[0012] The shape of the wearable device 10 is similar to that of ordinary eyeglasses, for example. The wearable device 10 has lenses L and a frame F that supports the lenses L. The lenses L include a left lens disposed in front of the left eye of the worker S and a right lens disposed in front of the right eye of the worker S. The left and right lenses L each have a half mirror. The half mirrors of the left and right lenses L transmit light representing real space R, thereby guiding the light representing real space R to the eyes of the worker S. The half mirrors of the left and right lenses L also reflect light representing a virtual object, which is guided by an optical member, toward the eyes of the worker S. The light of real space R that has passed through the half mirror and the light representing the virtual object that has been reflected by the half mirror are superimposed and enter the eyes of the worker S, causing the worker S to perceive the virtual object as being located in real space R.

[0013] Auxiliary information for assisting the worker S is displayed as a virtual object on the lens L of the wearable terminal 10. The auxiliary information is, for example, an attention-calling image PZ (see FIG. 10, etc.) that indicates a part of the device DV that requires attention during work. Furthermore, as the auxiliary information, for example, a work manual MN (see FIG. 4) that describes the work procedure may be displayed.

[0014] Fig. 3 is a diagram schematically illustrating the use state of the assistant terminal 20. The assistant terminal 20 is used by an assistant H who assists a worker S in performing a task. The assistant terminal 20 is an information processing terminal such as a personal computer, a laptop computer, a tablet terminal, or a smartphone. In Fig. 3, the assistant terminal 20 is illustrated as a personal computer.

[0015] The assistant terminal 20 includes a display device 201. The assistant terminal 20 displays a first image PC1 transmitted from the first camera 106A (see FIG. 5) of the wearable terminal 10 on the display device 201. The first image PC1 captures the field of view of the worker S. The first image PC1 shows the space in which the device DV is placed and the worker S performing work on the device DV. By displaying the first image PC1 on the assistant terminal 20, the assistant H can remotely advise and monitor the work being performed by the worker S.

[0016] For example, the assistant H can place a graphic such as a pointer PT on the first image PC1 or draw any graphic (such as a frame for specifying an area). The assistant terminal 20 transmits graphic data for projecting the graphic placed or drawn by the assistant H onto the lens L of the wearable terminal 10 to the wearable terminal 10 via the work support server 30. The wearable terminal 10 projects the graphic onto the lens L based on the graphic data.

[0017] In addition, assistant H can also delete the figure placed or drawn on the first image PC1. For example, if the work process progresses and the worker S starts working on a different part, or if the worker S is able to understand the intention of assistant H's instructions by showing the figure, leaving the figure displayed may actually interfere with the work. Therefore, assistant H deletes the figure placed or drawn on the first image PC1 as necessary. This also causes the figure projected on the lens L of the wearable terminal 10 to disappear.

[0018] 4 is a diagram schematically illustrating the visual field VS1 of a worker S wearing a wearable device 10. Similar to the first image PC1 shown in FIG. 3, the visual field VS1 includes devices DV1 and DV2. As an example of auxiliary information, a pointer PT is displayed in the same position as that of the assistant H. As an example of auxiliary information, a manual MN for the work being performed by the worker S is displayed. The outer edges of the pointer PT and the manual MN are shown with dotted lines to indicate that they are virtual objects.

[0019] The work support server 30 mediates the transmission and reception of data between the wearable device 10 and the assistant device 20. As will be described in detail later, the work support server 30 analyzes the first image PC1 transmitted from the wearable device 10 and determines whether or not the device DV shown in the first image PC1 contains a difficult-to-identify part RZ that is difficult to identify visually by the worker S with low accuracy, and a part requiring careful attention in the work (hereinafter referred to as a "caution part"). If a difficult-to-identify part RZ and a caution part are present, the work support server 30 displays an attention drawing image PZ calling attention to the part in at least one of the field of view (on the lens L) of the worker S and the first image PC1 displayed on the display device 201.

[0020] A-2. Wearable device 10 FIG. 5 is a block diagram showing the configuration of the wearable terminal 10. The wearable terminal 10 includes a projection device 101, a speaker 102, a microphone 103, an input device 104, a communication device 105, a first camera 106A, a second camera 106B, a 3D scanner 107, a storage device 108, a processing device 109, and a bus 110. Each component shown in FIG. 3 is stored in, for example, a frame F. The projection device 101, the speaker 102, the microphone 103, the input device 104, the communication device 105, the first camera 106A, the second camera 106B, the 3D scanner 107, the storage device 108, and the processing device 109 are interconnected by a bus 110 for communicating information. The bus 110 may be a single bus or multiple buses that are different for each element of the device.

[0021] The projection device 101 includes left and right lenses L, a display panel, and optical members. A pair of display panels and optical members may be provided, one on each side, corresponding to the left and right lenses L. The projection device 101 displays a projection image corresponding to a virtual object on the display panel based on control from the processing device 109. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The optical members guide light emitted from the display panel to the left and right lenses L.

[0022] The speaker 102 plays back audio data and outputs audio corresponding to the audio data. For example, when the auxiliary information is video information or audio information, the audio data played back by the speaker 102 is audio data included in the video information or audio data corresponding to the audio information. Furthermore, for example, when the worker S is conversing with the assistant H, the audio data played back by the speaker 102 is the speech of the assistant H picked up by the microphone 204 of the assistant terminal 20.

[0023] The microphone 103 picks up surrounding sounds and generates audio data. The sounds picked up by the microphone 103 are, for example, the speech of a worker S wearing the wearable device 10. The speaker 102 and the microphone 103 may not be included in the wearable device 10 and may be separate from the wearable device 10. In this case, the wearable device 10 is connected to the speaker 102 and the microphone 103 by an interface (not shown).

[0024] The input device 104 is an input device (for example, a keyboard, a mouse, a switch, a button, or a sensor) that accepts an input from the outside. In particular, in the wearable terminal 10, the input device 104 may be, for example, a sensor that detects the movement of the worker S. For example, when the sensor detects that the worker S has made a specific gesture, it may be recognized that a specific operation has been performed on the wearable terminal 10.

[0025] The communication device 105 includes a communication interface for communicating with other devices. The communication device 105 connects to a communication network N using wireless communication or wired communication, and communicates with the work support server 30 via the communication network N.

[0026] First camera 106A and second camera 106B generate images showing real space R. First camera 106A captures a first image PC1 that shows a range of real space R that is included in the field of view of worker S. In this embodiment, for the sake of convenience, it is assumed that the range of real space R that is included in the field of view of worker S is similar to the range shown in first image PC1. Second camera 106B captures a second image PC2 that shows the eyes of worker S. First image PC1 and second image PC2 are used to estimate an object of attention of worker S.

[0027] As described above, the first image PC1 captured by the first camera 106A includes a range of the real space R that is included in the field of view of the worker S. The worker S views the real space R through the lens L. Therefore, the real space R within the range captured in the first image PC1 and the virtual object projected onto the lens L are visually recognized by the worker S as overlapping. The correspondence between each pixel of the first image PC1 captured by the first camera 106A and each pixel of the transmissive display realized by the left and right lenses L is calibrated in advance. In other words, the position of an object captured in the first image PC1 on the left and right lenses L is known from the perspective of the worker S wearing the wearable terminal 10. Therefore, the first output control unit 112, which will be described later, can project a virtual object, the display position of which is determined based on the first image PC1, onto the left and right lenses L.

[0028] As described above, the second image PC2 captured by the second camera 106B captures the eyes of the worker S. The second image PC2 is used to detect the direction of the worker S's gaze. Specifically, for example, a "reference point" that does not move even when the worker S's gaze moves and a "moving point" that moves when the gaze moves are detected from the image of the worker S's eyes. The reference point may be, for example, the inner corner of the eye, and the moving point may be, for example, the iris. The gaze can be detected based on the position of the iris relative to the inner corner of the eye. For example, if the iris of the left eye is far from the inner corner of the eye, the worker S is detected as looking left. If the inner corner of the left eye and the iris are close to each other, the worker S is detected as looking right. Also, for example, the correspondence between the direction of the worker S's gaze and the position on the lens L may be calibrated in advance. Specifically, for example, before starting work, a small moving virtual object is displayed on the lens L, and the worker S is instructed to follow it with his or her eyes. A second image PC2 captured during this time (more specifically, the positional relationship between the reference point and the moving point in the second image PC2) is associated with the display position of the virtual object on the lens L. This makes it possible to use the second image PC2 to accurately detect which part of the lens L the worker S is looking at. The method for detecting the direction of the worker S's line of sight may also be a corneal reflex method using an infrared camera and an infrared LED.

[0029] A first image PC1 generated by the first camera 106A and a second image PC2 generated by the second camera 106B are transmitted to the work support server 30 via the communication device 105. The first camera 106A and the second camera 106B repeatedly capture images at predetermined intervals, and transmit the generated captured image information to the work support server 30 each time an image is captured. In this embodiment, the first camera 106A and the second camera 106B capture video and transmit the video to the work support server 30.

[0030] The first camera 106A and the second camera 106B each have an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens. The imaging lens of the first camera 106A is disposed, for example, on the bridge described above, facing the outside world toward which the face of the worker S is facing. The imaging lens of the second camera 106B is disposed, for example, near the eyes of the worker S in the frame F described above, facing the direction of the eyes of the worker S. Note that the second camera 106B may have two imaging lenses, one for each of the worker S's left and right eyes.

[0031] The imaging optical system may have various optical elements such as a prism, or may have a zoom lens, a focus lens, etc. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0032] The 3D scanner 107 acquires point cloud data for measuring the three-dimensional shape of an object in the vicinity of the worker S. The measurement range of the 3D scanner 107, for example, coincides with the field of view of the worker S or includes the entire area of ​​the field of view of the worker S. Therefore, the three-dimensional shape of an object present in the field of view of the worker S can be calculated based on the point cloud data.

[0033] In this embodiment, the 3D scanner 107 irradiates the measurement range with light from a light source of a light transmitting and receiving unit provided in the frame F. The 3D scanner 107 also receives reflected light from an object using a light receiving unit provided in the light transmitting and receiving unit. The time from light irradiation to reception of the reflected light is proportional to the distance to the object from which the light is reflected. The 3D scanner 107 measures the time until the light is received for each point within the measurement range and calculates the distance to each point within the measurement range.

[0034] In this embodiment, the positional relationship between the 3D scanner 107 and the first camera 106A is fixed. Therefore, the correspondence between the coordinates of the point cloud data acquired by the 3D scanner 107 and the coordinates of the first image PC1 captured by the first camera 106A is known. For example, by specifying a specific area in the first image PC1, the point cloud data of the specific area can be identified.

[0035] The point cloud data generated by the 3D scanner 107 is transmitted to the work support server 30. The 3D scanner 107 repeats measurements at predetermined measurement intervals, and transmits the point cloud data to the work support server 30 each time a measurement is taken.

[0036] The storage device 108 is a storage medium readable by the processing device 109. The storage device 108 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). The storage device 108 stores a program PG1. The program PG1 is a program for operating the wearable terminal 10.

[0037] The processing device 109 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0038] The processing device 109 reads the program PG1 from the storage device 108. The processing device 109 executes the program PG1 to function as a first input control unit 111 and a first output control unit 112. At least one of the first input control unit 111 and the first output control unit 112 may be configured by a circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0039] The first input control unit 111 transmits input information such as information input to the input device 104 or microphone 103, a first image PC1 captured by the first camera 106A, a second image PC2 captured by the second camera 106B, and point cloud data calculated by the 3D scanner 107 to the work support server 30.

[0040] The first output control unit 112 causes the projection device 101 to display an image and causes the speaker 102 to output sound based on control information transmitted from the work support server 30. The first output control unit 112 may also cause the projection device 101 to display an image and causes the speaker 102 to output sound based on auxiliary information transmitted from the assistant terminal 20.

[0041] A-3. Assistant terminal 20 6 is a block diagram showing the configuration of assistant terminal 20. Assistant terminal 20 includes display device 201, input device 202, speaker 203, microphone 204, communication device 205, storage device 206, processing device 207, and bus 208. Display device 201, input device 202, speaker 203, microphone 204, communication device 205, storage device 206, and processing device 207 are interconnected by bus 208 for communicating information. Bus 208 may be a single bus or may be multiple buses that are different for each element of the device, etc.

[0042] The display device 201 is a display device (for example, various display panels such as a liquid crystal panel or an organic EL panel) that displays information to the outside. The input device 202 is an input device (for example, a keyboard, a mouse, a switch, a button, a sensor, etc.) that receives input from the outside. The display device 201 and the input device 202 may be integrated into one device (for example, a touch panel).

[0043] The speaker 203 plays back audio data and outputs audio corresponding to the audio data. The audio data played back by the speaker 203 is, for example, audio data indicating the speech of the worker S picked up by the microphone 103 of the wearable device 10. The microphone 204 picks up surrounding audio and generates audio data. The audio picked up by the microphone 204 is, for example, the speech of the assistant H who uses the assistant terminal 20. The speaker 203 and the microphone 204 may not be included in the assistant terminal 20 and may be separate from the assistant terminal 20.

[0044] The communication device 205 includes a communication interface for communicating with other devices. The communication device 205 connects to the communication network N using wireless or wired communication, and communicates with the wearable terminal 10 and the work assistance server 30 via the communication network N.

[0045] The storage device 206 is a recording medium that can be read by the processing device 207. The storage device 206 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 206 stores a program PG2. The program PG2 is a program for operating the assistant terminal 20.

[0046] The processing unit 207 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0047] The processing device 207 reads the program PG2 from the storage device 206. By executing the program PG2, the processing device 207 functions as a second input control unit 211 and a second output control unit 212. At least one of the second input control unit 211 and the second output control unit 212 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0048] The second input control unit 211 transmits information input to the input device 202 or the microphone 204 to the work assistance server 30.

[0049] The second output control unit 212 displays the first image PC1 and the attention drawing image PZ on the display device 201 based on the first image data and control information transmitted from the work support server 30. The second output control unit 212 also outputs sound from the speaker 102 based on the voice data of the worker S transmitted from the work support server 30.

[0050] A-5. Work support server 30 7 is a block diagram showing the configuration of the work support server 30. The work support server 30 includes a communication device 301, a storage device 302, a processing device 303, and a bus 310. The communication device 301, the storage device 302, and the processing device 303 are connected to one another by the bus 310 for communicating information. The bus 310 may be a single bus, or may be multiple buses that are different for each element such as a device.

[0051] The communication device 301 includes a communication interface for communicating with other devices. The communication device 301 connects to a communication network N using wireless or wired communication, and communicates with the wearable terminal 10 and the assistant terminal 20 via the communication network N.

[0052] The storage device 302 is a storage medium that can be read by the processing device 303. The storage device 302 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 302 stores a program PG3. The program PG3 is a program for operating the work support server 30.

[0053] The storage device 302 also stores a work information database DB1 and a history information database DB2. The work information database DB1 stores work information related to work performed by the worker S. The work information may be, for example, a work manual or a collection of questions (Q&A) related to the work. The history information database DB2 stores history information related to work performed by the worker S. As will be described in detail later, the history information is, for example, a first image PC1 captured while the worker S is working, and the following information is associated with it: gaze information indicating the gaze point VP of the worker S while working; work content information indicating the content of the entire work; and process information indicating the process the worker S is performing at each point in the image. Note that the history information stored in the history information database DB2 may be information related not only to the worker S himself, but also to the history of the work performed by someone other than the worker S.

[0054] The work information database DB1 and the history information database DB2 may be stored on another web server (not shown) rather than on the work support server 30. Furthermore, the work information database DB1 and the history information database DB2 may be stored in a distributed manner across multiple servers rather than being stored together in one place.

[0055] The processing unit 303 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0056] The processing device 303 reads the program PG3 from the storage device 302. By executing the program PG3, the processing device 303 functions as an acquisition unit 311, a history processing unit 312, an estimation unit 313, a determination unit 314, and an output unit 315. At least one of the acquisition unit 311, the history processing unit 312, the estimation unit 313, the determination unit 314, and the output unit 315 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0057] A-4. Details of the processing device 303 Next, the acquisition unit 311, history processing unit 312, estimation unit 313, determination unit 314, and output unit 315, which are realized by the processing device 303 executing the program PG3, will be described.

[0058] The acquisition unit 311 acquires various types of information from the wearable device 10. The information acquired by the acquisition unit 311 is referred to as "acquired information." The acquired information may be, for example, first image data indicating a first image PC1 captured by the first camera 106A, second image data indicating a second image PC2 captured by the second camera 106B, and point cloud data generated by the 3D scanner 107. The acquired information may also be, for example, auxiliary information projected onto the lens L by the projection device 101. The auxiliary information preferably includes information (such as text or an image) projected onto the lens L as well as projection position information indicating the projection position of the auxiliary information on the lens L. The acquired information may also be, for example, audio data indicating audio input to the speaker 102 and audio data of audio output from the microphone 103. The acquired information may also be, for example, information indicating the content of an input from the worker S to the input device 104.

[0059] The history processing unit 312 generates history information to be stored in the history information database DB2. In this embodiment, the history processing unit 312 generates the history information by adding line-of-sight information, work content information, process information, and graphic information to the first image PC1.

[0060] The gaze information indicates the gaze point VP of the worker S at each time point in the first image PC1. The history processing unit 312 detects the gaze direction of the worker S at each time point during work, using the second image PC2 acquired by the acquisition unit 311. The history processing unit 312 adds the gaze information by mapping the detected gaze direction to the first image PC1.

[0061] The work content information indicates the overall content of the work being performed by the worker S shown in the first image PC1. The work content information may be indicated, for example, by text such as "initial setup work for the switchboard" or by symbols such as "XX-YY-S103." The work content information may be input by the worker S, for example, before starting or after completing the work. The work content information may also be estimated, for example, using a machine learning model that estimates the work content through image analysis. The work content information is used as an index for retrieving history information the next time the same (or similar) work is performed.

[0062] The process information indicates the process that the worker S is performing at each point in time in the first image PC1. The process information may be represented by text such as "Process 2: Check Initial Settings" or by a symbol such as "S002." The process information is estimated, for example, based on the page of the manual MN displayed to the worker S as auxiliary information. Specifically, for example, while the page of process 1 of the manual MN is displayed to the worker S, it is estimated that the worker S is performing process 1. Furthermore, it is estimated that the worker S will turn the page and display the page of process 2 of the manual MN when process 1 ends and process 2 begins. Therefore, the history processing unit 312 labels the time period in the first image PC1 when the page of process 1 of the manual MN is displayed to the worker S as "Process 1," and the time period when the page of process 2 of the manual MN is displayed as "Process 2." Furthermore, for example, the worker S himself may perform a predetermined operation at the time when the work process changes, thereby inserting an index indicating the change of process into the first image PC1 as process information. Furthermore, the process information may be estimated using a machine learning model that estimates the content of work through image analysis, for example.

[0063] The graphic information is information indicating a graphic that has been placed or drawn by the assistant H to assist with the work. As described above, the assistant H places or draws a graphic on the first image PC1 displayed on the display device 201 as necessary. The graphic information specifies the shape and position of the graphic placed or drawn on the first image PC1. The graphic information also includes information specifying the timing at which the graphic was placed or drawn and the timing at which the graphic was erased from the first image PC1.

[0064] The above description has been given of generating historical information about work performed by worker S. However, the historical information database DB2 also stores historical information about work performed by worker S or a person other than worker S in the past. Therefore, for example, when a work was performed by a person other than worker S, "worker S" in the above description can be read as "a person other than worker S." For example, the historical information may include, as gaze information, a gaze point VP of worker S or a person other than worker S when work was performed by worker S or a person other than worker S in the past. Furthermore, the historical information may include, as graphic information, at least one of the positions of a pointer PT displayed to worker S or a person other than worker S using the wearable device 10 (transmissive display) when work was performed by worker S or a person other than worker S in the past.

[0065] The estimation unit 313 estimates a difficult-to-identify portion RZ of an object that falls within the field of view of the worker S and that has low visual identification accuracy by the worker S. An object that falls within the field of view of the worker S is, for example, a device DV that is the target of work. In this embodiment, the estimation unit 313 estimates a portion of the object that has little difference in hue and a complex shape as the difficult-to-identify portion RZ.

[0066] The estimation unit 313 estimates areas with small hue differences based on the first image PC1 acquired from the wearable device 10. Areas with small hue differences are, for example, areas where there is little change in color on the surface of an object. As an example, the estimation unit 313 divides the first image PC1 into multiple areas. The estimation unit 313 calculates the variance of the RGB values ​​of each pixel included in each of the divided areas, and if the variance is less than a first predetermined value, estimates that the area has small hue differences.

[0067] The estimation unit 313 also estimates areas with complex shapes based on the point cloud data acquired from the wearable device 10. Areas with complex shapes are, for example, areas with many irregularities on the surface of the object. As an example, the estimation unit 313 converts the point cloud data into three-dimensional coordinates of the object surface to generate shape data indicating the three-dimensional shape of the object. The estimation unit 313 calculates the variance of coordinate values ​​in the depth direction in each of the divided areas, and if the variance is equal to or greater than a second predetermined value, estimates that the shape of the area is complex.

[0068] FIG. 8 is a diagram showing a schematic diagram of an indistinguishable portion RZ. FIG. 8 shows a device DV4 having multiple breakers B (B1 to B8). The portion of device DV4 including breakers B1 and B2 is not estimated to be an indistinguishable portion RZ because the breakers B are spaced apart from each other and have a different hue from the surrounding area. On the other hand, the portion of device DV4 including breakers B3 to B8 is estimated to be an indistinguishable portion RZ because the breakers B are arranged close to each other and have no different hue from the surrounding area.

[0069] In this embodiment, the work support server 30 calculates shape data indicating the shape of an object based on point cloud data, but the wearable terminal 10 may calculate the shape data and transmit the shape data to the work support server 30.

[0070] The shape of the object may also be detected from a captured image captured by first camera 106 A. In this case, for example, first camera 106 A may be a stereo camera having multiple cameras, and the three-dimensional shape of the object may be calculated using multiple captured images of the object captured from different angles.

[0071] The determination unit 314 determines the display position of the attention alert image PZ to be displayed superimposed on the object, based on the difficult-to-distinguish portion RZ estimated by the estimation unit 313 and history information indicating the past history of the work performed by the worker S. The determination unit 314 identifies the type of work currently being performed by the worker S, and determines the display position of the attention alert image PZ based on the history information corresponding to the type of work. More specifically, the determination unit 314 identifies the type of work currently being performed by the worker S, identifies a caution point in the work that requires attention based on the history information corresponding to the type of work, and displays the attention alert image PZ in the part where the caution point and the difficult-to-distinguish portion RZ overlap.

[0072] The type of work currently being performed by the worker S may be estimated, for example, based on the movement of the worker S captured in the first image PC1, or may be estimated from a page of the manual MN displayed as auxiliary information on the wearable terminal 10. Furthermore, the history information corresponding to the type of work currently being performed by the worker S may be identified, for example, based on the work content information and process information included in the history information.

[0073] In this embodiment, the attention area refers to, for example, a location where a graphic such as a pointer PT was placed or drawn by an assistant H during a previous task, and a gaze point VP where a person who performed the task (worker S or a person other than worker S) gazed during the previous task. As described above, the history information includes at least one of, for example, the gaze point of worker S or a person other than worker S when a task was previously performed by worker S or a person other than worker S, and the position (graphic information) of the pointer PT displayed to worker S or a person other than worker S using the wearable device 10. When the indistinguishable portion RZ overlaps with the position of the gaze point VP or the pointer PT, the determination unit 314 displays an attention drawing image PZ in a portion including the position of the gaze point VP or the pointer PT. The attention drawing image PZ is, for example, an image that distinguishes a portion of an object from other areas.

[0074] 9 is a diagram schematically illustrating an example of history information. For example, in operation A performed by operator S on device DV4, step 1 is an operation performed on breaker B1, step 2 is an operation performed on breaker B2, step 3 is an operation performed on breaker B3, and step 4 is an operation performed on breaker B8. Therefore, when setting operations were performed in the past, pointer PT1 was placed near breaker B1 while operation 1 was being performed, pointer PT2 was placed near breaker B2 while operation 2 was being performed, pointer PT3 was placed near breaker B3 while operation 3 was being performed, and pointer PT4 was placed near breaker B8 while operation 4 was being performed.

[0075] Of the pointers PT, which are caution points, the pointers PT3 and PT4 overlap the difficult-to-distinguish portion RZ. Therefore, when the worker S performs the task A on the device DV4 this time, the determination unit 314 does not display the attention drawing image PZ while the worker S is performing steps 1 and 2, but displays the attention drawing image PZ while the worker S is performing steps 3 and 4.

[0076] FIG. 10 is a diagram schematically illustrating an example of an attention drawing image PZ. While the worker S is performing step 3, an attention drawing image PZ1 is displayed around the breaker B3, and while the worker S is performing step 4, an attention drawing image PZ2 is displayed around the breaker B8. The range of the attention drawing image PZ is not limited to that shown in FIG. 10, and may be, for example, a frame that surrounds the entire difficult-to-distinguish portion RZ. Furthermore, the attention drawing image PZ is not limited to the frame shown in FIG. 10, and may be, for example, any image that produces a visual effect that is different from other areas of the attention drawing image PZ (for example, the image appears to be filtered with a predetermined color).

[0077] The determination unit 314 may also determine the outer edge of the attention drawing image PZ based on the positions of the gaze point VP and the pointer PT. FIG. 11 is a diagram schematically illustrating another example of history information and an attention drawing image PZ. In a past work, it is assumed that the pointer PT5 was placed near the breaker B3 while the worker S was performing step 5. It is also assumed that the gaze points VP of the worker S before and after the pointer PT5 was placed were VP1 to VP6. From this history information, it is estimated that in step 5, in addition to the breaker B3, the surrounding breakers B4 and B5 are also important. In this case, the determination unit 314 displays the attention drawing image PZ3 in a range including the breaker B3 indicated by the pointer PT and the gaze points VP1 to VP6.

[0078] The output unit 315 outputs control information for displaying the attention drawing image PZ at the position determined by the determination unit 314. The output unit 315 outputs control information for displaying the attention drawing image PZ to at least one of the worker S and the assistant H who assists the worker S in his work.

[0079] The control information for displaying the attention drawing image PZ to the worker S is control information for causing the projection device 101 to display the attention drawing image PZ so as to overlap a predetermined position of an object (device DV) in the real space R. By looking at the attention drawing image PZ, the worker S can recognize parts of the device DV that require attention during work. This can increase the worker S's awareness of the importance of attention during work and provide a criterion for determining whether it is okay to work alone or whether it is better to seek the judgment of an assistant H or the like, leading to a reduction in work errors.

[0080] The control information for displaying the attention drawing image PZ to the assistant H is control information for displaying the attention drawing image PZ superimposed on an object (device DV) on the first image PC1. By looking at the attention drawing image PZ, the assistant H can objectively recognize areas where he or she needs to provide particularly thorough assistance to the worker S. This makes it easier for the assistant H and the worker S to communicate with each other, leading to improved work efficiency.

[0081] A-5. Flowchart Fig. 12 is a flowchart showing the operation of the processing device 303. The processing shown in Fig. 12 is performed continuously while the worker S is working. The processing device 303 functions as an acquisition unit 311 and acquires various information from the wearable device 10 (step S100). The processing device 303 functions as a history processing unit 312 and generates history information centered on the first image PC1 (step S102). The generated history information is stored in a history information database DB2.

[0082] The processing device 303 functions as an estimation unit 313 and estimates the difficult-to-identify portion RZ in the device DV that is the work target (step S104). The processing device 303 also functions as a determination unit 314 and identifies the work that the worker S is currently performing and identifies areas that require attention in the current work based on the history information (step S106).

[0083] If there is an area where the difficult-to-distinguish part RZ and the caution part overlap (step S108: YES), the processing device 303 functions as the output unit 315, outputs control information for outputting an attention drawing image PZ to at least one of the worker S and the assistant H (step S110), and returns the processing to step S100. The control information is transmitted to at least one of the wearable terminal 10 and the assistant terminal 20, and the attention drawing image PZ is displayed on at least one of the wearable terminal 10 and the assistant terminal 20.

[0084] Furthermore, if the hard-to-identify portion RZ and the caution portion do not overlap (step S108: NO), the processing device 303 returns the process to step S100. Note that the case where the hard-to-identify portion RZ and the caution portion do not overlap includes the case where at least one of the hard-to-identify portion RZ and the caution portion does not exist.

[0085] A-6. Summary of implementation form As described above, the work support server 30 according to the embodiment estimates the difficult-to-identify portion RZ of the device DV to be worked on, which is difficult to visually identify by the worker S, and determines the display position of the attention drawing image PZ to be displayed superimposed on the device DV based on the difficult-to-identify portion RZ and history information indicating the past history of the work performed by the worker S. Therefore, it is possible to automatically display areas requiring attention during work. For example, appropriate support can be provided even when the assistant H cannot grasp the point that the worker S is confused about due to differences in the skill levels of the worker S and the assistant H, or when the assistant H is not highly skilled in the work performed by the worker S. Furthermore, for example, it is more likely that the work support server 30 can point out to the worker S areas that neither the worker S nor the assistant H has noticed.

[0086] Furthermore, the work support server 30 identifies the type of work currently being performed by the worker S, and determines the display position of the attention drawing image PZ based on the history information corresponding to the type of work. Since the position of the attention drawing image PZ is changed in accordance with the content of the work currently being performed by the worker S, the usefulness of the attention drawing image PZ can be improved.

[0087] Furthermore, the work support server 30 determines the outer edge of the attention drawing image PZ based on the gaze point VP of the worker S and the position of the pointer PT. Since the range of the attention drawing image PZ is determined from multiple perspectives, the usefulness of the attention drawing image PZ can be improved.

[0088] Furthermore, the task support server 30 estimates, as the difficult-to-identify portion RZ, a portion of the device DV that has little difference in hue and a complex shape. Therefore, the difficult-to-identify portion RZ can be estimated accurately with simple processing.

[0089] B: Other (1) In the above-described embodiments, ROM and RAM are exemplified as storage devices 108, 206, and 302, but storage devices 108, 206, and 302 may also be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray® discs), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy® disks, magnetic strips, databases, servers, or other suitable storage media.

[0090] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0091] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0092] (4) In the above-described embodiment, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0093] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0094] (6) Each function illustrated in Figures 5 to 7 is realized by any combination of at least one of hardware and software. Furthermore, there are no particular limitations on the method for realizing each functional block. That is, each functional block may be realized by using a single device that is physically or logically coupled, or may be realized by using two or more physically or logically separated devices that are connected directly or indirectly (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0095] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0096] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0097] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0098] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0099] (10) In the above-described embodiments, the portable device may be a mobile station (MS). Those skilled in the art may also refer to a mobile station as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0100] (11) In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0101] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0102] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" can include regarding some action as having been "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0103] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0104] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the nouns following these articles being plural.

[0105] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0106] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information). [Explanation of symbols]

[0107] 1...work support system, 10...wearable terminal, 20...assistant terminal, 30...work support server, 101...projection device, 102...speaker, 103...microphone, 104...input device, 105...communication device, 106A...first camera, 106B...second camera, 107...3D scanner, 108...storage device, 109...processing device, 111...first input control unit, 112...first output control unit, 201...display device, 202...input device, 203...speaker, 204...microphone, 205...communication device, 206...storage device, 207...processing device, 211...second input control unit, 212...second output control unit, 301...communication device, 302...storage device, 303...processing device, 311...acquisition unit, 312...history processing unit, 313...estimation unit, 314...determination unit, 315...output unit, DB1...work information database, DB2...history information database, DV (DV1 to DV4)...equipment, H...assistant, L...lens, N...communication network, PC1...first image, PZ (PZ1 to PZ3)...warning image, R...real space, RZ...difficult-to-identify part, S...worker.

Claims

1. an estimation unit that estimates a difficult-to-identify portion of an object that is within the field of view of the worker and that has low visual identification accuracy by the worker; a determination unit that determines a display position of an attention-drawing image to be displayed superimposed on the object based on the difficult-to-identify portion estimated by the estimation unit and history information indicating a past history of work performed by the worker; and an output unit that outputs control information for displaying the attention-attracting image at the position determined by the determination unit; A work support device comprising:

2. the determination unit identifies a type of work currently being performed by the worker, and determines a display position of the attention-drawing image based on history information corresponding to the type of work. The work support device according to claim 1.

3. the worker wears a transmissive display while performing the work; the history information includes at least one of a point of gaze of the worker or the person other than the worker when the work was performed by the worker or the person other than the worker in the past, and a position of a pointer displayed to the worker or the person other than the worker using the transmissive display; when the difficult-to-distinguish portion overlaps with the position of the gaze point or the pointer, the determination unit displays the attention-calling image in a portion including the position of the gaze point or the pointer. The work support device according to claim 1.

4. the attention-calling image is an image that distinguishes a partial area of ​​the object from other areas, the determination unit determines an outer edge of the attention drawing image based on the position of the gaze point and the pointer. The work support device according to claim 3.

5. the estimation unit estimates, as the difficult-to-identify portion, a portion of the object that has little difference in hue and a complex shape. The work support device according to claim 1.

6. the output unit outputs the control information for displaying the attention-calling image to at least one of the worker and an assistant assisting the worker in the work. The work support device according to claim 1.

7. Among the objects within the field of view of the worker, a difficult-to-identify portion that is difficult to identify by the worker's visual inspection with low accuracy is estimated; determining a display position of an attention drawing image to be displayed superimposed on the object based on the estimated difficult-to-identify portion and history information indicating a past history of work performed by the worker; outputting control information for displaying the attention-calling image at the determined position; Work support method.

Images