Work support device and work support method

Abstract

To provide a work support device and a work support method which offer work support information with appropriate contents to a worker.SOLUTION: A determination section 312 determines a search string on the basis of visual recognition information including information visually recognized by a worker S and visual line information indicating a direction of a visual line of the worker S. A search section 313 searches work information related to work performed by the worker S from a search object information group by using the search string. An output section 315 generates control information for presentation of the work information to the worker S on the basis of the work information searched by the search section 313, so as to output it.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a work assistance device and a work assistance method. [Background technology]

[0002] Conventionally, technologies have been developed to provide support to workers using MR (Mixed Reality) devices or AR (Augmented Reality) devices. For example, an assembly support system described in Patent Document 1 below includes an AR device (worker terminal), an instructor terminal, and an assembly support device. The assembly support device supports aircraft engine assembly work. An identifier acquisition unit acquires identifiers of aircraft engine parts. A work manual selection unit selects, from a work manual database, a work manual related to the part identified by the identifier acquired by the identifier acquisition unit. A display control unit displays at least a part of the work manual selected by the work manual selection unit on the worker terminal. The display control unit has a screen sharing mode that displays, on the instructor terminal, the same screen as the display screen of the worker terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-168870 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described conventional technology, the worker terminal supports the work by displaying a manual for a part specified by the worker and enabling conversation between the worker and an assistant. However, in the above-described conventional technology, the content of support for the worker is decided by the worker or the assistant, and is not actively decided by the system. However, for example, if the worker is a beginner at a task, he or she may not be able to determine what kind of support to receive. Furthermore, when the assistant decides on the content of support, the content of support decided by the assistant may be inappropriate due to differences in the worker and assistant's levels of expertise in the task. Furthermore, due to a future decline in the working population, it may be impossible to assign assistants to workers.

[0005] An object of the present disclosure is to provide workers with work support information with appropriate content. [Means for solving the problem]

[0006] A work assistance device according to a preferred aspect of the present disclosure includes: a determination unit that determines a search string based on visual information including information visually recognized by a worker and gaze information indicating the direction of the worker's gaze; a search unit that uses the search string to search for work information related to work to be performed by the worker from a group of search target information; and an output unit that generates control information for presenting the work information to the worker based on the work information searched by the search unit, and outputs the control information.

[0007] A work assistance method according to a preferred aspect of the present disclosure determines a search string based on visual information including information visually recognized by a worker and gaze information indicating the direction of the worker's gaze, uses the search string to search for work information related to work to be performed by the worker from a group of search target information, generates control information for presenting the work information to the worker based on the searched work information, and outputs the control information. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide appropriate work support information to a worker. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of a work support system 1 according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating a typical state in which the wearable terminal 10 is used. [Figure 3] 2 is a diagram schematically illustrating a state in which the assistant terminal 20 is used. FIG. [Figure 4] FIG. 2 is a block diagram showing the configuration of the wearable terminal 10. [Figure 5] FIG. 2 is a block diagram showing the configuration of an assistant terminal 20. [Figure 6] FIG. 2 is a block diagram showing the configuration of a work support server 30. [Figure 7] FIG. 2 is a diagram schematically illustrating a reproduction image G. [Figure 8] 10 is a flowchart showing the operation of the processing device 303. DETAILED DESCRIPTION OF THE INVENTION

[0010] A. Embodiment A-1. System Configuration 1 is a block diagram showing the configuration of a work support system 1 according to an embodiment. The work support system 1 includes a wearable device 10, an assistant terminal 20, and a work support server 30. The support server is an example of a work support device. The wearable device 10, the assistant terminal 20, and the work support server 30 are connected via a communication network N.

[0011] 2 is a diagram schematically illustrating a state in which the wearable terminal 10 is used. As shown in FIG. 2, the wearable terminal 10 is a see-through head-mounted display that is worn on the head of a worker S. In this embodiment, the worker S performs work on a device DV (devices DV1 to DV3 are shown in FIG. 2) while wearing the wearable terminal 10.

[0012] The shape of the wearable device 10 is similar to that of ordinary eyeglasses, for example. The wearable device 10 has lenses L and a frame F that supports the lenses L. The lenses L include a left lens disposed in front of the left eye of the worker S and a right lens disposed in front of the right eye of the worker S. The left and right lenses L each have a half mirror. The half mirrors of the left and right lenses L transmit light representing real space R, thereby guiding the light representing real space R to the eyes of the worker S. The half mirrors of the left and right lenses L also reflect light representing a virtual object, which is guided by an optical member, toward the eyes of the worker S. The light of real space R that has passed through the half mirror and the light representing the virtual object that has been reflected by the half mirror are superimposed and enter the eyes of the worker S, causing the worker S to perceive the virtual object as being located in real space R.

[0013] Auxiliary information JH (see FIG. 7) that assists the worker S is displayed as a virtual object on the lens L of the wearable terminal 10. In this embodiment, the auxiliary information JH is mainly described as information transmitted from the work support server 30 (more specifically, at least one of the work information searched by the work support server 30 and the aggregated information generated by the work support server 30). Note that the auxiliary information JH may be, for example, information selected by the worker S himself (for example, a page in the work manual specified by the worker S), or information selected by an assistant H (for example, a page in the work manual specified by the assistant H).

[0014] Fig. 3 is a diagram schematically illustrating the use state of the assistant terminal 20. The assistant terminal 20 is used by an assistant H who assists a worker S in performing a task. The assistant terminal 20 is an information processing terminal such as a personal computer, a laptop computer, a tablet terminal, or a smartphone. In Fig. 3, the assistant terminal 20 is illustrated as a personal computer.

[0015] The assistant terminal 20 includes a display device 201. The assistant terminal 20 displays an image PC1 transmitted from the first camera 106A (see FIG. 4) of the wearable terminal 10 on the display device 201. The image PC1 captures the field of view of the worker S. The image PC1 shows the space in which the device DV is placed and the worker S performing work on the device DV. By displaying the image PC1 on the assistant terminal 20, the assistant H can remotely advise and monitor the work being performed by the worker S.

[0016] A-2. Wearable device 10 FIG. 4 is a block diagram showing the configuration of the wearable terminal 10. The wearable terminal 10 includes a projection device 101, a speaker 102, a microphone 103, an input device 104, a communication device 105, a first camera 106A, a second camera 106B, a storage device 107, a processing device 108, and a bus 110. Each component shown in FIG. 4 is stored in, for example, a frame F. The projection device 101, the speaker 102, the microphone 103, the input device 104, the communication device 105, the first camera 106A, the second camera 106B, the storage device 107, and the processing device 108 are interconnected by a bus 110 for communicating information. The bus 110 may be a single bus or may be multiple buses that are different for each element of the device.

[0017] The projection device 101 includes left and right lenses L, a display panel, and optical members. A pair of display panels and optical members may be provided, one on each side, corresponding to the left and right lenses L. The projection device 101 displays a projection image corresponding to a virtual object on the display panel based on control from the processing device 108. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The optical members guide light emitted from the display panel to the left and right lenses L.

[0018] The speaker 102 plays back audio data and outputs audio corresponding to the audio data. For example, when the auxiliary information JH is video information or audio information, the audio data played back by the speaker 102 is audio data included in the video information or audio data corresponding to the audio information. Furthermore, for example, when the worker S is conversing with the assistant H, the audio data played back by the speaker 102 is the speech of the assistant H picked up by the microphone 204 of the assistant terminal 20.

[0019] The microphone 103 picks up surrounding sounds and generates audio data. The sounds picked up by the microphone 103 are, for example, the speech of a worker S wearing the wearable device 10. The speaker 102 and the microphone 103 may not be included in the wearable device 10 and may be separate from the wearable device 10. In this case, the wearable device 10 is connected to the speaker 102 and the microphone 103 by an interface (not shown).

[0020] The input device 104 is an input device (for example, a keyboard, a mouse, a switch, a button, or a sensor) that accepts an input from the outside. In particular, in the wearable terminal 10, the input device 104 may be, for example, a sensor that detects the movement of the worker S. For example, when the sensor detects that the worker S has made a specific gesture, it may be recognized that a specific operation has been performed on the wearable terminal 10.

[0021] The communication device 105 has a communication interface for communicating with other devices. The communication device 105 connects to the communication network N using wireless communication or wired communication, and communicates with the work support server 30 and the assistant terminal 20 via the communication network N.

[0022] The first camera 106A and the second camera 106B generate images showing the real space R. The first camera 106A captures a first image PC1 that shows the range of the real space R that is included in the field of view of the worker S. The first image PC1 becomes part of the visual recognition information, which will be described later. In this embodiment, for the sake of convenience, it is assumed that the range of the real space R that is included in the field of view of the worker S is similar to the range shown in the first image PC1. The second camera 106B captures a second image that shows the eyes of the worker S. The first image PC1 and the second image are used to estimate the target of attention of the worker S.

[0023] As described above, the first image PC1 captured by the first camera 106A includes a range of the real space R that is included in the field of view of the worker S. The worker S views the real space R through the lens L. Therefore, the real space R within the range captured in the first image PC1 and the virtual object projected onto the lens L are visually recognized by the worker S as overlapping. The correspondence between each pixel of the first image PC1 captured by the first camera 106A and each pixel of the transmissive display realized by the left and right lenses L is calibrated in advance. In other words, the position of an object captured in the first image PC1 on the left and right lenses L is known from the perspective of the worker S wearing the wearable terminal 10. Therefore, the first output control unit 112, which will be described later, can project a virtual object, the display position of which is determined based on the first image PC1, onto the left and right lenses L.

[0024] As described above, the second image captured by the second camera 106B captures the eyes of the worker S. The second image is used to detect the direction of the worker S's gaze. Specifically, for example, a "reference point" that does not move even when the worker S's gaze moves and a "moving point" that moves when the gaze moves are detected from the image of the worker S's eyes. The reference point may be, for example, the inner corner of the eye, and the moving point may be, for example, the iris. The gaze can be detected based on the position of the iris relative to the inner corner of the eye. For example, if the iris of the left eye is far from the inner corner of the eye, the worker S is detected as looking left. If the inner corner of the left eye and the iris are close to each other, the worker S is detected as looking right. Also, for example, the correspondence between the direction of the worker S's gaze and the position on the lens L may be calibrated in advance. Specifically, for example, before starting work, a small moving virtual object is displayed on the lens L, and the worker S is instructed to follow it with his or her eyes. The second image captured during this time (more specifically, the positional relationship between the reference point and the moving point in the second image) is associated with the display position of the virtual object on the lens L. This makes it possible to use the second image to accurately detect which part of the lens L the worker S is looking at. The method for detecting the direction of the worker S's line of sight may also be a corneal reflex method using an infrared camera and an infrared LED.

[0025] The first image PC1 generated by the first camera 106A and the second image generated by the second camera 106B are transmitted to the work support server 30 via the communication device 105. The first camera 106A and the second camera 106B repeatedly capture images at predetermined intervals, and transmit the generated captured image information to the work support server 30 each time an image is captured. In this embodiment, the first camera 106A and the second camera 106B capture video and transmit the video to the work support server 30.

[0026] The first camera 106A and the second camera 106B each have an imaging optical system and an imaging element. The imaging optical system is an optical system including at least one imaging lens. The imaging lens of the first camera 106A is disposed, for example, on the bridge described above, facing the outside world toward which the face of the worker S is facing. The imaging lens of the second camera 106B is disposed, for example, near the eyes of the worker S in the frame F described above, facing the direction of the eyes of the worker S. Note that the second camera 106B may have two imaging lenses, one for each of the worker S's left and right eyes.

[0027] The imaging optical system may have various optical elements such as a prism, or may have a zoom lens, a focus lens, etc. The imaging element is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor.

[0028] The storage device 107 is a storage medium readable by the processing device 108. The storage device 107 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). The storage device 107 stores a program PG1. The program PG1 is a program for operating the wearable terminal 10.

[0029] The processing device 108 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0030] The processing device 108 reads the program PG1 from the storage device 107. The processing device 108 executes the program PG1 to function as a first input control unit 111 and a first output control unit 112. At least one of the first input control unit 111 and the first output control unit 112 may be configured by a circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0031] The first input control unit 111 transmits input information such as information input to the input device 104 or microphone 103, a first image PC1 captured by the first camera 106A, and a second image captured by the second camera 106B to the work support server 30.

[0032] The first output control unit 112 causes the projection device 101 to display an image and causes the speaker 102 to output sound based on the auxiliary information JH transmitted from the work support server 30. The first output control unit 112 may also cause the projection device 101 to display an image and causes the speaker 102 to output sound based on the auxiliary information JH transmitted from the assistant terminal 20.

[0033] A-3. Assistant terminal 20 5 is a block diagram showing the configuration of assistant terminal 20. Assistant terminal 20 includes display device 201, input device 202, speaker 203, microphone 204, communication device 205, storage device 206, processing device 207, and bus 210. Display device 201, input device 202, speaker 203, microphone 204, communication device 205, storage device 206, and processing device 207 are interconnected by bus 210 for communicating information. Bus 210 may be a single bus or may be multiple buses that are different for each element of the device, etc.

[0034] The display device 201 is a display device (for example, various display panels such as a liquid crystal panel or an organic EL panel) that displays information to the outside. The input device 202 is an input device (for example, a keyboard, a mouse, a switch, a button, a sensor, etc.) that receives input from the outside. The display device 201 and the input device 202 may be integrated into one device (for example, a touch panel).

[0035] The speaker 203 plays back audio data and outputs audio corresponding to the audio data. The audio data played back by the speaker 203 is, for example, audio data indicating the speech of the worker S picked up by the microphone 103 of the wearable device 10. The microphone 204 picks up surrounding audio and generates audio data. The audio picked up by the microphone 204 is, for example, the speech of the assistant H who uses the assistant terminal 20. The speaker 203 and the microphone 204 may not be included in the assistant terminal 20 and may be separate from the assistant terminal 20.

[0036] The communication device 205 includes a communication interface for communicating with other devices. The communication device 205 connects to the communication network N using wireless or wired communication, and communicates with the wearable terminal 10 and the work assistance server 30 via the communication network N.

[0037] The storage device 206 is a recording medium that can be read by the processing device 207. The storage device 206 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 206 stores a program PG2. The program PG2 is a program for operating the assistant terminal 20.

[0038] The processing unit 207 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0039] The processing device 207 reads the program PG2 from the storage device 206. By executing the program PG2, the processing device 207 functions as a second input control unit 211 and a second output control unit 212. At least one of the second input control unit 211 and the second output control unit 212 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0040] The second input control unit 211 transmits information input to the input device 202 or the microphone 204 to the work assistance server 30.

[0041] The second output control unit 212 displays the first image PC1 and the attention drawing image PZ on the display device 201 based on the first image data and control information transmitted from the work support server 30. The second output control unit 212 also outputs sound from the speaker 102 based on the voice data of the worker S transmitted from the work support server 30.

[0042] A-4. Work support server 30 6 is a block diagram showing the configuration of the work support server 30. The work support server 30 includes a communication device 301, a storage device 302, a processing device 303, and a bus 310. The communication device 301, the storage device 302, and the processing device 303 are connected to one another by the bus 310 for communicating information. The bus 310 may be a single bus, or may be multiple buses that are different for each element such as a device.

[0043] The communication device 301 includes a communication interface for communicating with other devices. The communication device 301 connects to a communication network N using wireless communication or wired communication, and communicates with the wearable terminal 10 and the assistant terminal 20 via the communication network N.

[0044] The storage device 302 is a storage medium that can be read by the processing device 303. The storage device 302 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 302 stores a program PG2. The program PG2 is a program for operating the work support server 30.

[0045] The storage device 302 also stores a work information database DB. The work information database DB stores work information related to work performed by the worker S. The work information database DB may not be stored in the work support server 30, but may be stored in another web server (not shown). Furthermore, the work information may not be stored collectively in one place, but may be stored in a distributed manner across multiple servers.

[0046] The processing unit 303 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0047] The processing device 303 reads the program PG2 from the storage device 302. By executing the program PG2, the processing device 303 functions as an acquisition unit 311, a determination unit 312, a search unit 313, an aggregation unit 314, and an output unit 315. At least one of the acquisition unit 311, the determination unit 312, the search unit 313, the aggregation unit 314, and the output unit 315 may be configured by a circuit such as a DSP, an ASIC, a PLD, or an FPGA.

[0048] A-4. Details of the processing device 303 Next, the acquisition unit 311, the determination unit 312, the search unit 313, the aggregation unit 314, and the output unit 315, which are realized by the processing unit 303 executing the program PG2, will be described.

[0049] The acquisition unit 311 acquires various types of information from the wearable terminal 10. The information acquired by the acquisition unit 311 is referred to as "acquired information." The acquired information is, for example, a first image PC1 captured by the first camera 106A and a second image captured by the second camera 106B. The acquired information may also be, for example, auxiliary information JH projected onto the lens L by the projection device 101. The auxiliary information JH preferably includes information (such as text or an image) projected onto the lens L as well as projection position information indicating the projection position of the auxiliary information JH on the lens L. The acquired information may also be, for example, audio data indicating audio input to the speaker 102 and audio data of audio output from the microphone 103. The acquired information may also be, for example, information indicating the content of an input from the worker S to the input device 104.

[0050] The determination unit 312 determines a search string based on visual information including information visually recognized by the worker S and line-of-sight information indicating the direction of the worker S's line of sight. In this embodiment, the visual information includes, for example, at least one of the following visual information 1 to visual information 3. Visual information 1: Information indicating the real space R included in the field of view of the worker S. Visual information 2: Auxiliary information JH actively selected by the worker S or assistant H. Visual information 3: Auxiliary information JH displayed based on control information output by the output unit 315 described later.

[0051] The determination unit 312 generates visual information using the first image PC1 and auxiliary information JH from the acquired information. The determination unit 312 generates, as visual information, a reproduction image G that reproduces the field of view of the worker S, for example, by overlaying the auxiliary information JH on the first image PC1. Furthermore, as gaze information, the determination unit 312 detects the gaze direction of the worker S using the second image from the acquired information. The determination unit 312 estimates the gaze target of the worker S by mapping the estimated gaze direction to the reproduction image G.

[0052] FIG. 7 is a diagram schematically illustrating a reproduction image G. In the reproduction image G shown in FIG. 7, a device DV4 arranged in a real space R and auxiliary information JH1 and JH2 are displayed. The outer edges of the auxiliary information JH1 and JH2 are dotted lines to indicate that they are virtual objects. The device DV4 is a distribution board, and it is assumed that a worker S is performing setting work on the device DV4. The auxiliary information JH1 is a manual for the setting work being performed by the worker S. The manual may be stored in, for example, a work information database DB. The worker S selects a manual related to the work he or she is performing from the work information database DB and instructs that it be displayed on the lens L. The worker S turns the pages of the manual according to the progress of the work.

[0053] Furthermore, auxiliary information JH2 is a Q&A regarding the setting work being performed by worker S. As will be described in detail later, auxiliary information JH2 is work information searched by the work support server 30 (search unit 313) based on the work status of worker S. The part of the reproduced image G that worker S is gazing at is estimated from the second image. The gaze point of worker S is taken as VP. In the example of FIG. 7, gaze point VP is on earth leakage breaker B located in the center of device DV4. Therefore, it is estimated that the object of worker S's attention is earth leakage breaker B of the distribution board.

[0054] The determination unit 312 estimates the target of attention of the worker S based on the visual recognition information and the line of sight information, and determines at least one search string (search query) based on the target of attention. The determination unit 312 recognizes the name of the target of attention of the worker S, for example, by image recognition, and determines the name as a search string. Image recognition is performed, for example, using a machine learning model that, when an image is input, outputs the name of an object appearing in the image. There may be multiple search strings. For example, in the example of FIG. 7, as described above, the target of attention of the worker S is the earth leakage breaker B of the distribution panel. Therefore, the determination unit 312 determines, for example, "distribution panel" and "earth leakage breaker" as search strings. Furthermore, for example, because a switch SW is provided near the earth leakage breaker B, "distribution panel," "breaker," and "switch" may be determined as search strings.

[0055] Furthermore, the determination unit 312 may estimate the work process being performed by the worker S from, for example, a portion of the manual that is the auxiliary information JH1, and include a character string related to the work process in the search character string. The character string related to the work process may be, for example, the name of the work performed in the work process, the name of the device DV and part that is the target of the work, the name of the action included in the work, the name of the tool used in the work, etc.

[0056] For example, in the example of FIG. 7, the manual of auxiliary information JH1 displays the page "Work step 3: Reset current settings." Therefore, it is highly likely that worker S is performing work step 3, which is resetting current settings, among the configuration processes for the switchboard. The manual also lists the model number of device DV that is the target of the configuration (not shown). For example, assume that the model number of device DV4 is "XX-YY." In this case, the determination unit 312 determines, for example, "switchboard," "XX-YY," "configuration work," and "work step 3," or "switchboard," "XX-YY," and "reset current settings" as the search strings.

[0057] Also, for example, it is assumed that it is estimated from the gaze information that the worker S is reading a specific section (hereinafter referred to as the "specific section") of the manual, which is the auxiliary information JH1. In this case, the determination unit 312 may extract a search character string from the text written in the specific section. Specifically, for example, it is assumed that the specific section contains the statement "If you press and hold the reset switch, the red light will flash." In this case, the determination unit 312 determines, for example, "distribution board," "reset switch," "red light," and "flashing" as the search character strings.

[0058] Furthermore, the determination unit 312 may determine a search string based on at least one of the visual recognition information and the line-of-sight information, the worker S's movement (*1), the time elapsed since the start of the work (*2), the worker S's speech (*3), and the assistant H's speech (*4). As described above, when the worker S is displaying a manual as auxiliary information JH1, it is easy to identify the work process the worker S is performing. On the other hand, for example, when the manual is not displayed or the manual is not turned over for a long time, it is difficult to identify the work process the worker S is performing. Therefore, the determination unit 312 may identify the work process the worker S is performing based on the worker S's movement (e.g., pressing a switch, raising and lowering a lever, etc.) or the time elapsed since the start of the work, and determine a word or phrase related to the work process as a search string. The worker S's movement may be estimated by image recognition based on the worker S's body movement captured in the first image PC1, for example. Alternatively, if the acquired information includes a detection value from a sensor that detects the worker S's movement, etc., the movement may be estimated based on the detection value. Furthermore, for example, a standard required time may be set for each step of the work performed by the worker S, and the step currently being performed by the worker S may be estimated based on the time elapsed since the start of the work.

[0059] The determination unit 312 may also determine a search string by estimating the status of the work based on at least one of the speech of the worker S and the speech of the assistant H. Because speech can be easily converted into text, the status of the work (e.g., the difficulty the worker S is facing in the work) can be directly understood. The speech of the worker S may be, for example, a conversation with the assistant H or the monologue of the worker S. The speech of the assistant H may be, for example, a conversation with the worker S or the monologue of the assistant H. The determination unit 312 may, for example, perform speech recognition on the voice data acquired as the acquired information and convert the speech of the worker S or the speech of the assistant H into text. The determination unit 312 then extracts a character string included in the text and determines it as a search string. For example, if the worker S says to the assistant H, "The red light doesn't flash even when I press and hold the reset switch," the determination unit 312 determines, for example, "switchboard," "reset switch," "red light," and "doesn't flash," as search strings.

[0060] The search unit 313 uses the search string to search for task information related to the task performed by the worker S from among the search target information group. The search target information group is, for example, information stored in a task information database DB. The search target information group may also be information stored in a web server connected to the communication network N. In this embodiment, the search target information group is assumed to be text information.

[0061] The plurality of pieces of information included in the search target information group are converted in advance into a plurality of search target vectors that indicate each of the plurality of pieces of information. More specifically, the information (text information) included in the search target information group is divided into, for example, predetermined units, and each unit is vectorized. The predetermined unit may be, for example, one sentence, or an item (such as one section in a document or one question in a Q&A). A single document may be divided in different ways (for example, a first way is by sentence, a second way is by section, etc.), and each of these may be vectorized.

[0062] The search unit 313 converts the search string into a search string vector and selects information to be used as work information from the search target information group based on the similarities between the multiple search target vectors and the search string vector. Specifically, the search unit 313 selects, as work information, information corresponding to a search target vector that has the highest similarity to the search string vector from among the multiple search target vectors. Alternatively, the search unit 313 may select, as work information, a predetermined number of pieces of information corresponding to a predetermined number of search target vectors from among the multiple search target vectors in descending order of similarity to the search string vector.

[0063] The search unit 313 may change the search range in the search target information group based on at least one of visual recognition information, line-of-sight information, the movement of the worker S, the time elapsed since the start of the task, speech by the worker S, and speech by the assistant H. Specifically, for example, as shown in the example of FIG. 5, when the page "Work Step 3: Reset Current Settings" of the manual is displayed as auxiliary information JH1, it is highly likely that information prior to work step 3 in the search target information group does not contain information useful to the worker S. Therefore, the search unit 313 sets, for example, information on work step 3 and onward as the search range. In other words, the search unit 313 may exclude information related to work steps 1 and 2, which are prior to the work step currently being performed by the worker S, from the search range. Changing the search range in this way increases the possibility of quickly presenting information useful to the worker S.

[0064] When the search unit 313 searches for two or more pieces of work information, the aggregating unit 314 aggregates the two or more pieces of work information searched by the search unit 313 to generate one piece of aggregated information. For example, when the search unit 313 selects a predetermined number of pieces of information corresponding to a predetermined number of search target vectors in descending order of similarity to the search character string vector as work information, the aggregating unit 314 aggregates the multiple pieces of information to generate one piece of aggregated information. The aggregating unit 314 may aggregate information using a learning model used in so-called generative AI (Artificial Intelligence).

[0065] The aggregating unit 314 may select information to be included in one piece of aggregated information based on, for example, at least one of visual information, line-of-sight information, the movements of the worker S, the time elapsed since the start of the task, speech by the worker S, and speech by the assistant H. Specifically, for example, a search is performed using search strings such as "switchboard," "XX-YY," and "reset current settings," and multiple pieces of work information are obtained. At this time, for example, if the gaze point VP of the worker S is directed toward the red light for a long period of time (if the proportion of time that the gaze point VP of the worker S is on the red light is equal to or exceeds a predetermined proportion within a predetermined period of time), the aggregating unit 314 may generate aggregated information centered on information related to the red light from among the multiple pieces of work information.

[0066] Furthermore, the information included in the aggregated information may be changed based on, for example, the history of work performed by worker S in the past. For example, the types of work performed by worker S may be recorded, and the aggregated information may be generated so that the amount of information about types of work that worker S has performed in the past is small, and the amount of information about types of work that worker S has not performed in the past is large. The work history may be obtained, for example, from the work content (type of work) recorded in a work management system that manages work requests to worker S. Furthermore, the work history may be obtained from history information that associates, for example, a first image PC1 captured while worker S is working with gaze information indicating the gaze point VP of worker S while working, work content information indicating the content of the entire work, and process information indicating the process that worker S is performing at each point in the image.

[0067] It should be noted that aggregation of the work information does not necessarily have to be performed. For example, if the number of pieces of work information obtained as search results is very small, or if the worker S does not wish to aggregate the work information (for example, if the worker S wishes to be presented with a list of search results), the work information may be used as auxiliary information JH as is.

[0068] The output unit 315 generates control information for presenting the work information to the worker S based on the work information searched by the search unit 313, and outputs the control information. Furthermore, when the work information is aggregated by the aggregation unit 314, the output unit 315 outputs control information for presenting one piece of aggregated information to the worker S. The control information output by the output unit 315 is transmitted to the wearable terminal 10. The wearable terminal 10 displays the work information or the aggregated information as auxiliary information JH on the lens L based on the control information. "Control information for presenting the work information (or one piece of aggregated information) to the worker S" may be rephrased as "control information for outputting the work information (or one piece of aggregated information) on the wearable terminal 10."

[0069] Note that when the search unit 313 searches for two or more pieces of work information, instead of the aggregation unit 314 aggregating the two or more pieces of work information, the output unit 315 may select information to be included in the control information (i.e., information to be presented to the worker S) from the two or more pieces of work information. In this case, the output unit 315 may select information to be included in the control information from the two or more pieces of work information based on at least one of visual information, line-of-sight information, the movement of the worker S, the elapsed time since the start of the work, the speech of the worker S, and the speech of the assistant H.

[0070] Furthermore, in the above-described embodiment, the information included in the search target information group is text information, but the information included in the search target information group may be information other than text information (hereinafter referred to as "non-text information"). Non-text information is, for example, video, image, audio, etc. In this case, for example, text information (label) indicating the content of the information is assigned to the non-text information. The text information is vectorized in the same way as other text information. The search unit 313 performs a search by comparing the vector of the search target character string with the vector of the non-text information. Furthermore, when non-text information is included in the text information (for example, when a section of a work manual for work step 3 includes a video of work step 3 actually being performed), if the text information is selected as a search result, the non-text information may also be included as part of the search result.

[0071] A-5. Flowchart Fig. 8 is a flowchart showing the operation of the processing device 303. The processing shown in Fig. 8 is performed continuously while the worker S is performing the work. The processing device 303 functions as an acquisition unit 311 and acquires various information from the wearable device 10 (step S100). The processing device 303 functions as a determination unit 312 and determines a search string based on visual information and line-of-sight information (step S102). At this time, the processing device 303 may further determine the search string based on the movement of the worker S, the time elapsed since the start of the work, the speech of the worker S, the speech of the assistant H, etc.

[0072] The processing device 303 functions as a search unit 313 and searches for task information from the search target information group (step S104). Specifically, from the search target information group, information having a vector highly similar to the vector of the search target character string is selected as task information. At this time, the processing device 303 may determine the range of information to be included in the search target information group based on visual recognition information, line of sight information, the movements of the worker S, the elapsed time since the start of the task, the speech of the worker S, the speech of the assistant H, etc.

[0073] The processing device 303 functions as the aggregation unit 314 and aggregates the multiple pieces of work information searched for in step S104 (step S106). As described above, aggregation of work information does not necessarily have to be performed. In this case, the processing device 303 may select information to be included in the aggregated information based on visual information, line-of-sight information, the movements of the worker S, the elapsed time since the start of the work, speech by the worker S, speech by the assistant H, etc.

[0074] The processing device 303 functions as the output unit 315, outputs control information for outputting the aggregated information (or the work information if aggregation is not performed) on the wearable device 10 (step S108), and returns the process to step S100. Note that instead of aggregating multiple pieces of work information in step S106, the processing device 303 functioning as the output unit 315 may select work information to be included in the control information based on visual information, line-of-sight information, the movement of the worker S, the elapsed time since the start of the work, the speech of the worker S, the speech of the assistant H, etc.

[0075] The control information output by the output unit 315 is transmitted to the wearable terminal 10. The wearable terminal 10 displays the task information or the summarized information on the lens L as auxiliary information JH based on the control information.

[0076] A-6. Summary of implementation As described above, the work support server 30 according to the embodiment determines a search string based on visual information including information visually recognized by the worker S and gaze information indicating the direction of the worker S's gaze, and presents the work information searched for using the search string to the worker S. Therefore, information related to the work to be performed by the worker S can be automatically presented to the worker S. For example, compared to when the worker S or the assistant H searches for the information necessary for the work by themselves, work efficiency can be improved. Furthermore, for example, there is an increased possibility that useful information that the worker S and the assistant H are unaware of can be presented to the worker S.

[0077] Furthermore, the work support server 30 according to the embodiment presents consolidated information that aggregates the work information to the worker S. Therefore, the worker S can efficiently grasp the contents of the multiple pieces of work information.

[0078] Furthermore, the work support server 30 according to the embodiment determines the search string based on not only the visual recognition information and the line of sight information, but also at least one of the movement of the worker S, the time elapsed since the start of the work, the speech of the worker S, and the speech of the assistant H. This increases the possibility that the search string can be made more appropriate, and increases the possibility that useful work information for the worker S can be obtained.

[0079] Furthermore, the work support server 30 according to the embodiment changes the search range in the search target information group based on at least one of visual recognition information, line of sight information, the movement of the worker S, the time elapsed since the start of the work, the speech of the worker S, and the speech of the assistant H. This increases the possibility that useful information matching the current work content of the worker S will be obtained as a search result.

[0080] Furthermore, the work support server 30 according to the embodiment selects information to be included in the aggregated information from among the plurality of pieces of work information based on at least one of visual recognition information, line of sight information, the movement of the worker S, the time elapsed since the start of the work, the speech of the worker S, and the speech of the assistant H. This increases the possibility that useful information that matches the current situation of the worker S can be presented as aggregated information.

[0081] Furthermore, the work support server 30 according to the embodiment selects information to be included in the control information from among a plurality of pieces of work information based on at least one of visual recognition information, line of sight information, the movement of the worker S, the time elapsed since the start of the work, the speech of the worker S, and the speech of the assistant H. This increases the possibility of presenting useful information that matches the current situation of the worker S.

[0082] Furthermore, the work support server 30 according to the embodiment converts the search string into a search string vector, and selects information to be used as work information from the search target information group based on the similarity between the search target vector and multiple search target vectors. This increases the possibility that useful information for the worker S can be found from the large amount of information included in the search target information group.

[0083] B: Other (1) In the above-described embodiment, ROM and RAM are exemplified as storage devices 107 and 202, but storage devices 107 and 202 may also be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) discs), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media.

[0084] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0085] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0086] (4) In the above-described embodiment, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0087] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0088] (6) Each function illustrated in Figures 3 and 4 is realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0089] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0090] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0091] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0092] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0093] (10) In the above-described embodiments, the portable device may be a mobile station (MS). Those skilled in the art may also refer to a mobile station as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0094] (11) In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0095] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0096] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" can include regarding some action as having been "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0097] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0098] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0099] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0100] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information). [Explanation of symbols]

[0101] 1...work support system, 10...wearable terminal, 20...assistant terminal, 30...work support server, 101...projection device, 102...speaker, 103...microphone, 104...input device, 105...communication device, 106A...first camera, 106B...second camera, 107...storage device, 108...processing device, 111...first input control unit, 112...first output control unit, 201...display device, 202...input device, 203...speaker, 204...microphone, 2 05...communication device, 206...storage device, 207...processing device, 211...second input control unit, 212...second output control unit, 301...communication device, 302...storage device, 303...processing device, 311...acquisition unit, 312...determination unit, 313...search unit, 314...aggregation unit, 315...output unit, DV (DV1 to DV4)...equipment, F...frame, G...reproduced image, JH (JH1 to JH2)...auxiliary information, L...lens, N...communication network, R...real space, S...worker.

Claims

1. a determination unit that determines a search string based on visual recognition information including information visually recognized by a worker and line-of-sight information indicating a line-of-sight direction of the worker; a search unit that uses the search string to search for task information related to the task performed by the worker from among a group of search target information; an output unit that generates control information for presenting the work information to the worker based on the work information searched by the search unit, and outputs the control information; A work support device comprising:

2. The search unit searches for two or more pieces of work information, An aggregation unit that aggregates the two or more pieces of work information searched by the search unit and generates one aggregated piece of information, the output unit generates control information for presenting the one piece of consolidated information to the worker. The work support device according to claim 1.

3. the determination unit determines the search string based on at least one of the visual recognition information and the line of sight information, a movement of the worker, an elapsed time since the start of the work, a speech of the worker, and a speech of an assistant assisting the worker. The work support device according to claim 1.

4. the search unit changes a search range in the search target information group based on at least one of the visual recognition information, the line of sight information, the worker's movement, the elapsed time from the start of the work, the worker's speech, and the speech of an assistant assisting the worker. The work support device according to claim 1.

5. the aggregating unit selects information to be included in the one aggregated information from the two or more pieces of work information based on at least one of the visual recognition information, the line of sight information, the worker's movement, the elapsed time from the start of the work, the worker's speech, and the speech of an assistant assisting the worker. The work support device according to claim 2.

6. The search unit searches for two or more pieces of work information, the output unit selects information to be included in the control information from the two or more pieces of work information based on at least one of the visual recognition information, the line of sight information, the movement of the worker, the elapsed time from the start of the work, an utterance of the worker, and an utterance of an assistant assisting the worker. The work support device according to claim 1.

7. a plurality of pieces of information included in the search target information group are converted into a plurality of search target vectors each representing the plurality of pieces of information; the search unit converts the search character string into a search character string vector, and selects information to be the work information from the search target information group based on a similarity between the plurality of search target vectors and the search target vector. The work support device according to claim 1.

8. determining a search character string based on visual recognition information including information visually recognized by a worker and line-of-sight information indicating a line-of-sight direction of the worker; Using the search string, search for work information related to the work performed by the worker from among the search target information group; generating control information for presenting the work information to the worker based on the searched work information, and outputting the control information; Work support method.

Images