Attention extraction system and attention extraction method
The attention extraction system effectively addresses the challenge of accurately determining and providing caution information by using video and coordinate data to analyze the teacher's gaze mode and extract relevant images, enhancing the accuracy of attention information delivery.
Patent Information
- Application Number
- PCT/JP2024/015567
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-04-19
- Publication Date
- 2024-11-14
AI Technical Summary
Existing caution extraction systems fail to accurately grasp the state of a teacher's gaze and provide appropriate attention information, as they struggle to determine the gaze mode and extract relevant caution information based on gaze duration and shift.
An attention extraction system that uses video information and coordinate data to determine the teacher's gaze mode, sets a gaze area, and extracts the gaze image, associating it with the visual field range and viewpoint displacement, and stores it as caution information in a database, allowing for accurate identification of what the instructor is focusing on and under what circumstances.
Enables the accurate extraction and provision of attention information, improving the ability to grasp the teacher's gaze state and provide appropriate caution, with features like wide-field and alert modes for different gaze behaviors and determination of correct or incorrect work targets.
Smart Images

Figure JP2024015567_14112024_PF_FP_ABST
Abstract
Description
Attention extraction system and attention extraction method
[0001] The present invention relates to an attention extraction system and an attention extraction method for extracting attention information for a task.
[0002] Conventionally, as techniques relating to attention extraction, for example, a skill transfer system disclosed in Patent Document 1 and a wiring work support system disclosed in Patent Document 2 have been proposed.
[0003] The skill transfer system disclosed in Patent Document 1 discloses a technology in which first image data and gaze data of a first user's field of view are subjected to machine learning based on training data, feature data of the image is extracted, information on the first user's focus point in the first image is input as training data and registered in association with the feature data, second image data of a second user's field of view is sent to a server, data on the focus point associated with the feature data corresponding to the second image is read, support display data is created and sent to the second user's eyeglass terminal, and the second user's eyeglass terminal displays information on the focus point based on the support display data, superimposed on the second user's field of view.
[0004] Furthermore, the wiring work support system disclosed in Patent Document 2 includes an eye-tracking device that tracks the line of sight of the worker, an imaging device that captures images of the wiring or other objects being worked on, and image processing within the worker's visual range to interpret characters and colors.The patent document also discloses a portable work management device that has an image processing unit that provides the worker with information about the object stored in a data storage management unit, and that stores work completion signals from limit switches provided on the power tools and images of the object being worked on.
[0005] JP 2017-191490 A JP 2015-061339 A
[0006] Here, Patent Document 1 assumes that users will share their gazes and transfer skills between them. That is, it is difficult to grasp the gaze state based on the gaze duration and gaze shift, extract appropriate attention information, and provide it to the worker. Furthermore, there is no description or suggestion of identifying attention behavior that indicates what the instructor gazed at and in what situation.
[0007] Furthermore, Patent Document 2 is premised on associating the line of sight of the worker with the work object and providing the worker with information about the object. In other words, it is difficult to grasp the state of gaze based on the gaze duration and gaze shift of the viewpoint, extract appropriate attention information, and provide it to the worker. Furthermore, there is no description or suggestion of identifying attention behavior that indicates what the instructor gazed at and in what situation.
[0008] The present invention was devised in consideration of the above-mentioned problems, and its purpose is to provide an attention extraction system and an attention extraction method that grasp the instructor's gaze state and extract and provide appropriate attention information.
[0009] The attention extraction system of the first invention is an attention extraction system that extracts attention information for work, and is characterized by comprising: an acquisition means that acquires video information in the field of view of an instructor performing the work and coordinate information indicating the viewpoint at which the instructor is gazing within the field of view, linking the information to the work in chronological order; an identification means that determines the viewpoint displacement of the instructor within the field of view based on the video information and coordinate information acquired by the acquisition means, and identifies the instructor's gaze mode; an extraction means that sets a gaze area based on the gaze mode identified by the identification means, and extracts a gaze image of the work object at which the instructor is gazing within the gaze area; and a storage means that links the gaze image extracted by the extraction means to the field of view, the viewpoint displacement, and the gaze mode, and stores the gaze image in a database as attention information for the work.
[0010] The attention extraction system of the second invention is characterized in that, in the first invention, the acquisition means further includes a judgment means for judging the correctness of the work object contained in the gaze image, and a display means for displaying the judgment result of the judgment means.
[0011] The attention extraction system of the third invention is the same as that of the second invention, further comprising a database storing the correlation between previously acquired past gaze image information and reference information indicating the correctness of the work object linked to the gaze image, the judgment means referring to the database to judge the correctness of the work object and acquire corresponding information from the database according to the judgment result, and the display means further outputting the corresponding information acquired by the judgment means.
[0012] The attention extraction system according to the fourth invention is characterized in that, in the second invention, it further comprises an input means for inputting the correspondence information output by the judgment means, and the storage means links the correspondence information input by the input means with the gaze image and stores it as an attention data set including setting conditions for recognizing it as a gaze target.
[0013] The attention extraction system of the fifth invention is characterized in that, in the second invention, the video information acquired by the acquisition means includes recording date and time information, recording location information, and recording control information related to the operation of acquiring the video information, and the display means displays the recording date and time information, recording location information, and recording control information in the center of the field of view before the acquisition of the video information by the acquisition means, and switches to displaying only the recording control information in a corner of the field of view while the video information is being acquired.
[0014] The attention extraction system according to a sixth aspect of the present invention is the second aspect of the present invention, wherein the display means further comprises an acquisition display area that displays instructions to the worker performing the work to select the type of the acquisition means that acquires the video information, the type of the attention dataset stored in the database, and when to start the work, and an attention information display area that switches between displaying corresponding information corresponding to the worker's gaze image and attention information based on the attention dataset after the selection, wherein the attention information displayed in the attention display area includes nudge information that is displayed at least at the start, middle, or end of the work depending on the progress of the worker's work, and the information including at least any of the corresponding information, the attention information, and the nudge information displayed in the attention information display area is displayed in a manner that is appropriate to the gaze state of the worker performing the work, based on the attention dataset and the result of the judgment by the judgment means.
[0015] The attention extraction method of the seventh invention is characterized in that it has a computer execute the following steps: an acquisition step for acquiring, in chronological order, video information in the visual field of an instructor performing the task and coordinate information indicating the instructor's viewpoint within the visual field, and linking the information to the task; an identification step for identifying the instructor's gaze mode based on the video information and coordinate information acquired by the acquisition step, such that if the time-series coordinate information indicating the instructor's viewpoint displacement within the visual field is concentrated at the center of the visual field, it is set to a wide field of view mode; and if the time-series coordinate information is dispersed outside the center of the visual field, it is set to an alert mode; an extraction step for setting a gaze area based on the gaze mode identified by the identification step, and extracting a gaze image of a work object that the instructor is gazing at within the gaze area; and a storage step for linking the gaze image extracted by the extraction step to the visual field, the viewpoint displacement, and the gaze mode, and storing the gaze image in a database as attention information for the task.
[0016] According to the first to seventh inventions, the identifying means determines the instructor's viewpoint displacement within the field of view based on the video information and coordinate information, and identifies the instructor's gaze mode. Therefore, the extracting means can set a gaze area based on the identified gaze mode, and extract a gaze image of the work object gazed at by the instructor in the gaze area. This makes it possible to extract a gaze image showing what and in what situation the instructor gazed, based on the gaze duration and gaze displacement of the viewpoint, thereby accurately grasping the state of gaze and providing appropriate attention information.
[0017] In particular, according to the second aspect of the present invention, the gaze area includes the instructor's wide-field mode and alert mode. Therefore, it is possible to set the wide-field mode, in which the instructor's viewpoint displacement is concentrated at the center of the field of view, and the alert mode, in which the instructor's viewpoint displacement is dispersed outside the center of the field of view. This makes it possible to extract a gaze image that shows what the instructor is gazing at and in what situation.
[0018] In particular, according to the third aspect of the present invention, the acquisition means further includes a determination means. Therefore, it is possible to determine whether the task object included in the gaze image is correct or incorrect. This makes it possible to accurately grasp the gaze states of the instructor and the worker, and to provide appropriate warning information.
[0019] In particular, according to the fourth aspect of the present invention, the database stores the correlation between previously acquired past gaze image information and reference information indicating the correctness of the work object associated with the gaze image. Therefore, the determination means can refer to the database to determine the correctness of the work object, and the display means can output corresponding information according to the determination result. This makes it possible to accurately grasp the worker's gaze state and provide appropriate warning information.
[0020] In particular, according to the fifth aspect of the present invention, the input means inputs the correspondence information output by the determination means. Therefore, the storage means can associate the correspondence information with the gaze image and store it as an attention data set including setting conditions for recognizing it as a gaze target. This makes it possible to accurately grasp the gaze state and provide appropriate attention information.
[0021] In particular, according to the sixth aspect of the present invention, the display means switches the information displayed in the field of view between before and during acquisition of video information. Therefore, before acquisition of video information, recording date and time information, recording position information, and recording control information of the instructor's work are displayed in the center of the field of view, and during acquisition, only the recording control information is displayed in the corner of the field of view. This makes it possible to acquire video information for the instructor performing the work and extract the gaze image without imposing a burden on the instructor.
[0022] In particular, according to the seventh aspect of the present invention, the display means switches between the acquisition display area and the warning display area. Therefore, based on the warning data set and the determination result, the corresponding information, warning information, or nudge information can be displayed according to the gaze state of the worker performing the work. This makes it possible to accurately grasp the gaze state of the worker and provide appropriate warning information.
[0023] According to the eighth aspect of the present invention, the identifying step determines the instructor's viewpoint displacement within the field of view based on the video information and coordinate information, and identifies the instructor's gaze mode. Therefore, the extracting step can set a gaze area based on the identified gaze mode, and extract a gaze image of the work object gazed at by the instructor in the gaze area. This makes it possible to extract a gaze image showing what and in what situation the instructor gazed, based on the gaze duration and gaze displacement of the viewpoint, thereby accurately grasping the gaze state and providing appropriate attention information.
[0024] FIG. 1 is a schematic diagram showing an example of the configuration of an attention extraction system according to this embodiment. FIG. 2 is a schematic diagram showing an example of attention extraction by an instructor and task evaluation by an operator in the attention extraction system 100 according to this embodiment. FIGS. 3(a) to 3(c) are schematic diagrams showing an example of an attention extraction method according to this embodiment. FIG. 4(a) is a schematic diagram showing an example of the configuration of an attention extraction device, and FIG. 4(b) is a schematic diagram showing an example of the functions of the attention extraction device. FIG. 5 is a schematic diagram showing an example of a database according to this embodiment. FIG. 6 is a schematic diagram showing a first modified example of the database according to this embodiment. FIG. 7 is a schematic diagram showing an example of a data table stored in the database according to this embodiment. FIG. 8 is a schematic diagram showing an example of a flowchart illustrating the attention extraction method according to this embodiment. FIGS. 9(a) to 9(e) are schematic diagrams showing an example of the attention extraction method according to this embodiment. FIGS. 10(a) and 10(b) are schematic diagrams showing an example of a display on the operator device according to this embodiment. FIG. 11 is a schematic diagram showing an example of a display on the attention extraction device according to this embodiment. 12A to 12E are schematic diagrams showing examples of displays on the worker terminal in this embodiment.
[0025] An example of an attention extraction system and an attention extraction method according to an embodiment of the present invention will be described below with reference to the drawings.
[0026] (Present embodiment: attention extraction system 100 and attention extraction method) An example of the configuration of the attention extraction system 100 in this embodiment will be described with reference to Figures 1 and 2. Figure 1 is a schematic diagram showing an example of the configuration of the attention extraction system 100 in this embodiment, and Figure 2 is a schematic diagram showing an example of attention extraction by an instructor and task evaluation by a worker in the attention extraction system 100 in this embodiment.
[0027] The attention extraction system 100 is used to extract the object of a task that the instructor should pay attention to, using image information in the visual field of the instructor performing the task and coordinate information indicating the viewpoint at which the instructor is gazing within the visual field. The attention extraction system 100 can acquire image information and coordinate information under various conditions within the instructor's field of view for various types of information including image information, coordinate information, and task information related to the task.
[0028] 1 , the attention extraction system 100 includes an attention extraction device 1, an instructor device 2, a worker device 3, and a server 4, and for example, a plurality of instructor devices 2 and a plurality of worker devices 3 may be provided in a work area 50. The attention extraction system 100 may transmit and receive various information to and from the attention extraction device 1, the instructor device 2, the worker device 3, the server 4, other user devices (not shown), and the like via, for example, a known communication network 5.
[0029] The attention extraction system 100 acquires a viewpoint displacement 2b of the field of view that the instructor is gazing at, which is included in the instructor's field of view 2a, via an instructor device 2 worn by the instructor, for example. The information acquired by the attention extraction system 100 from the instructor includes, for example, video information (video within the field of view where the work is being performed), coordinate information (viewpoint, position information, etc.), work information (work date and time, work instructions, process information, etc.), instructor information (instructor ID, device ID, etc.), and may also include various information about the work that the instructor is performing (assigned work, group work, etc.).
[0030] 2, the attention extraction system 100 acquires, in chronological order, image information within the visual field of the instructor performing a task and coordinate information indicating the viewpoint at which the instructor is gazing within the visual field, in association with the task performed by the instructor, from an instructor device 2 worn by the instructor. The image information and coordinate information may be acquired using, for example, known eye-tracking technology, a head-mounted display, or a technology for acquiring a viewpoint provided in smart glasses, etc.
[0031] The attention extraction system 100 acquires video information and coordinate information, for example, using the aforementioned technology provided in the instructor device 2, identifies the instructor's gaze mode in the attention extraction device 1, and extracts a gaze image based on the identified gaze mode. The attention extraction system 100, for example, associates the extracted instructor's gaze image with reference information associated with the gaze image, and stores the association in a database as an attention dataset. The attention extraction system 100 may, for example, refer to the database to acquire attention information associated with the reference information, and display the attention information on the operator device 1 of the operator using the attention dataset.
[0032] The attention extraction system 100 then evaluates the worker's work based on, for example, an attention dataset selected by the worker, video information of the work object 6 included in the worker's field of view 3a, numerical information and coordinate information of the viewpoint displacement at the end of the field of view, and identifies, for example, the work object 6 as gaze information based on the evaluation results. It also obtains corresponding information such as "(1) Check" and "(2) Adjust," and displays this information together with the actual display by superimposing it on the field of view 3a of the worker device 3. Details of the configurations of the attention extraction device 1, instructor device 2, and worker device 3 will be described later.
[0033] Here, the identification of the instructor's gaze mode will be described with reference to Fig. 3. The attention extraction system 100 identifies the instructor's gaze mode using an attention extraction method such as those shown in Figs. 3(a) to 3(c). As shown in Fig. 3(a), for example, the attention extraction system 100 identifies the instructor's gaze mode in the instructor's visual field based on video information in the visual field 2a acquired by the instructor device 2 and coordinate information (x-axis, y-axis) of the instructor's viewpoint displacement.
[0034] The attention extraction system 100 determines the instructor's gaze displacement within the field of view 2a from the coordinates based on the video information and coordinate information acquired via the instructor device 2, and identifies the instructor's gaze mode. For example, as shown in FIG. 3B, the attention extraction system 100 may identify the instructor's gaze displacement as a "wide field of view mode" if the instructor's gaze displacement is characterized by a "narrow displacement range (concentrated)" and "few points of attention for the gaze," respectively. Furthermore, as shown in FIG. 3C, the attention extraction system 100 may identify the instructor's gaze displacement as a "vigilant mode" if the instructor's gaze displacement is characterized by a "wide displacement range (dispersed)" and "many points of attention for the gaze," respectively.
[0035] The attention extraction system 100 sets a gaze area where the instructor gazes based on the identified gaze mode, extracts a gaze image of the work object where the instructor gazes in the set gaze area, links the extracted gaze image with the field of view, viewpoint displacement, and gaze mode, and stores it in a database of the server 4 as attention information for the instructor's work. Note that there may be multiple instructors and workers, and for example, multiple instructors and workers may work on the same process, and the attention extraction system 100 may assign, for example, one work process of the instructor so that multiple workers share the work.
[0036] The assignment of instructors and workers by the attention extraction system 100 may be determined based on, for example, the number of steps in the work process, the difficulty level, the skills of the workers, the work deadline, etc., or may be done by, for example, an evaluator via the attention extraction device 1. It is arbitrary how many workers are assigned to which tasks and how the workers are arranged, and may be specified as appropriate.
[0037] <Attention extraction device 1> The attention extraction device 1 determines the displacement of the instructor's viewpoint within the field of view based on, for example, video information within the field of view of the work performed by the instructor acquired by the instructor device 1 and coordinate information indicating the viewpoint at which the instructor is gazing within the field of view, and identifies the instructor's gaze mode.
[0038] The gaze area identified by the identification means includes, for example, a wide field of view mode in which the instructor's viewpoint displacement is within a field of view radius of 5 to 20 degrees and is concentrated at the center of the field of view range, and an alert mode in which the viewpoint displacement is within a field of view radius of 4 degrees and is dispersed outside the center of the field of view range.
[0039] The attention extraction device 1 identifies a case where the instructor's gaze displacement characteristics are "narrow displacement range (concentrated)" and "few points of attention for the viewpoint" as a "wide field of view mode," as shown in, for example, the above-mentioned Figure 3(b), and on the other hand, identifies a case where the instructor's gaze displacement characteristics are "wide displacement range (dispersed)" and "many points of attention for the viewpoint" as an "alert mode," as shown in, for example, the above-mentioned Figure 3(c).
[0040] The attention extraction device 1 sets a gaze area based on the specified gaze mode, for example, and extracts a gaze image of the work object gazed at by the instructor in the gaze area. The attention extraction device 1 associates the extracted gaze image with the field of view, viewpoint displacement, and gaze mode, for example, and stores it in a database as attention information for the work.
[0041] 11 shows an example of the attention extraction device screen 1a of the attention extraction device 1 in the attention extraction system 100. Fig. 11 is a screen operated by, for example, an evaluator, and displays various information acquired by the instructor device 1.
[0042] The attention extraction device 1 refers to a database, for example, and displays on the attention extraction device screen 1a a gaze monitoring area that includes at least a gaze display area 1b (viewpoint movement distance graph, gaze tracking, gaze area, gaze target, etc.) that displays coordinate information, gaze area, gaze image, and gaze shift in chronological order, a target setting area 1c (gazing target evaluation, recognizer generation, etc.) that displays setting information for setting the gaze image displayed in the gaze display area 1b, and a judgment setting area 1d (corresponding information setting, etc.) that displays judgment conditions for the setting information displayed in the target setting area 1c.
[0043] The attention extraction device 1 receives, for example, conditions and numerical values related to various settings and adjustments from an evaluator via a setting item menu displayed on the attention extraction device screen 1 a. Based on the received conditions and numerical values, for example, the attention extraction device 1 sets and adjusts the display and conditions in the gaze display area 1 b, the target setting area 1 c, and the judgment setting area 1 d.
[0044] The attention extraction device 1 determines whether the task object included in the instructor's gaze image is correct or incorrect in accordance with various conditions received from the evaluator via, for example, a setting item menu displayed on the attention extraction device screen 1 a. The attention extraction device 1 may display the determination result on the instructor device 2 or the worker device 3.
[0045] The attention extraction device 1 accepts input of correspondence information from the evaluator via, for example, a setting item menu displayed on the attention extraction device screen 1 a. In addition to accepting input of new correspondence information, the attention extraction device 1 may also update or delete existing conditions, settings, correspondence information, etc., and store them in the database of the server 4.
[0046] The attention extraction device 1 may, for example, link the inputted corresponding information with the gaze image, generate an attention dataset including setting conditions for recognizing it as a gaze target, and store the generated attention dataset in a database of the server 4.
[0047] 4A is a schematic diagram showing an example of the configuration of the attention extraction device 1. A single-board computer such as a Raspberry Pi (registered trademark) may be used as the attention extraction device 1, or a known electronic device such as a personal computer (PC) may be used. The attention extraction device 1 includes, for example, a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage unit 104, and I / Fs 105 to 107. The components 101 to 107 are connected via an internal bus 110.
[0048] The CPU 101 controls the entire attention extraction device 1. The ROM 102 stores operation code for the CPU 101. The RAM 103 is a working area used when the CPU 101 is operating. The storage unit 104 stores various information such as learning models and databases. As the storage unit 104, a known data storage medium such as an SD memory card, a hard disk drive (HDD), a solid state drive (SSD), etc. may be used.
[0049] The I / F 105 is a known interface for transmitting and receiving various information to and from the instructor device 2, the worker device 3, the server 4, the communication network 5, etc., which are connected depending on the application. For example, a plurality of I / Fs 105 may be provided.
[0050] The I / F 106 is a known interface for transmitting and receiving various types of information to and from an input unit 108 connected depending on the application. For example, a keyboard is used as the input unit 108, and an administrator or the like who manages the attention extraction system 100 inputs or selects various types of information or control commands for the attention extraction device 1 via the input unit 108.
[0051] The I / F 107 is a known interface for transmitting and receiving various information to and from a display unit 109 connected depending on the application. The display unit 109 outputs various information stored in the storage unit 104, the processing status of the attention extraction device 1, etc. As the display unit 109, for example, a display is used, and it may be of a touch panel type, for example. In this case, the display unit 109 may be configured to include the input unit 108.
[0052] Note that, for example, the same I / F may be used as I / F 105 to I / F 107, or, for example, a plurality of I / Fs may be used as each of I / F 105 to I / F 107. Furthermore, at least one of the instructor device 2, the worker device 3, the server 4, the communication network 5, the input unit 108, and the display unit 109 may be detached depending on the situation.
[0053] <Storage Unit 14 (Database)> The storage unit 14 stores various databases, for example, in the saving unit 104. The storage unit 14 stores associations between previously acquired past evaluation target information and reference information linked to the past evaluation target information, and stores, for example, a learning model having associations. The storage unit 14 may be provided in, for example, the instructor device 2, the worker device 3, and the server 4.
[0054] The memory unit 14 records the instructor's attention behavior (points of attention, viewpoint, etc.) when, for example, covering the instructor's work. The attention extraction system 100 may determine the gaze mode based on information such as a gaze image, a field of view, and viewpoint displacement acquired via the instructor device 2. For example, the attention extraction system 100 may determine a "wide field of view mode" when the instructor's gaze displacement characteristics are "narrow displacement range (concentrated)" and "few viewpoint points of attention." On the other hand, as shown in FIG. 3( c) above, the attention extraction system 100 may determine a "vigilance mode" when the instructor's gaze displacement characteristics are "wide displacement range (dispersed)" and "many viewpoint points of attention." This information may be stored in the memory unit 14 of the server 4 as attention information for the instructor's work.
[0055] The storage unit 14 stores various information related to the instructor's work, such as information acquired by gestures, voice, gaze, and other input devices, via various input interfaces (not shown) of the instructor device 2. The attention extraction system 100 associates various information acquired via the input devices with information about the instructor and information about the instructor's work, and stores the information in the storage unit 14. The attention extraction system 100 may record various information appropriately in response to recording instructions, such as recording start and recording end, issued by the instructor via the instructor device 2.
[0056] The memory unit 14 may sequentially record, for example, via the instructor device 2, the coordinate position set for the viewpoint supplemented by the instructor, changes in the coordinate position, the rate of change, etc., as well as, for example, the position and orientation of the head in the workspace coordinate system for each spatial layout of the location where each task is performed. The attention extraction system 100 may acquire various information recorded for each spatial layout using an imaging device such as a well-known 360° camera, surveillance camera, or network camera, and store the information in the memory unit 14 in association with the coordinate position of the instructor's head and information indicating changes in the coordinate position.
[0057] Furthermore, after recording to the database is completed, the storage unit 14 may record various data recorded in association with the image data and the photographed data as an attention data set for the instructor's work. The attention data set is, for example, evaluated by an evaluator via the attention extraction device 1 in terms of attention and know-how, and is registered (recorded) in the storage unit 4 as correspondence information for the attention object recognizer and the monitoring of the attention object.
[0058] Here, for example, the attention object recognizer is a device generated using attention object video evaluated by the attention extraction system 100, and may be, for example, a publicly known image processing program such as pattern matching, or an image recognition model. The image processing program or image recognition model may be installed in, for example, the worker device 3 that the worker wears while performing work. This enables the worker to evaluate attention information for work, for example, based on time-series video information and coordinate information acquired via the worker device 3 while performing work.
[0059] The storage unit 14 may further include a database storing, for example, correlations between previously acquired past gaze image information and reference information indicating the correctness of the work object associated with the gaze image. The storage unit 14 may be referenced by, for example, the determination unit 15 and used when determining the correctness of the work object in the instructor's work. The determination unit 15 may, for example, obtain corresponding correspondence information according to the determination result. The display unit 16 may further output corresponding correspondence information from the storage unit 14 according to the content determined by the determination unit 15.
[0060] In addition, the memory unit 14 may be configured to register, for example, the gaze target recognizer acquired by the instructor device 2 and evaluated as attention know-how by the evaluation unit 15 and the corresponding information in monitoring the gaze target as an attention data set for work in the worker device 13.
[0061] The storage unit 14 may store, for example, past evaluation target information and reference information acquired by the instructor device 2. The correlation is constructed by machine learning using a plurality of learning data sets, for example, with the past evaluation target information and the reference information as a set of learning data. As a learning method, for example, deep learning such as a convolutional neural network is used.
[0062] In this case, for example, the association indicates the degree of connection between many-to-many information (a plurality of data included in the past evaluation target information, versus a plurality of data included in the reference information).
[0063] The correlation is updated as appropriate during the machine learning process. That is, the correlation represents a function optimized based on, for example, past image data A, corresponding information A, and reference information. Therefore, an evaluation result for the evaluation target information is generated using the correlation constructed based on all past evaluation results of the evaluation target information. This makes it possible to generate an optimal evaluation result even when the evaluation target information has a configuration state in which it is combined with other image data A, corresponding information A, etc.
[0064] Furthermore, the evaluation target information stored in the storage unit 14 may be the same as or similar to past evaluation target information, or may be dissimilar. This allows the attention extraction system 100 to quantitatively generate optimal evaluation results. Furthermore, the attention extraction system 100 can improve generalization capabilities when performing machine learning, thereby improving evaluation accuracy for unknown evaluation target information.
[0065] The correlation may include multiple correlation degrees indicating the degree of connection between multiple data included in the past evaluation target information and multiple data included in the reference information. For example, when the learning model is constructed using a neural network, the correlation degrees can be associated with weight variables.
[0066] The past evaluation target information indicates information of the same type as the above-mentioned evaluation target information, and includes, for example, a plurality of pieces of evaluation target information acquired when the image data A was evaluated in the past.
[0067] The reference information is linked to past evaluation target information and indicates information related to image data A and corresponding information A. The reference information indicates, for example, the scope of work for image data A, work instructions, work procedures, details of subsequent processes and related work, details of shared work and alternative work of other workers working in the same area, evaluations based on various work relationships and mutual influences (e.g., "check," "adjust," "pay attention (WATCH!)," "judge," "respond," etc.), and may also include various information related to the progress of the work process, checklists, images of interest, priority of displayed information, combinations of displayed and hidden information, numerical values, data, distributions, etc. The specific content included in the reference information can be set arbitrarily.
[0068] The correlation may indicate the degree of connection between the past evaluation target information and the reference information, as shown in Fig. 5, for example. In this case, by using the correlation, the degree of relationship between each of the multiple data included in the reference information ("Reference A" to "Reference C" in Fig. 5) and each of the multiple data included in the past evaluation target information ("Image Data A" to "Image Data C" in Fig. 5) can be linked and stored. Therefore, for example, multiple data included in the reference information can be linked to one piece of data included in the past evaluation target information via the correlation, thereby enabling the generation of multifaceted evaluation results.
[0069] The correlation has, for example, multiple correlations linking multiple data included in the past evaluation target information with multiple data included in the reference information. The correlation is expressed in three or more levels, such as a percentage, a 10-point scale, or a 5-point scale, and is expressed, for example, by line characteristics (e.g., thickness, etc.). For example, "image data A" included in the past evaluation target information shows a correlation AA of "80%" with "reference A" included in the reference information, and a correlation AB of "55%" with "reference B" included in the reference information. In other words, the "correlation" indicates the degree of connection between each piece of data; for example, the higher the correlation, the stronger the connection between the pieces of data. Note that when constructing the correlation using the above-mentioned machine learning, the correlation may be set to have three or more levels of correlation.
[0070] For example, as shown in Fig. 6, the past evaluation target information may be divided into past gaze images of the work object and past correspondence information of the work object, and the divided information may be stored in the database. In this case, the correlation degree is calculated based on the relationship between the combination of image data of the past gaze images and past correspondence information, and the reference information. In addition to the above, the past evaluation target information may also be divided into past countermeasure information and stored in the database.
[0071] For example, the combination of "image data a" included in the past gaze image and "correspondence information a" included in the past correspondence information indicates a correlation degree AAA of "85%" with "Reference A" and a correlation degree ABA of "25%" with "Reference B." In this case, the past target data and the past correspondence information can be stored independently. This makes it possible to improve accuracy and expand the range of options when generating evaluation results.
[0072] The past evaluation target information may include, for example, composite data and a similarity. The composite data is represented by three or more levels of similarity between the composite data and the past target data or the past corresponding information. The composite data may be stored in the database in the form of a numerical value, a matrix, a histogram, or the like, or may be stored in the form of an image, a character string, or the like.
[0073] An example of the database in this embodiment will now be described with reference to Fig. 7. The database stores, as attention datasets, various types of information related to users of the attention extraction system 100, such as instructors (instructing workers), evaluators, and workers, as well as information related to the content and processes of various tasks and multiple datasets used by workers.
[0074] In the attention data set, various information such as information on each data table, tasks by instructors and workers, and evaluations by evaluators are associated with the respective data. The evaluations by the evaluators are input, for example, by the input unit 17. The attention data set stores at least, for example, a "task information table," a "task procedure table," a "correspondence information table," a "teaching task record table," a "viewpoint record table," a "viewpoint record data table," and a "gazing target table."
[0075] <<Work Information Table>> The "Work Information Table" stores (contains) data for identifying the work to be performed, for example. The "Work Information Table" stores, for example, a "Work ID" and a "Work Name" for identifying the work to be performed by an instructor or worker.
[0076] <<Work Procedure Table>> The "work procedure table" stores (contains) data related to the procedures of work to be performed by, for example, an instructor or a worker. The "work procedure table" stores, for example, a "work procedure ID" that identifies the procedure of the work, and "work ID," "work procedure name," and "work sequence" that are linked to the "work procedure ID" and correspond to each other. The "work procedure table" is linked to the "work information table" by, for example, the "work ID."
[0077] <<Corresponding Information Table>> The “corresponding information table” stores (contains) data for identifying information corresponding to the work to be displayed by the display means, for example, when a worker performs work, after the judgment means has judged whether the work object is correct or incorrect based on the image information and viewpoint coordinate information acquired by the acquisition means.
[0078] The "corresponding information table" stores, for example, a "corresponding information ID" and a "work procedure ID" for identifying information for responding to work, "advance" indicating nudge actions or the presentation of advance confirmation information, and a "display type" indicating a display type such as an "interrupt type" that is displayed depending on the work situation, including oversights.
[0079] In the "display type," a trigger indicating the start timing, the display time, or the end of the display is set, such as in the timing setting of an animation effect. If the display type is, for example, "interrupt," information indicating the determination timing for interruption is stored as the "display / determination start condition." The determination start condition may, for example, specify a trigger for interruption determination.
[0080] Furthermore, the "corresponding information table" stores "determination criteria" that indicate the criteria and thresholds that the instructor uses to determine whether or not the object has been seen when the "interrupt determination" process is started, and the display content that is displayed as the corresponding information as "corresponding information content."
[0081] The "correspondence information content" may be stored with correspondence information created at the work site, for example, by specifying a file name, or by using Markdown, etc. The "correspondence information content" may be stored with one-off instructions, such as contacting a superior or an experienced worker, and may be generated by inputting the instructions via an editing screen.
[0082] Furthermore, the "corresponding information table" stores "work result record display conditions" that indicate conditions such as the timing for displaying content that records the results of each work procedure, such as work reports and checklists, and "work result record content" that indicates content for recording work results, such as work reports and checklists.
[0083] The attention extraction system 100 may refer to the "work result record display conditions" and determine by image comparison that the gaze target has reached an appropriate state based on a signal, for example, a gesture or voice input. The attention extraction system 100 may store various information and data in the "work result record display conditions" as conditions for storing the determination results, for example.
[0084] The attention extraction system 100 may require the input of the results of the content in the "work result record content," for example. The attention extraction system 100 may determine, for example, this result input and determine that one work procedure has been completed based on the result of the determination. The "correspondence information table" is linked to the "work procedure table," for example, by the "work procedure ID."
[0085] <<Teaching Work Record Table>> The "Teaching Work Record Table" stores (contains), for example, various information related to work performed by an instructor. The "Teaching Work Record Table" stores, for example, a "Teaching Work Record ID" for identifying the recorded teaching work, a "Teaching Work Date and Time" indicating the date and time when the teaching work was performed, a "Teaching Worker" indicating the instructor who performed the teaching work, a "Teaching Work Location" indicating the location or area where the teaching work was performed, and a "Work ID" linked to each other. The "Teaching Work Memory Table" is linked to the "Work Information Table" by, for example, the "Work ID."
[0086] <<Viewpoint Record Table>> The “viewpoint record table” stores information and data related to the viewpoint from which the instructor performs work, acquired by, for example, the acquisition unit 11. The “viewpoint record table” stores a “viewpoint record ID” that identifies the viewpoint record of the work performed by the instructor, and data on the coordinate information of the viewpoint obtained by eye tracking, as a “viewpoint record data file.”
[0087] The viewpoint record table may store stream data acquired in millisecond units, which allows the attention extraction system 100 to store the actual file name of data such as JavaScript Object Notation (JSON) instead of directly storing the data in a database.
[0088] The "viewpoint record table" also stores a "viewpoint range record video file" that indicates video information data within the instructor's field of view, as recorded by an outer camera, for example. The "viewpoint record table" also stores a "teaching work record ID" that identifies the history of teaching work by the instructor. The "viewpoint record table" is linked to the "teaching work record table" by, for example, the "teaching work record ID."
[0089] <<Viewpoint Recording Data Table>> The "viewpoint recording data table" stores, for example, multiple pieces of detailed information about the coordinate information of the viewpoint acquired by the acquisition unit 11 when an instructor performs a task. The "viewpoint recording data table" stores a "viewpoint identification data file" that identifies a file storing the viewpoint recording data, and a "viewpoint recording elapsed time" that indicates the time of viewpoint identification recorded by the instructor. The "viewpoint recording elapsed information" stores (retains) the elapsed time since the viewpoint was recorded, with the start of recording when the instructor started the task as 00:00:00, for example. The "viewpoint recording elapsed information" may be recorded, for example, at unit time intervals (e.g., one-second intervals) together with various position information. The "viewpoint recording data table" is linked to the "teaching work record table" by, for example, the "viewpoint identification data file."
[0090] The "viewpoint recording data table" also stores position information of the teaching operator in the workspace as "teaching operator position information (X, Y, Z)." The "teaching operator position information (X, Y, Z)" may, for example, record the position of the HMD worn by the teaching operator, i.e., the three-dimensional coordinates of the head position information, and may also store information such as "teaching operator orientation information (rad)," "teaching operator eyeball position information (x, y, z)," and "teaching operator eyeball angle information (rad)" acquired by the acquisition unit 11. The "viewpoint recording data table" is linked to the "teaching work record table" by, for example, a "viewpoint recording data file."
[0091] <<Gaze Target Table>> The "gaze target table" stores (contains) various information related to the object that the instructor is gazing at, for example. A plurality of various pieces of information set by the evaluator are stored. The "gaze target table" stores, for example, a "gaze target ID" that identifies the object that the instructor is gazing at, a "field of view range recording video file," a "viewpoint recording data file," a "viewpoint recording elapsed time," and a "work procedure ID."
[0092] In addition, the "gaze target table" stores "gaze field range image" as a still image extracted from the "gaze field range recording video file" at a moment during the viewpoint recording elapsed time. In addition, the "gaze target position (x, y)" indicating the position of the gaze target extracted by the extraction unit 13, the "gaze target range (w, h)" indicating the range, and the "gaze target image" indicating the image of the gaze target may be stored as attention information for the work, based on the gaze mode identified by the identification unit 12 based on the video information and coordinate information associated in the "gaze target recording table".
[0093] The attention information may be generated by the identification unit 12 and the extraction unit 13 based on information acquired by the acquisition unit 11, for example, and then evaluated, set, and recorded by an evaluator. The "gaze target table" is linked to the "viewpoint recording table" by the "gaze target ID," the "work procedure table" by the "work procedure ID," and the "viewpoint recording data table" by the "viewpoint recording data file," for example.
[0094] Fig. 4(b) is a schematic diagram showing an example of the functions of the attention extraction device 1. The attention extraction device 1 includes, for example, an acquisition unit 11, an identification unit 12, an extraction unit 13, a storage unit 14, a determination unit 15, a display unit 16, an input unit 17, and a monitoring display unit 18. Note that each function shown in Fig. 5(b) is realized by the CPU 101 using the RAM 103 as a working area to execute a program stored in the storage unit 104 or the like.
[0095] <<Acquisition Unit 11 (Acquisition Means)>> The acquisition unit 11 acquires video information within the visual field of an instructor performing a task and coordinate information indicating the viewpoint at which the instructor gazes within the visual field, in chronological order, by linking the information to the task. The acquisition unit 11 is used, for example, when performing acquisition step S110, which will be described later. The timing at which the acquisition unit 11 acquires the video information and coordinate information from the instructor device 2 can be set arbitrarily. The acquisition unit 11 links the acquired video information and coordinate information to the task performed by the instructor and stores them in chronological order in the storage unit 104, for example, via the memory unit 14.
[0096] <<Identification Unit 12 (Identification Means)>> The identification unit 12 obtains the instructor's viewpoint displacement in the field of view range and identifies the instructor's gaze mode based on the video information and coordinate information acquired by the acquisition unit 11. The identification unit 12 is used, for example, when performing an identification step S120 described below. For example, as shown in FIG. 3 above, the identification unit 12 identifies the instructor's gaze mode in the field of view range based on the video information in the field of view range 2a acquired by the instructor device 2 and the coordinate information (x-axis, y-axis) of the instructor's viewpoint displacement.
[0097] Next, if the information acquired by the acquisition unit 11 has characteristics or tendencies different from the already stored "wide field mode" and "alert mode," the identification unit 12 may identify the information as a new mode. The identification unit 12 can arbitrarily set various modes using the attention extraction device 1. For example, via the storage unit 14, the identification unit 12 changes the acquired existing mode and also stores the new mode in the storage unit 104 in chronological order, linking it with the instructor and the work performed by the instructor. Note that the identification unit 12 may identify the various modes of the instructor by, for example, an evaluator setting the various modes based on information acquired by the acquisition unit 11 and stored in the database.
[0098] <<Extraction unit 13 (extraction means)>> The extraction unit 13 sets a gaze area based on the gaze mode identified by the identification unit 12, and extracts gaze images of the work object 6 gazed at by the instructor in the gaze area. For example, when there are multiple gaze modes identified by the identification unit 12, the extraction unit 13 may determine the gaze images of the work object 6 included in the set gaze area based on each gaze mode, and extract them in chronological order.
[0099] The extraction unit 13 extracts gaze images of the work object 6 included in the set gaze area based on, for example, a gaze mode, and if there is an order to the instructor's work or confirmation, it extracts them in chronological order with emphasis on the order, or if there is no order, it may link multiple gaze images to one work and extract them.
[0100] The extraction unit 13 may output the extraction result to the storage unit 14, the attention extraction device 1, or the like, for example, via the communication network 5. The extraction unit 13 links the extracted gaze image with the instructor, the task performed by the instructor, the gaze mode, etc., via the storage unit 14, for example, and stores the linked image in the storage unit 104. Note that the linking of the gaze image with the gaze mode, etc., by the extraction unit 13 may be set by the evaluator based on various information stored in the storage unit 104.
[0101] <<Storage Unit 14 (Storage Means, Database)>> The storage unit 14 associates the gaze image extracted by the extraction unit 13 with the field of view range, viewpoint displacement, and gaze mode, and stores the association in a database as attention information for the instructor's work. The storage unit 14 stores various information in the storage unit 104 or retrieves various information from the storage unit 104.
[0102] The storage unit 14 stores multiple correlations between previously acquired past gaze image information and reference information indicating the correctness of the work object linked to the gaze image. The storage unit 14 stores or retrieves various information according to the processing content of, for example, the acquisition unit 11, the identification unit 12, the extraction unit 13, the determination unit 15, the display unit 16, the input unit 17, and the monitoring display unit 18.
[0103] <<Determination Unit 15 (Determination Means)>> The determination unit 15 determines the correctness of a work object included in a work gaze image acquired, for example, via the worker's worker device 3, using a work-related attention dataset. The determination unit 15 refers to a database stored in the storage unit 14 or the storage unit 104, and determines the correctness of a work performed by the worker in relation to gaze information (for example, work object 6) based on video information acquired via the worker's worker device 1 by referring to the database.
[0104] The determination unit 15 acquires various types of response information from a database depending on, for example, the result of the determination. The determination unit 15 acquires information related to the work that the worker should perform, such as the target information ("(1) Confirm OK; 01234" and "(2) Adjustment Procedure: XXX") displayed in the field of view 3a of the worker device 3 in FIG. 2 described above. The determination unit 15 transmits, for example, the acquired response information to the display unit 16 of the worker device 3.
[0105] For example, when multiple workers are performing the same task in the same task area, the determination unit 15 may collectively determine whether the task objects of the multiple workers are correct. In this case, for example, the multiple workers may set a common task attention data set in advance in each of the worker devices 3. For example, the determination unit 15 may acquire multiple pieces of coordinate information for each task workspace layout from the coordinate information acquired by the acquisition unit 11 based on the relationship between coordinate information indicating the position and orientation of each worker's head and coordinate information indicating the task workspace, and determine the tasks and task objects of each worker comprehensively or on a group basis.
[0106] Furthermore, the determination unit 15 acquires status information such as whether the original worker is facing the correct direction toward the work object that the original worker should be working on, and if not (unaware of it), whether there are other workers who can provide support, whether there are other workers facing the work object that the original worker should be working on, etc. Based on the acquired information and the information on the work process of each worker, for example, the determination unit 15 may transmit corresponding information to the worker device 3 of another worker working in the vicinity who can work on the work object appropriately, even if the original worker is not paying attention.
[0107] <<Display Unit 16 (Display Means)>> The display unit 16 outputs various pieces of correspondence information acquired by, for example, the determination unit 15 to the worker device 3 of the worker performing the corresponding work, using an attention data set generated based on the work of the instructor. The display unit 16 (display portion 109 of the worker device 3) displays, for example, the various pieces of correspondence information transmitted from the determination unit 15 in the field of view 3a of the worker device 3 worn by the worker, superimposing the display of the correspondence information on an actual image.
[0108] For example, when multiple workers are working on the same task in the same work area, the display unit 16 appropriately displays the exercise information obtained from the judgment unit 15 according to the work situation and work position of each worker.
[0109] The display unit 16 also displays on the instructor device 2 a display that extracts attention information for the task, as shown in Fig. 9 . Fig. 9 is a schematic diagram showing an example of an attention extraction method in this embodiment. Before the acquisition of video information by the observer unit 11, the display unit 16 displays recording date and time information, recording position information, and recording control information in the center of the field of view 2a, as shown in Fig. 9(a). Furthermore, while the acquisition unit 11 is acquiring video information, the display unit 16 may switch to displaying only the recording control information (e.g., "recording paused," "recording ended," etc.) in a corner of the field of view 2a.
[0110] 9(c) to 9(e), the display unit 16 may display indications for each operation in the center of the field of view 2a of the instructor device 2. These indications by the display unit 16 may be, for example, the work process planned by the instructor, the results of the work performed by the instructor, and the timing of the end or completion of the work detected by a position detection sensor provided in advance in the instructor device 2.
[0111] The display unit 16 displays, for example, displays such as those shown in Figures 10(a) and (b) and Figures 12(a) to (e) on the worker device 3. Figures 10(a) and (b) are schematic diagrams showing an example of extracted attention displayed when a worker performs work via the worker device 3, for example, using the above-mentioned attention dataset in this embodiment, and Figures 12(a) to (e) are schematic diagrams showing an example of display on the worker device 3 (worker terminal, smartphone, display device, etc.) held by a worker in this embodiment.
[0112] The display unit 16 displays the determination result by the determination unit 15, for example, on the gaze target 6a included in the field of view 3a of the worker device 3 worn by the worker, or on a display device held by the worker, in accordance with the determination result by the determination unit 15. Furthermore, the display unit 16 refers to a database and displays corresponding information in accordance with the determination result acquired by the determination unit 15.
[0113] 10(a) is a display of the visual field 3a when, for example, a worker is working and there is no corresponding information for the worker. Also, FIG. 10(b) is a display of the visual field 3a when, for example, a worker is working and there is corresponding information for the worker. The corresponding information displayed in the visual field 3a includes points that the worker should actually pay attention to (correct target for gaze in the visual field 3a: circle + "WATCH!"), procedures and related information that the worker should confirm or refer to (center of the visual field 3a: "Check-3-A," "Check," "Judgment," "Response," "Reference," etc.), and the work status and check results of the worker or co-workers (left side of the visual field 3a: work location, work content, work progress, etc.).
[0114] Furthermore, the display unit 16 may further include an acquisition display area that displays, for example, at the start of work by the worker, the type of acquisition unit 11 that acquires the video information, the type of attention data set stored in the database, and instructions for selecting when to start the work, and an attention information display area that switches between displaying corresponding information corresponding to the worker's gaze image and attention information based on the attention data set selected by the worker.
[0115] Furthermore, the display unit 16 may appropriately display the warning information to be displayed in the field of view 3 a of the worker device 3 as nudge information, for example, at least at the start, middle, or end of work depending on the progress of the worker's work. The display unit 16 refers to the warning database based on, for example, the warning data set and the result of the determination by the determination unit 15, and acquires response information, warning information, nudge information, or the like, and distributes and displays the acquired various types of information according to the gaze state of the worker performing the work.
[0116] The nudge information displayed by the display unit 16 may be displayed as an approach to the worker, such as a nudge default (unconsciously encouraging), a device (naturally encouraging), a nudge that makes the worker aware of the nudge but leaves the choice up to the worker, labeling (intentionally encouraging), or an incentive (encouraging with rewards). The nudge information may be linked to the work process or work progress, the worker's personality, temperament, evaluation, work difficulty, expected completion time, etc., and may be extracted from a database as appropriate and displayed according to the worker's actual work progress.
[0117] Furthermore, the display unit 16 may display the nudge information in a random pattern, such as asking a question, such as "Are you OK?" or "Wait a moment?", in addition to directly displaying the reference information raised as a display candidate as a result of the determination by the determination unit 15. The display unit 16 may also display the nudge information at the center of the field of view 3a of the worker device 3 or at a corresponding location of the gaze target 6a within the field of view 3a.
[0118] Furthermore, the display unit 13 may mix or randomly change the display output methods (visual, auditory, haptics (vibration), nudge information approach (labeling), and nudge information technique (talking, presenting simply, making people wonder, changing completely)) as appropriate depending on the skills of the worker and the progress of the work.
[0119] Furthermore, the display unit 16 may display, for example, heuristics, normality (maintaining the status quo), or variations as related biases for the nudge information to be displayed to prevent overlooking of reference information due to familiarity, boredom, or an expectation to maintain the status quo. The display unit 16 may include, for example, a means (not shown) for grasping the overall flow and procedures of the work being performed by the worker as a prerequisite process for displaying the nudge information, and may further include a means (not shown) for detecting deviations from appropriate attention.
[0120] Furthermore, when a worker finishes a task, the display unit 16 may refer to the database and display words of appreciation to the worker, and the type of nudge information and timing of displaying the information can be determined arbitrarily. This makes it possible to vary the response information displayed to the worker, preventing the worker from overlooking the reference information due to familiarity, boredom, or an expectation to maintain the status quo.
[0121] Furthermore, the display unit 16 displays, for example, a display as shown in Fig. 11 on the attention extraction device 1. Fig. 11 displays, for example, the gaze monitoring area of the attention extraction device 1 in this embodiment as the attention extraction device screen 1a. As described above, the display unit 16 displays, for example, the coordinate information, gaze area, gaze image, and gaze shift in chronological order in the gaze display area 1b when an evaluator refers to a database via the attention extraction device 1 regarding the work performed by an instructor.
[0122] 11, the display unit 16 displays, in a target setting area 1c, setting information for setting the gaze image displayed in the gaze display area 1b. Furthermore, the display unit 16 displays, for example, a judgment condition of the setting information displayed in the target setting area 1c in a judgment setting area 1d.
[0123] The display unit 16 displays, for example, setting information for setting the gaze image displayed in the gaze display area 1b in the target setting area 1c. Furthermore, the display unit 16 displays, for example, judgment conditions for the setting information displayed in the target setting area 1c in the judgment setting area 1d.
[0124] <<Input Unit 17 (Input Means)>> The input unit 17 inputs, for example, response information output by the determination unit 16. The input unit 17 receives, for example, conditions and numerical values related to various settings and adjustments from the evaluator, as well as, for example, via a setting item menu displayed on the attention extraction device screen 1a, for example, displayed on the attention extraction device 1. As a result, for example, the acquisition unit 11, the identification unit 12, the extraction unit 13, the storage unit 14, the determination unit 15, the display unit 16, and the monitoring and display unit 18 perform various processes based on the various conditions and numerical values received by the input unit 17.
[0125] The corresponding information input by the input unit 17 is linked to the gaze image and stored in the database of the server 4 as an attention data set including setting conditions for recognizing it as a gaze target.
[0126] <<Monitoring Display Unit 18 (Monitoring Display Means)>> The monitoring display unit 18 displays, for example, various pieces of information that an evaluator refers to and sets regarding work performed by an instructor as a gaze monitoring area on the attention extraction device screen 1a. The monitoring display unit 18 displays, for example, a gaze display area 1b that displays coordinate information, gaze areas, gaze images, and gaze shifts stored in a database in chronological order, a target setting area 1c that displays setting information for setting the gaze images displayed in the gaze display area, and a judgment setting area 1d that displays judgment conditions for the setting information displayed in the target setting area.
[0127] The attention extraction device 1 acquires video information related to a task via, for example, the instructor's instructor device 2. The task performed by the instructor may be selected in advance for each of various attention datasets set according to the task of the worker, and a gaze dataset for the task performed by the instructor may be generated using the attention dataset corresponding to the selected task.
[0128] The attention data sets extracted by the attention extraction device 1 are selected by, for example, workers A to C, and the work of the workers A to C is evaluated using the selected attention data sets. Note that an evaluator may monitor the work status of the workers A to C who are performing work using, for example, the attention data sets via the attention extraction device 1. The attention extraction device 1 may, for example, acquire the work status of the workers A to C together with the work using the gaze data sets, and may acquire images and position information of the work target using the worker device 1 worn by each worker, and display them on the display unit 16 of the attention extraction device 1.
[0129] The evaluator may refer to the work status of workers A to C displayed on the attention extraction device screen 1a displayed on the attention extraction device 1, for example, and if a task or evaluation other than that in the attention dataset occurs, for example, if worker C is not performing the original task (skipping the task order and being in front of another task), and worker B is in a position where he can confirm the original work of worker C, the evaluator may "ignore" the task corresponding to the gaze information that worker C should be focusing on, and may instruct worker B in real time to "perform" the task corresponding to the gaze information that he should be focusing on.
[0130] <Teacher Device 2> The teacher device 2 is worn by the instructor of the work, as well as by, for example, an expert or qualified person, and acquires video information seen by the instructor's eyes while the instructor is working. The teacher device 2 may be, for example, a known eye-tracking technology, a head-mounted display, or smart glasses, and may acquire audio and ambient sound information, temperature and humidity, position information, spatial information, and the like in addition to video information.
[0131] The instructor device 2 is connected to, for example, the attention extraction device 1, other instructor devices 2, the worker device 3, and the server 4 in a state where data communication is possible, and may also have, for example, the attention extraction device 1 built in.
[0132] <Worker Device 3> The worker device 3 is worn by the worker performing the work or by a person other than the instructor, for example, and acquires video information seen by the worker's eyes through the worker's work. The worker device 3 may be, for example, a known eye-tracking technology, a head-mounted display, or smart glasses, and may acquire audio and ambient sound information, temperature and humidity, position information, spatial information, and the like in addition to video information.
[0133] The worker device 3 is connected to, for example, the attention extraction device 1, the instructor device 2, other worker devices 3, and the server 4 in a state where data communication is possible, and may also have, for example, the attention extraction device 1 built in.
[0134] <Communication Network 5> The communication network 5 indicates, for example, an Internet network to which the attention extraction device 1, the instructor device 2, and the worker device 3 are connected via communication circuits, and may be configured as an optical fiber communication network. The communication network 5 can be realized by a known communication network such as a wired communication network or a wireless communication network.
[0135] <Learning Model> The learning model is generated as a database by, for example, machine learning. The learning model acquires, as a pair of learning data, a learning target image including a gaze image captured using the instructor device 2 and reference information indicating the accuracy of the gaze image captured using the instructor device 2, in multiple pairs for each task in the instructor device 2. The learning model is generated by machine learning using the multiple learning data, from a database in which associations between the multiple learning target images and the multiple pieces of reference information are stored.
[0136] 8 is a schematic diagram showing an example of an attention extraction method according to this embodiment. The attention extraction method includes an acquisition step S110, a specification step S120, an extraction step S130, a storage step S140, and a determination step S150. The attention extraction method can be implemented using an attention extraction system 100.
[0137] <Acquisition Step S110> In the acquisition step S110, for example, video information and coordinate information are acquired in chronological order, linked to the work. In the acquisition step S110, the information is acquired using, for example, an instructor device 2 equipped with a known camera or imaging device. In addition, in the acquisition step S110, the video information and coordinate information may be acquired in chronological order, linked to the work, from, for example, the worker device 3. In the acquisition step S110, the video information and the like are acquired using, for example, a device selected by the instructor and the worker.
[0138] <Identification Step S120> In the identification step S120, a viewpoint displacement of the instructor is obtained and a gaze mode of the instructor is identified. In the identification step S120, a viewpoint displacement in the instructor's visual field range is obtained and the gaze mode of the instructor is identified based on, for example, the image information and the coordinate information acquired in the acquisition step S110.
[0139] In the identification step S120, for example, if the instructor's viewpoint displacement is within the instructor's field of view radius of 5 to 20 degrees and is characterized by being concentrated at the center of the field of view range, the mode is identified as "wide field of view mode." In addition, in the identification step S120, for example, if the instructor's viewpoint displacement is within the instructor's field of view radius of 4 degrees and is characterized by being dispersed outside the center of the field of view range, the mode is identified as "alert mode."
[0140] In the identification step S120, the gaze modes in the field of view 2a of the instructor are identified as a "wide field of view mode" and an "alert mode" based on, for example, image information in the field of view 2a acquired by the instructor device 2 and coordinate information (x-axis, y-axis) of the instructor's viewpoint displacement. In the identification step S120, the gaze modes are identified as a "wide field of view mode" and an "alert mode" based on, for example, the coordinate information of the instructor's viewpoint displacement, but if other characteristics or tendencies can be confirmed, the confirmed gaze mode may be set as a new one.
[0141] <Extraction Step S130> In the extraction step S130, for example, a gaze area is set and an image of the work object gazed at by the instructor is extracted. In the extraction step S130, the gaze area of the instructor is set based on the gaze mode (e.g., "wide field mode" or "alert mode") specified in the specification step S120. In the extraction step S130, for example, in the set gaze area, an image of the work object gazed at by the instructor is extracted from the instructor's video information.
[0142] In the extraction step S130, if there are multiple gaze images in the video information of the instructor's gaze area, the images may be extracted in the order of gaze or the duration of gaze. In the extraction step S130, images of the work object gazed upon by the instructor are extracted from the video information in the gaze area where the instructor gazes, for example, according to conditions set by the evaluator. This makes it possible to grasp the instructor's gaze state and extract appropriate attention information.
[0143] <Storage step S140> In the storage step S140, for example, the gaze image is linked to the field of view, the viewpoint displacement, and the gaze mode, and stored as attention information. In the storage step S140, for example, the gaze image extracted in the extraction step S130 is linked to the field of view, the viewpoint displacement, and the gaze mode, and stored in the database as attention information for the work, as described above.
[0144] <Determination Step S150> In the determination step S150, the correctness of the task object included in the gaze image acquired via the worker device 3 is determined using the attention dataset, for example. In the determination step S150, the correctness of the task object is determined based on the gaze image acquired by the worker device 3 in the acquisition step S110, for example. In the determination step S150, in addition to an evaluation using the attention dataset for each selected task, for example, the determination may be made based on an evaluation by an evaluator monitoring the task via the attention extraction device 1, for example.
[0145] In the determination step S150, for example, when the gaze image is acquired by the worker device 3 in the acquisition step S110, the correctness of the work object on which the worker is working may be determined based on an attention data set selected in advance on the worker device 3. The determination step S150 may determine the work of multiple workers in the same work area.
[0146] The determination step S150 may be configured to simultaneously determine the work of multiple workers using, for example, the same or a common attention data set, thereby making it possible to determine whether a worker has overlooked something, skipped a step, or performed a work step out of order, depending on the work position and work situation of each worker.
[0147] The determination step S150 refers to various data tables stored in a database, such as a "task information table," a "task procedure table," a "correspondence information table," a "teaching task record table," a "viewpoint record table," a "viewpoint record data table," and a "gaze target table," and determines whether the task target is correct or incorrect based on the various data tables and gaze data sets, and acquires various correspondence information stored in the corresponding task or reference link according to the determination result. The correspondence information acquired by the determination step S150 may be displayed on the worker device 3 of the corresponding teaching worker or other workers, for example.
[0148] Furthermore, in the determination step S150, additions or updates of corresponding information in various data tables stored in the database are input via, for example, the input unit 17. The input unit 17 can appropriately input, for example, work processes, determination conditions, corresponding work, reference links, and the like for specific targets and determination targets stored in each of the various data tables.
[0149] This completes the operation of the attention extraction device 1 in this embodiment.
[0150] Furthermore, according to this embodiment, for example, the attention extraction device 1 acquires video information via the acquisition unit 11 of the instructor device 2. The video information may include, for example, recording date and time information of the instructor's work, recording position information, and recording control information related to the operation of acquiring the video information.
[0151] Furthermore, according to this embodiment, for example, the display unit 11 switches the display of the instructor device 2 as shown in Fig. 9. For example, before the acquisition unit 11 acquires video information, the display unit 11 displays the recording date and time information, recording position information, and recording control information in the center of the field of view. Furthermore, while the acquisition unit 11 is acquiring video information, the display unit 11 switches to displaying only the recording control information in a corner of the field of view. This makes it possible to perform a display that does not interfere with the instructor's work.
[0152] Furthermore, according to this embodiment, the display unit 16 displays attention information. The display information is nudge information based on nudge theory at the start, middle, or end of work, depending on information such as the progress and status of the work performed by the worker. This makes it possible to eliminate biases that affect interactions with conventional computer systems and information, and by displaying nudge displays based on nudge theory, it becomes possible to enhance the effectiveness of notifications from the attention extraction system device 1.
[0153] Although the embodiments of the present invention have been described, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. The novel embodiments described above can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The above-described embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as set forth in the claims.
[0154] DESCRIPTION OF SYMBOLS 1: Attention extraction device 1a: Attention extraction device screen 1b: Gaze display area 1c: Target setting area 1d: Judgment setting area 1e: Work space map 1f: Work proxy flag 2: Instructor device 2a: Field of view range (instructor) 2b: Gaze mode 3: Worker device 3a: Field of view range (worker) 4: Server 5: Communication network 6: Work target 6a: Gaze target 10: Housing 11: Acquisition unit 12: Identification unit 13: Extraction unit 14: Storage unit (database) 15: Judgment unit 16: Display unit 17: Input unit 18: Monitoring display unit 50: Work area 100: Attention extraction system S110: Acquisition step S120: Identification step S130: Extraction step S140 : Storage step S150: Determination step
Claims
1. An attention extraction system that extracts attention information for a task, comprising: an acquisition means for acquiring video information in the visual field of an instructor performing the task and coordinate information indicating the viewpoint at which the instructor gazes within the visual field, in a time series, by linking the information to the task; an identification means for identifying the instructor's gaze mode based on the video information and the coordinate information acquired by the acquisition means, wherein the coordinate information in the time series indicating the viewpoint displacement of the instructor within the visual field is concentrated at the center of the visual field, as a wide field mode; and the coordinate information in the time series indicating the viewpoint displacement of the instructor within the visual field, as a warning mode; an extraction means for setting a gaze area based on the gaze mode identified by the identification means, and extracting a gaze image of a work object at which the instructor gazes within the gaze area; and a storage means for linking the gaze image extracted by the extraction means to the visual field, the viewpoint displacement, and the gaze mode, and storing the gaze image in a database as attention information for the task.
2. The attention extraction system of claim 1, further comprising: a determination means for determining whether the work performed by the worker with respect to the work object is correct or incorrect based on video information contained in the determination image and object information related to the work to be performed by the worker that is stored in advance in a database; and a display means for displaying the determination result of the determination means.
3. The attention extraction system of claim 2, further comprising a database storing correlation between previously acquired past gaze image information and reference information indicating the correctness or incorrectness of the work object linked to the gaze image, wherein the judgment means refers to the database to judge the correctness or incorrectness of the work object and acquires corresponding information from the database according to the judgment result, and the display means further outputs the corresponding information acquired by the judgment means.
4. The attention extraction system of claim 2, further comprising an input means for inputting the corresponding information output by the judgment means, wherein the storage means links the corresponding information input by the input means with the gaze image and stores it as an attention data set including setting conditions for recognizing it as a gaze target.
5. The attention extraction system of claim 2, wherein the video information acquired by the acquisition means includes recording date and time information, recording position information, and recording control information related to the operation of acquiring the video information, and the display means displays the recording date and time information, recording position information, and recording control information in the center of the field of view before the video information is acquired by the acquisition means, and switches to only the recording control information and displays it in a corner of the field of view while the video information is being acquired.
6. The display means further comprises an acquisition display area for displaying instructions to the worker performing the work to select the type of the acquisition means for acquiring the video information, the type of attention data set including setting conditions for recognizing it as a gaze target stored in the database, and the start of the work, and an attention information display area for switching between displaying corresponding information corresponding to the worker's gaze image and attention information based on the attention data set after the selection, wherein the attention information displayed in the attention display area includes nudge information displayed at least at the start of work, in the middle of work, or at the end of work, depending on the progress of the worker's work, and the information including at least any of the corresponding information, the attention information, and the nudge information displayed in the attention information display area is displayed in a manner distributed according to the gaze state of the worker performing the work, based on the attention data set and the result of judgment by the judgment means.
7. An attention extraction method for extracting attention information for a task, comprising: an acquisition step of acquiring, in chronological order, image information in the visual field of an instructor performing the task and coordinate information indicating the viewpoint at which the instructor gazes within the visual field, linking the image information and coordinate information acquired by the acquisition step; a specification step of specifying the instructor's gaze mode based on the image information and the coordinate information acquired by the acquisition step, such that a wide-field mode is set when the coordinate information in the time series indicating the viewpoint displacement of the instructor within the visual field is concentrated at the center of the visual field, and a vigilance mode is set when the coordinate information is dispersed outside the center of the visual field; an extraction step of setting a gaze area based on the gaze mode specified by the specification step, and extracting a gaze image of a work object at which the instructor gazes within the gaze area; and a storage step of linking the gaze image extracted by the extraction step to the visual field, the viewpoint displacement, and the gaze mode, and storing the gaze image in a database as attention information for the task.
Citation Information
Patent Citations
Work support system, work support method, and terminal for work
JP2013097466A
Methods for Evaluating Human Performance in Aviation
US20160027336A1
Technique managing device, and technique managing method
WO2010137165A1
