Caution extraction system and caution extraction method
Patent Information
- Application Number
- JP2023078265
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2043-05-11
AI Technical Summary
【0017】 第1発明~第7発明によれば、特定手段は、映像情報及び座標情報に基づいて、視野範囲における教示者の視点変位を求め、教示者の注視モードを特定する。このため、抽出手段において、特定された注視モードに基づいて注視領域を設定し、注視領域において教示者が注視した作業対象の注視画像を抽出することができる。これにより、視点の注視時間と注視変移に基づき、教示者が何を、どのような状況で注視したかを示す注視画像を抽出することができ、注視の状態を正確に把握し、適した注意情報を提供することができる。
Smart Images

Figure 2024162591000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an attention extraction system and an attention extraction method for extracting attention information for an operation. [Background technology]
[0002] Conventionally, as techniques relating to attention extraction, for example, a skill transfer system disclosed in Patent Document 1 and a wiring work support system disclosed in Patent Document 2 have been proposed.
[0003] The skill transfer system disclosed in Patent Document 1 discloses a technology in which first image data and gaze data of a first user's field of view are subjected to machine learning based on teacher data, feature data of the images is extracted, information on the first user's focus point in the first image is input as teacher data and registered in association with the feature data, second image data of a second user's field of view is sent to a server, focus point data associated with the feature data corresponding to the second image is read, support display data is created and sent to the second user's glasses terminal, and the second user's glasses terminal displays information on the focus point based on the support display data, superimposed on the second user's field of view.
[0004] In addition, the wiring work support system disclosed in Patent Document 2 includes an eye tracking device that tracks the line of sight of the worker, an image capturing device that captures the wiring or the like to be worked on, and images of the worker's visual range are processed to decipher characters and colors.A portable work management device is disclosed that has an image processing unit that provides the worker with information about the object stored in the data storage management unit, and that stores the work completion signal from a limit switch provided on the work tool and images of the object to be worked on. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2017-191490 A [Patent Document 2] JP 2015-061339 A Summary of the Invention [Problem to be solved by the invention]
[0006] Here, in Patent Document 1, it is assumed that users will transfer skills and the like between them by linking their gazes. In other words, it is difficult to grasp the state of gaze based on the gaze time and gaze shift of the viewpoint, extract appropriate attention information, and provide it to the worker. Furthermore, there is no description or suggestion of the specification of attention behavior that indicates what and in what situation the instructor gazed.
[0007] In addition, in Patent Document 2, it is assumed that the line of sight of the worker is linked to the work object and information about the object is provided to the worker. In other words, it is difficult to grasp the state of gaze based on the gaze time and gaze shift of the viewpoint, extract appropriate attention information, and provide it to the worker. Furthermore, there is no description or suggestion of the specification of attention behavior that indicates what and in what situation the instructor gazed.
[0008] The present invention has been devised in consideration of the above-mentioned problems, and its purpose is to provide an attention extraction system and an attention extraction method that grasp the instructor's state of gaze and extract and provide appropriate attention information. [Means for solving the problem]
[0009] The attention extraction system of the first invention is an attention extraction system that extracts attention information for a task, and is characterized in that it comprises an acquisition means that chronologically acquires video information in the field of view of a teacher performing the task and coordinate information indicating the viewpoint at which the teacher gazes in the field of view, linking the video information and coordinate information acquired by the acquisition means, a determination means that calculates the viewpoint displacement of the teacher in the field of view and identifies the teacher's gaze mode based on the video information and coordinate information acquired by the acquisition means, an extraction means that sets a gaze area based on the gaze mode identified by the determination means and extracts a gaze image of a work object at which the teacher gazes in the gaze area, and a storage means that links the gaze image extracted by the extraction means to the field of view, the viewpoint displacement, and the gaze mode, and stores it in a database as attention information for the task.
[0010] The attention extraction system of the second invention is characterized in that, in the first invention, the gaze area identified by the identification means includes at least a wide field of view mode in which the viewpoint displacement is concentrated at the center of the field of view range, and an alert mode in which the viewpoint displacement is dispersed outside the center of the field of view range.
[0011] The attention extraction system of the third invention is characterized in that, in the first invention, the acquisition means further includes a judgment means for judging the correctness of the work object contained in the gaze image, and a display means for displaying the judgment result of the judgment means.
[0012] The attention extraction system of the fourth invention is characterized in that, in any of the first to third inventions, it further includes a database in which correlation between previously acquired past gaze image information and reference information indicating the correctness or incorrectness of the work object linked to the gaze image is stored, the judgment means refers to the database to judge the correctness or incorrectness of the work object and acquires corresponding information from the database according to the judgment result, and the display means further outputs the corresponding information acquired by the judgment means.
[0013] The attention extraction system according to a fifth aspect of the present invention is the system according to the first aspect of the present invention, further comprising an input means for inputting the corresponding information output by the determination means, and the storage means links the corresponding information input by the input means with the gaze image and stores it as an attention data set including setting conditions for recognizing the gaze image as the gaze target.
[0014] The attention extraction system of the sixth invention is characterized in that, in any of the first to third inventions, the video information acquired by the acquisition means includes recording date and time information, recording position information, and recording control information related to the acquisition operation of the video information, and the display means displays the recording date and time information, recording position information, and recording control information in the center of the field of view before the acquisition of the video information by the acquisition means, and switches to only the recording control information and displays it in a corner of the field of view while the video information is being acquired.
[0015] The attention extraction system according to the seventh invention is any of the first to third inventions, wherein the display means further comprises an acquisition display area that displays instructions to the worker performing the work to select the type of the acquisition means for acquiring the video information, the type of the attention data set stored in the database, and the start of the work, and an attention information display area that switches between displaying corresponding information corresponding to the worker's gaze image and attention information based on the attention data set after the selection, wherein the attention information displayed in the attention display area includes nudge information that is displayed at least at the start of work, in the middle of work, or at the end of work, depending on the progress of the worker's work, and the information including at least any of the corresponding information, the attention information, and the nudge information displayed in the attention information display area is displayed in a manner that is distributed according to the gaze state of the worker performing the work, based on the attention data set and the result of the judgment by the judgment means.
[0016] The attention extraction method of the eighth invention is an attention extraction method for extracting attention information for a task, and is characterized in that it has a computer execute the following steps: an acquisition step for chronologically acquiring video information in the field of view of a teacher performing the task and coordinate information indicating the viewpoint at which the teacher gazes in the field of view, linking the video information and coordinate information acquired by the acquisition step; a determination step for calculating the viewpoint displacement of the teacher in the field of view and identifying the teacher's gaze mode based on the video information and coordinate information acquired by the acquisition step; an extraction step for setting a gaze area based on the gaze mode identified by the determination step and extracting a gaze image of a work object at which the teacher gazes in the gaze area; and a storage step for linking the gaze image extracted by the extraction step to the field of view, the viewpoint displacement, and the gaze mode, and storing it in a database as attention information for the task. Effect of the Invention
[0017] According to the first to seventh inventions, the identifying means obtains the instructor's viewpoint displacement in the field of view based on the image information and the coordinate information, and identifies the instructor's gaze mode. Therefore, the extracting means can set a gaze area based on the identified gaze mode, and extract a gaze image of the work object gazed by the instructor in the gaze area. This makes it possible to extract a gaze image showing what and in what situation the instructor gazed based on the gaze time and gaze shift of the viewpoint, and it is possible to accurately grasp the gaze state and provide appropriate attention information.
[0018] In particular, according to the second invention, the gaze area includes the wide field of view mode and the alert mode of the instructor. Therefore, it is possible to set the wide field of view mode in which the instructor's viewpoint displacement is concentrated at the center of the field of view range, and the alert mode in which the instructor's viewpoint displacement is dispersed outside the center of the field of view range. This makes it possible to extract a gaze image that shows what the instructor gazed at and in what situation.
[0019] In particular, according to the third aspect of the present invention, the acquisition means further includes a determination means. Therefore, it is possible to determine whether the task object included in the gaze image is correct or incorrect. This makes it possible to accurately grasp the gaze states of the instructor and the worker, and provide appropriate attention information.
[0020] In particular, according to the fourth aspect of the present invention, the database stores the correlation between the previously acquired past gaze image information and the reference information indicating the correctness of the work object associated with the gaze image. Therefore, the determination means can refer to the database to determine the correctness of the work object, and the display means can output corresponding information according to the determination result. This makes it possible to accurately grasp the gaze state of the worker and provide appropriate attention information.
[0021] In particular, according to the fifth aspect of the present invention, the input means inputs the corresponding information output by the determination means. Therefore, the storage means can associate the corresponding information with the gaze image and store it as an attention data set including a setting condition for recognizing it as a gaze target. This makes it possible to accurately grasp the gaze state and provide appropriate attention information.
[0022] In particular, according to the sixth aspect of the invention, the display means switches the information displayed in the field of view before and during acquisition of video information. Therefore, before acquisition of video information, recording date and time information, recording position information, and recording control information of the instructor's work are displayed in the center of the field of view, and during acquisition, only the recording control information is displayed in the corner of the field of view. This makes it possible to acquire video information of the instructor performing the work and extract the gaze image without imposing a burden on the instructor.
[0023] In particular, according to the seventh aspect of the present invention, the display means switches between the acquisition display area and the attention display area. Therefore, based on the attention data set and the determination result, the corresponding information, the attention information, or the nudge information can be displayed in accordance with the gaze state of the worker performing the work. This makes it possible to accurately grasp the gaze state of the worker and provide appropriate attention information.
[0024] According to the eighth aspect of the invention, the identifying step determines the instructor's viewpoint displacement in the visual field range based on the image information and the coordinate information, and identifies the instructor's gaze mode. Therefore, in the extracting step, a gaze area can be set based on the identified gaze mode, and a gaze image of the work object gazed by the instructor in the gaze area can be extracted. This makes it possible to extract a gaze image showing what and in what situation the instructor gazed based on the gaze time and gaze shift of the viewpoint, and it is possible to accurately grasp the gaze state and provide appropriate attention information. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of an attention extraction system according to this embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of attention extraction by an instructor and task evaluation by a worker in the attention extraction system 100 according to this embodiment. [Diagram 3] 3(a) to (c) are schematic diagrams showing an example of an attention extraction method in this embodiment. [Figure 4] FIG. 4(a) is a schematic diagram showing an example of the configuration of an attention extraction device, and FIG. 4(b) is a schematic diagram showing an example of the function of the attention extraction device. [Diagram 5] FIG. 5 is a schematic diagram showing an example of a database in this embodiment. [Figure 6] FIG. 6 is a schematic diagram showing a first modified example of the database in this embodiment. [Figure 7] FIG. 7 is a schematic diagram showing an example of a data table stored in a database in this embodiment. [Figure 8] FIG. 8 is a schematic diagram showing an example of a flowchart illustrating the attention extraction method according to this embodiment. [Figure 9] 9(a) to (e) are schematic diagrams showing an example of an attention extraction method in this embodiment. [Figure 10] 10(a) and 10(b) are schematic diagrams showing an example of a display on the worker device in this embodiment. [Figure 11] FIG. 11 is a schematic diagram showing an example of a display of the attention extraction device in this embodiment. [Figure 12] 12(a) to 12(e) are schematic diagrams showing an example of a display on a worker terminal in this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Hereinafter, an example of an attention extraction system and an attention extraction method according to an embodiment of the present invention will be described with reference to the drawings.
[0027] (Embodiment: Attention Extraction System 100 and Attention Extraction Method) An example of the configuration of the attention extraction system 100 in this embodiment will be described with reference to Figures 1 and 2. Figure 1 is a schematic diagram showing an example of the configuration of the attention extraction system 100 in this embodiment, and Figure 2 is a schematic diagram showing an example of attention extraction by an instructor and task evaluation by an operator in the attention extraction system 100 in this embodiment.
[0028] The attention extraction system 100 is used to extract the object of the work to which the instructor pays attention, using image information in the visual field of the instructor performing the work and coordinate information indicating the viewpoint of the instructor in the visual field. The attention extraction system 100 can obtain image information and coordinate information under various conditions in the instructor's view and range for various information including image information, coordinate information, and work information related to the work.
[0029] 1, the attention extraction system 100 includes an attention extraction device 1, an instructor device 2, a worker device 3, and a server 4, and for example, a plurality of instructor devices 2 and a plurality of worker devices 3 may be provided in a work area 50. The attention extraction system 100 may transmit and receive various information to and from the attention extraction device 1, the instructor device 2, the worker device 3, the server 4, and other user devices (not shown), for example, via a known communication network 5.
[0030] The attention extraction system 100 acquires a viewpoint displacement 2b of the visual field of the instructor that is included in the visual field range 2a of the instructor via, for example, an instructor device 2 worn by the instructor. The information acquired by the attention extraction system 100 from the instructor includes, for example, video information (video within the visual field range where the work is performed), coordinate information (viewpoint, position information, etc.), work information (work date and time, work instructions, process information, etc.), instructor information (instructor ID, device ID, etc.), and may also include various information on the work performed by the instructor (assigned work, group work, etc.).
[0031] The attention extraction system 100 acquires, in a time series, image information in the visual field of the instructor performing an operation and coordinate information indicating the viewpoint that the instructor is gazing at in the visual field, from an instructor device 2 worn by the instructor, in association with the operation performed by the instructor, as shown in Fig. 2. The image information and coordinate information may be acquired using, for example, a known eye tracking technology, a head-mounted display, or a technology for acquiring a viewpoint provided in smart glasses or the like.
[0032] The attention extraction system 100 acquires video information and coordinate information, for example, by the aforementioned technology provided in the instructor device 2, identifies the instructor's gaze mode in the attention extraction device 1, and extracts a gaze image based on the identified gaze mode. The attention extraction system 100, for example, associates the extracted instructor's gaze image with reference information associated with the gaze image, and stores it in a database as an attention dataset. The attention extraction system 100 may, for example, refer to the database, acquire attention information associated with the reference information, and display it on the operator device 1 of the operator who uses the attention dataset.
[0033] The attention extraction system 100 then evaluates the work of the worker based on, for example, an attention data set selected by the worker, image information of the work target 6 included in the visual field 3a of the worker, numerical information of the viewpoint displacement and coordinate information of the visual field, and identifies, for example, the work target 6 as gaze information based on the evaluation result, and further obtains corresponding information such as "(1) Check" and "(2) Adjust", and displays it together with the actual display by superimposing it on the visual field 3a of the worker device 3. Details of the configurations of the attention extraction device 1, the instructor device 2, and the worker device 3 will be described later.
[0034] Here, the identification of the instructor's gaze mode will be described with reference to Fig. 3. The attention extraction system 100 identifies the instructor's gaze mode by, for example, an attention extraction method as shown in Figs. 3(a) to (c). The attention extraction system 100 identifies the instructor's gaze mode in the visual field range based on image information in the visual field range 2a acquired by the instructor device 2 and coordinate information (x-axis, y-axis) of the instructor's viewpoint displacement, as shown in Fig. 3(a), for example.
[0035] The attention extraction system 100 obtains the instructor's gaze displacement in the visual field range 2a from the coordinates based on the image information and coordinate information acquired via the instructor device 2, and identifies the instructor's gaze mode. For example, as shown in FIG. 3(b), the attention extraction system 100 may identify the instructor's gaze displacement as a "wide visual field mode" when the instructor's gaze displacement is characterized by a "narrow displacement range (concentration)" and "few points of interest in the visual field." Also, the attention extraction system 100 may identify the instructor's gaze displacement as a "vigilance mode" when the instructor's gaze displacement is characterized by a "wide displacement range (dispersion)" and "many points of interest in the visual field," as shown in FIG. 3(c).
[0036] The attention extraction system 100 sets a gaze area gazed at by the instructor based on the specified gaze mode, extracts a gaze image of the work object gazed at by the instructor in the set gaze area, links the extracted gaze image with the visual field range, viewpoint displacement, and gaze mode, and stores it in a database of the server 4 as attention information for the instructor's work. Note that there may be multiple instructors and workers, and for example, multiple instructors and workers may work on the same process, and the attention extraction system 100 may assign, for example, one work process of the instructor to multiple workers in a shared manner.
[0037] The allocation of instructors and workers by the attention extraction system 100 may be determined based on, for example, the number of steps in a work process, the difficulty level, the skills of the workers, and the deadline for the work. Alternatively, the allocation may be made by, for example, an evaluator via the attention extraction device 1. It is arbitrary as to how many workers are assigned to which tasks and how the workers are arranged, and may be specified as appropriate.
[0038] <Caution extraction device 1> The attention extraction device 1 determines the displacement of the instructor's viewpoint within the field of view based on, for example, video information in the field of view of the instructor's work acquired by the instructor device 1 and coordinate information indicating the viewpoint at which the instructor is gazing within the field of view, and identifies the instructor's gaze mode.
[0039] The gaze area identified by the identification means includes, for example, a wide field of view mode in which the instructor's viewpoint displacement is within a field of view radius of 5 degrees to 20 degrees and concentrated at the center of the field of view range, and an alert mode in which the displacement is within a field of view radius of 4 degrees and dispersed outside the center of the field of view range.
[0040] The attention extraction device 1 identifies a case in which the instructor's gaze displacement characteristics are "narrow displacement range (concentrated)" and "few points of attention" as a "wide field of view mode," as shown in, for example, the above-mentioned Figure 3(b), and identifies a case in which the instructor's gaze displacement characteristics are "wide displacement range (dispersed)" and "many points of attention" as an "alert mode," as shown in, for example, the above-mentioned Figure 3(c).
[0041] The attention extraction device 1 sets a gaze area based on the specified gaze mode, for example, and extracts a gaze image of a work object gazed at by an instructor in the gaze area. The attention extraction device 1 links the extracted gaze image, for example, to a visual field range, a viewpoint displacement, and a gaze mode, and stores it in a database as attention information for the work.
[0042] 11 shows an example of an attention extraction device screen 1a of the attention extraction device 1 in the attention extraction system 100. Fig. 11 is a screen operated by, for example, an evaluator, and various information acquired by the instructor device 1 is displayed.
[0043] The attention extraction device 1, for example, refers to a database and displays on the attention extraction device screen 1a a gaze monitoring area including at least a gaze display area 1b (eye gaze movement distance graph, gaze tracking, gaze area, gaze target, etc.) that displays coordinate information, gaze area, gaze image, and gaze shift in time series, a target setting area 1c (gazing target evaluation, recognizer generation, etc.) that displays setting information for setting the gaze image displayed in the gaze display area 1b, and a judgment setting area 1d (corresponding information setting, etc.) that displays judgment conditions for the setting information displayed in the target setting area 1c.
[0044] The attention extraction device 1 receives, for example, various settings and adjustment-related conditions and values from an evaluator via a setting item menu displayed on the attention extraction device screen 1a. The attention extraction device 1 sets and adjusts the display and conditions in the gaze display area 1b, the target setting area 1c, and the judgment setting area 1d based on, for example, the various conditions and values received.
[0045] The attention extraction device 1 determines whether the task object included in the instructor's gaze image is correct or incorrect according to various conditions received from the evaluator via, for example, a setting item menu displayed on the attention extraction device screen 1a. The attention extraction device 1 may display the determination result on the instructor device 2 or the worker device 3.
[0046] The attention extraction device 1 accepts input of correspondence information from the evaluator via, for example, a setting item menu displayed on the attention extraction device screen 1a. The attention extraction device 1 may accept input of new correspondence information, as well as update or delete existing conditions, settings, correspondence information, etc., and store them in a database of the server 4.
[0047] The attention extraction device 1 may, for example, link the inputted corresponding information with the gaze image, generate an attention dataset including setting conditions for recognizing it as a gaze target, and store the generated attention dataset in a database of the server 4.
[0048] 4(a) is a schematic diagram showing an example of the configuration of the attention extraction device 1. As the attention extraction device 1, a single board computer such as Raspberry Pi (registered trademark) may be used, or a known electronic device such as a personal computer (PC) may be used. The attention extraction device 1 includes, for example, a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage unit 104, and I / Fs 105 to 107. The components 101 to 107 are connected by an internal bus 110.
[0049] The CPU 101 controls the entire attention extraction device 1. The ROM 102 stores the operation code of the CPU 101. The RAM 103 is a working area used when the CPU 101 operates. The storage unit 104 stores various information such as a learning model and a database. As the storage unit 104, a known data storage medium such as an SD memory card, a hard disk drive (HDD), a solid state drive (SSD), etc. may be used.
[0050] The I / F 105 is a known interface for transmitting and receiving various information to and from the instructor device 2, the worker device 3, the server 4, the communication network 5, etc., which are connected depending on the purpose. For example, a plurality of I / Fs 105 may be provided.
[0051] The I / F 106 is a known interface for transmitting and receiving various information to and from an input unit 108 connected depending on the purpose. For example, a keyboard is used as the input unit 108, and an administrator or the like who manages the attention extraction system 100 inputs or selects various information or control commands for the attention extraction device 1 through the input unit 108.
[0052] The I / F 107 is a known interface for transmitting and receiving various information to and from a display unit 109 connected depending on the purpose. The display unit 109 outputs various information stored in the storage unit 104, the processing status of the attention extraction device 1, and the like. For example, a display is used as the display unit 109, and may be, for example, a touch panel type. In this case, the display unit 109 may be configured to include the input unit 108.
[0053] For example, the same I / F may be used as I / F 105 to I / F 107, or a plurality of I / Fs may be used as each of I / F 105 to I / F 107. At least one of the instructor device 2, the operator device 3, the server 4, the communication network 5, the input unit 108, and the display unit 109 may be detached depending on the situation.
[0054] <Memory unit 14 (database)> The storage unit 14 stores various databases, for example, in the storage unit 104. The storage unit 14 stores associations between previously acquired past evaluation target information and reference information linked to the past evaluation target information, and stores, for example, a learning model having associations. The storage unit 14 may be provided in, for example, the instructor device 2, the worker device 3, and the server 4.
[0055] The memory unit 14 records the instructor's attention behavior (points of attention, viewpoint, etc.) when, for example, observing the instructor's work. The attention extraction system 100 may determine the gaze mode based on information such as a gaze image, a field of view, and a viewpoint displacement acquired via the instructor device 2. The attention extraction system 100 may determine, for example, a "wide field of view mode" when the instructor's gaze displacement has a "narrow displacement range (concentration)" and "few points of attention for the viewpoint," and may determine, for example, as shown in FIG. 3(c) above, a "vigilance mode" when the instructor's gaze displacement has a "wide displacement range (dispersion)" and "many points of attention for the viewpoint," and store the determination in the memory unit 14 of the server 4 as attention information for the instructor's work.
[0056] The storage unit 14 stores various information related to the instructor's work acquired by gestures, voice, gaze, and other input devices, for example, through various input system interfaces (not shown) provided in the instructor device 2. The attention extraction system 100 stores various information acquired through the input devices, for example, in the storage unit 14, by linking the various information acquired through the input devices with information related to the instructor and information related to the instructor's work. The attention extraction system 100 may record various information appropriately in response to recording instructions, such as start and end of recording, issued by the instructor through the instructor device 2.
[0057] The storage unit 14 may sequentially record the coordinate position set for the viewpoint supplemented by the instructor via the instructor device 2, the change in the coordinate position, the change speed, etc., and may also record, for example, the position and orientation of the head in the work space coordinates for each spatial layout of the place where each task is performed. The attention extraction system 100 may acquire various information recorded for each spatial layout using an imaging device such as a known 360° camera, a surveillance camera, or a network camera, and store the information in the storage unit 14 in association with the coordinate position of the instructor's head and information indicating the change in the coordinate position.
[0058] In addition, after recording to the database, the storage unit 14 may record various data recorded in association with the image data and the photographed data as an attention data set for the instructor's work. The attention data set is, for example, evaluated by an evaluator via the attention extraction device 1 for attention and know-how, and is registered (recorded) in the storage unit 4 as correspondence information in the attention object recognizer and the monitoring of the attention object.
[0059] Here, for example, the attention object recognizer is a device generated using attention object video evaluated by the attention extraction system 100, and may be, for example, a known image processing program such as pattern matching, or an image recognition model. The image processing program or image recognition model may be installed in, for example, the worker device 3 worn by the worker when performing work. This enables the worker to evaluate attention information for work based on, for example, time-series video information and coordinate information acquired via the worker device 3 when performing work.
[0060] The storage unit 14 may further include a database in which the correlation between the gaze image information previously acquired and the reference information indicating the correctness of the work object associated with the gaze image is stored. The storage unit 14 may be referred to by the judgment unit 15, for example, and used when judging the correctness of the work object in the instructor's work. The judgment unit 15 may acquire corresponding information according to the judgment result, for example. The display unit 16 may further output the corresponding information from the storage unit 14 according to the content judged by the judgment unit 15, for example.
[0061] In addition, the memory unit 14 may be configured to register, for example, the gaze target recognizer and corresponding information in monitoring the gaze target, which is acquired by the instructor device 2 and evaluated as attention know-how by the evaluation unit 15, as an attention data set for the work in the worker device 13.
[0062] The storage unit 14 may store, for example, past evaluation target information and reference information acquired by the instructor device 2. The association is constructed by machine learning using a plurality of learning data, for example, with the past evaluation target information and the reference information as a set of learning data. As a learning method, for example, deep learning such as a convolutional neural network is used.
[0063] In this case, for example, the association indicates the degree of connection between many-to-many information (a plurality of data included in the past evaluation target information, versus a plurality of data included in the reference information).
[0064] The association is appropriately updated during the machine learning process. That is, the association indicates a function optimized based on, for example, past image data A, corresponding information A, and reference information. Therefore, an evaluation result for the evaluation target information is generated using the association constructed based on all past evaluation results of the evaluation target information. This makes it possible to generate an optimal evaluation result even when the evaluation target information has a configuration state in which it is combined with other image data A, corresponding information A, etc.
[0065] Furthermore, the evaluation target information stored in the storage unit 14 may be the same as or similar to past evaluation target information, or may be dissimilar. This allows the attention extraction system 100 to quantitatively generate optimal evaluation results. The attention extraction system 100 can improve generalization ability when performing machine learning, and can improve evaluation accuracy for unknown evaluation target information.
[0066] The association may have multiple association degrees indicating the degree of connection between multiple data included in the past evaluation target information and multiple data included in the reference information. For example, when the learning model is constructed by a neural network, the association degrees can correspond to weight variables.
[0067] The past evaluation target information indicates information of the same type as the evaluation target information described above. The past evaluation target information includes, for example, a plurality of pieces of evaluation target information acquired when the image data A was evaluated in the past.
[0068] The reference information is linked to past evaluation target information and indicates information related to image data A and corresponding information A. The reference information indicates, for example, the scope of work for image data A, work instructions, work procedures, details of subsequent processes and related work, details of shared work and alternative work of other workers working in the same area, evaluations based on various work relationships and mutual influences (for example, "check," "adjust," "pay attention (WATCH!)," "judge," "respond," etc.), and may also include various information related to the progress of the work process, checklists, images of interest, priority of displayed information, combinations of display and non-display, numerical values, data, distribution, etc. The specific contents included in the reference information can be set arbitrarily.
[0069] The association may indicate the degree of connection between the past evaluation target information and the reference information, for example, as shown in Fig. 5. In this case, by using the association, the degree of relationship between each of the multiple data included in the past evaluation target information ("Image Data A" to "Image Data C" in Fig. 5) and the multiple data included in the reference information ("Reference A" to "Reference C" in Fig. 5) can be linked and stored. Therefore, for example, multiple data included in the reference information can be linked to one data included in the past evaluation target information via the association, thereby realizing the generation of a multifaceted evaluation result.
[0070] The association has, for example, a plurality of associations linking a plurality of data included in the past evaluation target information with a plurality of data included in the reference information. The association is indicated in three or more stages, such as a percentage, 10 stages, or 5 stages, and is indicated, for example, by the characteristics of the line (for example, thickness, etc.). For example, "image data A" included in the past evaluation target information indicates an association degree AA "80%" with "reference A" included in the reference information, and indicates an association degree AB "55%" with "reference B" included in the reference information. That is, the "association degree" indicates the degree of connection between each data, and for example, the higher the association degree, the stronger the connection between each data. Note that when constructing the association by the above-mentioned machine learning, the association may be set to have three or more stages of association.
[0071] For example, as shown in Fig. 6, the past evaluation target information may be divided into a past gaze image of the work target and past corresponding information of the work target, and stored in the database. In this case, the degree of association is calculated based on the relationship between the combination of image data of the past gaze image and the past corresponding information, and the reference information. In addition to the above, the past evaluation target information may be divided into past countermeasure information and stored in the database.
[0072] For example, the combination of "image data a" included in the past gaze image and "corresponding information a" included in the past corresponding information indicates a correlation degree AAA "85%" with "reference A" and a correlation degree ABA "25%" with "reference B". In this case, the past target data and the past corresponding information can be stored independently. This makes it possible to improve accuracy and expand the range of options when generating evaluation results.
[0073] The past evaluation target information may include, for example, composite data and a similarity. The composite data is indicated by three or more levels of similarity between the past target data or the past corresponding information. The composite data may be stored in the database in the form of a numerical value, a matrix, a histogram, or the like, or may be stored in the form of an image, a character string, or the like.
[0074] Here, an example of the database in this embodiment will be described with reference to Fig. 7. In the database, various information on the instructor (instructor), evaluator, worker, etc., who are users of the attention extraction system 100, as well as information on the contents and steps of various tasks, and multiple data sets used by the workers are stored as attention data sets.
[0075] In the attention data set, various information such as information of each data table, work of an instructor, an operator, etc., and an evaluation by an evaluator are associated with each data. The evaluation by the evaluator is input, for example, by the input unit 17. In the attention data set, at least, for example, a "work information table", a "work procedure table", a "correspondence information table", a "teaching work record table", a "viewpoint record table", a "viewpoint record data table", a "gaze target table", etc. are stored.
[0076] <<Work information table>> The "task information table" stores (contains) data for identifying the task to be performed, for example. The "task information table" stores, for example, a "task ID" and a "task name" for identifying the task to be performed by an instructor or worker.
[0077] <<Work procedure table>> The "work procedure table" stores (contains) data on the procedures of work to be performed by, for example, an instructor or a worker. The "work procedure table" stores, for example, a "work procedure ID" that identifies the procedure of the work, and a "work ID," "work procedure name," and "work sequence" that are linked to the "work procedure ID" and correspond to each other. The "work procedure table" is linked to the "work information table" by, for example, the "work ID."
[0078] <<Correspondence information table>> The "corresponding information table" stores (contains) data for identifying information corresponding to the work displayed by the display means, for example, when a worker performs work, after the judgment means judges whether the work object is correct or incorrect based on the image information acquired by the acquisition means and the coordinate information of the viewpoint.
[0079] The "corresponding information table" stores, for example, a "corresponding information ID" for identifying information for responding to work, a "work procedure ID," "advance" indicating an approach through a nudge or the presentation of advance confirmation information, and a "display type" indicating a display type such as an "interrupt type" that is displayed depending on the work situation, including oversights.
[0080] In the "display type", a trigger for start timing, a trigger for display time, or a trigger for display end is set, such as an animation effect timing setting. When the display type is, for example, "interrupt", information indicating the judgment timing for interrupt is stored as the "display / judgment start condition". The judgment start condition may, for example, specify a trigger for interrupt judgment.
[0081] Furthermore, the "corresponding information table" stores, when moving to "interrupt judgment", "judgment criteria" that indicate the standards or thresholds that the instructor uses to judge "saw" or "did not see", and the display content that is displayed as corresponding information as "corresponding information content".
[0082] The "corresponding information content" may store, for example, corresponding information created at the work site by designation using a file name, Markdown, etc. The "corresponding information content" may store, for example, a one-off instruction such as contacting a superior or a veteran, and may be generated by inputting the instruction via an editing screen.
[0083] Furthermore, the "Corresponding Information Table" stores "Work Result Record Display Conditions," which indicate conditions such as the timing for displaying content that records the results of each work procedure, such as work reports and checklists, and "Work Result Record Content," which indicates content for recording work results, such as work reports and checklists.
[0084] The attention extraction system 100 may refer to the "work result record display conditions" and determine by image comparison that the gaze target has become an appropriate state based on, for example, a signal by gesture or voice input. The attention extraction system 100 may store various information and data in the "work result record display conditions" as conditions for storing the determination result, for example.
[0085] The attention extraction system 100 may require, for example, the input of the result of the content in the "work result recording content." The attention extraction system 100 may, for example, determine this result input and determine that one work procedure is completed based on the result of the determination. The "correspondence information table" is linked to the "work procedure table" by, for example, the "work procedure ID."
[0086] <<Teaching work record table>> In the "teaching work record table", for example, various information related to work performed by an instructor is stored. In the "teaching work record table", for example, a "teaching work record ID" for identifying the recorded teaching work, a "teaching work date and time" indicating the date and time when the teaching work was performed, a "teaching worker" indicating the instructor who performed the teaching work, a "teaching work location" indicating the place or area where the teaching work was performed, and a "work ID" are stored, each linked to each other. The "teaching work memory table" is linked to the "work information table" by, for example, the "work ID".
[0087] <<Viewpoint record table>> The "viewpoint record table" stores (contains) information and data related to the viewpoint from which the instructor works, acquired by, for example, the acquisition unit 11. The "viewpoint record table" stores, as a "viewpoint record data file", a "viewpoint record ID" that identifies the viewpoint record worked by the instructor, and data on the coordinate information of the viewpoint obtained by eye tracking.
[0088] The viewpoint record table may store stream data acquired in milliseconds. This allows the attention extraction system 100 to store the actual file name of data such as JavaScript Object Notation (JSON) without directly storing the data in a DB.
[0089] The "viewpoint record table" also stores a "viewpoint range record video file" that indicates video information data in the instructor's field of view recorded by, for example, an outer camera. The "viewpoint record table" also stores a "teaching work record ID" that identifies the history of the teaching work by the instructor. The "viewpoint record table" is linked to the "teaching work record table" by, for example, the "teaching work record ID."
[0090] <<Viewpoint record data table>> The "viewpoint recording data table" stores detailed information on the coordinate information of the viewpoint acquired by the acquisition unit 11 when the instructor performs work, for example. The "viewpoint recording data table" stores a "viewpoint identification data file" that identifies a file that stores the viewpoint recording data, and a "viewpoint recording elapsed time" that indicates the time of the viewpoint identification recorded by the instructor. The "viewpoint recording elapsed information" stores (holds) the elapsed time of recording the viewpoint as a record, for example, with the start of recording the instructor's work as 00:00:00. The "viewpoint recording elapsed information" may be recorded, for example, at unit time intervals (for example, 1 second intervals) together with various position information. The "viewpoint recording data table" is linked to the "teaching work record table" by, for example, the "viewpoint identification data file".
[0091] In addition, the "viewpoint recording data table" stores the position information of the teaching operator in the work space as "teaching operator position information (X,Y,Z)". In the "teaching operator position information (X,Y,Z)", for example, the position of the HMD worn by the teaching operator, i.e., the three-dimensional coordinates of the head position information, is recorded, and further, information such as "teaching operator orientation information (rad)", "teaching operator eyeball position information (x,y,z)", and "teaching operator eyeball angle information (rad)" acquired by the acquisition unit 11 may be stored together. The "viewpoint recording data table" is linked to the "teaching work record table" by, for example, a "viewpoint recording data file".
[0092] <<Gaze target table>> In the "gaze target table", for example, various information related to the object gazed upon by the instructor is stored. A plurality of various pieces of information set by the evaluator are stored. In the "gaze target table", for example, a "gaze target ID" that identifies the object gazed upon by the instructor, a "visual field range recording video file", a "viewpoint recording data file", a "viewpoint recording elapsed time", and a "work procedure ID" are stored.
[0093] In addition, the "gaze target table" stores a "visual field range image at time of gaze" as a still image extracted from the "visual field range recording video file" at a moment during the gaze recording elapsed time. In addition, the "gaze target position (x, y)" indicating the position of the gaze target extracted by the extraction unit 13, the "gaze target range (x, h)" indicating the range, and the "gaze target image" indicating the image of the gaze target may be stored as attention information for the work, based on the gaze mode identified by the identification unit 12 based on the video information and coordinate information associated in the "gaze target recording table".
[0094] The attention information may be generated by the identification unit 12 and the extraction unit 13 based on information acquired by the acquisition unit 11, for example, and then evaluated, set, and recorded by an evaluator. The "gaze target table" is linked, for example, to the "viewpoint recording table" by the "gaze target ID," the "work procedure table" by the "work procedure ID," and the "viewpoint recording data table" by the "viewpoint recording data file."
[0095] Fig. 4(b) is a schematic diagram showing an example of the functions of the attention extraction device 1. The attention extraction device 1 includes, for example, an acquisition unit 11, a specification unit 12, an extraction unit 13, a storage unit 14, a determination unit 15, a display unit 16, an input unit 17, and a monitoring display unit 18. Each function shown in Fig. 5(b) is realized by the CPU 101 using the RAM 103 as a working area and executing a program stored in the storage unit 104 or the like.
[0096] <<Acquisition unit 11 (acquisition means)>> The acquisition unit 11 acquires image information in the visual field of the instructor performing the task and coordinate information indicating the viewpoint at which the instructor gazes within the visual field in a chronological order, linking the information to the task. The acquisition unit 11 is used, for example, when performing an acquisition step S110 described later. The timing at which the acquisition unit 11 acquires the image information and coordinate information from the instructor device 2 can be set arbitrarily. The acquisition unit 11 stores the acquired image information and coordinate information in the storage unit 104 in a chronological order, linking the information to the task performed by the instructor, for example, via the storage unit 14.
[0097] <<Specifying part 12 (specific means)>> The identification unit 12 obtains the instructor's viewpoint displacement in the visual field range and identifies the instructor's gaze mode based on the image information and coordinate information acquired by the acquisition unit 11. The identification unit 12 is used, for example, when performing an identification step S120 described later. The identification unit 12 identifies the instructor's gaze mode in the visual field range based on the image information in the visual field range 2a acquired by the instructor device 2 and the coordinate information (x-axis, y-axis) of the instructor's viewpoint displacement, as shown in FIG. 3 described above, for example.
[0098] Next, if the information acquired by the acquisition unit 11 has characteristics or tendencies different from the already stored "wide field mode" and "alert mode," the identification unit 12 may identify it as a new mode. The identification unit 12 can arbitrarily set various modes using the attention extraction device 1. The identification unit 12, for example, via the storage unit 14, saves the new mode in chronological order in the storage unit 104 in addition to making changes to the existing acquired mode, linking it with the instructor and the work performed by the instructor. The identification unit 12 may identify the various modes of the instructor by, for example, an evaluator setting the various modes based on information acquired by the acquisition unit 11 and stored in a database.
[0099] <<Extraction unit 13 (extraction means)>> The extraction unit 13 sets a gaze area based on the gaze mode identified by the identification unit 12, and extracts gaze images of the work target 6 gazed by the instructor in the gaze area. For example, when there are multiple gaze modes identified by the identification unit 12, the extraction unit 13 may determine gaze images of the work target 6 included in the set gaze area based on each gaze mode, and extract them in chronological order.
[0100] The extraction unit 13 extracts gaze images of the work target 6 included in the set gaze area based on, for example, a gaze mode, and if there is an order to the instructor's work or confirmation, it extracts them in chronological order with an emphasis on the order.If there is no order, it may also extract multiple gaze images linked to one work.
[0101] The extraction unit 13 may output the extraction result to the storage unit 14, the attention extraction device 1, or the like, for example, via the communication network 5. The extraction unit 13 stores the extracted gaze image in the storage unit 104, for example, via the storage unit 14, by linking it with the instructor, the work performed by the instructor, the gaze mode, and the like. Note that the linking of the gaze image and the gaze mode by the extraction unit 13 may be set by the evaluator based on various information stored in the storage unit 104.
[0102] <<Storage unit 14 (storage means, database)>> The storage unit 14 associates the gaze image extracted by the extraction unit 13 with the visual field range, the viewpoint displacement, and the gaze mode, and stores the image in a database as attention information for the instructor's work. The storage unit 14 stores various information in the storage unit 104, or retrieves various information from the storage unit 104.
[0103] The storage unit 14 stores a plurality of associations between the past gaze image information acquired in advance and the reference information indicating the correctness or incorrectness of the work object associated with the gaze image. The storage unit 14 stores or retrieves various information according to the processing contents of the acquisition unit 11, the identification unit 12, the extraction unit 13, the determination unit 15, the display unit 16, the input unit 17, and the monitoring display unit 18, for example.
[0104] <<Judgment unit 15 (judgment means)>> The determination unit 15 determines whether a task object included in a task gaze image acquired, for example, via the worker's worker device 3, is correct or incorrect by using a task-related attention dataset. The determination unit 15 refers to a database stored in the storage unit 14 or the storage unit 104, and determines whether a task performed by the worker with respect to gaze information (for example, task object 6) is correct or incorrect based on video information acquired via the worker's worker device 1 by referring to the database.
[0105] The determination unit 15 acquires various types of corresponding information from a database, for example, depending on the result of the determination. The determination unit 15 acquires information related to the work that the worker should handle, such as the target information displayed in the visual field 3a of the worker device 3 in FIG. 2 ("(1) Confirmation OK; 01234", "(2) Adjustment procedure: XXX"). The determination unit 15 transmits the acquired corresponding information to the display unit 16 of the worker device 3, for example.
[0106] For example, when multiple workers perform the same task in the same work area, the determination unit 15 may collectively determine whether the tasks of the multiple workers are correct. In this case, for example, the multiple workers set a common task attention data set in advance in each of the worker devices 3. For example, the determination unit 15 may obtain multiple pieces of coordinate information for each work workspace layout based on the relationship between the coordinate information indicating the position and direction of the head of each worker and the coordinate information indicating the work workspace from the coordinate information obtained by the obtaining unit 11, and may determine the tasks and tasks of each worker in an integrated manner or on a group basis.
[0107] Furthermore, the determination unit 15 acquires status information such as whether the original worker is facing the correct direction toward the work object that the original worker should be working on, and if not (unaware), whether there are other workers who can support the original worker, whether there are other workers facing toward the work object that the original worker should be working on, etc. Based on the acquired information and the information on the work process of each worker, for example, the determination unit 15 may transmit corresponding information to the worker device 3 of another worker who is working together nearby and can work on the work object appropriately even if the original worker is not looking at it.
[0108] <<Display section 16 (display means)>> The display unit 16 outputs various types of correspondence information acquired by, for example, the determination unit 15 to the worker device 3 of the worker performing the corresponding work, using an attention data set generated based on the work of the instructor. The display unit 16 (display portion 109 of the worker device 3) displays, for example, various types of correspondence information transmitted from the determination unit 15 in the visual field range 3a of the worker device 3 worn by the worker, superimposing the corresponding information on the actual image.
[0109] For example, when multiple workers are working on the same task in the same work area, the display unit 16 appropriately displays the exercise information obtained from the judgment unit 15 according to the work situation and work position of each worker.
[0110] Furthermore, the display unit 16 displays on the instructor device 2 a display for extracting attention information for the task, for example, as shown in Fig. 9. Fig. 9 is a schematic diagram showing an example of an attention extraction method in this embodiment. For example, before the acquisition of video information by the observer unit 11, the display unit 16 displays recording date and time information, recording position information, and recording control information in the center of the visual field range 2a, as shown in Fig. 9(a). Furthermore, while the acquisition unit 11 is acquiring video information, the display unit 16 may switch to displaying only the recording control information (for example, "recording paused", "recording ended", etc.) in a corner of the visual field range 2a.
[0111] 9(c) to 9(e), the display unit 16 may display indications for each operation in the center of the visual field 2a of the instructor device 2. The display unit 16 may display, for example, the work process planned by the instructor, the work results performed by the instructor, and the timing of the end or end of the work detected by a position detection sensor provided in advance in the instructor device 2.
[0112] The display unit 16 displays, for example, displays as shown in Fig. 10(a) and (b) and Fig. 12(a) to (e) on the worker device 3. Fig. 10(a) and (b) are schematic diagrams showing an example of attention extraction displayed when a worker performs work via the worker device 3 using, for example, the above-mentioned attention data set in this embodiment, and Fig. 12(a) to (e) are schematic diagrams showing, for example, an example of display on the worker device 3 (worker terminal, smartphone, display device, etc.) held by a worker in this embodiment.
[0113] The display unit 16 displays the result of the determination by the determination unit 15, for example, on a gaze target 6a included in the visual field 3a of the worker device 3 worn by the worker, or on a display device held by the worker, in accordance with the result of the determination by the determination unit 15. Furthermore, the display unit 16 refers to a database and displays corresponding information in accordance with the result of the determination acquired by the determination unit 15.
[0114] Fig. 10(a) is a display of the visual field 3a when, for example, a worker is working and there is no corresponding information for the worker. Fig. 10(b) is a display of the visual field 3a when, for example, a worker is working and there is corresponding information for the worker, and the corresponding information displayed in the visual field 3a includes points that should be focused on (correct target for focusing on the visual field 3a: circle mark + "WATCH!"), procedures and related information that the worker should confirm or refer to (center of the visual field 3a: "Check-3-A", "Check", "Judgment", "Response", "Reference", etc.), and the work status and check results of the worker or co-workers (left side of the visual field 3a: work location, work content, work progress, etc.).
[0115] Furthermore, the display unit 16 may further include an acquisition display area that displays, for example, at the start of work by a worker, the type of acquisition unit 11 that acquires video information, the type of attention data set stored in the database, and instructions for selecting each to start the work, and an attention information display area that switches between displaying corresponding information corresponding to the worker's gaze image and attention information based on the attention data set selected by the worker.
[0116] Furthermore, the display unit 16 may appropriately display the attention information to be displayed in the visual field 3a of the worker device 3 as nudge information, for example, at least one of the timings of the start, middle, or end of work according to the progress of the worker's work. The display unit 16 refers to the attention database based on, for example, the attention data set and the result of the determination by the determination unit 15, acquires response information, attention information, nudge information, or the like, and distributes and displays the acquired various information according to the gaze state of the worker performing the work.
[0117] The nudge information displayed by the display unit 16 may be, for example, a display of a nudge default (unconsciously encouraging) that does not make the subject aware of it, a mechanism (naturally encouraging), a nudge that makes the worker aware of it but leaves the choice up to the worker, labeling (intentionally encouraging), and an incentive (encouraging with rewards) as an approach to display to the worker. The display of the nudge information may be linked, for example, to the work process or work progress, the worker's personality, temperament, evaluation, difficulty of the work, expected completion time, etc., and may be appropriately extracted from a database and displayed according to the worker's actual work progress.
[0118] Further, the display unit 16 may display the nudge information in a random manner, such as by directly displaying the reference information raised as a display candidate as a result of the determination by the determination unit 15, or by randomly displaying a pattern that asks a question, such as "Are you OK?" or "Wait a minute?". Furthermore, the display unit 16 may display the nudge information at the center of the visual field 3a of the worker device 3, or at a corresponding location of the gaze target 6a within the visual field 3a.
[0119] In addition, the display unit 13 may mix or randomly change the display output methods (visual, auditory, haptics (vibration), nudge information approach (labeling), and nudge information technique (speaking to the worker, presenting it succinctly, making the worker wonder, changing it completely) as appropriate depending on the worker's skills and the progress of the work.
[0120] Furthermore, the display unit 16 may display, for example, heuristics, normality (maintaining the status quo), and variations in nudge information to prevent overlooking of reference information due to familiarity, boredom, and expectations for maintaining the status quo as related biases for the nudge information to be displayed. The display unit 16 may include, for example, a means (not shown) for grasping the overall flow and procedure of the work being performed by the worker as a prerequisite process for displaying nudge information, and may further include a means (not shown) for detecting deviations from due attention.
[0121] Furthermore, when a worker finishes a task, the display unit 16 may refer to the database and display words of appreciation to the worker, and the type of nudge information and timing of display are optional. This makes it possible to vary the response information displayed to the worker, for example, and to prevent the worker from overlooking the reference information due to familiarity, boredom, or expectation of maintaining the status quo.
[0122] Furthermore, the display unit 16 displays, for example, a display as shown in Fig. 11 on the attention extraction device 1. Fig. 11 displays, for example, a gaze monitoring area of the attention extraction device 1 in this embodiment as an attention extraction device screen 1a. As described above, the display unit 16 displays, for example, coordinate information, gaze area, gaze image, and gaze shift in chronological order in the gaze display area 1b when an evaluator refers to a database via the attention extraction device 1 for the work performed by an instructor.
[0123] The display unit 16 displays, in a target setting area 1c, setting information for setting the gaze image displayed in the gaze display area 1b, as shown in Fig. 11. Furthermore, the display unit 16 displays, for example, a judgment condition of the setting information displayed in the target setting area 1c in a judgment setting area 1d.
[0124] The display unit 16 displays, for example, setting information for setting the gaze image displayed in the gaze display area 1b in the target setting area 1c. Furthermore, the display unit 16 displays, for example, a judgment condition of the setting information displayed in the target setting area 1c in the judgment setting area 1d.
[0125] <<Input unit 17 (input means)>> The input unit 17 inputs, for example, the correspondence information output by the judgment unit 16. The input unit 17 accepts, for example, conditions and numerical values related to various settings and adjustments from the evaluator, for example, via the attention extraction device screen 1a displayed on the attention extraction device 1, as well as, for example, via a setting item menu displayed on the attention extraction device screen 1a. As a result, for example, the acquisition unit 11, the identification unit 12, the extraction unit 13, the storage unit 14, the judgment unit 15, the judgment unit 16, and the monitoring display unit 18 perform various processes based on the various conditions and numerical values accepted by the input unit 17.
[0126] The corresponding information input by the input unit 17 is linked to the gaze image, and is stored in the database of the server 4 as an attention data set including setting conditions for recognizing it as a gaze target.
[0127] <<Monitoring display section 18 (monitoring display means)>> The monitoring display unit 18 displays, for example, various pieces of information that an evaluator refers to and sets while an instructor is working, as a gaze monitoring area on the attention extraction device screen 1a. The monitoring display unit 18 displays, for example, a gaze display area 1b that displays coordinate information, a gaze area, a gaze image, and a gaze shift stored in a database in chronological order, a target setting area 1c that displays setting information for setting the gaze image displayed in the gaze display area, and a judgment setting area 1d that displays a judgment condition for the setting information displayed in the target setting area.
[0128] The attention extraction device 1 acquires video information related to a task, for example, via an instructor device 2 of an instructor. The task performed by the instructor may be selected in advance for each of various attention data sets set according to the task of the worker, for example, and a gaze data set for the task performed by the instructor may be generated using the attention data set corresponding to the selected task.
[0129] The attention data set extracted by the attention extraction device 1 is selected by, for example, the workers A to C, and the work of the workers A to C is evaluated using the selected attention data set. The evaluator may monitor the work status of the workers A to C who perform the work using, for example, the attention data set via the attention extraction device 1. The attention extraction device 1 may acquire, for example, the work status of the workers A to C together with the work using the gaze data set, and may acquire images and position information of the work target by the worker device 1 worn by each worker, and display them on the display unit 16 of the attention extraction device 1.
[0130] The evaluator may, for example, refer to the work status of workers A to C displayed on the attention extraction device screen 1a displayed on the attention extraction device 1, and if a task or evaluation other than that in the attention dataset occurs, for example, if worker C is not performing his / her original task (skipping the task order and being in front of another task), and worker B is in a position where he / she can confirm the original work of worker C, the evaluator may "ignore" the task corresponding to the gaze information that worker C should be focusing on, and instruct worker B in real time to "perform the task corresponding to the gaze information that he / she should be focusing on."
[0131] <Teacher device 2> The teacher device 2 is worn by the teacher of the work, as well as by, for example, an expert or a qualified person, and acquires video information seen by the teacher's eyes through the teacher's work. The teacher device 2 may be, for example, a known eye tracking technology, a head-mounted display, or smart glasses, and may acquire audio and surrounding sound information, temperature and humidity, position information, spatial information, and the like, in addition to video information.
[0132] The instructor device 2 is connected to, for example, the attention extracting device 1, other instructor devices 2, the worker device 3, and the server 4 in a state capable of data communication, and may also have, for example, the attention extracting device 1 built therein.
[0133] <Worker device 3> The worker device 3 is worn by the worker performing the work, or by a person other than the instructor, and acquires image information seen by the worker's eyes through the worker's work. The worker device 3 may be, for example, a known eye tracking technology, a head-mounted display, or smart glasses, and may acquire voice and surrounding sound information, temperature and humidity, position information, spatial information, and the like, in addition to the image information.
[0134] The worker device 3 is connected to, for example, the attention extracting device 1, the instructor device 2, other worker devices 3, and a server 4 in a state capable of data communication, and may also have, for example, the attention extracting device 1 built therein.
[0135] <Communication Network 5> The communication network 5 indicates, for example, an Internet network to which the attention extraction device 1, the instructor device 2, and the worker device 3 are connected via communication circuits, and may be configured as an optical fiber communication network. The communication network 5 can be realized by a known communication network such as a wired communication network or a wireless communication network.
[0136] <Learning model> The learning model generates a database by, for example, machine learning. The learning model acquires, as a pair of learning data, a learning target image including a gaze image captured by the instructor device 2 and reference information indicating the correctness of the gaze image captured by the instructor device 2, for each task in the instructor device 2. The learning model is generated by machine learning using the multiple learning data, using a database in which associations between multiple learning target images and multiple pieces of reference information are stored.
[0137] 8 is a schematic diagram showing an example of an attention extraction method in this embodiment. The attention extraction method includes an acquisition step S110, a specification step S120, an extraction step S130, a storage step S140, and a determination step S150. The attention extraction method can be implemented using an attention extraction system 100.
[0138] <Acquisition step S110> In the acquisition step S110, for example, image information and coordinate information are acquired in chronological order by linking them with the work. In the acquisition step S110, the information is acquired using an instructor device 2 equipped with a known camera, imaging device, or the like. In addition, in the acquisition step S110, the image information and coordinate information may be acquired in chronological order by linking them with the work from an operator device 3. In the acquisition step S110, for example, the image information is acquired by a device selected by the instructor and the operator.
[0139] <Specification step S120> In the identification step S120, the instructor's viewpoint displacement is obtained and the instructor's gaze mode is identified. In the identification step S120, the instructor's viewpoint displacement in the visual field range is obtained and the instructor's gaze mode is identified based on the image information and the coordinate information acquired in the acquisition step S110, for example.
[0140] In the identification step S120, for example, if the instructor's viewpoint displacement is within the instructor's visual field radius of 5 degrees to 20 degrees and has a characteristic of concentrating at the center of the visual field range, it is identified as a "wide visual field mode." In addition, in the identification step S120, for example, if the instructor's viewpoint displacement is within the instructor's visual field radius of 4 degrees and has a characteristic of dispersing outside the center of the visual field range, it is identified as an "alert mode."
[0141] In the identification step S120, the gaze modes in the visual field range of the instructor are identified as a "wide-field mode" and an "alert mode" based on, for example, image information in the visual field range 2a acquired by the instructor device 2 and coordinate information (x-axis, y-axis) of the instructor's viewpoint displacement. In the identification step S120, the gaze modes are identified as a "wide-field mode" and an "alert mode" based on, for example, coordinate information of the instructor's viewpoint displacement, but if other characteristics or tendencies can be confirmed, the confirmed gaze modes may be set as new ones.
[0142] <Extraction step S130> In the extraction step S130, for example, a gaze area is set, and a gaze image of the work object gazed by the instructor is extracted. In the extraction step S130, for example, a gaze area gazed by the instructor is set based on the gaze mode (for example, "wide-field mode" or "alert mode") specified in the specification step S120. In the extraction step S130, for example, in the set gaze area, a gaze image of the work object gazed by the instructor is extracted from the video information of the instructor.
[0143] In the extraction step S130, if there are multiple gaze images in the video information of the instructor's gaze area, they may be extracted in the order of gaze or the gaze time. In the extraction step S130, the gaze image of the work object is extracted from the video information in the gaze area where the instructor gazes, according to the conditions set by the evaluator, for example. This makes it possible to grasp the instructor's gaze state and extract appropriate attention information.
[0144] <Storage Step S140> In the storage step S140, for example, the gaze image is linked to the field of view, the viewpoint displacement, and the gaze mode, and stored as attention information. In the storage step S140, for example, the gaze image extracted in the extraction step S130 is linked to the field of view, the viewpoint displacement, and the gaze mode, and stored in the database as attention information for the work, as described above.
[0145] <Determination Step S150> In the determination step S150, the correctness of the task object included in the gaze image acquired, for example, via the worker device 3 is determined using the attention dataset. In the determination step S150, the correctness of the task object is determined, for example, based on the gaze image acquired by the worker device 3 in the acquisition step S110. In addition to evaluation using the attention dataset for each selected task, the determination step S150 may be performed based on evaluation by an evaluator monitoring the task, for example, via the attention extraction device 1.
[0146] In the determination step S150, for example, when the gaze image is acquired by the worker device 3 in the acquisition step S110, the correctness of the work object on which the worker is working may be determined based on an attention data set selected in advance in the worker device 3. The determination step S150 may be configured to determine the work of multiple workers in the same work area.
[0147] The judgment step S150 may be configured to judge the work of multiple workers simultaneously using, for example, the same or a common attention data set, so that judgment can be made according to the work position and work situation of each worker, for example, for a work process that a certain worker overlooked, skipped, or performed out of order.
[0148] In the judgment step S150, for example, various data tables such as a "task information table", a "task procedure table", a "correspondence information table", a "teaching task record table", a "viewpoint record table", a "viewpoint record data table", and a "gaze target table" stored in a database are referenced to judge whether the task target is correct or incorrect based on the various data tables and gaze data sets, and various correspondence information stored in the corresponding task or reference link is acquired according to the judgment result. The correspondence information acquired in the judgment step S150 may be displayed on the worker device 3 of the corresponding teaching worker or other workers, for example.
[0149] Furthermore, in the judgment step S150, addition or updating of corresponding information of various data tables stored in the database is input, for example, via the input unit 17. The input unit 17 can appropriately input, for example, the work processes, judgment conditions, corresponding work, reference links, and the like of the specific targets and judgment targets stored in each of the various data tables.
[0150] This ends the operation of the attention extraction device 1 in this embodiment.
[0151] Furthermore, according to this embodiment, for example, the attention extraction device 1 acquires video information via the acquisition unit 11 of the instructor device 2. The video information may include, for example, recording date and time information of the instructor's work, recording position information, and recording control information related to the operation of acquiring the video information.
[0152] Furthermore, according to this embodiment, for example, the display unit 11 switches the display of the instructor device 2 as shown in Fig. 9. For example, before the acquisition unit 11 acquires video information, the display unit 11 displays the recording date and time information, recording position information, and recording control information in the center of the field of view. Also, while the acquisition unit 11 is acquiring video information, the acquisition unit 11 switches to display only the recording control information in a corner of the field of view. This makes it possible to perform a display that does not interfere with the instructor's work.
[0153] According to the present embodiment, the display unit 16 displays attention information. The display information is nudge information based on nudge theory at the start, middle, or end of work depending on information such as the progress and status of the work performed by the worker. This makes it possible to eliminate bias that affects interactions with conventional computer systems and information, and the effect of notification from the attention extraction system device 1 can be improved by displaying nudge information based on nudge theory.
[0154] Although the embodiment of the present invention has been described, the above-mentioned embodiment is presented as an example and is not intended to limit the scope of the invention. The above-mentioned novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above-mentioned embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0155] 1: Attention extraction device 1a: Attention extractor screen 1b: Gaze display area 1c: Target setting area 1d: Judgment setting area 1e:Workspace map 1f: Work proxy flag 2: Teacher device 2a: Field of view (instructor) 2b: Attention mode 3: Operator equipment 3a: Field of view (worker) 4: Server 5: Communication network 6: Work target 6a:Gaze target 10: Housing 11: Acquisition part 12: Specific part 13:Extraction part 14: Memory section (database) 15: Judgment section 16:Display section 17: Input section 18: Monitoring display section 50:Work area 100: Attention Extraction System S110: Acquisition step S120: Specific step S130: Extraction step S140: Memory step S150: Judgment step
Claims
1. An attention extraction system that extracts attention information for a task, An acquisition means for acquiring image information in a visual field range of an instructor performing the task and coordinate information indicating a viewpoint at which the instructor is gazing within the visual field range in a time series in association with the task; a determination unit that determines a viewpoint displacement of the instructor within the field of view based on the image information and the coordinate information acquired by the acquisition unit, and determines a gaze mode of the instructor; an extraction means for setting a gaze area based on the gaze mode specified by the specification means, and extracting a gaze image of a work object gazed at by the instructor in the gaze area; a storage means for linking the gaze image extracted by the extraction means with the visual field range, the viewpoint displacement, and the gaze mode, and storing the linked image in a database as attention information for the work; An attention extraction system comprising:
2. the gaze area identified by the identification means includes at least a wide field of view mode in which the viewpoint displacement is concentrated at the center of the field of view range, and a vigilance mode in which the viewpoint displacement is dispersed outside the center of the field of view range; The attention extraction system of claim 1 .
3. The acquisition means further includes a determination means for determining whether the task object included in the gaze image is correct or incorrect, A display means for displaying the determination result of the determination means; The attention extraction system of claim 1 .
4. Further comprising a database storing associations between previously acquired past gaze image information and reference information indicating the correctness or incorrectness of the work object associated with the gaze image; The determination means refers to the database, determines whether the task object is correct or incorrect, and obtains corresponding information corresponding to a result of the determination from the database; the display means further outputs the correspondence information acquired by the determination means; 4. The attention extraction system according to claim 1, wherein the attention extraction system further comprises:
5. The apparatus further includes an input unit for inputting the corresponding information output by the determination unit, the storage means associates the corresponding information input by the input means with the gaze image, and stores the association information as an attention data set including setting conditions for recognizing the gaze image as a gaze target; The attention extraction system of claim 1 .
6. the image information acquired by the acquiring means includes recording date and time information of the work by the instructor, recording position information, and recording control information related to an acquisition operation of the image information, the display means displays the recording date and time information, the recording position information, and the recording control information in the center of the field of view before the video information is acquired by the acquisition means, and switches to displaying only the recording control information in a corner of the field of view while the video information is being acquired; 4. The attention extraction system according to claim 1, wherein the attention extraction system further comprises:
7. The display means includes: an acquisition display area for displaying to a worker performing the work an instruction for selecting a type of the acquisition means for acquiring the image information, a type of the attention data set stored in the database, and a start of the work, and an attention information display area for displaying corresponding information corresponding to the gaze image of the worker and attention information based on the attention data set after the selection, The warning information displayed in the warning display area includes nudge information displayed at least at any one of a start of work, a middle of work, and an end of work, according to the progress of the work of the worker; Displaying information including at least one of the corresponding information, the attention information, and the nudge information displayed in the attention information display area in a divided manner according to the gaze state of a worker performing the work, based on the attention data set and a result of the determination by the determination means; 4. The attention extraction system according to claim 1, wherein the attention extraction system further comprises:
8. An attention extraction method for extracting attention information for an operation, comprising: an acquisition step of acquiring image information in a visual field range of an instructor performing the task and coordinate information indicating a viewpoint at which the instructor gazes within the visual field range in a time series in association with the task; a determination step of determining a viewpoint displacement of the instructor in the field of view range based on the image information and the coordinate information acquired by the acquisition step, and determining a gaze mode of the instructor; an extraction step of setting a gaze area based on the gaze mode identified by the identification step, and extracting a gaze image of a work object gazed at by the instructor in the gaze area; a storage step of associating the gaze image extracted by the extraction step with the visual field range, the viewpoint displacement, and the gaze mode, and storing the image in a database as attention information for the work; to have a computer execute The attention extraction method is characterized by:
Citation Information
Patent Citations
Work support system, work support method, and terminal for work
JP2013097466A
Technique managing device, and technique managing method
WO2010137165A1
Wire connection work support system
JP2015061339A
Skill transmission system and method
JP2017191490A
