Attention extraction system and attention extraction method

Abstract

The invention provides an attention extraction system and an attention extraction method for extracting attention information for a job. In an attention extraction system (100) that extracts attention information for a job, an attention extraction device (1) is provided with: an acquisition means that associates video information and coordinate information with the job and acquires in time series; a specifying means for determining a viewpoint shift of the teacher in the field of view and specifying a gaze pattern of the teacher; an extraction means for setting a gaze region in accordance with the gaze pattern and extracting a gaze image of the work object gazed by the teacher in the gaze region; and a storage means that associates the extracted gaze image with the field of view, the viewpoint shift, and the gaze pattern and stores the same in a database as attention information for the job.

Description

Technical Field

[0001] The present invention relates to an attention extraction system and an attention extraction method for extracting attention information on a task. Background Art

[0002] Conventionally, as technologies related to attention extraction, for example, a skill transfer system disclosed in Patent Document 1 and a wiring work support system disclosed in Patent Document 2 have been proposed.

[0003] In the skill inheritance system disclosed in patent document 1, the following technology is disclosed: based on teacher data, machine learning is performed on first image data and line of sight data of the first user's field of view, feature data of the image is extracted and input as teacher data, information on the first user's focus area in the first image is registered in association with the feature data, second image data of the second user's field of view is sent to a server, data on the focus area associated with the feature data equivalent to the second image is read out, auxiliary display data is created, and sent to the eyeglass terminal of the second user, and in the eyeglass terminal of the second user, a display is performed based on the auxiliary display data so that the information on the focus area overlaps with the field of view of the second user.

[0004] In addition, in the wiring operation assisting system disclosed in Patent Document 2, text and color are recognized by a sight tracking device that tracks the sight of the operator, a photographing device that photographs the wiring of the operation object, and image processing of the operator's visual range. A portable operation management device is disclosed, which has an image processing unit that guides the information of the object stored in the data storage management unit to the operator, and stores the operation end signal from the limit switch provided on the operation tool and the image of the operation object.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2017-191490

[0008] Patent Document 2: Japanese Patent Application Publication No. 2015-061339 Summary of the invention

[0009] Problems to be solved by the invention

[0010] Here, Patent Document 1 assumes that the users can coordinate their gazes and pass on skills to each other. That is, it is difficult to grasp the state of gaze based on the gaze time and gaze shift of the viewpoint and extract appropriate attention information to provide to the operator. Furthermore, there is no record or suggestion on the determination of attention actions that indicate what the instructor is looking at and in what situation.

[0011] In addition, Patent Document 2 assumes that the operator's line of sight is associated with the work object and the information of the object is guided to the operator. That is, it is difficult to grasp the state of gaze based on the gaze time and gaze shift of the viewpoint and extract appropriate attention information to provide to the operator. Moreover, there is no record or suggestion on the determination of the attention action indicating what the instructor is looking at and in what situation.

[0012] Therefore, the present invention is proposed in view of the above-mentioned problems, and its object is to provide an attention extraction system and an attention extraction method that can grasp the state of the instructor's gaze and realize the extraction and provision of appropriate attention information.

[0013] Means for solving problems

[0014] The attention extraction system of the first invention extracts attention information about a job, and is characterized in that the attention extraction system comprises: an acquisition unit, which associates image information within the field of view of a teacher performing the job and coordinate information indicating the viewpoint at which the teacher is gazing within the field of view with the job and acquires them in a time series; a determination unit, which calculates the viewpoint displacement of the teacher within the field of view based on the image information and the coordinate information acquired by the acquisition unit, and determines the gaze pattern of the teacher; an extraction unit, which sets a gaze area based on the gaze pattern determined by the determination unit, and extracts a gaze image of the job object at which the teacher is gazing in the gaze area; and a storage unit, which associates the gaze image extracted by the extraction unit with the field of view, the viewpoint displacement, and the gaze pattern, and stores it in a database as attention information in the job.

[0015] The attention extraction system of the second invention is characterized in that, in the first invention, the acquisition unit further includes a determination unit that determines whether the work object included in the attention image is correct, and the attention extraction system further includes a display unit that displays the determination result of the determination unit.

[0016] The attention extraction system of the third invention is characterized in that, in the second invention, there is also a database, which stores the correlation between the past attention image information acquired in advance and the reference information indicating the correctness of the work object associated with the attention image, the judgment unit refers to the database, judges the correctness of the work object, and obtains corresponding information corresponding to the judgment result from the database, and the display unit also outputs the corresponding information obtained by the judgment unit.

[0017] The attention extraction system of the fourth invention is characterized in that, in the second invention, it also has an input unit, which inputs the corresponding information output by the judgment unit, and the storage unit associates the corresponding information input by the input unit with the attention image and stores it as an attention data set containing set conditions for identifying it as an attention object.

[0018] The attention extraction system of the fifth invention is characterized in that, in the second invention, the image information acquired by the acquisition unit includes the recording date and time information, the recording position information and the recording control information related to the acquisition operation of the image information of the work performed by the instructor, and the display unit displays the recording date and time information, the recording position information and the recording control information in the center of the field of view before the acquisition unit acquires the image information, and during the acquisition of the image information, only the recording control information is switched to be displayed in the corner of the field of view.

[0019] The attention extraction system of the sixth invention is characterized in that, in the second invention, the display unit further has an attention information display area, which switches the display of an acquisition display area and an attention display area, wherein the acquisition display area displays, for the operator performing the operation, the category of the acquisition unit that acquires the image information, the category of the attention data set stored in the database, and instructions for selecting the start of the operation, respectively, and the attention display area displays corresponding information and attention information corresponding to the operator's gaze image based on the attention data set after the selection, so that the attention information displayed in the attention display area includes boosting information, and the boosting information is displayed at at least any time at the beginning, middle or end of the operation according to the operation process of the operator, and the information displayed in the attention information display area, including at least any of the corresponding information, the attention information or the boosting information, is allocated for display based on the attention data set and the result of the judgment by the judgment unit and according to the gaze status of the operator performing the operation.

[0020] The attention extraction method of the seventh invention is characterized in that a computer is caused to execute the following steps: an acquisition step, in which image information within the field of view of a teacher performing the operation and coordinate information indicating the viewpoint at which the teacher is gazing within the field of view are associated with the operation and acquired in a time series; a determination step, in which, based on the image information and coordinate information acquired by the acquisition step, the gaze mode of the teacher is determined to be a wide field of view mode when the coordinate information of the time series indicating the viewpoint displacement of the teacher within the field of view is concentrated at the center of the field of view, and the gaze mode of the teacher is determined to be a warning mode when the coordinate information of the time series indicating the viewpoint displacement of the teacher within the field of view is dispersed to a position outside the center of the field of view; an extraction step, in which a gaze area is set according to the gaze mode determined by the determination step, and a gaze image of the operation object at which the teacher is gazing in the gaze area is extracted; and a storage step, in which the gaze image extracted by the extraction step is associated with the field of view, the viewpoint displacement and the gaze pattern and stored in a database as attention information in the operation.

[0021] Effects of the Invention

[0022] According to the first to seventh inventions, the determination unit determines the instructor's gaze pattern by finding the instructor's viewpoint shift within the visual field based on the image information and the coordinate information. Therefore, in the extraction unit, the gaze area can be set according to the determined gaze pattern, and the gaze image of the work object that the instructor is gazing at in the gaze area can be extracted. Thus, based on the gaze time and gaze shift of the viewpoint, it is possible to extract a gaze image that indicates what the instructor is gazing at under what circumstances, and it is possible to accurately grasp the state of gaze and provide appropriate attention information.

[0023] In particular, according to the second invention, the gaze area includes a wide field of view mode and a warning mode of the instructor. Therefore, it is possible to set a wide field of view mode in which the instructor's viewpoint shift is concentrated at the center of the field of view and a warning mode in which the instructor's viewpoint shift is dispersed to a position outside the center of the field of view. Thus, it is possible to extract a gaze image showing what the instructor is looking at and under what circumstances.

[0024] In particular, according to the third invention, the acquisition unit further includes a determination unit. Therefore, it is possible to determine whether the work object included in the gaze image is correct. Thus, the state of gaze of the instructor and the operator can be accurately grasped, and appropriate attention information can be provided.

[0025] In particular, according to the fourth invention, the database stores the association between the previously acquired past gaze image information and the reference information indicating the correctness of the work object associated with the gaze image. Therefore, the judgment unit refers to the database to judge the correctness of the work object, and the corresponding information corresponding to the judgment result can be output through the display unit. As a result, the state of the operator's gaze can be accurately grasped, and appropriate attention information can be provided.

[0026] In particular, according to the fifth invention, the input unit inputs the corresponding information output by the determination unit. Therefore, the storage unit can associate the corresponding information with the gaze image and store it as an attention data set including setting conditions for identifying the gaze object. Thus, the state of the gaze can be accurately grasped and appropriate attention information can be provided.

[0027] In particular, according to the sixth invention, the display unit switches the information displayed in the visual field before and during the acquisition of the image information. Therefore, before the acquisition of the image information, the recording date and time information, the recording position information, and the recording control information of the teacher's work are displayed in the center of the visual field, and during the acquisition, only the recording control information is displayed in the corner of the visual field. As a result, the acquisition of the image information of the teacher performing the work and the extraction of the gaze image can be performed without placing a burden on the teacher.

[0028] In particular, according to the seventh invention, the display unit switches and displays the acquisition display area and the attention display area. Therefore, it is possible to allocate and display corresponding information, attention information, or boosting information according to the attention status of the operator performing the operation based on the attention data set and the judgment result. As a result, it is possible to accurately grasp the operator's attention status and provide appropriate attention information.

[0029] According to the eighth invention, the determination step determines the instructor's gaze pattern by finding the instructor's viewpoint shift within the visual field based on the image information and the coordinate information. Therefore, in the extraction step, the gaze area can be set according to the determined gaze pattern, and the gaze image of the work object that the instructor is gazing at in the gaze area can be extracted. Thus, based on the gaze time and gaze shift of the viewpoint, it is possible to extract a gaze image that indicates what the instructor is gazing at under what circumstances, and it is possible to accurately grasp the state of gaze and provide appropriate attention information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram showing an example of the structure of the attention extraction system in this embodiment.

[0031] Figure 2 1 is a schematic diagram showing an example of attention extraction by a teacher and work evaluation by a worker in the attention extraction system 100 according to the present embodiment.

[0032] Figure 3 (a)~ Figure 3 (c) is a schematic diagram showing an example of the attention extraction method in this embodiment.

[0033] Figure 4 (a) is a schematic diagram showing an example of the structure of the attention extraction device, Figure 4 (b) is a schematic diagram showing an example of the function of the attention extraction device.

[0034] Figure 5 It is a schematic diagram showing an example of a database in this embodiment.

[0035] Figure 6 It is a schematic diagram showing a first modified example of the database in this embodiment.

[0036] Figure 7 It is a schematic diagram showing an example of a data table stored in the database in this embodiment.

[0037] Figure 8 It is a schematic diagram showing an example of a flowchart of the attention extraction method in this embodiment.

[0038] Fig. 9 (a)~ Fig. 9 (e) is a schematic diagram showing an example of the attention extraction method in this embodiment.

[0039] Fig.10 (a) and Fig.10 (b) is a schematic diagram showing an example of a display of the operator device in this embodiment.

[0040] Fig.11 It is a schematic diagram showing an example of the display of the attention extracting device in this embodiment.

[0041] Fig.12 (a)~ Fig.12 (e) is a schematic diagram showing an example of a display of the operator terminal in this embodiment. DETAILED DESCRIPTION

[0042] Hereinafter, an example of an attention extraction system and an attention extraction method in an embodiment of the present invention will be described with reference to the drawings.

[0043] (Present embodiment: attention extraction system 100 and attention extraction method)

[0044] Reference Figure 1 and Figure 2 , an example of the structure of the attention extraction system 100 in this embodiment is described. Figure 1is a schematic diagram showing an example of the structure of the attention extraction system 100 in this embodiment, Figure 2 1 is a schematic diagram showing an example of the teacher's attention extraction and the worker's work evaluation in the attention extraction system 100 according to the present embodiment.

[0045] The attention extraction system 100 is used to extract the object of the operation that the instructor is paying attention to, using the image information within the visual field of the instructor who is performing the operation and the coordinate information indicating the viewpoint that the instructor is paying attention to within the visual field. In the attention extraction system 100, for various information including the image information, the coordinate information, and the operation information related to the operation, the image information and the coordinate information can be obtained within the visual field of the instructor under various conditions.

[0046] For example Figure 1 As shown, the attention extraction system 100 includes an attention extraction device 1, an instructor device 2, an operator device 3, and a server 4. For example, the instructor device 2 and the operator device 3 may be provided in a plurality in the work area 50. The attention extraction system 100 may also transmit / receive various information to the attention extraction device 1, the instructor device 2, the operator device 3, the server 4, and other user devices (not shown) via a known communication network 5, for example.

[0047] The attention extraction system 100 acquires the viewpoint shift 2b of the visual field destination that the instructor is looking at, which is included in the visual field 2a of the instructor, through the instructor device 2 worn by the instructor. The information acquired by the attention extraction system 100 from the instructor may include, for example, image information (images within the visual field where the work is performed), coordinate information (viewpoint, position information, etc.), work information (work date and time, work instructions, process information, etc.), instructor information (instructor ID, device ID, etc.), and various information (assigned work, group work, etc.) performed by the instructor.

[0048] For example, Figure 2 As shown, the attention extraction system 100 associates the image information within the visual field of the instructor who is performing the operation and the coordinate information indicating the viewpoint that the instructor is looking at within the visual field with the operation performed by the instructor from the instructor device 2 worn by the instructor and acquires them in a time series. For acquiring the image information and the coordinate information, for example, a known eye tracking technology, a technology for acquiring viewpoints provided by a head mounted display or smart glasses, etc., etc. may be used.

[0049] The attention extraction system 100 acquires the image information and the coordinate information by, for example, the above-mentioned technique possessed by the instructor device 2, determines the instructor's gaze pattern in the attention extraction device 1, and extracts the gaze image according to the determined gaze pattern. The attention extraction system 100 associates the extracted instructor's gaze image with the reference information associated with the gaze image, and stores the image as an attention data set in a database. The attention extraction system 100 may also refer to the database, acquire the attention information associated with the reference information, and display the information on the operator device 1 of the operator who uses the attention data set.

[0050] Afterwards, the attention extraction system 100 evaluates the operator's work based on, for example, the image information of the work object 6 included in the operator's field of view 3a, the numerical information and coordinate information of the viewpoint shift of the field of view destination, etc., based on the attention data set selected by the operator, and based on the evaluation result, for example, the work object 6 is determined as the attention information, and then corresponding information such as "(1) confirmation" and "(2) adjustment" is obtained, and is superimposed on the field of view 3a of the operator device 3 and displayed together with the actual display. The details of each structure of the attention extraction device 1, the teaching device 2, and the operator device 3 are described later.

[0051] Here, refer to Figure 3 , the instructor's gaze pattern is described. The attention extraction system 100 is, for example, Figure 3 (a)~ Figure 3 The attention extraction method shown in (c) of FIG. 1 determines the instructor's gaze pattern. For example, Figure 3 As shown in (a), the attention extraction system 100 determines the gaze pattern within the visual field of the instructor based on the image information in the visual field 2a acquired by the instructor device 2 and the coordinate information (x-axis, y-axis) of the instructor's viewpoint shift.

[0052] The attention extraction system 100 obtains the video information and coordinate information acquired through the instructor device 2, calculates the viewpoint displacement of the instructor in the visual field 2a according to the coordinates, and determines the instructor's gaze pattern. Figure 3 As shown in (b), when the characteristics of the instructor's line of sight shift are "narrow shift range (concentrated)" and "few points of attention at the viewpoint", the attention extraction system 100 can be determined as "wide field of view mode". Figure 3 As shown in (c), when the characteristics of the instructor's line of sight shift are "large shift range (dispersion)" and "many viewpoints of attention", the attention extraction system 100 can determine it as the "alert mode".

[0053] The attention extraction system 100 sets the attention area that the instructor is looking at according to the determined attention pattern, extracts the attention image of the work object that the instructor is looking at in the set attention area, associates the extracted attention image with the visual field, viewpoint shift, and attention pattern, and stores it in the database of the server 4 as the attention information in the instructor's work. In addition, there may be multiple instructors and operators, for example, multiple instructors and operators may work on the same process, and the attention extraction system 100 may, for example, assign one work process of the instructor in a manner that multiple operators share the work.

[0054] The allocation of instructors and operators by the attention extraction system 100 may be determined based on, for example, the number and difficulty of the work process, the skills of the operators, the delivery date of the work, etc., or may be determined by, for example, an evaluator via the attention extraction device 1. How many operators are assigned to which work and how to configure the operators are arbitrary and can be determined appropriately.

[0055] <Attention Extraction Device 1>

[0056] The attention extraction device 1 calculates the viewpoint displacement of the instructor within the field of view and determines the instructor's gaze pattern based on, for example, the image information within the field of view of the instructor's work acquired by the instructor device 1 and the coordinate information indicating the viewpoint the instructor is looking at within the field of view.

[0057] The gaze area determined by the determination unit includes, for example: a wide field of view mode, in which the instructor's viewpoint shift is contained within a field of view radius of 5 degrees to 20 degrees and is concentrated at the center of the field of view; and a warning mode, in which the instructor's viewpoint shift is contained within a field of view radius of 4 degrees and is dispersed to a position outside the center of the field of view.

[0058] For example, as mentioned above Figure 3 As shown in (b), the attention extraction device 1 determines the situation where the characteristics of the instructor's line of sight shift are "narrow shift range (concentrated)" and "few points of attention at the viewpoint" as the "wide field of view mode". On the other hand, for example, as described above Figure 3 As shown in (c), the "alert mode" is determined when the characteristics of the instructor's line of sight shift are "large shift range (dispersion)" and "many points of attention of the viewpoint".

[0059] The attention extraction device 1 sets a gaze area according to the determined gaze pattern, and extracts a gaze image of the work object that the instructor is gazing at in the gaze area. The attention extraction device 1 associates the extracted gaze image with the visual field, viewpoint shift, and gaze pattern, and stores it in a database as attention information during the work.

[0060] Here, Fig.11An example of the attention extraction device screen 1 a of the attention extraction device 1 in the attention extraction system 100 is shown in FIG. Fig.11 For example, the screen is operated by the evaluator and displays various information acquired by the teaching device 1 .

[0061] The attention extraction device 1, for example, refers to a database and displays a gaze monitoring area on the attention extraction device screen 1a, and the gaze monitoring area at least includes: a gaze display area 1b (viewpoint movement distance curve graph, viewpoint tracking, gaze area, gaze object, etc.), which displays coordinate information, gaze area, gaze image and gaze shift in a time series; an object setting area 1c (gaze object evaluation, identifier generation, etc.), which displays setting information for setting the gaze image displayed in the gaze display area 1b; and a judgment setting area 1d (corresponding information setting, etc.), which displays the judgment conditions of the setting information displayed in the object setting area 1c.

[0062] The attention extraction device 1 receives various conditions and values ​​related to settings and adjustments from the evaluator, for example, via a setting item menu displayed on the attention extraction device screen 1a. The attention extraction device 1 performs display, setting and adjustment of conditions in the attention display area 1b, the object setting area 1c and the judgment setting area 1d, for example, based on the received various conditions and values.

[0063] The attention extraction device 1 determines whether the work object included in the teacher's attention image is correct or not according to various conditions received from the evaluator, for example, through a setting item menu displayed on the attention extraction device screen 1a. The attention extraction device 1 may also display the determination result on the teacher device 2 or the worker device 3.

[0064] The attention extraction device 1 receives input of corresponding information from the evaluator, for example, via a setting item menu displayed on the attention extraction device screen 1a. In addition to receiving input of new corresponding information, the attention extraction device 1 can also update and delete existing conditions and settings, corresponding information, etc., and store them in the database of the server 4.

[0065] For example, the attention extraction device 1 may associate the input corresponding information with the attention image, generate an attention data set including setting conditions for identifying the attention object, and store the generated attention data set in the database of the server 4 .

[0066] Figure 4(a) is a schematic diagram showing an example of the structure of the attention extraction device 1. As the attention extraction device 1, in addition to being able to use a single-board computer such as Raspberry Pi (registered trademark), for example, a well-known electronic device such as a personal computer (PC) can also be used. The attention extraction device 1 has, for example, a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage unit 104, and I / Fs 105 to 107. The respective structures 101 to 107 are connected via an internal bus 110.

[0067] The CPU 101 controls the entire attention extraction device 1. The ROM 102 stores the action code of the CPU 101. The RAM 103 is a work area used by the CPU 101 when it is working. The storage unit 104 stores various information such as learning models and databases. As the storage unit 104, in addition to an SD memory card, for example, a well-known data storage medium such as a HDD (Hard Disk Drive) and an SSD (Solid State Drive) is used.

[0068] The I / F 105 is a well-known interface for transmitting and receiving various information with the teaching device 2, the operator device 3, the server 4, the communication network 5, etc. connected according to the purpose. For example, a plurality of I / Fs 105 may be provided.

[0069] The I / F 106 is a well-known interface for transmitting and receiving various information to and from an input unit 108 connected according to the application. The input unit 108 uses, for example, a keyboard, and an administrator or the like who manages the attention extraction system 100 inputs or selects various information or control commands of the attention extraction device 1 through the input unit 108.

[0070] The I / F 107 is a well-known interface for transmitting and receiving various information to and from a display unit 109 connected according to the application. The display unit 109 outputs various information stored in the storage unit 104 and the processing status of the attention extraction device 1. As the display unit 109, for example, a display is used, and for example, a touch panel type may be used. In this case, the display unit 109 may include the input unit 108.

[0071] In addition, for example, the same interface may be used as I / F 105 to I / F 107, or, for example, a plurality of interfaces may be used as each of I / F 105 to I / F 107. In addition, at least any one of the teaching device 2, the operator device 3, the server 4, the communication network 5, the input unit 108, and the display unit 109 may be removed according to the situation.

[0072] <Storage Unit 14 (Database)>

[0073] The storage unit 14 stores various databases in the storage unit 104, for example. The storage unit 14 stores the association between the previously acquired past evaluation target information and the reference information associated with the past evaluation target information, for example, a learning model having the association. The storage unit 14 may be provided in the teaching device 2, the operator device 3, and the server 4, for example.

[0074] The storage unit 14 records the attention actions (attention points, viewpoints, etc.) of the instructor when acquiring the instructor's work. The attention extraction system 100 may also determine the attention mode based on information such as the attention image, the visual field, and the viewpoint shift acquired through the instructor device 2. The attention extraction system 100 may also set the instructor's visual field shift characteristics as "narrow (concentrated) shift range" and "few attention points" as "wide visual field mode". On the other hand, as described above, Figure 3 As shown in (c), the characteristics of the instructor's line of sight shift are determined as "wide shift range (dispersion)" and "many points of attention in the viewpoint" are determined as "alert mode" and stored in the storage unit 14 of the server 4 as attention information in the instructor's work.

[0075] The storage unit 14 stores information related to the teacher's work, which is obtained through gestures, voices, sight lines, and other input devices, for example, through various interfaces (not shown) of the input system of the teacher device 2. The attention extraction system 100, for example, associates various information obtained through the input device with information about the teacher and information about the teacher's work, and stores them in the storage unit 14. The attention extraction system 100 can also appropriately record various information according to recording instructions such as recording start and recording end implemented by the teacher through the teacher device 2.

[0076] In the storage unit 14, for example, the coordinate position of the viewpoint setting supplemented by the instructor, the change of the coordinate position, the change speed, etc. are sequentially recorded via the instructor device 2. In addition, for example, the position and orientation of the head in the work space coordinates can be recorded for each spatial layout of the place where each operation is performed. The extraction system 100 can also obtain various information recorded for each spatial layout through a camera such as a well-known 360° camera, a surveillance camera, or a network camera, and store it in the storage unit 14 in association with the information indicating the coordinate position of the instructor's head and the change of the coordinate position.

[0077] In addition, the storage unit 14 may also record the photographic data and various data recorded in association with the photographic data as a data set of attention in the teacher's work after recording in the database is completed. The data set of attention is, for example, evaluated by the evaluator on the attention points or skills via the attention extraction device 1, and is registered (recorded) in the storage unit 4 as corresponding information in the monitoring of the attention object identifier and the attention object.

[0078] Here, for example, the attention object identifier refers to a device generated using the attention object image evaluated by the attention extraction system 100, and may be, for example, a well-known image processing program such as pattern matching or an image recognition model. The image processing program or image recognition model may also be installed in the operator device 3 worn by the operator when performing the work. Thus, the operator can evaluate the attention information for the work based on the time series image information and coordinate information acquired through the operator device 3 when performing the work.

[0079] In addition, the storage unit 14 may further include a database storing, for example, the association between the previously acquired past gaze image information and the reference information indicating the correctness of the work object associated with the gaze image. The storage unit 14 may be used, for example, when the determination unit 15 refers to and determines the correctness of the work object in the work of the instructor. The determination unit 15 may, for example, acquire the corresponding correspondence information based on the result of the determination. The display unit 16 may further output the corresponding correspondence information from the storage unit 14 based on the content determined by the determination unit 15.

[0080] In addition, for example, the correspondence information during monitoring of the attention object identifier and the attention object acquired by the teaching device 2 and evaluated as the attention skill by the evaluation unit 15 can be registered in the storage unit 14 as an attention data set of the work in the worker device 13 .

[0081] The storage unit 14 may store, for example, past evaluation object information and reference information acquired by the teaching device 2. The association is constructed by, for example, machine learning using a plurality of learning data using the past evaluation object information and reference information as a set of learning data. As a learning method, deep learning such as a convolutional neural network is used.

[0082] In this case, for example, the relevance indicates the degree of association between many-to-many information (a plurality of data included in past evaluation target information and a plurality of data included in reference information).

[0083] The correlation is updated appropriately during the machine learning process. That is, the correlation, for example, represents a function optimized based on past image data A, corresponding information A, and reference information. Therefore, the evaluation result of the evaluation object information is generated using the correlation constructed based on all the results of the evaluation of the evaluation object information in the past. As a result, even if the evaluation object information has a configuration state combined with other image data A, corresponding information A, etc., it is possible to generate the best evaluation result.

[0084] In addition, the evaluation object information stored in the storage unit 14 may be the same as or similar to the past evaluation object information, or may be different. Thus, the attention extraction system 100 can quantitatively generate the best evaluation result. In addition, the attention extraction system 100 can improve the generalization ability when performing machine learning, and can improve the evaluation accuracy for unknown evaluation object information.

[0085] Furthermore, the correlation may include, for example, a plurality of correlation degrees indicating the degree of correlation between a plurality of data included in past evaluation target information and a plurality of data included in reference information. For example, when the learning model is constructed by a neural network, the correlation degree may correspond to a weight variable.

[0086] The past evaluation target information indicates the same type of information as the above-mentioned evaluation target information. The past evaluation target information includes, for example, a plurality of pieces of evaluation target information acquired when the image data A was evaluated in the past.

[0087] The reference information is associated with the past evaluation object information and indicates information related to the image data A and the corresponding information A. For example, in addition to indicating the scope of work for the image data A, work instructions, work steps, the content of subsequent processes or related work, the content of shared work or substitute work of other workers working in the same area, and evaluations based on various work relationships and mutual influences (such as "confirmation", "adjustment", "attention (attention!)", "judgment", "correspondence", etc.), the reference information may also include various information related to the progress of the work process, the inspection list, the focus image, the priority of the display information, the combination of display and non-display, numerical values ​​and data, distribution, etc. In addition, the specific content included in the reference information can be set arbitrarily.

[0088] For example, Figure 5 As shown in FIG. 1 , the correlation may also indicate the degree of correlation between the past evaluation object information and the reference information. In this case, by using the correlation, it is possible to compare multiple data (in Figure 5 "Reference A" to "Reference C" in the above) are respectively related to a plurality of data (in Figure 5 Therefore, for example, through the association, one data included in the past evaluation target information can be associated with multiple data included in the reference information, and various evaluation results can be generated.

[0089] For example, the correlation has multiple correlation degrees that respectively associate multiple data included in the past evaluation object information with multiple data included in the reference information. The correlation degree is expressed in three or more levels such as percentage, 10 levels or 5 levels, for example, by the characteristics of the line (such as thickness, etc.). For example, the "image data A" contained in the past evaluation object information shows the correlation degree AA "80%" with the "reference A" contained in the reference information, and the correlation degree AB "55%" with the "reference B" contained in the reference information. That is, the "correlation degree" indicates the degree of association between each data, for example, the higher the correlation degree, the stronger the association between each data. In addition, when constructing the correlation through the above-mentioned machine learning, it can also be set that the correlation has a correlation degree of three or more levels.

[0090] Past evaluation object information can also be Figure 6As shown, the past evaluation object information is divided into the fixation image of the past work object and the corresponding information of the past work object and stored in the database. In this case, the correlation degree is calculated based on the relationship between the combination of the image data of the past fixation image and the past corresponding information and the reference information. In addition to the above information, the past evaluation object information may also be divided into past countermeasure information and stored in the database.

[0091] For example, the combination of "image data a" included in the past fixation image and "correspondence information a" included in the past correspondence information shows a correlation degree AAA "85%" with "reference A" and a correlation degree ABA "25%" with "reference B". In this case, the data can be stored in a manner that allows the past object data and the past correspondence information to be independently stored. Therefore, when generating the evaluation result, the accuracy can be improved and the range of options can be expanded.

[0092] The past evaluation object information may include, for example, synthetic data and similarity. The synthetic data is represented by a similarity of more than three levels with the past object data or the past corresponding information. In addition to being stored in the database in the form of numerical values, matrices, or histograms, the synthetic data may also be stored in the form of images or character strings, for example.

[0093] Here, refer to Figure 7 An example of a database in the present embodiment is described. In the database, in addition to various information related to, for example, an instructor (teaching operator), an evaluator, an operator, etc., who are users of the attention extraction system 100, information related to the content and process of various operations and a plurality of data sets for operators to use are stored as attention data sets.

[0094] The attention data set associates various information such as information of each data table, work of the instructor, operator, etc., and evaluation of the evaluator with each data. The evaluation of the evaluator is input, for example, by the input unit 17. The attention data set stores at least, for example, a "work information table", "work step table", "correspondence information table", "teaching work record table", "viewpoint record table", "viewpoint record data table", and "attention object table".

[0095] <<Job Information Table>>

[0096] In the "job information table", for example, data for identifying a job to be performed is stored (saved). In the "job information table", for example, a "job ID" and a "job name" for identifying a job performed by a teacher or a worker are stored.

[0097] <<Working Steps Table>>

[0098] In the "operation step table", for example, data related to the steps of the operation performed by the instructor or the operator is stored (saved). In the "operation step table", for example, a "operation step ID" for identifying the steps of the operation is stored, and in association with the "operation step ID", "operation ID", "operation step name", and "operation sequence" are stored in correspondence. The "operation step table" is associated with the "operation information table" through the "operation ID", for example.

[0099] <<Corresponding information table>>

[0100] In the "corresponding information table", for example, data is stored (saved) for identifying information corresponding to the work displayed by the display unit after the judgment unit determines the correctness of the work object based on the image information and viewpoint coordinate information acquired by the acquisition unit when the operator is performing the work.

[0101] The "corresponding information table" stores, for example, a "corresponding information ID" for identifying information corresponding to a job, a "job step ID", "preliminary" indicating a nudge-based urging or a prompt to confirm information in advance, and a "display type" indicating a display type such as an "interruption type" displayed based on a job status headed by missed detection.

[0102] In the "display type", for example, a trigger indicating the start timing, a trigger indicating the display time, or a trigger indicating the end of the display is set, such as the animation effect timing setting. When the display type is, for example, "interruption", information indicating the timing for determining the interruption is stored as "display / determination start condition". The determination start condition may also be, for example, a trigger for specifying the interruption determination.

[0103] Furthermore, in the "corresponding information table", the "determination criteria" indicating the criteria and thresholds used by the instructor to determine "observed" or "not observed" when transferring to the "interruption determination" and the display content displayed as corresponding information are stored as "corresponding information content".

[0104] In the "corresponding information content", for example, in addition to the designation of the file name, the corresponding information created at the workplace can also be stored by marking, etc. In the "corresponding information content", for example, single instructions such as contacting a superior or senior person can also be stored, which can be generated by inputting through the editing screen.

[0105] In addition, the "corresponding information table" stores, for example, "job result record display conditions" which indicate conditions such as the timing for displaying the contents of the results recorded for each job step, such as reports and check lists representing the job, and "job result record contents" which indicate the contents of the job results, such as reports and check lists used to record the job.

[0106] The attention extraction system 100 may also refer to the "operation result record display condition" and determine that the attention object has reached an appropriate state by image comparison, for example, based on a signal based on a gesture or voice input. The attention extraction system 100 may also store various information and data in the "operation result record display condition" as conditions for storing the determination result.

[0107] In addition, the attention extraction system 100 may also make the result input of the content in the "work result record content" mandatory. The attention extraction system 100 may also, for example, determine this result input and determine that one work step is completed based on the result of the determination. The "corresponding information table" is associated with the "work step table" through, for example, the "work step ID".

[0108] <<Teaching Work Record Form>>

[0109] In the "teaching operation record table", for example, various information related to the operation performed by the teacher is stored (saved). In the "teaching operation record table", for example, a "teaching operation record ID" for identifying the recorded teaching operation, a "teaching operation date and time" indicating the date and time when the teaching operation was performed, a "teaching operator" indicating the teacher who performed the teaching operation, a "teaching operation location" indicating the place and area where the teaching operation was performed, and a "job ID" are stored in association with each other. The "teaching operation storage table" is associated with the "job information table" through the "job ID", for example.

[0110] <<Viewpoint Record Table>>

[0111] In the "viewpoint record table", for example, information and data related to the viewpoints at which the instructor works, acquired by the acquisition unit 11, are stored (saved). In the "viewpoint record table", data of the "viewpoint record ID" for identifying the viewpoint record at which the instructor works and the coordinate information of the viewpoint based on eye tracking are stored as a "viewpoint record data file".

[0112] The "viewpoint record table" may store stream data acquired in milliseconds. Thus, the attention extraction system 100 can store the actual file name of data such as JavaScript (registered trademark) object notation (JSON) instead of directly storing the data in a DB.

[0113] In addition, in the "viewpoint record table", for example, a "viewing range recording image file" is stored, which is data indicating image information within the visual range of the instructor recorded by an external camera. In addition, in the "viewpoint record table", a "teaching operation record ID" for identifying the history of teaching operations performed by the instructor is stored. The "viewpoint record table" is associated with the "teaching operation record table" through, for example, a "teaching operation record ID".

[0114] <<Viewpoint Record Data Table>>

[0115] In the "viewpoint record data table", for example, detailed information of coordinate information of multiple viewpoints acquired by the acquisition unit 11 when the instructor performs the operation is stored. In the "viewpoint record data table", a "viewpoint identification data file" that identifies the file that stores the viewpoint record data and a "viewpoint record elapsed time" that indicates the time of viewpoint identification recorded by the instructor are stored. The "viewpoint record elapsed information" sets the start of the record of the start of the operation performed by the instructor to 00:00:00, and stores (keeps) the elapsed time of the recorded viewpoint as a record. The "viewpoint record elapsed information" can be recorded together with various position information at unit time intervals (for example, 1 second intervals, etc.). The "viewpoint record data table" is associated with the "teaching operation record table" through, for example, the "viewpoint identification data file".

[0116] In addition, in the "viewpoint record data table", the position information of the teaching operator in the work space is stored as "teaching operator position information (X, Y, Z)". In the "teaching operator position information (X, Y, Z)", in addition to recording the three-dimensional coordinates of the position of the HMD worn by the teaching operator, that is, the position information of the head, for example, "teaching operator orientation information (rad)", "teaching operator eye position information (x, y, z)", "teaching operator eye angle information (rad)" and other information acquired by the acquisition unit 11 can also be stored. The "viewpoint record data table" is associated with the "teaching operation record table" through, for example, a "viewpoint record data file".

[0117] <<List of objects to be watched>>

[0118] In the "attention object table", for example, various information related to the object that the instructor is gazing at is stored (saved). A plurality of various information set by the evaluator is stored. In the "attention object table", for example, "attention object ID", "viewing range recording image file", "viewpoint recording data file", "viewpoint recording elapsed time" and "operation step ID" for identifying the object that the instructor is gazing at are stored.

[0119] In addition, in the "attention object table", in addition to storing the "attention field range image" as a still image obtained by cutting out the viewpoint record at a moment in time from the "viewpoint record image file", for example, the "attention object position (x, y)" representing the position related to the attention object extracted by the extraction unit 13, the "attention object range (w, h)" representing the range, and the "attention object image" representing the image of the attention object can also be stored as attention information in the operation based on the attention mode determined by the determination unit 12 based on the image information and coordinate information associated in the "viewpoint record table".

[0120] The attention information may be evaluated, set, and recorded by an evaluator after being generated by the determination unit 12 and the extraction unit 13, for example, based on the information acquired by the acquisition unit 11. The "attention object table" is associated with the "viewpoint record table" through the "attention object ID", associated with the "work step table" through the "work step ID", and associated with the "viewpoint record data table" through the "viewpoint record data file".

[0121] Figure 4 (b) is a schematic diagram showing an example of the function of the attention extraction device 1. The attention extraction device 1 includes, for example, an acquisition unit 11, a determination unit 12, an extraction unit 13, a storage unit 14, a determination unit 15, a display unit 16, an input unit 17, and a monitoring display unit 18. Figure 5 Each function shown in (b) is implemented by the CPU 101 executing a program stored in the storage unit 104 or the like using the RAM 103 as a work area.

[0122] <<Acquisition Unit 11 (Acquisition Means)>>

[0123] The acquisition unit 11 associates the image information within the visual field of the instructor who performs the operation and the coordinate information indicating the viewpoint that the instructor is looking at within the visual field with the operation and acquires them in time series. The acquisition unit 11 is used, for example, when implementing the acquisition step S110 described later. The timing at which the acquisition unit 11 acquires the image information and the coordinate information from the instructor device 2 can be arbitrarily set. The acquisition unit 11 associates the acquired image information and coordinate information with the operation performed by the instructor via the storage unit 14, for example, and stores them in the storage unit 104 in time series.

[0124] <<Determination Unit 12 (Determination Means)>>

[0125] The determination unit 12 determines the viewpoint displacement of the instructor within the visual field based on the image information and coordinate information acquired by the acquisition unit 11, and determines the instructor's gaze pattern. The determination unit 12 is used, for example, when implementing the determination step S120 described later. The determination unit 12 is, for example, as described above. Figure 3As shown, the gaze pattern in the visual field of the instructor is determined based on the image information in the visual field 2a acquired by the instructor device 2 and the coordinate information (x-axis, y-axis) of the instructor's viewpoint shift.

[0126] Next, if the information acquired by the acquisition unit 11 has characteristics and tendencies different from the stored "wide field of view mode" and "alert mode", the determination unit 12 may also determine it as a new mode. The determination unit 12 can arbitrarily set various modes through the attention extraction device 1. For example, the determination unit 12 associates the new mode with the instructor and the work performed by the instructor through the storage unit 14 in addition to the change of the existing mode that has been acquired, and stores it in the storage unit 104 in a time series. In addition, the determination unit 12 may determine the various modes of the instructor, for example, by the evaluator setting various modes based on the information acquired by the acquisition unit 11 and stored in the database.

[0127] <<Extraction unit 13 (extraction means)>>

[0128] The extraction unit 13 sets a gaze area according to the gaze pattern determined by the determination unit 12, and extracts a gaze image of the work object 6 that the instructor is gazing at in the gaze area. For example, when there are a plurality of gaze patterns determined by the determination unit 12, the extraction unit 13 may discriminate the gaze images of the work object 6 included in the set gaze area according to each gaze pattern, and extract them in time series.

[0129] The extraction unit 13 extracts the gaze image of the work object 6 included in the set gaze area based on the gaze pattern, for example. In the case where there is sequence in the work and confirmation of the instructor, the extraction is performed in a time series in an order that emphasizes the sequence. In the case where there is no sequence, multiple gaze images can also be associated with one work and extracted.

[0130] The extraction unit 13 may output the extraction result to the storage unit 14 and the attention extraction device 1, for example, via the communication network 5. The extraction unit 13 may associate the extracted gaze image with the instructor, the work performed by the instructor, the gaze pattern, etc., via the storage unit 14, for example, and store the associated gaze image in the storage unit 104. In addition, the association with the gaze image and the gaze pattern, etc., based on the extraction unit 13 may be performed by the evaluator based on various information stored in the storage unit 104.

[0131] <<Storage Unit 14 (Storage Means, Database)>>

[0132] The storage unit 14 associates the gaze image extracted by the extraction unit 13 with the visual field, viewpoint shift, and gaze pattern, and stores them in a database as attention information for the instructor's work. The storage unit 14 stores various information in the storage unit 104 or retrieves various information from the storage unit 104 .

[0133] The storage unit 14 stores associations between a plurality of previously acquired past fixation image information and reference information indicating the correctness of a work object associated with the fixation image. The storage unit 14 stores or retrieves various information, for example, based on the processing contents of the acquisition unit 11, the determination unit 12, the extraction unit 13, the determination unit 15, the display unit 16, the input unit 17, and the monitor display unit 18.

[0134] <<Determination Unit 15 (Determination Means)>>

[0135] The determination unit 15 determines the correctness of the work object included in the attention image of the work acquired through the operator device 3 of the operator, for example, using the attention data set related to the work. The determination unit 15 refers to the database stored in the storage unit 14 or the storage unit 104, and based on the image information acquired through the operator device 1 of the operator, refers to the database to determine the correctness of the work performed by the operator with respect to the attention information (for example, the work object 6, etc.).

[0136] The determination unit 15 obtains various corresponding information from the database, for example, based on the determination result. Figure 2 The object information displayed in the field of view 3a of the operator device 3 ("(1) Confirmation OK: 01234", "(2) Adjustment steps: XXX", etc.) is information related to the work that the operator should handle. The determination unit 15, for example, sends the acquired corresponding information to the display unit 16 of the operator device 3.

[0137] Regarding the determination unit 15, for example, when multiple operators perform the same operation in the same operation area, the correctness of the operation objects of the multiple operators can also be determined collectively. In this case, for example, multiple operators pre-set a common attention data set for the operation in each operator device 3. For example, the determination unit 15 can also obtain multiple operation space layouts for each operation based on the coordinate information obtained by the acquisition unit 11, based on the relationship between the coordinate information indicating the position and orientation of the head of each operator and the coordinate information indicating the operation space of the operation, and determine the operation and operation object of each operator comprehensively or in groups.

[0138] Furthermore, the determination unit 15 obtains information on whether the original operator is facing the correct direction relative to the work object that the original operator is supposed to work on, and if the operator is not facing the correct direction (not noticing), whether there are other operators who can support the operator, whether there are other operators facing the work object that the original operator is supposed to work on, etc. For example, the determination unit 15 can also send corresponding information to the operator devices 3 of other operators who are working together around the work object and can work properly, based on the obtained information and the information on the work process of each operator, even if the original operator is not paying attention.

[0139] <<Display unit 16 (display means)>>

[0140] The display unit 16 uses the attention data set generated based on the teacher's work to output various corresponding information, such as obtained by the determination unit 15, to the operator device 3 of the operator who performs the corresponding work. The display unit 16 (display part 109 of the operator device 3) displays various corresponding information sent from the determination unit 15 in the visual field 3a of the operator device 3 worn by the operator, for example, in an overlapping manner with the actual image.

[0141] For example, when a plurality of workers share and perform the same work in the same work area, the display unit 16 appropriately displays the corresponding information acquired by the determination unit 15 according to the work status and work position of each worker.

[0142] In addition, the display unit 16 displays the extracted Fig. 9 The display contents of the precautionary information of the work shown are displayed on the teaching device 2 . Fig. 9 1 is a schematic diagram showing an example of the attention extraction method in the present embodiment. The display unit 16, for example, before the acquisition unit 11 acquires the image information, Fig. 9 As shown in (a), the recording date and time information, the recording position information, and the recording control information are displayed in the center of the visual field 2a. In addition, the display unit 16 may, for example, switch to only the recording control information (e.g., "recording temporarily stopped", "recording ended", etc.) and display it in a corner of the visual field 2a during the acquisition of the image information by the acquisition unit 11.

[0143] In addition, the display unit 16 may also be as follows, for example Fig. 9 (c)~ Fig. 9 As shown in (e), the display contents indicating each operation are displayed in the center of the visual field 2a of the teaching device 2. The display unit 16 may display these contents based on, for example, the operation steps predetermined by the teaching device, the results of the operation performed by the teaching device, and the timing of division and completion of the operation detected by the position detection sensor pre-installed in the teaching device 2.

[0144] The display unit 16 displays, for example, Fig.10 (a) and Fig.10 (b) and Fig.12 (a)~ Fig.12 The display content shown in (e) is displayed on the operator device 3. Fig.10 (a) and Fig.10 (b) is a schematic diagram showing an example of a caution extraction displayed when an operator performs a work via the operator device 3 using the caution data set in the present embodiment, for example. Fig.12 (a)~ Fig.12 (e) is a schematic diagram showing an example of a display of the operator device 3 (operator terminal, smartphone, display device, etc.) held by the operator in the present embodiment.

[0145] The display unit 16 displays the determination result of the determination unit 15, for example, on the gaze object 6a included in the visual field 3a of the operator device 3 worn by the operator or on a display device held by the operator, based on the determination result of the determination unit 15. Furthermore, the display unit 16 displays corresponding information corresponding to the determination result acquired by the determination unit 15 by referring to the database.

[0146] Fig.10 (a) is a display of the visual field range 3a in which, for example, the operator is working and there is no corresponding information for the operator. Fig.10 (b) is a display of the field of view 3a of the state in which the operator has corresponding information for the operator during the operation, and the points that should be paid attention to are displayed in the field of view 3a as corresponding information (the correct focus object of the field of view 3a: ○ mark + "Attention!"), the steps or related information that the operator should confirm or refer to (the center of the field of view 3a: "Check-3-A", "Check", "Judgement", "Correspondence", "Reference destination", etc.), and the working status / inspection results of the operator himself or the co-operator (the left side of the field of view 3a: the working place, the working content, the working process, etc.).

[0147] Furthermore, the display unit 16 may also have an attention information display area, which switches to display an acquisition display area and an attention display area. For example, when the operator starts the operation, the acquisition display area is displayed. The acquisition display area displays the category of the acquisition unit 11 that acquires the image information, the category of the attention data set stored in the database, and instructions for selecting the start of the operation. In this attention display area, based on the attention data set selected by the operator, corresponding information corresponding to the operator's gaze image and attention information are displayed.

[0148] Furthermore, the display unit 16 can appropriately display the attention information displayed in the visual field 3a of the operator device 3 as boost information, for example, information displayed at at least one time of the start of the work, the middle of the work, or the end of the work, according to the work process of the operator. For example, the display unit 16 refers to the attention database based on the attention data set and the determination result of the determination unit 15, obtains the corresponding information, attention information, or boost information, and displays the obtained various information according to the attention status of the operator performing the work.

[0149] The boost information displayed on the display unit 16 may be displayed to the operator in the form of default (unconsciously driven), push (naturally driven), boost that makes the operator aware but leaves the operator to choose, marking (intentionally driven), and reward (driven by reward) as an approach to display to the operator. The display of the boost information may be appropriately extracted from the database and displayed according to the actual progress of the operator's work, for example, in association with the work process and progress of the work, the operator's personality, temperament, evaluation, difficulty of the work, and expected completion time.

[0150] In addition, the display unit 16 may randomly display the type of questions such as "Is it okay?", "Wait a minute?", etc., in addition to directly displaying the reference information that lists the display of the boost information as the display candidate of the result of the determination of the determination unit 15. In addition, the display unit 16 may randomly display the visual field 3a of the operator device 3 at the center or at the corresponding part of the attention object 6a in the visual field 3a.

[0151] In addition, the display unit 13 can, for example, mix the display output method or randomly change the display output method (visual, auditory, vibration feedback (vibration), a method of boosting information (marking), a method of boosting information (dialogue, direct questioning, making the operator think "?", sudden change), and appropriately set and display it according to the operator's skills and the progress of the work.

[0152] In addition, the display unit 16 can, for example, perform heuristic, normality (maintaining the status quo), and display of reference information for adding changes to the displayed boosting information as a related bias to prevent the user from missing the reference information due to habit, boredom, or expectation of maintaining the status quo. The display unit 16 can, for example, have a unit (not shown) for grasping the overall flow and steps of the work being performed by the operator as a prerequisite process for displaying the boosting information, and can also have a unit (not shown) for detecting deviations from the content that should be noted.

[0153] In addition, the display unit 16 can refer to the database and display the words for the worker's work, etc., when the worker's work is completed. It is arbitrary what kind of boosting information is displayed at what time. Thus, for example, the corresponding information displayed to the worker can be displayed in a variable manner, and it can be prevented that the worker misses the reference information due to habit, boredom, and expectation of maintaining the status quo.

[0154] In addition, the display unit 16 displays, for example, Fig.11 The display content shown is displayed on the attention extraction device 1. Fig.11 For example, the attention monitoring area of ​​the attention extracting device 1 in this embodiment is displayed as the attention extracting device screen 1a. As described above, in the display unit 16, for example, for a task based on a teacher, the evaluator refers to the database via the attention extracting device 1 and displays the coordinate information, the attention area, the attention image, and the attention shift in a time series in the attention display area 1b.

[0155] The display unit 16 is, for example, Fig.11 As shown, setting information for setting the attention image displayed in the attention display area 1b is displayed in the target setting area 1c. In addition, the display unit 16 displays, for example, the determination condition of the setting information displayed in the target setting area 1c in the determination setting area 1d.

[0156] The display unit 16 displays, for example, setting information for setting the attention image displayed in the attention display area 1b in the target setting area 1c. In addition, the display unit 16 displays, for example, the determination condition of the setting information displayed in the target setting area 1c in the determination setting area 1d.

[0157] <<Input section 17 (input means)>>

[0158] The input unit 17 inputs, for example, the corresponding information output by the determination unit 16. The input unit 17 receives, for example, the conditions and numerical values ​​related to various settings and adjustments from the evaluator through the attention extraction device screen 1a displayed on the attention extraction device 1, for example, through the setting item menu displayed on the attention extraction device screen 1a. Thus, for example, the acquisition unit 11, the determination unit 12, the extraction unit 13, the storage unit 14, the determination unit 15, the display unit 16, and the monitoring display unit 18 perform various processes based on the various conditions and numerical values ​​received by the input unit 17.

[0159] The corresponding information inputted by the input unit 17 is associated with the attention image and stored in the database of the server 4 as an attention data set including setting conditions for recognizing it as an attention object.

[0160] <<Monitoring display unit 18 (monitoring display means)>>

[0161] The monitoring display unit 18 displays various information that the evaluator refers to and sets based on the work of the instructor as a gaze monitoring area on the attention extraction device screen 1a. The monitoring display unit 18 displays, for example: a gaze display area 1b that displays the coordinate information, the gaze area, the gaze image, and the gaze shift stored in the database in time series; an object setting area 1c that displays setting information for setting the gaze image displayed in the gaze display area; and a judgment setting area 1d that displays the judgment conditions of the setting information displayed in the object setting area.

[0162] The attention extraction device 1 acquires video information related to the work, for example, via the instructor's instructor device 2. The work performed by the instructor can be pre-selected according to each of various attention data sets set according to the operator's work, and the attention data set of the instructor's work is generated by the attention data set corresponding to the selected work.

[0163] The attention data set extracted by the attention extraction device 1 is selected by, for example, workers A to C, and the work of workers C is evaluated by the selected attention data set. In addition, the evaluator can also monitor the work status of workers A to C who use the attention data set to perform the work through the attention extraction device 1. The attention extraction device 1 can, for example, obtain the work status of workers A to C together with the work using the attention data set, for example, by obtaining the image and position information of the work object through the worker device 1 worn by each worker, and display it on the display unit 16 of the attention extraction device 1.

[0164] For example, the evaluator can refer to the working status of operators A~C displayed on the attention extraction device screen 1a displayed on the attention extraction device 1. When a job or evaluation different from the attention data set is generated, for example, when operator C is in a situation where he cannot perform the original job (skipping the job step and is in front of other job objects), and when operator B is in a position where he can confirm the original job of operator C, the job for the attention information that operator C should have paid attention to is set to "ignore", and the job for the attention information that operator B should have paid attention to is set as "agent" and instructed in real time.

[0165] <Teacher Device 2>

[0166] The teaching device 2 is worn by, for example, an expert or a qualified person in addition to the teaching person who is working, and acquires the image information observed by the teaching person's eyes through the teaching person's work. The teaching device 2 may be, for example, a well-known eye tracking technology, a head-mounted display, or smart glasses, and may acquire sound, surrounding sound information, temperature, humidity, position information, spatial information, etc. together with the image information.

[0167] The teaching device 2 may be connected to the attention extracting device 1 , other teaching devices 2 , the operator device 3 , and the server 4 in a data-communicable state, or may have the attention extracting device 1 built therein, for example.

[0168] <Operator Device 3>

[0169] The operator device 3 is worn by a person other than the operator who performs the operation, for example, a person other than the instructor, and acquires the image information observed by the operator's eyes through the operator's operation. The operator device 3 may be, for example, a well-known eye tracking technology, a head-mounted display, or smart glasses, and may acquire sound, surrounding sound information, temperature, humidity, position information, space information, etc. together with the image information.

[0170] For example, the operator device 3 may be connected to the attention extraction device 1 , the teaching device 2 , other operator devices 3 , and the server 4 in a data-communicable state, or may have the attention extraction device 1 built therein.

[0171] <Communication Network 5>

[0172] The communication network 5 is, for example, the Internet connecting the attention extraction device 1, the teaching device 2, and the operator device 3 via a communication circuit, and may be formed of an optical fiber communication network. The communication network 5 may be realized by a known communication network such as a wireless communication network in addition to a wired communication network.

[0173] <Learning Model>

[0174] The learning model generates a database by machine learning, for example. The learning model uses, for example, a learning target image including a gaze image captured by the teaching device 2 and reference information indicating the correctness of the gaze image captured by the teaching device 2 as a pair of learning data, and acquires a plurality of learning target images for each operation in the teaching device 2. The learning model is generated by machine learning using a plurality of learning data, from a database storing the associations between a plurality of learning target images and a plurality of reference information.

[0175] Figure 8 1 is a schematic diagram showing an example of the attention extraction method in the present embodiment. The attention extraction method includes an acquisition step S110, a determination step S120, an extraction step S130, a storage step S140, and a determination step S150. The attention extraction method can be implemented using the attention extraction system 100.

[0176] <Acquisition Step S110>

[0177] The acquisition step S110, for example, associates the image information and coordinate information with the work and acquires them in time series. The acquisition step S110, for example, uses the instructor device 2 having a known camera, a photographing device, etc. to acquire. In addition, the acquisition step S110, for example, may also associate the image information and coordinate information with the work from the operator device 3 and acquire them in time series. The acquisition step S110, for example, uses the device selected by the instructor and the operator to acquire the image information, etc.

[0178] <Determination Step S120>

[0179] The determination step S120 obtains the viewpoint displacement of the instructor and determines the instructor's gaze pattern. The determination step S120 obtains the viewpoint displacement within the instructor's field of view and determines the instructor's gaze pattern, for example, based on the image information obtained in the acquisition step S110 and the coordinate information.

[0180] In the determination step S120, for example, when the viewpoint displacement of the instructor is included in the instructor's field of vision radius of 5 degrees to 20 degrees and has the characteristic of being concentrated in the center of the field of vision, the "wide field of vision mode" is determined. In addition, in the determination step S120, for example, when the viewpoint displacement of the instructor is included in the instructor's field of vision radius of 4 degrees or less and has the characteristic of being dispersed to a position outside the center of the field of vision, the "alert mode" is set.

[0181] The determination step S120 determines the gaze mode in the visual field of the instructor as "wide visual field mode" and "alert mode" based on the image information in the visual field 2a acquired by the instructor device 2 and the coordinate information (x-axis, y-axis) of the instructor's viewpoint shift. The determination step S120 determines the gaze mode as "wide visual field mode" and "alert mode" based on the coordinate information of the instructor's viewpoint shift, for example. However, if other characteristics and tendencies can be confirmed, the confirmed gaze mode can be reset.

[0182] <Extraction Step S130>

[0183] The extraction step S130, for example, sets a gaze area and extracts a gaze image of the work object that the instructor is gazing at. The extraction step S130, for example, sets a gaze area that the instructor is gazing at based on the gaze mode (e.g., "wide field of view mode" or "alert mode") determined in the determination step S120. The extraction step S130, for example, extracts a gaze image of the work object that the instructor is gazing at from the video information of the instructor in the set gaze area.

[0184] Regarding the extraction step S130, for example, if there are multiple gaze images in the video information of the instructor's gaze area, the images may be extracted in the order of gaze or the time of gaze. The extraction step S130 extracts the gaze image of the work object from the video information in the gaze area of ​​the instructor's gaze, for example, based on the conditions set by the evaluator. In this way, the instructor's gaze state can be grasped and appropriate attention information can be extracted.

[0185] <Storage Step S140>

[0186] The storage step S140 associates the gaze image with the visual field, the viewpoint shift, and the gaze pattern, and stores the associated gaze image as attention information. The storage step S140 associates the gaze image extracted by the extraction step S130 with the visual field, the viewpoint shift, and the gaze pattern, and stores the associated gaze image as attention information in the operation in the database as described above.

[0187] <Determination Step S150>

[0188] The determination step S150 determines the correctness of the work object included in the attention image acquired via the operator device 3, for example, using the attention data set. The determination step S150 determines the correctness of the work object based on the attention image acquired by the operator device 3 in the acquisition step S110, for example. The determination step S150 may perform determination by evaluation of an evaluator who is monitoring the work, for example, via the attention extraction device 1, in addition to evaluation using the attention data set for each selected work.

[0189] In the determination step S150, for example, when the attention image is acquired by the operator device 3 in the acquisition step S110, the correctness of the work object performed by the operator can be determined based on the attention data set selected in advance in the operator device 3. The determination step S150 can determine the work of multiple operators in the same work area.

[0190] The determination step S150 can determine the work of multiple operators simultaneously using the same or common attention data set, so that, for example, if an operator misses or skips a process or a process in a different order, determinations can be made according to the work position and work status of each operator.

[0191] The determination step S150 refers to various data tables such as "operation information table", "operation step table", "corresponding information table", "teaching operation record table", "viewpoint record table", "viewpoint record data table", "attention object table" and the like stored in the database, and determines the correctness of the operation object according to various data tables and attention data sets, and obtains various corresponding information stored in the corresponding operation and reference link according to the determination result. The corresponding information obtained by the determination step S150 can be displayed on the operator device 3 of the corresponding teaching operator and other operators, for example.

[0192] In the determination step S150, for example, addition and update of corresponding information of various data tables stored in the database are inputted via the input unit 17. The input unit 17 can appropriately input, for example, work steps, determination conditions, corresponding work, reference links, etc. of the determination object and determination object respectively stored in various data tables.

[0193] Thus, the operation of the attention extraction device 1 in this embodiment is completed.

[0194] According to this embodiment, for example, the attention extraction device 1 acquires the video information via the acquisition unit 11 of the instructor device 2. The video information may include, for example, recording date and time information of the instructor's work, recording position information, and recording control information related to the acquisition operation of the video information.

[0195] In addition, according to this embodiment, for example, the display unit 11 is Fig. 9 The display of the teaching device 2 is switched as shown. For example, the display unit 11 displays the recording date and time information, the recording position information, and the recording control information in the center of the visual field before the acquisition unit 11 acquires the image information. In addition, the acquisition unit 11 switches only the recording control information to be displayed in the corner of the visual field during the acquisition of the image information. In this way, a display that does not hinder the work of the teaching can be performed.

[0196] In addition, according to the present embodiment, the display unit 16 displays attention information. The display information is based on information such as the operation process and operation status of the operator, and the boost information is displayed based on the theory of boost at the start, middle or end of the operation. In this way, the tendency to affect the dialogue with the conventional computer system or information can be eliminated, and the effect of the notification from the attention extraction system device 1 can be improved by displaying the boost information based on the theory of boost.

[0197] The embodiments of the present invention have been described, but the above embodiments are presented as examples and are not intended to limit the scope of the invention. The above novel embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the subject matter of the invention. The above embodiments and their variations are included in the scope and subject matter of the invention, and are included in the invention described in the claims and the scope of their equivalents.

[0198] Description of symbols

[0199] 1: attention extraction device; 1a: attention extraction device screen; 1b: attention display area; 1c: object setting area; 1d: judgment setting area; 1e: work space diagram; 1f: work agent mark; 2: instructor device; 2a: field of view (instructor); 2b: attention mode; 3: operator device; 3a: field of view (operator); 4: server; 5: communication network; 6: work object; 6a: attention object; 10: shell; 11: acquisition unit; 12: determination unit; 13: extraction unit; 14: storage unit (database); 15: judgment unit; 16: display unit; 17: input unit; 18: monitoring display unit; 50: work area; 100: attention extraction system; S110: acquisition step; S120: determination step; S130: extraction step; S140: storage step; S150: judgment step.

Claims

1. A system for extracting attention information for a task, characterized in that: The attention extraction system has: an acquisition unit that associates image information within a field of view of a teacher performing the operation and coordinate information indicating a viewpoint that the teacher is looking at within the field of view with the operation and acquires the information in a time series; a determination unit that determines the gaze mode of the instructor as a wide field of view mode when the coordinate information of a time series representing the viewpoint displacement of the instructor within the field of view converges at the center of the field of view, and determines the gaze mode of the instructor as a warning mode when the coordinate information of a time series representing the viewpoint displacement of the instructor within the field of view disperses to a position outside the center of the field of view, based on the image information and the coordinate information acquired by the acquisition unit; an extracting unit that sets a gaze area according to the gaze pattern determined by the determining unit, and extracts a gaze image of the work object that the instructor gazes at in the gaze area; as well as A storage unit is configured to associate the gaze image extracted by the extraction unit with the visual field range, the viewpoint shift, and the gaze pattern, and store the associated gaze image in a database as attention information in the task.

2. The attention extraction system according to claim 1, characterized in that The acquisition unit further includes a determination unit, which acquires a gaze image of an operator performing an operation, and determines whether the operation performed by the operator is correct or not based on image information included in the gaze image and object information related to the operation that the operator should handle, which is pre-stored in a database. The attention extraction system further includes a display unit that displays a determination result of the determination unit.

3. The attention extraction system according to claim 2, characterized in that The attention extraction system further includes a database storing a correlation between previously acquired past attention image information and reference information indicating the correctness of the work object associated with the attention image. The determination unit refers to the database to determine whether the operation object is correct or not, and acquires corresponding information corresponding to the determination result from the database. The display unit also outputs the corresponding information acquired by the determination unit.

4. The attention extraction system according to claim 2, characterized in that The attention extraction system further comprises an input unit, which inputs the corresponding information output by the determination unit. The storage unit associates the corresponding information input through the input unit with the attention image and stores the associated information as an attention data set including setting conditions for recognizing the attention object.

5. The attention extraction system according to claim 2, characterized in that The image information acquired by the acquisition unit includes recording date and time information of the work performed by the instructor, recording position information, and recording control information related to the acquisition operation of the image information. The display unit displays the recording date and time information, the recording position information, and the recording control information in the center of the field of view before the acquisition unit acquires the image information, and during the acquisition of the image information, only switches to the recording control information and displays it in a corner of the field of view.

6. The attention extraction system according to claim 2, characterized in that The display unit further includes an attention information display area, which switches between an acquisition display area and an attention display area, wherein the acquisition display area displays, for the operator performing the operation, the type of the acquisition unit that acquires the image information, the type of the attention data set stored in the database and the setting conditions for identifying the object of attention, and instructions for selecting the start of the operation, respectively, and the attention display area displays corresponding information corresponding to the operator's attention image and attention information based on the attention data set after the selection. The caution information displayed in the caution display area includes boost information, which is displayed at at least any timing of the start of the work, the middle of the work, or the end of the work according to the work progress of the operator, The information displayed in the attention information display area, including at least any of the corresponding information, the attention information or the boost information, is allocated for display based on the attention data set and the result of the judgment by the judgment unit and according to the attention status of the operator performing the operation.

7. A method for extracting attention information for a task, characterized in that: The attention extraction method enables the computer to perform the following steps: An acquisition step of associating image information within a field of view of a teacher performing the operation and coordinate information indicating a viewpoint that the teacher is looking at within the field of view with the operation and acquiring the information in a time series; a determination step of determining the gaze mode of the instructor as a wide field of view mode when the coordinate information of a time series representing the viewpoint displacement of the instructor within the field of view converges at the center of the field of view, and determining the gaze mode of the instructor as a warning mode when the coordinate information of a time series representing the viewpoint displacement of the instructor within the field of view disperses to a position outside the center of the field of view, based on the image information and the coordinate information acquired by the acquisition step; an extracting step of setting a gaze area according to the gaze pattern determined by the determining step, and extracting a gaze image of the work object gazed by the instructor in the gaze area; as well as A storing step of associating the gaze image extracted by the extracting step with the visual field range, the viewpoint shift, and the gaze pattern, and storing the associated gaze image in a database as attention information in the task.

Images