Vision line tracking system, vision line tracking method, and vision line tracking program

Abstract

To provide a vision line tracking system, a vision line tracking method, and a vision line tracking program which cause an attention degree of parts to which various operators give attention to be confirmed with a high level of actual feeling.SOLUTION: In a vision line tracking method with a vision line tracking system, vision line information is acquired while a user such as a skilled person, while wearing a viewing operation terminal, performs an inspection operation for an object to be observed which is a substantial object such as mechanical equipment. Then, vision line tracking processing is started which tracks a vision line of the user performing an inspection with careful viewing of a spot thought to be in need thereof from the inspection of the object to be observed. Then, viewpoint confirmation processing is started which is preparation of creation of vision line data. Then, the skilled person as an educator, while wearing the viewing operation terminal, observes the object to be observed or a three-dimensional object of the object to be observed for a check and an inspection, creating the vision line data. Then, the vision line data is stored in a vision line information database of a storage unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an eye-tracking system, an eye-tracking method, and an eye-tracking program. [Background technology]

[0002] Patent document 1 discloses a gaze point conversion device that includes a reference image database that stores multiple reference images obtained by photographing an object from different shooting positions in the left and right directions, a corresponding point search means that extracts feature points of a scene image input from an eye tracking device, and then, for each reference image stored in the reference image database, uses the feature points of the scene image and the reference image to match the feature points of the scene image with the feature points of the reference image, thereby performing a process of searching for corresponding points between the scene image and the reference image, and selects a reference image based on the results of the search for corresponding points, and a gaze point conversion means that performs a process of converting the gaze point into coordinates on a reference image of the object using the corresponding points on the reference image selected by the corresponding point search means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-186720 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional configuration, it was necessary to display the gaze analysis result on the display unit of the information processing device to confirm it. For example, even if the information resulting from tracking the gaze of an expert who picks up an object and inspects it is displayed on the display unit of the information processing device and an unskilled person watches it, there is a restriction that the unskilled person only watches an image in which the viewpoint point to be watched is displayed. In this case, while an unskilled person or other training subject can obtain knowledge about the skill of an expert in a certain technology from a video or image, it is difficult to improve the realism of the experience. Therefore, there is a problem that the sense of realism is low for an unskilled person or other training subject, and the effect of training on skills is limited. Therefore, there has been a demand for a technology that can confirm the degree of attention of various users, such as an expert educator and an unskilled training subject, with a high realism.

[0005] The present invention has been made in consideration of the above, and its object is to provide an eye-tracking system, an eye-tracking method, and an eye-tracking program that can confirm the attention level of parts that various users are paying attention to with a high degree of accuracy. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the above-mentioned objectives, the gaze tracking system of the present invention is an gaze tracking system that tracks the gaze of a user, acquires gaze information of the user, and uses the acquired gaze information to derive at least one area on which the user is focusing and a level of attention to the area, and superimposes the derived result on an observation target that the user is viewing or an image of the observation target captured by a specified imaging means.

[0007] In one embodiment of the gaze tracking system of the present invention, in the above invention, the gaze information is information generated based on the intersection coordinates between a three-dimensional mesh defined on the surface of the object in a three-dimensional space set relative to a predetermined origin and the user's gaze or the imaging direction of the imaging means.

[0008] In one aspect of the present invention, in the above-mentioned invention, the gaze tracking system is such that the observation target is a real object in real space or a virtually generated observation target object, and the intersection coordinates are set based on the position coordinates of the surface of the real object or the outer surface part of the observation target object.

[0009] In one aspect of the gaze tracking system of the present invention, in the above invention, the gaze information includes information on the time that the user's gaze or the image capture by the imaging means remained, or the number of times that the user's gaze was directed or the image capture by the imaging means was performed.

[0010] In the gaze tracking system according to one aspect of the present invention, in the above invention, the superimposition of the derivation result on the target object or an image of the target object is performed by using a derivation result object.

[0011] In the gaze tracking system according to one aspect of the present invention, in the above invention, the derived result object represents the derived result by color, shape, or size.

[0012] In one aspect of the present invention, the gaze tracking system is configured in the above invention such that the gaze information or the derived result object is stored in a memory unit accessible by multiple different terminals and can be shared instantly or in a timely manner among the multiple terminals.

[0013] In one aspect of the present invention, the gaze tracking system evaluates the gaze information based on the gaze information and previously generated educational information, and notifies the user of the evaluation result derived by the evaluation, in the above invention.

[0014] The gaze tracking method of the present invention is a gaze tracking method executed by a processor having hardware, which acquires gaze information of a user that tracks the user's gaze, uses the acquired gaze information to derive at least one area that the user is focusing on and a level of attention to the area, and superimposes the derived result on an object that the user is viewing or an image of the object captured by a specified imaging means.

[0015] The gaze tracking program of the present invention causes a processor having hardware to acquire gaze information of a user by tracking the user's gaze, use the acquired gaze information to derive at least one area on which the user is focusing and the level of attention to the area, and superimpose the derived result on an object being viewed by the user or an image of the object captured by a specified imaging means. Effect of the Invention

[0016] According to the gaze tracking system, the gaze tracking method, and the gaze tracking program of the present invention, it becomes possible to confirm the degree of attention of various workers to parts that they are paying attention to with a high degree of actuality. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of an eye-tracking system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the viewing operation terminal according to one embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart illustrating a gaze tracking method using the gaze tracking system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart illustrating a gaze tracking method using the gaze tracking system according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing a state in which viewpoint tiles are superimposed when a user (educator) wearing a viewing and operation terminal according to one embodiment of the present invention views an object. [Figure 6] FIG. 6 is a diagram showing a state in which viewpoint tiles are superimposed when a user (education subject) wearing a viewing operation terminal according to one embodiment of the present invention views an object. [Figure 7] FIG. 7 is a diagram showing an example of display of test results and comments displayed on the display screen of the projection unit of the viewing and operation terminal according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings of the embodiment, the same or corresponding parts are denoted by the same reference numerals. Furthermore, the present invention is not limited to the embodiment described below.

[0019] First, in describing the embodiment of the present invention, in order to facilitate understanding of the present invention, the inventor will explain the intensive study he conducted to solve the above problem. First, according to the knowledge of the inventor, in conventional gaze tracking, it is necessary to confirm the gaze analysis result on a display such as a personal computer, and there is a problem that the sense of realism is low and the educational effect is limited. For example, even if an unskilled person watches the gaze tracking result of an expert who picks up an object and inspects it on a display such as a personal computer, he or she is merely watching an educational video in which the viewpoint point to be focused on is displayed, and although knowledge can be obtained, it cannot be said that the educational effect is high. There is a problem that the real feeling of the educational subject's re-experience is low. In addition, in the case of gaze tracking that includes the movement of the subject, the movement situation and the head direction are significantly different for each subject, so there is a problem that it is difficult to compare the gaze tracking results. Specifically, in the case of gaze tracking that includes the movement of the subject, it is not guaranteed that all the subjects will stop at the same place, and it is difficult to obtain a common scene for comparison, such as a different moving speed.

[0020] Therefore, the present inventor has devised a technique capable of recording the viewpoints gazed upon by an educator such as an expert in a given skill or an educational subject such as an unskilled person as three-dimensional position information, thereby conceiving a method for improving the educational effect when teaching a skill with a high sense of realism. Furthermore, the present inventor has conceived of achieving a high educational effect by realizing the attention point of an expert with a high sense of realism when the educator visits an actual site, holds an object in his / her hand, or makes the object a three-dimensional object. This allows the area that the educator is paying attention to to be determined as the area that the educational subject should pay attention to. Here, the area can be a range having a settable predetermined area defined by a plane of a predetermined shape along the gaze point or movement line of the gaze of the educator or educational subject directed toward the object. The educational effect on the unskilled person can be scored using the number of times the expert passes through the area while gazing at it. The embodiment described below was devised by the present inventor through the above-mentioned intensive studies.

[0021] First, a gaze tracking system according to an embodiment of the present invention will be described below. Fig. 1 is a schematic diagram showing a schematic configuration of the gaze tracking system according to this embodiment.

[0022] (Configuration of eye tracking system) As shown in FIG. 1, the gaze tracking system 1 includes a viewing operation terminal 20 (20A, 20B) that is worn or used by a user. The gaze tracking system 1 may further include a processing server 10 connected via a network 2. In this case, the viewing operation terminal 20 can communicate with the processing server 10 via the network 2. In this specification, the viewing operation terminal refers to a device that can be worn or used by a user and that can view a virtual image, such as a smartphone or a headset. In addition, by using multiple viewing operation terminals 20A, 20B, multiple users can check the viewpoint while experiencing a sense of realism.

[0023] The network 2 is configured by appropriately combining wired communication and wireless communication, and is composed of communication networks such as the Internet network and mobile phone network. The network 2 is composed of one or more combinations of, for example, a dedicated line, a public communication network such as the Internet, for example, a LAN (Local Area Network), a WAN (Wide Area Network), a telephone communication network such as a mobile phone, a public line, a VPN (Virtual Private Network), etc. The processing server 10 and the viewing operation terminal 20 are connected via the network 2.

[0024] The processing server 10 collects various information from the viewing operation terminal 20, acquires image data and video data of the shape of the target object, and generates image data and video data and transmits them to the viewing operation terminal 20. Note that since video data can be treated as multiple image data, image data includes the concept of video data.

[0025] The processing server 10 includes a control unit 11, a communication unit 12, and a storage unit 13. The processing server may also include an input / output unit. In this case, the input / output unit may be configured to include, for example, a touch panel display, a speaker microphone, a button, a switch, a jog dial, or the like. The input / output unit as an output unit can be configured to notify the outside of predetermined information by displaying characters, figures, or the like on a screen of a display such as a liquid crystal display, an organic EL display, or a plasma display, or outputting sound from a speaker, according to the control by the control unit 11. The input / output unit includes a printer that outputs predetermined information by printing it on a printing paper or the like. The various information stored in the storage unit 13 can be confirmed, for example, on a display of the input / output unit installed in a predetermined office or the like. The input / output unit as an input unit is configured to be selected from, for example, a keyboard or a touch panel keyboard incorporated inside the input / output unit to detect touch operations on a display panel, a voice input device that enables calls with the outside, a switch, or a jog dial. By inputting predetermined information from the input / output unit of the processing server 10, the viewing operation terminal 20 can also be remotely controlled.

[0026] Specifically, the control unit 11 includes a processor such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field-Programmable Gate Array), and a main memory unit such as a RAM (Random Access Memory) or a ROM (Read Only Memory).

[0027] The communication unit 12 is, for example, a LAN (Local Area Network) interface board or a wireless communication circuit for wireless communication, and is configured using a communication module capable of wired or wireless communication. The LAN interface board or the wireless communication circuit can be connected to a network 2 such as the Internet, which is a public communication network. The communication unit 12 may also be capable of communicating with the outside in accordance with a predetermined communication standard, for example, 4G, 5G, Wi-Fi (Wireless Fidelity) (registered trademark), Bluetooth (registered trademark), or the like. The communication unit 12 can be connected to the network 2 and communicate with the viewing operation terminal 20, etc.

[0028] The storage unit 13 is composed of a storage medium selected from a volatile memory such as a RAM, a non-volatile memory such as a ROM, an erasable programmable ROM (EPROM), a hard disk drive (HDD, Hard Disk Drive), and a removable medium. The removable medium is, for example, a universal serial bus (USB) memory, or a disk recording medium such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray (registered trademark) disc (BD). The storage unit 13 may also be configured using a computer-readable recording medium such as an externally mountable memory card. This allows the data in the storage unit 13 to be transmitted to and received from the processing server 10 and to be detachable. In addition, the information stored in the storage unit 13 of the processing server 10 can be configured to be exchanged between the storage unit 13 and another device, for example, a viewing operation terminal 20 described later, by using, for example, an SD card or a USB memory. Furthermore, it is also possible to transmit and receive information between the storage unit 13 and another device, for example, a storage unit of a viewing operation terminal 20 described later, by communication using local Wi-Fi.

[0029] The storage unit 13 can store an operating system (OS), various programs, various tables, various databases, and the like for executing the operations of the processing server 10. Here, the various programs include the line of sight processing program and the object image generating program according to this embodiment. These various programs can also be recorded on computer-readable recording media such as hard disks, flash memories, CD-ROMs, DVD-ROMs, and flexible disks, and widely distributed.

[0030] The control unit 11 loads a program stored in the storage unit 13 into a working area of ​​the main storage unit, executes the program, and controls each component unit through the execution of the program. This allows the control unit 11 to realize functions that meet a predetermined purpose, specifically, the functions of the gaze processing unit 111 and the object image generating unit 112. The storage unit 13 stores a gaze information database 131. When constructing an education system using the gaze information database 131, it is also possible to store an education information database. The processing by the gaze processing unit 111 and the object image generating unit 112, and the gaze information database 131 will be described in detail later.

[0031] In the gaze tracking system 1, the plurality of viewing operation terminals 20A, 20B may be capable of communicating with each other via the network 2. In addition, another server capable of communicating with the viewing operation terminals 20A, 20B via the network 2 may be provided.

[0032] (Viewing and operation terminal) Next, a description will be given of the configuration of the viewing operation terminal 20. FIG.

[0033] As shown in FIG. 2, the viewing and operating terminals 20A and 20B constituting the gaze tracking system 1 may be, for example, a head mounted display (HMD) for so-called mixed reality (MR), such as HoloLens (registered trademark) (manufactured by Microsoft), or AR glasses for performing augmented reality (AR). That is, the gaze tracking system 1 mainly employs an augmented reality (AR) or mixed reality (MR) space in which a real space and a virtual space are superimposed, and outputs predetermined information to a user in a manner that the user can see, or outputs virtual digital content in an operable manner according to the user's gesture, etc. The user can view information in which an image is superimposed on a transparent, i.e., see-through, real space by wearing goggles, etc. In other words, the viewing and operating terminals 20A and 20B grasp the shape of the real world, such as the shape and position of an object that actually exists, and the generated digital image is superimposed on the shape of the real world grasped by the viewing and operating terminals 20A and 20B. This allows the user to experience the MR space, and the virtual information is superimposed on the real space and can be visually recognized by the user. Note that the eye tracking system 1 can also adopt a virtual reality (VR) space (VR) in which reality is projected into a virtual space.

[0034] The viewing operation terminal 20 displays images, videos, text information, etc. that can be viewed stereoscopically by superimposing the real world on a virtual world (digital space) for the user. The viewing operation terminal 20 is a viewing operation terminal that has an eye tracking function and can project augmented reality. The viewing operation terminal 20 can virtually display images, videos, text information, etc. within the user's visual field. In this specification, virtual images, videos, text information, etc. may be collectively referred to as virtual images.

[0035] As shown in FIG. 2, the viewing operation terminal 20 as an eye tracking device includes a control unit 21, a communication unit 22, a storage unit 23, a projection unit 24, a position acquisition sensor 25, a wearing sensor 26, an eye gaze sensor 27, and an imaging device 28. The viewing operation terminal 20 may also include an audio input device and an operation unit. The audio input device is configured using a microphone, an A / D conversion circuit that converts the audio input to the microphone into audio data, and an amplifier circuit that amplifies the audio data, and can be configured to receive an input of a user's voice and output audio data corresponding to the received voice to the control unit 21. The operation unit receives an input of a user's operation and outputs a signal corresponding to the received operation to the control unit 21. The operation unit can be configured using a button, a switch, a jog dial, a touch panel, or the like.

[0036] 2, the control unit 21 has a physical configuration similar to that of the above-mentioned control unit 11, and is configured using a processor having a memory and any one of hardware such as a CPU, a GPU, an FPGA, a DSP, and an ASIC. The control unit 21 controls the operation of each unit constituting the viewing operation terminal 20. The control unit 21 includes an acquisition unit 211, a determination unit 212, a generation unit 213, and an output control unit 214. In one embodiment, the control unit 21 can function as a processor of the gaze tracking system 1.

[0037] The acquisition unit 211 acquires various information such as gaze information from the processing server 10 via the communication unit 22 and the network 2. The acquisition unit 211 can also acquire various information from the storage unit 23, the projection unit 24, the position acquisition sensor 25, the wearing sensor 26, the gaze sensor 27, and the imaging device 28. For example, the acquisition unit 211 can acquire necessary information from the user's physical information, user information, and position information. The user's physical information includes behavior information indicating behavior, user information, and gaze information. The acquisition unit 211 can also acquire various information from an external server via the communication unit 22 and the network 2.

[0038] The determination unit 212 makes a determination based on various information acquired by the acquisition unit 211. Specifically, the determination unit 212 can determine, for example, whether or not motion data based on the user's physical information and mainly on line-of-sight information has been input. The determination unit 212 may also determine whether or not predetermined information, for example, an origin image, has been input from the acquisition unit 211 of the viewing operation terminal 20. The determination unit 212 may also have a trained model or a learning model that includes input parameters for making a determination and output parameters indicating a determination result, which are generated by machine learning using a predetermined input / output data set. In this case, the determination unit 212 can make a predetermined determination based on output parameters obtained by inputting the input parameters into the trained model.

[0039] The generating unit 213 generates virtual objects such as an operation panel and an icon that are visible from the user's viewpoint. The output control unit 214 of the control unit 21 causes the projection unit 24 to output a user-prescribed virtual image based on the user's line of sight information detected by the line of sight sensor 27 and the program stored in the storage unit 23. The projection unit 24 is made of a medium that can transmit natural light. That is, the output control unit 214 outputs the virtual image to the projection unit 24 so as to be superimposed on the real space that the user is viewing, in a state in which the real space is visible. In other words, the output control unit 214 controls the output of the virtual image generated by the generating unit 213 to the projection unit 24. The details of the virtual image generated by the generating unit 213 will be described later.

[0040] The communication unit 22 is configured using a communication module capable of wireless communication. Under the control of the control unit 21, the communication unit 22 transmits and receives various information to and from the processing server 10 in accordance with the above-mentioned predetermined communication standard.

[0041] The storage unit 23 has a physical configuration similar to that of the above-mentioned storage unit 13. The storage unit 23 stores a gaze information database 231. The gaze information database 231 is a database similar to the gaze information database 131 in the storage unit 13 of the processing server 10, and can be synchronized with each other.

[0042] The projection unit 24 as a display unit is composed of a see-through medium configured to be able to transmit natural light. The projection unit 24 may display a virtual image by projecting it onto the lens of the viewing operation terminal 20B under the control of the control unit 21. In this case, the projection unit 24 is composed of a pair of left and right display panels having a predetermined parallax. The display panel is composed of a liquid crystal or organic EL (Electro Luminescence) display, or a display screen of a smartphone or the like.

[0043] The position acquisition sensor 25 can be configured to include a three-axis gyro sensor, a three-axis acceleration sensor, and a three-axis geomagnetic sensor (electronic compass). The position acquisition sensor 25 reads an image (hereinafter, origin image) including preset information such as a two-dimensional code, and outputs it to the control unit 21. The control unit 21 determines and recognizes the origin image such as a two-dimensional code by the determination unit 212. The control unit 21 can set the acquired origin image as the origin of the gaze position coordinates to be acquired later. By setting a three-dimensional direction with the origin image as the origin, the position acquisition sensor 25 can calculate the gaze position of the viewing operation terminal 20 as coordinates in a three-dimensional space, so that the control unit 21 can generate position information of the gaze of the wearer of the viewing operation terminal 20. That is, the position acquisition sensor 25 calculates the position information of the viewing operation terminal 20 relative to the position where the origin O is set, and outputs the calculated position information to the control unit 21. The wearing sensor 26 detects the wearing state of the user, and outputs the detection result to the control unit 21. The position acquisition sensor 25 can also detect behavior information related to the behavior of the user wearing the viewing operation terminal 20. The detection result detected by the position acquisition sensor 25 is output to the control unit 21. Specifically, when the position acquisition sensor 25 detects the behavior of the user, it detects the angular velocity and acceleration occurring in the viewing operation terminal 20 as behavior information, and outputs the detection result to the control unit 21. Furthermore, the position acquisition sensor 25 can also detect the absolute direction as behavior information by detecting geomagnetism, and output the detection result to the control unit 21.

[0044] The gaze sensor 27 detects the direction of the gaze of the user wearing the viewing operation terminal 20, and outputs the detection result to the control unit 21. The gaze sensor 27 is configured using an optical system, an image sensor such as a CCD or CMOS, a memory, and a processor having hardware such as a CPU. The gaze sensor 27 detects a part of the user's eye that does not move, such as the inner corner of the eye, as a reference point, and detects a part of the eye that moves, such as the iris, as a moving point, for example, using well-known template matching. The gaze sensor 27 detects the direction of the user's gaze based on the positional relationship between these reference points and moving points. The gaze sensor 27 tracks the gaze of the user who wears the terminal, and records the intersection coordinates (also called viewpoint coordinates) between the above-mentioned three-dimensional mesh and the gaze, using the origin image consisting of the above-mentioned two-dimensional code as the origin. At that time, gaze information such as the accumulated dwell time of the gaze and the number of times the gaze is directed to the same place can also be stored in the gaze information database 231 of the storage unit 23. It is also possible to derive the intersection coordinates by a method other than the three-dimensional mesh, and it is also possible to derive the intersection coordinates based on the relative position from the origin image.

[0045] At least one imaging device 28 as an imaging means is provided in the viewing operation terminal 20. The imaging device 28 generates imaging data by imaging the user's line of sight under the control of the control unit 21, and outputs the imaging data to the control unit 21. The imaging device 28 is configured using, but is not limited to, an optical system composed of one or more lenses and an image sensor such as a CCD or a CMOS. It is also possible to configure the viewing operation terminal 20 without providing the imaging device 28. The viewing operation terminal 20 images and senses the surroundings of the user who is the wearer, for example, the user's visual field area, and constantly or sequentially generates a three-dimensional mesh space and stores it in the storage unit 23. Specifically, for example, the imaging device 28 is configured to be able to mesh an object in the vicinity of the user.

[0046] The user's actions can be grasped by the control unit 21 when an image captured by the imaging device 28 is determined by the determination unit 212 of the control unit 21 or detected by the position acquisition sensor 25. In addition, the acquisition unit 211 of the control unit 21 can acquire the direction in which the user is facing and relative position information, and can acquire this information as line of sight information.

[0047] (Eye tracking method) Next, a gaze tracking method in the gaze tracking system 1 according to this embodiment will be described. 3 is a flowchart showing an outline of the processing executed by the gaze tracking system 1. In the following explanation, various information is transmitted and received between the communication unit 12 of the processing server 10 and the communication unit 22 of the viewing operation terminal 20 directly or via the network 2, but a detailed explanation of this point will be omitted. Also, various processes performed by the control unit 21 of the viewing operation terminal 20 may be performed by the control unit 11 of the processing server 10 instead. In this case, the processing related to the gaze is performed by the gaze processing unit 111, and the generation of various images is performed by the object image generation unit 112.

[0048] The flowchart shown in FIG. 3 is executed by a user, such as an expert in inspection technology, wearing the viewing operation terminal 20. As shown in FIG. 3, first, in step ST1, gaze information by a user, such as an expert, is acquired. That is, the user wears the viewing operation terminal 20 and performs an inspection operation on an observation target 100 (see FIG. 5), which is an actual object such as a mechanical facility. Here, a two-dimensional code is placed in advance near the observation target 100, which is an actual object, as an origin image that becomes the origin O of a spatial coordinate system. When a user wearing the viewing operation terminal 20 directs his / her gaze toward the two-dimensional code, the imaging device 28 of the viewing operation terminal 20 captures an image of the two-dimensional code. Here, the position where the user directs his / her gaze is measured by the gaze sensor 27. The acquisition unit 211 of the control unit 21 outputs image data of the acquired two-dimensional code to the determination unit 212. The determination unit 212 analyzes the two-dimensional code to check whether there is a problem with the recognition, and outputs the confirmation result to the generation unit 213. The generating unit 213 sets the position of the two-dimensional code as the origin O, and also sets the X-, Y-, and Z-coordinate axes of the three-dimensional space to fix it in the three-dimensional space. This defines and associates the position coordinates (x, y, z) in the three-dimensional space within the range that the imaging device 28 of the viewing operation terminal 20 can capture.

[0049] After that, in step ST2, the gaze tracking system 1 starts the gaze tracking process to track the gaze of the user. For example, an expert user inspects the observation target 100 and gazes at a place that he / she considers necessary. The gaze processing unit 111 determines the intersection point between the gaze information related to the user's gaze and the three-dimensional space based on the three-dimensional information as the intersection coordinate. At this time, the determination unit 212 determines the position coordinate (x, y, z) of the place where the user's gaze stays for a predetermined time or more as the intersection coordinate. The predetermined time can be arbitrarily set by the user of the gaze tracking system 1.

[0050] Furthermore, the accumulated dwell time, which is the accumulated dwell time during which the user's gaze is paused and which can be set as the attention level, can be derived based on the number of frames in the imaging data acquired by the imaging device 28. That is, the control unit 21 and the gaze processing unit 111 can set a three-dimensional mesh for the imageable area of ​​the real space captured by the imaging device 28, and measure the accumulated dwell time based on the number of frames in the imaging data during which the gaze is paused on each three-dimensional mesh.

[0051] In addition, the number of times the gaze passes through the three-dimensional mesh set in the imageable area of ​​the real space (number of passes) is derived. That is, the control unit 21 and the gaze processing unit 111 can measure the number of times the gaze of the user passes through each of the three-dimensional meshes set in the imageable area of ​​the real space captured by the imaging device 28, based on the number of frames in the captured data. The number of passes of the gaze can also be set as the attention level.

[0052] The determination unit 212 generates gaze information by associating the intersection coordinates, the accumulated dwell time, and the number of passes, and stores the information in the gaze information database 231 of the storage unit 23. The gaze information may be composed of the intersection coordinates and the accumulated dwell time, or may be composed of the intersection coordinates and the number of passes. The control unit 21 repeatedly executes the process of acquiring the intersection coordinates, measuring at least one of the accumulated dwell time and the number of passes, and storing the generated gaze information in the storage unit 23 until the gaze tracking process is completed. At this time, information on the appearance of the observation target 100 is also acquired as necessary. The gaze tracking process is terminated when a series of inspections by the user on the observation target 100 is completed. During the execution of the gaze tracking process, the viewpoint tile 120 described later may be generated and projected into the augmented reality space.

[0053] Next, the process proceeds to step ST3, where control unit 21 starts preparation for creating line-of-sight data, that is, a viewpoint confirmation process. Control unit 21 reads out line-of-sight information from line-of-sight information database 231 of storage unit 23 in which the line-of-sight information has been stored in steps ST1 and ST2.

[0054] In the viewpoint confirmation process, the generation unit 213 of the control unit 21 generates, for example, a thin tile object (hereinafter, viewpoint tile) in a square (rectangular) shape so that it can be superimposed on the part of the observed object 100 corresponding to the position where the user directs his / her gaze. FIG. 5 is a diagram showing a state in which the viewpoint tile is superimposed when a user (educator) wearing the viewing operation terminal 20 according to this embodiment looks at an object. The generation unit 213 generates the viewpoint tile 120 based on the gaze information, with the two-dimensional code shown in FIG. 5 as the origin O. When the viewpoint tile 120 is generated in the processing server 10, it is generated by the object image generation unit 112.

[0055] The generating unit 213 projects the generated viewpoint tile 120 to the projection unit 24 under the control of the output control unit 214. In other words, the fact that the line of sight hits the intersection coordinates described above as viewpoint tracking is projected and displayed as a mixed reality space by an object to the projection unit 24. As a result, as shown in FIG. 5, the viewpoint tile 120 is displayed superimposed on the position where the user directs his / her line of sight toward the observation target 100 in step ST2. Note that the viewpoint tile 120 as a derived result object is not necessarily limited to a rectangular shape, and can be changed to various shapes such as a circular shape, an elliptical shape, a triangular shape, and a polygonal shape, and the size of the viewpoint tile 120 can also be changed to various sizes. That is, for example, the size of the viewpoint tile 120 can be increased, decreased, or set to an intermediate size between these, so that it can be changed to a so-called large, medium, or small size and displayed, or the shape can be changed. In this way, by changing the size or the shape and displaying it, the user can visually recognize the importance of the result based on the change in the size and shape of the derived result object. Furthermore, without being limited to the viewpoint tiles 120, it is also possible to directly display numerical values ​​or symbols on the projection unit 24 so that the user can intuitively recognize the derived results.

[0056] In the example shown in FIG. 5, the viewpoint tile 120 can be generated as a plane obtained by enlarging a plane parallel to the surface of the observed object 100 at the position where the user directs his / her gaze. This allows the user, who is the wearer of the viewing operation terminal 20, to clearly recognize the position where the user directs his / her gaze, and the viewpoint tile 120 allows the user to recognize the location where the gaze falls, which is called a viewpoint confirmation function, by recognizing the viewpoint tile 120 generated at the intersection coordinates where the gaze falls. Here, the viewpoint tile 120 at the position where the user inadvertently directs his / her gaze can be automatically or manually removed in advance. The intersection coordinates of the gaze and the viewpoint tile 120 may be collected instantly (in real time).

[0057] In addition, since the viewpoint tile 120 is superimposed and projected on the observation target 100 as a mixed reality space, the relative relationship between the observation target 100 and the viewpoint tile 120 does not change even if the user moves or moves his / her line of sight. That is, the viewpoint tile 120 that the user visually recognizes through the projection unit 24 of the viewing operation terminal 20 is fixed and projected onto the observation target 100. In other words, the viewing operation terminal 20 is attached, the two-dimensional code is recognized by the imaging device 28 to set the origin, and a three-dimensional coordinate fixed with respect to the real space is set. Therefore, even if the user moves, the position of the viewpoint tile 120 does not change, so that the viewpoint tile 120 can be observed and analyzed from various positions. Information such as the display position of the viewpoint tile 120 and the shape of the viewpoint tile 120 is stored in the line of sight information database 231 of the storage unit 23 as the user's line of sight data. When stored in the processing server 10, the user's line of sight data is stored in the line of sight information database 131 of the storage unit 13.

[0058] Furthermore, the acquisition unit 211 acquires an accumulated dwell time, which is an accumulated dwell time of how long the gaze stayed at a position where the user repeatedly directed the gaze. Here, the accumulated dwell time can be specifically derived by recording three-dimensional coordinates (x, y, z) for every arbitrarily specified minute time Δt, for example, about 1 / 60 seconds, and color-coding based on the total number of counts in a specified spatial range. The determination unit 212 determines whether or not to store the intersection coordinate information in the storage unit 23 according to the accumulated dwell time of the gaze at the intersection coordinate. When the determination unit 212 determines that the accumulated dwell time of the gaze at a predetermined intersection coordinate is less than a predetermined time, for example, less than 3 seconds, it determines not to store the intersection coordinate information in the storage unit 23. On the other hand, when the determination unit 212 determines that the accumulated dwell time of the gaze at a predetermined intersection coordinate is equal to or longer than a predetermined time, for example, equal to or longer than 3 seconds, it determines to store the intersection coordinate information in the storage unit 23, and stores the intersection coordinate information in the gaze information database 231 of the storage unit 23.

[0059] The generating unit 213 may change the display color of the viewpoint tile 120 according to the cumulative dwell time of the line of sight at the intersection coordinates. Here, the color of the viewpoint tile 120 according to the cumulative dwell time of the line of sight may be, for example, light blue for a short time of 3 seconds or less, green for more than 3 seconds or less than 5 seconds, yellow for more than 5 seconds or less than 20 seconds, and red for more than 20 seconds (for example, viewpoint tile 120a in FIG. 5). The process of color-coding the viewpoint tiles 120 based on such a predetermined condition is referred to as heat mapping in this specification, and the collection of the obtained viewpoint tiles 120 is referred to as a heat map. This allows the user to recognize at which position of the observation target 100 the line of sight stayed for how long. In addition, the display of the viewpoint is not limited to a tile shape, and a conventionally known display such as a planar heat map is also possible.

[0060] The acquisition unit 211 also acquires the number of passes of the line of sight at the position where the user repeatedly directs his / her line of sight. The determination unit 212 may determine whether or not to store the information of the intersection coordinates in the storage unit 23 according to the number of passes of the line of sight based on the three-dimensional mesh at the intersection coordinates. When the number of passes of the line of sight at a predetermined intersection coordinate is a predetermined number or more, for example, three or more, the determination unit 212 determines to store the information of the intersection coordinates in the storage unit 23, and stores the information of the intersection coordinates in the line of sight information database 231 of the storage unit 23. The generation unit 213 may also change the display color of the viewpoint tile 120 according to the number of passes of the line of sight at the intersection coordinates. This allows the user to recognize how many passes the line of sight has passed at which position of the observation target 100.

[0061] 5, an object to be observed such as the observed object 100 shown in FIG. 5 may be an observed object consisting of a three-dimensional object (also called a 3D object) which is a virtual image. In this case, the observed object may be generated by the object image generating unit 112. That is, the acquiring unit 211 acquires information on the appearance of the observed object 100 to be observed and transmits it to the processing server 10. The object image generating unit 112 of the processing server 10 generates an observed object based on the information on the appearance of the observed object 100 transmitted from the viewing operation terminal 20, and stores it in the storage unit 13 so as to be readable.

[0062] The generated observation target object is projected on the projection unit 24 of the viewing operation terminal 20. The observation target of the viewpoint tile 120 described above is not limited to a real object, and can be a three-dimensional virtual object generated from image data such as CAD data or scanning data. That is, the gaze tracking target is not limited to a real object or space, and may be a virtual object such as a three-dimensional object, a point cloud, an image, or an avatar based on electronic image data such as CAD data, and the virtual object may be stationary, moving, or deformed. Specifically, for example, the projection unit 24 projects the observation target object to be virtually placed in place of the observation target 100 at an arbitrary position such as a desk or a square, and the surface coordinates defined as the surface of the observation target object are set based on the spatial coordinates, and the intersection between the user's gaze and the position coordinates of the outer surface part of the observation target object can be set as the intersection coordinates of which part is gazed at. Information on the intersection coordinates with the observation target object can be recorded in the storage unit 13, or analyzed or evaluated by the gaze processing unit 111 of the control unit 11.

[0063] Next, the process proceeds to step ST3, where a user, such as an expert, wearing the viewing operation terminal 20 creates gaze data. The expert, who is an educator, wears the viewing operation terminal 20 and observes the observation target 100 or a three-dimensional object of the observation target 100 to perform inspection and testing. The user wearing the viewing operation terminal 20 visually recognizes a plurality of viewpoint tiles 120 that are generated by the control units 11 and 21 in steps ST1 and ST2 and are displayed superimposed on the observation target 100. The expert sets an area to be watched as a point area or the like based on the viewpoint tiles 120, which are derived result objects displayed in a heat map and projected on the projection unit 24. The point area is not limited to a spatial area, and may be any of points, lines such as straight lines and curves, surfaces such as planes and curved surfaces, and three-dimensional shapes such as spheres, rectangular parallelepipeds, and any shape. The point area is an attention area that serves as a criterion for grading in a test mode to be described later.

[0064] Specifically, a conventionally known virtual operation panel can be projected and displayed on the projection unit 24 of the viewing operation terminal 20. The user who wears the viewing operation terminal 20 can perform a predetermined operation on the observation target 100 or a three-dimensional object displayed on the projection unit 24 by operating the operation panel displayed on the projection unit 24. The predetermined operation is inputting educational information such as designation of a point area, text data explaining the setting reason linked to the point area, importance, weighting, and score. That is, the user who wears the viewing operation terminal 20 can set the point area by the operation panel, and input and associate various educational information such as the setting reason and score. Note that, if necessary, the viewing operation terminal 20 may be provided with a haptic feedback (tactile feedback) function according to the state. In this manner, the point area is set and the educational information is associated with the point area, and gaze data is created.

[0065] Next, the process proceeds to step ST4. In step ST4, the gaze data obtained by steps ST1 to ST3 is stored in the gaze information database 231 of the storage unit 23. The gaze information database 131 of the processing server 10 can be appropriately synchronized with the gaze information database 231 of the viewing operation terminal 20. The gaze information acquired from the gaze information database 131 of the processing server 10 can be overwritten or added to the gaze information database 231 of the viewing operation terminal 20. This completes the gaze data collection process. The gaze data can also be stored in the gaze information database 231 as educational data. This allows the multiple viewing operation terminals 20 (20A, 20B) as terminals to access the gaze information database 231 to acquire the gaze data, thereby allowing the gaze data to be shared instantly, at any time, or at any time among the multiple viewing operation terminals 20.

[0066] Next, a method of educating an unskilled person (a person to be trained) based on gaze information of an expert (a trainer) when inspecting mechanical equipment using the gaze tracking method according to this embodiment will be described. FIG. 4 is a flow chart showing an example of an education method using the gaze tracking method according to this embodiment. FIG. 6 is a diagram showing a state in which a viewpoint tile is superimposed on an observed object 100 when an unskilled person (a person to be trained) looks at an observed object 100 as a user wearing a viewing operation terminal 20 according to this embodiment. In this specification, an expert means a user who is familiar with a technology and can make a predetermined judgment on a predetermined technology, but the level of skill of a user to be called an expert can be arbitrarily set by the user of the gaze tracking system, etc.

[0067] The flowchart shown in Fig. 4 is an example in which a training subject, such as an unskilled person, becomes a user who wears the viewing operation terminal 20. As shown in Fig. 4, in step ST11, first, the unskilled person who is the training subject wears the viewing operation terminal 20. In step ST11, the user operates the operation panel projected on the projection unit 24 to display an education menu on the projection unit 24. The user uses his / her finger or voice to indicate a required part, i.e., an education menu, in the space in which the virtual operation panel is superimposed and displayed.

[0068] Next, the process proceeds to step ST12, and when the acquisition unit 211 of the control unit 21 determines that the user has selected the education mode and selected the observation target, the control unit 21 reads out the gaze data of the educator (expert) associated with the selected observation target 100 from the gaze information database 231 of the storage unit 23. Thereafter, the process proceeds to step ST13, and the user inputs user information using the operation panel. The user inputs the necessary user information using his / her finger, voice, or the like into the space in which the virtual operation panel is superimposed and displayed.

[0069] Next, the process proceeds to step ST14. In step ST14, the viewing operation terminal 20 worn by the user acquires and stores the gaze information of the unskilled person in the test mode using the read gaze data. That is, the unskilled user wears the viewing operation terminal 20 and performs a test of, for example, an inspection task of the observed object 100.

[0070] As shown in Fig. 6, a two-dimensional code is placed in advance near the observation target 100 to be inspected as an origin image that becomes the origin O of a spatial coordinate system. When an unskilled user wearing the viewing operation terminal 20 gazes at the two-dimensional code, the imaging device 28 of the viewing operation terminal 20 captures an image of the two-dimensional code. The control unit 21 of the viewing operation terminal 20 sets the position of the two-dimensional code as the origin O, and sets the X-, Y-, and Z-coordinate axes of a three-dimensional space with the position of the two-dimensional code as the origin O according to the read-out line-of-sight data. As a result, position coordinates (x, y, z) in the three-dimensional space are defined and associated within a range visible by the imaging device 28 of the viewing operation terminal 20.

[0071] After that, the gaze tracking system 1 starts a gaze tracking process for tracking the gaze of the user. The unskilled user gazes at a place that he / she considers necessary to inspect the observed object 100. The determination unit 212 determines the position coordinates (x, y, z) at the place where the user's gaze stays for a predetermined time or more as intersection coordinates. The determination unit 212 repeatedly executes a process of storing the intersection information as gaze information in the gaze information database 231 of the storage unit 23 until the gaze tracking process ends. When a series of inspections by the user on the observed object 100 ends, the gaze tracking process in the test mode ends. The determination unit 212 can also set a time limit from the start to the end of the test mode. In the gaze tracking process in step ST14, the viewing operation terminal 20 can measure the frequency of directing the gaze to a point area that is a predetermined attention area.

[0072] Next, when the test mode ends, the process proceeds to step ST15 and the scoring mode starts. In the test mode, the judgment unit 212 of the control unit 21 calculates a score, which is an evaluation result, by comparing the number of times and cumulative dwell time that an educational subject such as an unskilled person who took the test directed his / her gaze at a noteworthy part with the gaze information of an expert, that is, performs scoring. In this case, the score can be output by tallying up the degree of agreement with the gaze information of the expert and the points set in the point area. For example, by comparing FIG. 5, which is an example of a case where an expert user inspects the observation target 100, with FIG. 6, which is an example of a case where an unskilled person inspects the observation target 100, it is possible to extract the degree to which the area checked by the unskilled person differs from the area checked by the expert. In addition, the degree of agreement of the heat map may be calculated as a score by comparing the gaze information of the unskilled person with the gaze information for the teacher generated by the visual recognition of the expert. The calculated score information can be associated with the user information and stored in the storage unit 23 as educational information. This makes it possible to provide a more realistic educational tool for experts to inexperienced people, and to further improve the effectiveness of education using the eye tracking system 1.

[0073] Furthermore, when an inexperienced person who is the subject of training wears the viewing operation terminal 20 and uses the viewpoint confirmation function, the viewpoints to which attention should be paid are displayed by the viewpoint tiles 120, superimposed on the actual appearance of the observed object 100, so that the necessary inspection points can be confirmed with a high sense of realism. Furthermore, by filtering the viewpoints using information such as the cumulative gaze dwell time and the number of gaze turns described above, it is also possible to extract important viewpoints (intersection coordinates) and project them on the projection unit 24.

[0074] FIG. 7 is a diagram showing an example of the display of the scoring result displayed on the display screen of the projection unit 24 of the viewing operation terminal 20 according to the present embodiment. As shown in FIG. 7, the output control unit 214 of the control unit 21 outputs, as the point area 130a, a notation of the score set for the point to be noted based on the viewpoint of the expert. Also, the output control unit 214 displays the point of the part that the expert thinks should be paid attention to based on the viewpoint of the expert. The text data displayed in association with the point area 130b is text data such as notes entered by the expert when generating the gaze data based on the gaze information of the expert. The point area 130c displays the score of the unskilled person (trainee) based on the gaze data of the expert and the unskilled person. Thereby, the unskilled person who is the trainee is indirectly taught important checkpoints by the expert, and can further confirm the reason why the expert gazed. The display data of the scoring result as shown in FIG. 7 may be captured by the acquisition unit 211 after being projected on the projection unit 24, and automatically stored as an image in the gaze information database 231.

[0075] According to the embodiment described above, the viewpoint of the educator can be recorded as three-dimensional position information, so that the expert's attention point can be observed, and a high educational effect can be obtained. Conventionally, eye tracking was possible only in a place where the inspector could see the observed object 100, and the actual observed object 100 was required for creating data on the gaze information of the expert and for testing and grading of the unskilled, so there were time and place restrictions. In contrast, in the above-mentioned embodiment, the observed object 100 itself can be generated as a three-dimensional object that is a virtual image, so that even in a place where the observed object 100 does not exist, it is possible to obtain data on the gaze information of the expert and to test and grade the unskilled. Therefore, it is possible to train the observed object 100 in visual inspection using the eye tracking system 1 at any place, and to educate the unskilled about visual inspection. Since the actual observed object 100 is not required for creating data on the gaze information of the expert and for testing and grading of the unskilled, there are no time or place restrictions, and it is possible to further optimize educational opportunities.

[0076] According to one embodiment, the eye tracking function of the gaze sensor 27 of the viewing operation terminal can be used to obtain information on which part of the field of view the expert is gazing at, and this can be used to convert into know-how information on what the expert is focusing on, for example, during a witness inspection at the time of factory shipment or on-site acceptance. Furthermore, a series of visual confirmation tasks displayed as a heat map using three-dimensional objects such as the viewpoint tiles 120 can be saved as image data including video, and the reason and circumstances of the expert's gaze during the visual confirmation task can be added to the image data as data that can be recognized by other users, making it possible to use the gaze tracking system 1 as an educational tool.

[0077] (Recording medium) In the above-mentioned embodiment, the program executed by the processing server 10 or the viewing operation terminal 20 can be recorded in a recording medium readable by a computer or other machine or device such as a viewing operation terminal (hereinafter referred to as a computer, etc.). By making a computer, etc. read and execute the program from the recording medium, the computer, etc. functions as an eye tracking system. Here, a recording medium readable by a computer, etc. refers to a non-transient recording medium that accumulates information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from a computer, etc. Among such recording media, those that can be removed from a computer, etc. include, for example, a flexible disk, a magneto-optical disk, a CD-ROM, a CD-R / W, a DVD, a BD, a DAT, a magnetic tape, and a memory card such as a flash memory. In addition, a hard disk, a ROM, etc. are examples of recording media fixed to a computer, etc. Furthermore, an SSD can be used as a recording medium that can be removed from a computer, etc., and as a recording medium fixed to a computer, etc.

[0078] In addition, the processing server, viewing operation terminal, and programs to be executed by the processing server and viewing operation terminal according to one embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading via the network.

[0079] Although one embodiment of the present invention has been specifically described above, the present invention is not limited to the above-described embodiment, and various modifications based on the technical concept of the present invention are possible. For example, the virtual images and operations given in the above-described embodiment are merely examples, and different virtual images and operations may be used.

[0080] (Other embodiments) In the above-mentioned embodiment, an example was described in which a glasses-type viewing operation terminal that can be worn by a user was used, but the above-mentioned embodiment is not limited thereto, and can be applied to various viewing operation terminals. For example, a contact lens-type viewing operation terminal having an imaging function, a brain chip-type viewing operation terminal that directly transmits information to the user's brain, or a helmet-shaped terminal with a visor may be configured to project and display an image on the visor. In the above-mentioned embodiment, the viewing operation terminal projects an image onto the user's retina, but the image may be projected and displayed on the lens of glasses, for example.

[0081] It is also possible to use a smartphone equipped with an imaging device as a terminal similar to the viewing operation terminal 20. In this case, the imaging direction in which the imaging device of the smartphone faces the observation target 100 or the like is set as the line of sight of the user described above, so that the viewing operation terminal 20 described above can be replaced with a smartphone.

[0082] In addition, in one embodiment, the above-mentioned "unit" can be read as a "circuit" etc. For example, a control unit can be read as a control circuit.

[0083] In the description of the flowcharts in this specification, the order of processing between steps is clearly indicated using expressions such as "first," "then," and "continue," but the order of processing required to implement this embodiment is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed as long as there is no contradiction.

[0084] Further advantages and modifications may readily occur to those skilled in the art. The invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Thus, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof.

[0085] For example, it is possible to configure the processing server 10 and the viewing operation terminal 20 as a single viewing operation terminal. In this case, the control units 11 and 21 are shared, the storage units 13 and 23 are shared, and the eye gaze tracking system 1 is configured by the viewing operation terminal.

[0086] Also, for example, in the above-mentioned embodiment, the viewing operation terminal 20 such as a headset is configured to display the real space in a transparent state, i.e., in a see-through state, but the present invention is not limited to this, and it is also possible to adopt, for example, a video see-through configuration. Here, the video see-through method is a method in which imaging data captured by an imaging device 28 provided in the viewing operation terminal 20 is instantly (real-time) projected onto the viewing operation terminal 20, for example, to allow the real space to be indirectly recognized by the imaging device 28, and a virtual object is projected in a centralized manner. In this case, it is preferable to adopt a non-transparent display as the projection unit 24.

[0087] In the embodiment described above, the projected image and viewpoint tile 120 of the augmented reality space may be supplemented by the processing server 10, etc., and may be configured to be viewable by any display device, not limited to the viewing operation terminal 20. That is, the augmented reality image, virtual reality image, and mixed reality image may also be displayed on a monitor, which is a display unit of an information processing device. This allows multiple people to view the monitor and check the movement of the line of sight by the viewing operation terminal 20.

[0088] In the embodiment described above, it is also possible to set an area to be focused on as a point area, and score the educational effect of the subject using the number of times the point area was gazed at, etc. Also, a function may be provided to specify and set an area to be focused on in advance by analyzing the setting of the point area by an expert, and score how frequently the gaze of a user such as an inexperienced person was directed to the area to be focused on. [Explanation of symbols]

[0089] 1. Eye Tracking System 2 Network 10 Processing Server 11,21 Control section 12,22 Communications Department 13,23 Storage part 20, 20A, 20B Viewing and operation terminal 24 Projection section 25 Position acquisition sensor 26 Wearable Sensor 27 Eye gaze sensor 28 Imaging Device 100 Observation Targets 111 Line of Sight Processing Unit 112 Object image generation unit 120,120a Viewpoint tiles 130a, 130b, 130c Point Area 131 Gaze Information Database 211 Acquisition Department 212 Judgment section 213 Generation part 214 Output control section 231 Gaze Information Database

Claims

1. An eye tracking system for tracking a user's gaze, comprising: The method acquires line-of-sight information of the user, derives at least one area that the user is paying attention to and a degree of attention to the area using the acquired line-of-sight information, and superimposes the derived result on an object being observed by the user or an image of the object being observed captured by a predetermined imaging means. Eye tracking system.

2. The line of sight information is information generated based on the intersection coordinates between a three-dimensional mesh defined on the surface of the object in a three-dimensional space set with respect to a predetermined origin and the line of sight of the user or the imaging direction of the imaging means. The eye-tracking system of claim 1 .

3. the observation target is a real object in a real space or a virtually generated observation target object, The intersection coordinates are set based on the position coordinates of the surface of the actual object or the outer surface of the observation target object. The eye-tracking system of claim 2 .

4. The line of sight information includes information on the time during which the line of sight of the user or the image capturing by the imaging means remained, or the number of times the line of sight of the user was directed or the number of times the image capturing by the imaging means was performed. The eye-tracking system of claim 1 .

5. The derived result is superimposed on the object or an image of the object using a derived result object. The eye-tracking system of claim 1 .

6. The derived result object expresses the derived result by color, shape, or size. The eye-tracking system of claim 5 .

7. The line-of-sight information or the derived result object is stored in a storage unit accessible by a plurality of different terminals, and is configured to be shareable among the plurality of terminals immediately or at any time. An eye-tracking system according to claim 5 or 6.

8. Evaluating the gaze information based on the gaze information and pre-generated educational information; Informing the user of an evaluation result derived by the evaluation. The eye-tracking system of claim 1 .

9. 1. A method for eye tracking implemented by a processor having hardware, comprising: Acquire gaze information of the user by tracking the gaze of the user; Using the acquired gaze information, deriving at least one area that the user is paying attention to and a degree of attention to the area; The derived result is superimposed on an image of the object visually recognized by the user or the object captured by a predetermined imaging means. Eye-tracking method.

10. A processor having hardware, Acquiring gaze information of a user by tracking the gaze of the user; Using the acquired gaze information, deriving at least one area that the user is paying attention to and a degree of attention to the area; The derived result is superimposed on an image of the object visually recognized by the user or the object captured by a predetermined imaging means. An eye-tracking program that does just that.

Images