Gaze position analysis system and method

Abstract

To provide a gazing point analysis system capable of automatically associating a gazing point of a user on a 3D model corresponding to each object in a three-dimensional space.SOLUTION: The gazing point analysis system includes: a spatial model creation unit that creates a spatial model, which is a three-dimensional model of the surrounding space from multiple captured images; an object position posture estimation unit that matches the spatial model with the object model and arranges the object model on the spatial model according to the position and posture obtained by the matching; and a gazing point calculation unit that calculates the gazing point on the object model based on the gaze direction in the arranged object model and the spatial model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a gaze position analysis system and a gaze position analysis method. [Background technology]

[0002] When analyzing a user's interests, work status, etc. from gaze information acquired using a gaze measurement device worn by the user, the user moves freely in a three-dimensional space in which various objects are placed. For this reason, it is desirable to be able to confirm where the user's gaze is located in the three-dimensional space and how the gaze transition is occurring.

[0003] Patent documents 1 and 2 disclose technologies that target an eye gaze measurement device worn by a user and analyze the eye gaze information acquired while the user moves freely in three-dimensional space as the gaze position in three-dimensional space.

[0004] Patent document 1 discloses a technology that generates a virtual three-dimensional space using images taken from multiple different shooting positions, calculates the shooting positions in the virtual three-dimensional space, and calculates the user's gaze position and gaze time in the virtual three-dimensional space from the user's line of sight obtained at the time the image was taken.

[0005] On the other hand, Patent Document 2 discloses a technique for identifying an object that a user has focused on based on the position of a display device in three-dimensional space, the user's line of sight, and the position of the object in three-dimensional space. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-135737 A [Patent Document 2] JP 2018-195319 A Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, the virtual three-dimensional space generated from multiple images may contain sparse data or may only be partially generated, making it difficult to determine what the subject was looking at.

[0008] On the other hand, in Patent Document 2, since it is assumed that the line of sight of the user in the three-dimensional space and the position of the object in the three-dimensional space can be obtained at all times, there are problems in that the required equipment is large-scale and the scope of application is limited. In addition, it is necessary to clarify the positions of all objects to be analyzed in advance, which requires a lot of effort for advance preparation.

[0009] When analyzing gaze position in three-dimensional space, in addition to being able to see how the gaze position has shifted throughout the entire three-dimensional space, it is desirable to be able to focus only on a specific object and easily see how the gaze position has shifted over that object.

[0010] In this case, if it is possible to associate the gaze position on a three-dimensional model of each object that exists in three-dimensional space, it is expected that a more detailed analysis will be possible. Also, if each object model can be automatically positioned in three-dimensional space, it is thought that the pre-settings required for the analysis can be simplified.

[0011] An object of the present invention is to automatically associate a user's gaze position on a three-dimensional model corresponding to each object existing in a three-dimensional space in a gaze position analysis system. [Means for solving the problem]

[0012] A gaze position analysis system according to one aspect of the present invention has a spatial model creation unit, an object position and orientation estimation unit, and a gaze position calculation unit, and is a system that automatically matches an object model gaze position, which is a user's gaze position on an object model, onto an object model that is a three-dimensional model of an object existing in space. The spatial model creation unit acquires a first-person video that is an image similar to the user's field of vision, acquires a first-person video gaze position, which is the gaze position of the user on the first-person video, creates a spatial model from the first-person video that is a three-dimensional model of the space in the range in which the user is directed, and calculates the gaze position of the first-person video in the spatial model. the object position and orientation estimation unit estimates the position and orientation of the object in the spatial model by matching the object model with the spatial model, places the object model in the spatial model using the position and orientation of the object in the spatial model, calculates the gaze direction in the spatial model using the shooting position and orientation of the first-person video in the spatial model and the first-person video gaze position, and the gaze position calculation unit calculates the object model gaze position by finding an intersection of the gaze direction and the object model in the spatial model.

[0013] One embodiment of the gaze position analysis system of the present invention is a gaze position analysis system that automatically matches a user's gaze position on an object model, which is a three-dimensional model of an object existing in space, and is characterized by having a spatial model creation unit that creates a spatial model, which is a three-dimensional model of the surrounding space, from multiple captured images, an object position and orientation estimation unit that matches the spatial model with the object model and places the object model on the spatial model based on the position and orientation obtained by the matching, and a gaze position calculation unit that calculates the gaze position on the object model based on the placed object model and the gaze direction in the spatial model. Effect of the Invention

[0014] According to one aspect of the present invention, in a gaze position analysis system, the gaze position of a user can be automatically associated with a three-dimensional model corresponding to each object existing in a three-dimensional space. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a diagram showing the configuration of a general computer when the gaze position analysis system according to the embodiment of the present invention is executed by the general computer. [Diagram 2] 1 is a diagram illustrating a basic configuration of a gaze detection device assumed in the present invention. [Diagram 3] FIG. 1 is a diagram illustrating an example of a format of first-person video data. [Figure 4] FIG. 11 is a diagram illustrating an example of a format of gaze position data. [Diagram 5] FIG. 2 is a diagram illustrating an example of a format of object model data. [Figure 6] FIG. 2 is a diagram illustrating an example of a format of spatial model data. [Figure 7] FIG. 13 is a diagram illustrating an example of a format of object placement data. [Figure 8] FIG. 2 is a diagram illustrating the contents of a process executed by a spatial model creation program. [Figure 9] FIG. 11 is a diagram illustrating an example of a flowchart of a process executed by a gaze position calculation program. [Figure 10] 11A to 11C are diagrams illustrating the contents of a process executed by a gaze position calculation program. [Figure 11] FIG. 1 is a diagram showing an example of an AR marker placed on an actual object. [Figure 12] FIG. 1 is a diagram showing an example of an object model whose shape and size change. [Figure 13] FIG. 2 is a diagram showing an example of an object model and a gaze position on the object model. [Figure 14] FIG. 13 is a diagram showing an example of a case where the gaze position is always displayed facing the user. [Figure 15]13A and 13B are diagrams illustrating an example of a case where a gaze position is displayed by restricting a method for adjusting the position and orientation of an object model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0017] FIG. 1 is a diagram showing the configuration of a general computer when the gaze position analysis system according to the embodiment of the present invention is executed by the general computer. The gaze measurement device 101 in FIG. 1 is an input device for measuring a first-person image, which is an image similar to the user's visual field, and the gaze position on the first-person image, and recording them in a database for storing first-person image data 110 and gaze position data 111, respectively; a commonly used device known as an "eye tracker" or the like can be used.

[0018] In particular, the gaze measurement device 101 in the present invention is assumed to be a portable device that can be worn by a user, allowing the gaze direction to be measured while the user is able to move freely in space.

[0019] FIG. 2 shows a basic configuration of a gaze detection device 101 that is worn by a user. 2, 201 is a photographing device for acquiring a first-person video of a user, and a device equivalent to a camera generally used in a personal computer or the like can be used. 202 is a glasses-type device equipped with a sensor for detecting the movement of the user's eyes and measuring the gaze direction. 203 is a terminal for recording the acquired first-person video data 110 and gaze position data 111 in a database. The terminal 203 may transmit data to the information processing device 102.

[0020] The gaze measurement device 101 is not limited to a portable type, and may be a stationary type device as long as it is capable of measuring the gaze position while the user is able to move freely through the space and is capable of acquiring the gaze position on a first-person image.

[0021] An information processing device 102 in FIG. 1 is an information processing device for executing each program in the gaze position analysis system.

[0022] The input device 103 includes general computer input devices such as a keyboard, buttons, a mouse, or a touch panel for controlling the start and end of the system.

[0023] The output device 104 is a means for displaying the results of gaze position analysis and the operating status of the system to the user, and includes the screen of a smartphone or tablet terminal, or a display device for a general computer.

[0024] Further, a storage device 105 stores each program in the gaze position analysis system. The storage device 105 includes a space model creation program 106, an object position and orientation estimation program 107, a gaze position calculation program 108, and a gaze position display program 109.

[0025] Here, the information processing device 102 functions as a space model unit by executing processing according to a space model creation program 106. Moreover, the information processing device 102 functions as an object position / posture estimation unit by executing processing according to an object position / posture estimation program 107. Moreover, the information processing device 102 functions as a gaze position calculation unit by executing processing according to a gaze position calculation program 108. Moreover, the information processing device 102 functions as a gaze position display unit by executing processing according to a gaze position display program 109.

[0026] Data of first-person video, which is a video similar to the user's field of vision, is stored in the database of first-person video data 110. It is assumed that the first-person video data 110 is acquired by the gaze measurement device 101 as described above.

[0027] FIG. 3 shows an example of the format of the first-person video data 110. The data name 301 in Fig. 3 is a name given to the first-person video data, and can be any string of characters and symbols. The first-person video data 110 includes multiple images captured at predetermined time intervals or at any timing in the order in which they were captured, and the number of data 302 in Fig. 3 indicates the number of images included in the data.

[0028] Time 1 in 303 indicates the time when the first image was captured, image 1 in 304 indicates the first image captured, and shooting position and orientation 1 in 305 indicates the position and orientation of the camera that captured the first image. Time n in 306 indicates the time when the nth image was captured, image n in 307 indicates the nth image, and shooting position and orientation n in 308 indicates the position and orientation of the camera that captured the nth image. Note that the shooting positions and orientations 305 and 308 are data calculated by the space model creation program 106, as will be described later, and are blank when each image is captured.

[0029] Furthermore, any format may be used for the first-person video data 110 as long as the image at each time in the data can be easily acquired, such as a commonly used video format. The gaze position data 111 database stores the gaze position data acquired by the gaze measurement device 101.

[0030] FIG. 4 shows an example of the format of the gaze position data 111. As described above, since it is assumed that the gaze position is expressed as position coordinates on the first-person video data 110, the gaze position data 111 has corresponding first-person video data 110. Therefore, the correspondence between the gaze position data 111 and the first-person video data 110 is expressed by describing the same name as the data name 301 in the corresponding first-person video data in the data name 401.

[0031] Furthermore, the gaze position data 111 includes a plurality of position coordinates acquired at a predetermined time interval or at an arbitrary timing in the order of acquisition. The number of data 402 in Fig. 4 indicates the number of position coordinates included in the data. 403 indicates the time when the gaze position coordinates were acquired for the first time, 404 indicates the position coordinates of the gaze position acquired for the first time, and 405 indicates the name of the object when the gaze position data is associated with an object model by the gaze position calculation program 108.

[0032] 406 is the time when the position coordinates of the gaze position are acquired for the nth time, 407 is the position coordinates of the gaze position acquired for the nth time, and 408 is the name of the object when the gaze position data is associated with an object model by the gaze position calculation program 108. At the time when the gaze position data is acquired, there is no associated object model, so 405 and 408 are left blank.

[0033] Furthermore, when the gaze position data 111 is acquired, the gaze position is a position coordinate on the first-person image, i.e., two-dimensional coordinate data, but when the gaze position data 111 is associated with an object model, it becomes a position coordinate on the object model and is rewritten into three-dimensional coordinate data.

[0034] The above-described gaze position data 111 is premised on the assumption that the gaze position data acquired by the gaze measurement device 101 and the gaze position data associated with the object model are managed as the same data. However, the gaze position data acquired by the gaze measurement device 101 and the gaze position data associated with the object model may be managed as separate data.

[0035] The database of object model data 112 stores three-dimensional models of objects that exist in the space within the range of the user's line of sight. In the present invention, it is assumed that the three-dimensional model of each object is represented as a collection of points that represent the shape of the object, that is, as point cloud data.

[0036] Object models created using general 3D CAD and the like are often represented as a collection of polygons, but object models represented as polygons can be easily converted into point cloud models. For example, first, for each polygon, divide the polygon into triangles with a line connecting a vertex to a vertex other than the adjacent vertex, and then for each triangle, divide the triangle with a line connecting a vertex to the midpoint of the opposite side, and finally select the vertices of all triangles, so that the object model represented by polygons can be represented as point cloud data.

[0037] FIG. 5 shows an example of the format of the object model stored in the database of the object model data 112. 501 in Fig. 5 is a name given to an object, and can be any string of characters and symbols. 502 indicates the number of points included in the object model, 503 indicates the position coordinates of the first point, and 504 indicates the position coordinates of the nth point. The format shown in Fig. 5 includes only the position coordinates of the points, but it is also possible to include information associated with each point, such as information about color.

[0038] A spatial model, which is a three-dimensional model of the space within the range of the user's line of sight, is stored in the database of spatial model data 113. The spatial model is data created by the spatial model creation program 106 using first-person video data 110, and is assumed to be represented as point cloud data in the same manner as the above-mentioned object model.

[0039] FIG. 6 shows an example of the format of a spatial model stored in the database of the spatial model data 113. The model name 601 in Fig. 6 is the name given to the spatial model, and can be any string of characters and symbols. 602 is the name of the first-person video data used to create the spatial model, 603 is the number of points included in the spatial model, 604 is the position coordinate of the first point, and 605 is the position coordinate of the nth point. The format shown in Fig. 6 includes only the position coordinates of the points, but information associated with each point, such as information about color, may also be included.

[0040] The database of object arrangement data 114 stores data relating to object models that are matched to a space model and arranged on the space model by the object position and orientation estimation program 107 .

[0041] FIG. 7 shows an example of the format of the object placement data 114 stored in the database of the object placement data 114. Assuming that the object placement data 114 is stored for each spatial model, the name of the spatial model corresponding to the model name 701 in Fig. 7 is described. The number of objects 702 indicates the number of object models placed in the target spatial model.

[0042] 7, 703 denotes the name of the first object placed on the spatial model, 704 denotes the position and orientation of the first object on the spatial model, and 705 denotes the time when the first object was placed on the spatial model, 706 denotes the name of the nth object placed on the spatial model, 707 denotes the position and orientation of the nth object on the spatial model, and 708 denotes the time when the nth object was placed on the spatial model.

[0043] In the present invention, analysis processing of the gaze position on each object is performed using a space model creation program 106, an object position and orientation estimation program 107, and a gaze position calculation program 108 from first-person video data 110 and gaze position data 111 acquired from a gaze measurement device 101. In particular, in this embodiment, it is assumed that processing is performed while first-person video data 110 and gaze position data 111 are acquired from the gaze measurement device 101 in real time.

[0044] For this purpose, first, the spatial model creation program 106 constantly reads newly saved data from the data acquired from the gaze measurement device 101 and stored in the database of first-person video data 110, and performs processing to create a spatial model.

[0045] As a technology for creating a spatial model represented by point cloud data from video data, a combination of the well-known SLAM (Simultaneous Localization and Mapping) and MVS (Multi-View Stereo) methods can be used.

[0046] SLAM is a technology that creates coarse point cloud data by analyzing the correspondence between images using multiple consecutive images that correspond to the movement of the camera, while the MVS method is a technology that creates more detailed, dense point cloud data by using the analysis results of SLAM.

[0047] Fig. 8 shows an example of creating a spatial model from point cloud data using SLAM and MVS.

[0048] In Fig. 8, 801 indicates an object that exists within the range where the user moves his / her line of sight, and for simplicity, it is assumed that there are no other objects in the vicinity. 802 and 804 indicate the camera positions when the first-person video was captured, and 803 and 805 show the range of the camera's field of view and the camera's attitude when the first-person video was captured from each camera position.

[0049] As shown in 802 and 804, by using images of the same object or the same location taken from multiple locations, a spatial model represented by point cloud data as shown in 806 is created.

[0050] 8 shows only two camera positions, but generally more images are used to create a more accurate space model 806. By using the above-mentioned techniques, the space model 806 can be created in real time in response to the movement of the user's gaze position. In addition to the above-mentioned techniques, any technique may be used in the space model creation program 106 as long as it can create a space model similarly represented by point cloud data.

[0051] In addition, by using SLAM technology, the position and orientation of the camera that captured the first-person video on the created spatial model can also be calculated at the same time. As will be described later, the position and orientation of the camera is information required for calculating the gaze position on the object model. For this reason, when using a spatial model creation technology that cannot calculate the position and orientation of the camera, it is necessary to use a separate means for acquiring the position and orientation of the camera on the spatial model. For example, any technology that can acquire the position and orientation in space can be used, such as using a sensor that acquires the position and orientation.

[0052] The object position and orientation estimation program 107 is a program for matching the point cloud data of the object model to a portion of the point cloud data in the spatial model created by the spatial model creation program 106, i.e., for adjusting the position and orientation so that they match well, and for obtaining the position and orientation of the object model in the spatial model.

[0053] The object position and orientation estimation program 107 may use well-known techniques such as an ICP (Iterative Closest Point) algorithm or an NDT (Normal Distribution Transform) algorithm. Alternatively, any technique may be used as long as it can match point cloud data and determine the position and orientation of an object model in a spatial model. Furthermore, the object position and orientation estimation program 107 has its processing timing controlled by a gaze position calculation program 108, which will be described later.

[0054] The gaze position calculation program 108 is a program that uses gaze position data 111 acquired by the gaze measurement device 101, object model data 112, and space model data 113 created by the space model creation program 106 to find the gaze position on the object model.

[0055] The flow of processing of the gaze position calculation program 108 will be described with reference to the flowchart of FIG. 9, new gaze position data 111 is acquired from the gaze position data 111 stored in the database of the gaze position data 111. Alternatively, new gaze position data may be acquired directly from the gaze measurement device 101.

[0056] In step 902, the gaze direction on the spatial model is calculated using the newly acquired gaze position data 111 and the shooting positions and attitudes 305 and 308 calculated when creating the spatial model by the spatial model creation program 106 and recorded in the first-person video data 110.

[0057] The gaze direction on the spatial model is expressed by a vector indicating the starting point of gaze and the line of sight from that point. The gaze direction on the spatial model is calculated by first acquiring the shooting position and orientation for the first-person video data at the same time as the newly acquired gaze position data 111 from the first-person video data 110.

[0058] If there is no shooting position and orientation at the exact same time, for example, a method may be used in which the shooting position and orientation for the first-person video data 110 corresponding to the time before or after the time of the gaze position data 111 is acquired, and the shooting position and orientation obtained by performing interpolation based on the relationship with the time of the gaze position data 111 is used. Next, the acquired shooting position and orientation is used to perform coordinate conversion of the gaze position on the first-person video to the gaze position on the spatial model.

[0059] As described above, the gaze position data 111 is represented by the gaze position on the first-person video, but by applying information such as the camera's viewing angle and focal length to a well-known pinhole camera model, the positional relationship between the shooting position, the first-person video, and the gaze position can be obtained in the same size as the actual space. Furthermore, by performing coordinate conversion on the obtained positional relationship using the shooting position and orientation obtained from the first-person video data 110, the positional relationship between the camera's shooting position, the first-person video, and the gaze position on the spatial model can be expressed as shown in FIG.

[0060] In FIG. 10, 1001 indicates a spatial model, 1002 indicates the shooting position of a camera that shot a first-person video on the spatial model 1001, and 1003 indicates the shooting range of the first-person video on the spatial model 1001, which corresponds to the shooting posture of the camera.

[0061] Also, 1004 is a gaze position on the first-person image associated with the spatial model 1001. The gaze direction on the spatial model 1001 can be obtained as a vector 1005 that starts at 1002 and passes through the gaze position 1004 on the first-person image in FIG.

[0062] In step 903 in FIG. 9, it is determined whether or not an object model placed on the spatial model 1001 exists in the gaze direction on the spatial model 1001 obtained in step 902. If it is determined that an object model placed on the spatial model 1001 exists in the gaze direction on the spatial model 1001, the process proceeds to step 906; if it is determined that an object model does not exist, the process proceeds to step 904.

[0063] The method of determination is as follows: first, information on the object model placed in the target space model 1001 is obtained from the object placement data 114, and point cloud data of the corresponding object model is read from the object model data 112 and placed on the space model. Next, a vector representing the gaze direction on the space model 1001 (hereinafter, gaze direction vector), for example, the distance between vector 1005 in Fig. 10 and each point of the object model placed on the space model 1001 is obtained, and the distance closest to the gaze direction vector is selected.

[0064] If the selected distance is equal to or less than a predetermined threshold, it can be determined that the target object model is in the gaze direction. For example, the threshold for the distance can be selected as the maximum distance or half the maximum distance between points in the point cloud data constituting the object model. In addition, as a determination method, it may be determined that the target object model is in the gaze direction when the number of points on the spatial model whose distance from the gaze direction vector is equal to or less than a predetermined threshold is equal to or more than a predetermined threshold.

[0065] In addition, in addition to the point cloud data, polygon data, which is data in a format represented by a collection of polygons, may also be stored as the object model to be stored in the database of object model data 112, and the polygon data for the corresponding object model may also be arranged on spatial model 1001.If there is a point where each polygon arranged on spatial model 1001 intersects with a vector representing the line of sight arranged on the spatial model, it may be determined that the target object model is in the gaze direction.

[0066] Furthermore, if there are multiple objects placed on the spatial model 1001, the point closest to the above-mentioned gaze direction vector 1005 or the position on the gaze direction vector of the polygon that intersects with the gaze direction vector 1005 can be obtained, and the starting point of the gaze direction vector 1005, i.e., the point or object model including the polygon closest to the shooting position, can be selected as the object model existing in the gaze direction.

[0067] Furthermore, a virtual space that can express the same positional relationship as that of spatial model 1001 may be prepared, and each time an object is placed on spatial model 1001, an object model may be placed in the virtual space, and the above-mentioned determination process may be executed in the virtual space.

[0068] In step 904, the object model 1004 is matched to point cloud data on the space model 1001 that exists in the gaze direction on the space model 1001, using the object position and orientation estimation program 107. This process is a process for making it possible to replace a part of the space model 1001 with the target object model 1006 by adjusting the position and orientation of the object model 1006 in various ways with respect to the space model 1001 in Fig. 10.

[0069] As the point cloud data on the spatial model 1001 existing in the line of sight direction, for example, point cloud data on the spatial model whose distance from the gaze direction vector 1005 is equal to or less than a predetermined distance can be selected. As the predetermined distance, the maximum size of the object model to be matched or half the maximum size can be used.

[0070] Alternatively, the point cloud data may be present within a range of a specific shape of a predetermined size with the gaze direction vector 1005 as the central axis. Also, the position of each point in the point cloud data on the gaze direction vector may be obtained, and points within a predetermined range may be selected centered on the point where the point cloud data is most concentrated on the gaze direction vector. In addition to the above, any method may be used as long as it is possible to select point cloud data to be matched with the gaze direction vector as the center.

[0071] The object model to be matched to the point cloud data on the spatial model can be all object models stored in the database of object model data 112, or a well-known QR code (registered trademark), AR marker, symbol or character string representing the name of the object, etc. can be placed on the actual object and read using well-known image recognition technology or character recognition technology, and a corresponding object model can be selected from the object model data 112 for matching.

[0072] An example of an AR marker placed on an actual object is shown in 1101 in Fig. 11. Alternatively, well-known point cloud features (three-dimensional features) may be extracted from the point cloud data on the space model and the point cloud data of the object model, and an object model having point cloud features similar to the point cloud data on the space model may be selected from the object model data 112 for matching.

[0073] Furthermore, when performing assembly work, the shape and size of an object present at a specific location change. In such a case, as shown by 1201, 1202 and 1203 in FIG. 12, object models showing the state of the object for each work step and their sequential relationships are stored in object model data 112, and matching with the spatial model is performed in sequence, thereby making it possible to match the object model according to the work step with the spatial model.

[0074] Specifically, for example, if no object model is matched to the point cloud data on the space model present in the gaze direction, the object model corresponding to the first step is selected and matching is performed.

[0075] On the other hand, if an object model corresponding to one of the steps has already been matched to the point cloud data on the spatial model present in the gaze direction, the object model corresponding to the next step is selected and matched with the point cloud data on the spatial model. If the object model corresponding to the next step is matched to the spatial model with a higher degree of match than the degree of match between the already matched object model and the spatial model, the already matched object model is replaced with the object model corresponding to the next step.

[0076] The degree of coincidence may be the same as that used in the process in step 905 described later. Furthermore, matching of the object model to the space model corresponding to the next step may be performed when the gaze position moves onto an already matched object model or when a change in the space model is detected in the space model creation program 106.

[0077] To detect a change in the spatial model, the spatial model creation program 106 compares a spatial model generated using a newly acquired first-person video image and a predetermined number of first-person video images immediately preceding it with the spatial model generated previously, and if the difference between the two is greater than or equal to a predetermined threshold, it is determined that the spatial model has changed.

[0078] The difference between the spatial models can be calculated, for example, by searching for the closest point in the other spatial model for each point in one spatial model, calculating the distance to the searched point, and averaging the distances for all points in one spatial model. Alternatively, the number of points whose distance is equal to or greater than a predetermined threshold value may be used. Alternatively, any method may be used as long as it can calculate the difference between point groups.

[0079] In step 905, it is determined whether the object model has been correctly matched to the point cloud data on the spatial model that exists in the gaze direction on the spatial model. If the match is correct, the matching result of the object model is stored in the database of object arrangement data 114 and then the process proceeds to step 906; if not, the process proceeds to step 908.

[0080] A method for determining whether an object model has been correctly matched to point cloud data on a spatial model involves calculating the degree of match between the point cloud data on the spatial model and the object model placed on the spatial model based on the results of the matching process, and determining that the object model has been correctly matched if the degree of match is greater than a predetermined value.

[0081] If there are multiple object models that can be matched to the point cloud data on the spatial model, the highest degree of match is selected, and if the selected degree of match is greater than a predetermined value, it is determined that the object models have been matched correctly, and that the object model corresponding to the selected degree of match has been matched to the point cloud data on the spatial model.

[0082] As the degree of agreement, for example, a point in the point cloud data on the spatial model with the smallest distance to each point in the object model arranged on the spatial model using the matching result is searched for, the number of points whose distance is smaller than a predetermined threshold is calculated, and the ratio of the number of points calculated to the number of points in the object model can be used. As the degree of agreement between the point cloud data on the spatial model and the object model, in addition to the above, any index can be used as long as it can determine the quality of the matching result between the point cloud data.

[0083] In step 906, the gaze position of the object model arranged on the space model is calculated using the matching result in step 904. Since the object model arranged on the space model and the gaze direction vector on the space model are 1005 in Fig. 10, it is possible to use the same process as that in step 903 described above, that is, the process of determining whether or not an object model arranged on the space model exists in the gaze direction on the space model.

[0084] However, in step 903, when the object model is point cloud data, attention was paid to whether there was a point whose distance from the gaze direction vector 1005 was less than a predetermined threshold value. However, in calculating the gaze position, a point whose distance from the gaze direction vector is less than a predetermined threshold value and which is closest to the shooting position 1002 is selected, and this point is set as the gaze position.

[0085] Alternatively, if the distance from the gaze direction vector 1005 is below a predetermined threshold and there are multiple points within a predetermined range from the point closest to the shooting position 1002, the gaze position may be the average of the point closest to the shooting position 1002 and the points within a predetermined range from that point.

[0086] Alternatively, if the object model is represented as a collection of polygons, the intersections between the gaze direction vector and each polygon in the object model are found, and the intersection closest to the shooting position 1002 is determined as the gaze position.

[0087] Since the gaze position found as described above is the gaze position on the spatial model, in step 906, the found gaze position is subjected to coordinate conversion using the position and orientation of the object model on the spatial model obtained by the matching result in step 904. As a result, the gaze position 1004 on the spatial model 1001 is converted to a gaze position 1007 on the object model 1006, i.e., in the coordinate system of the object model.

[0088] Furthermore, when an object model is placed on the space model in step 905, the gaze position on the placed object model may be calculated for past gaze position data in step 906. This is because, in the initial stage of processing of the gaze position calculation program 108, there may be a small amount of point cloud data for each object on the space model, and it may be impossible to place the object model on the space model.

[0089] In step 907, the gaze position on the object model obtained in step 906 is stored in the gaze position data 111. When storing the gaze position of the object model, the shooting position and orientation of the first-person video on the space model, the gaze position in the space model, the position and orientation of the object model on the space model, etc. may also be stored together.

[0090] In step 908, if an instruction to end the process is received from the input device 103 or the like, the process ends;

[0091] In the above-mentioned gaze position calculation program 108, first-person video data 110 and gaze position data 111 are acquired in real time, and the object model data 112 is placed on the spatial model while generating spatial model data 113, and the gaze position on the object model is calculated. However, after acquisition of the target first-person video data 110 and gaze position data 111 has been completed, the processing of the gaze position calculation program 108 may be executed while sequentially reading the data stored in the first-person video data 110 and gaze position data 111.

[0092] In this case, the gaze position on the object model may be calculated after creating a space model, or after creating a space model and arranging an object model on the space model. Also, in this case, the object model may be manually arranged in a location on the space model where no object model is arranged, and then the above-mentioned processing may be performed.

[0093] Alternatively, after executing the processing of the gaze position calculation program 108, the gaze position calculation program 108 may be executed again using the space model data 113 and the object arrangement data 114 generated after executing the gaze position calculation program 108. This is assumed to be the case in which, in the initial stage of the processing of the gaze position calculation program 108, there is little point cloud data for each object on the space model, and it is not possible to arrange the object model on the space model.

[0094] In such a case, there may be a gaze position on the object model that is not calculated among the gaze positions stored in the gaze position data 111. In this case, the gaze position calculation program 108 is executed after making the following changes.

[0095] The first change is that the gaze position data is acquired in step 901 by reading the data in the order in which it was saved from the gaze position data 111. Furthermore, a process is added to determine whether the gaze position on the object model has been calculated for the acquired gaze position data, and to proceed to step 908 if it has been calculated.

[0096] The second change is that in step 903, if an object model is not positioned in the gaze direction on the spatial model, the process proceeds to step 908 instead of step 904.

[0097] The third change is to delete step 904 and step 905. By making the above three changes to the gaze position calculation program 108, it is possible to calculate the gaze position on the object model for a gaze position where the gaze position on the object model has not been calculated. In this case, the above-mentioned processing may be performed after manually placing an object model in a location where no object model is placed on the space model.

[0098] The gaze position display program 109 is a program that displays gaze position data in response to an instruction from a user. The gaze position is displayed by a method well known in gaze analysis, such as a heat map method that is a frequency distribution of gaze positions, or a method that visually displays the length of time that the gaze position remained within a certain range by the size of a circle, etc.

[0099] In addition to displaying the gaze position on the first-person image, the gaze position on the object model is also displayed. As a method for displaying the gaze position on the object model, a method can be used in which a virtual space model, which is a virtual space corresponding to the space model, is prepared, an object model is arranged in the virtual space model based on data stored in the object arrangement data 114, and the gaze position is displayed on the object model arranged in the virtual space model. Alternatively, a method can be used in which only a specific object model is selected, and the gaze position is displayed only on the selected object model.

[0100] Furthermore, when the gaze position is displayed on the object model, the virtual space model or the object model may be adjusted and displayed so that the gaze position always faces the user directly.

[0101] In Fig. 13, 1301 indicates an example of an object model, and 1302, 1303, 1304, and 1305 indicate examples of gaze positions on the object model. Fig. 14 shows an example in which the gaze positions shown in Fig. 13 are always displayed facing the user.

[0102] 14, 1401 is an object model in which the gaze point 1302 is displayed facing the user, and 1402 is the gaze point displayed on 1401. 1403 is an object model in which the gaze point 1303 is displayed facing the user, and 1404 is the gaze point displayed on 1403. 1405 is an object model in which the gaze point 1304 is displayed facing the user, and 1406 is the gaze point displayed on 1405. 1407 is an object model in which the gaze point 1305 is displayed facing the user, and 1408 is the gaze point displayed on 1407.

[0103] In the display method shown in Fig. 14, the gaze positions are displayed by adjusting the object model so that each gaze position faces the user completely. To achieve this, for example, if the object model is point cloud data, a point in a predetermined range including the gaze position is selected, the direction of a normal line of a plane including the gaze position is obtained from the selected point, and the position and orientation of the object model are adjusted so that the obtained normal line faces the user side. If the object model is expressed as a collection of polygons, a polygon including the gaze position is selected from the object model, and the position and orientation of the object model are adjusted so that the normal line faces the user side.

[0104] In addition, when displaying the gaze position, a restriction may be placed on the method of adjusting the position and orientation of the displayed object model. Furthermore, by adjusting the degree to which the gaze position faces the user, a large change in the orientation of the object model when displaying the gaze position may be suppressed.

[0105] For example, the first gaze position is displayed so as to face the user directly, and the position and orientation of the object model are adjusted for subsequent gaze positions so that the degree to which the gaze position faces the user does not exceed a predetermined range. The degree to which the gaze position faces the user can be determined by the angle or inner product between the normal direction of the plane including the gaze position and the direction facing the user, i.e., the direction perpendicular to the screen.

[0106] FIG. 15 shows an example of a display in which the gaze position is displayed using the above-mentioned method. 15 shows a case where an object model is displayed with its vertical coordinate axis tilted toward the user, and only rotation around the vertical coordinate axis of the object model is permitted as a method of adjusting the position and orientation when displaying the gaze position. The position and orientation of an object model 1501 is adjusted so that an initial gaze position 1502 faces the user directly.

[0107] However, in FIG. 15, the coordinate axis in the vertical direction of the object model is inclined, and the gaze position cannot be completely directly faced to the user, so the position and orientation are adjusted so that it faces as directly as possible. Specifically, the normal of the plane including the gaze position is obtained as described above, and the position and orientation of the object model are adjusted so that the difference between the obtained normal and the direction facing the user, i.e., the direction perpendicular to the screen, becomes the smallest. In the object model 1503, the gaze position 1504 is displayed in a state in which the degree to which it faces the user is lower than that of the gaze position 1502, that is, the gaze position can be confirmed, but it is not directly faced to the user. The gaze position 1506 on the object model 1505 and the gaze position 1508 on the object model 1507 are also displayed in the same manner as the gaze position 1504.

[0108] As described above, the gaze position analysis system in an embodiment of the present invention generates a spatial model, which is a three-dimensional model of the space in the range in which the user's gaze is directed from a first-person video, which is an image similar to the user's field of vision, and calculates the shooting position and orientation of the first-person image in the spatial model, calculates the gaze direction on the spatial model using the shooting position and orientation of the first-person video on the spatial model and the gaze position on the first-person video, places the object model in the spatial model based on the position and orientation of the object on the spatial model estimated by matching the object model, which is a three-dimensional model of the object, with the spatial model, and calculates the gaze position on the object model by finding the intersection between the gaze direction on the spatial model and the object model.

[0109] According to an embodiment of the present invention, by using a gaze measurement device worn by a user, it is possible to automatically match the user's gaze position onto three-dimensional models of various objects existing in the space, based on the gaze information of the user measured while moving freely within the space being measured. [Explanation of symbols]

[0110] 101 Eye Tracking Device 102 Information processing equipment 103 Input Device 104 Output Device 105 Storage device 106 Spatial model creation program 107 Object position and orientation estimation program 108 Gaze position calculation program 109 Gaze position display program 110 First-person video data 111 Gaze position data 112 Object model data 113 Spatial Model Data 114 Object placement data

Claims

1. A gaze position analysis system having a space model creation unit, an object position and orientation estimation unit, and a gaze position calculation unit, which automatically associates an object model gaze position, which is a gaze position of a user on an object model, with an object model, which is a three-dimensional model of an object existing in a space, The spatial model creation unit includes: Acquire a first-person image that is an image similar to the user's visual field; A first-person image gaze position, which is a gaze position of the user on the first-person image, is acquired; creating a spatial model, which is a three-dimensional model of the space in the range in which the user is looking, from the first-person image; Calculating a shooting position and orientation of the first-person video in the spatial model; The object position and orientation estimation unit estimating a position and orientation of the object in the spatial model by matching the object model with the spatial model; placing the object model in the spatial model using a position and orientation of the object in the spatial model; Calculating a gaze direction in the spatial model using the shooting position and orientation of the first-person image and the first-person image gaze position in the spatial model; The gaze position calculation unit A gaze position analysis system, comprising: a gaze position analysis unit that calculates a gaze position of the object model by determining an intersection between the gaze direction and the object model in the spatial model.

2. The object position and orientation estimation unit The gaze position analysis system of claim 1, characterized in that if there is data in the spatial model to which the object model is not matched in the gaze direction, the position and orientation of the object in the spatial model is estimated by matching the object model with the data in the spatial model to which the object model is not matched.

3. The object position and orientation estimation unit selecting data in the spatial model that exists within a predetermined range based on the gaze direction from the data in the spatial model; 2. The gaze position analysis system according to claim 1, wherein the position and orientation of the object in the selected spatial model are estimated by matching the object model with data in the selected spatial model.

4. The object position and orientation estimation unit Reading a preset image pattern, character string, or symbol string from the first-person video; selecting the object model corresponding to the image pattern, the character string, or the symbol string; calculating a position of the image pattern, the character string or the symbol string in the spatial model from the spatial model, a shooting position and orientation of the first-person video in the spatial model, and a position of the image pattern, the character string or the symbol string in the first-person video; selecting data in the spatial model that exists within the predetermined range based on a position in the spatial model of the image pattern, the character string, or the symbol string; 4. The gaze position analysis system according to claim 3, wherein the position and orientation of the object in the selected spatial model are estimated by matching the selected object model with data in the selected spatial model.

5. The object position and orientation estimation unit Calculating a difference between data representing a spatial model created from the first-person image at a certain time and data of a range of the spatial model that corresponds to the first-person image at the same time that has already been created; If the calculated difference is greater than a predetermined threshold, replacing the data in the corresponding range on the spatial model with data representing the spatial model created from the first-person video; performing matching of the object model with the data of the replaced portion; 2. The gaze position analysis system according to claim 1, wherein the object model that was matched to the data before the replacement is replaced with the newly matched object model.

6. further comprising an object model storage unit configured to store, for the object whose shape or structure changes, the object model corresponding to each process of the change and an order relationship of the change, The object position and orientation estimation unit The gaze position analysis system according to claim 1, characterized in that, for a location where the object model corresponding to the changing object is matched, each time a change in the spatial model is detected, the object model corresponding to the process of the change of the object is selected from the object model storage unit, and the selected object model is matched with the spatial model.

7. The gaze position calculation unit When any of the object models is newly matched to any location on the spatial model, The gaze position analysis system according to claim 1 , further comprising: determining whether or not the object model gaze position is on the object model newly matched on the spatial model with respect to the previous object model gaze position.

8. The object position and orientation estimation unit The user manually matches the object model to a portion of the spatial model to which the object model has not been matched; The gaze position calculation unit The gaze position analysis system according to claim 1, characterized in that for all of the object model gaze positions already acquired, it is determined whether the object model gaze position is on the object model that the user manually matched on the spatial model.

9. The gaze position calculation unit The gaze position analysis system according to claim 1, characterized in that at least one of the name of the object model, the shooting position and orientation of the first-person image in the spatial model, the first-person image gaze position in the spatial model, and the position and orientation of the object model in the spatial model is stored in correspondence with the object model gaze position on the object model.

10. a gaze position display unit that displays a gaze position of the object model on the object model, The gaze position display unit is The gaze position analysis system according to claim 1, characterized in that, when displaying the gaze position of the object model on the object model, the position and orientation of the object model are adjusted so that the gaze position of the object model at each time is always displayed directly in front.

11. a gaze measurement device that captures the first-person image of the user and measures a gaze position of the first-person image while the user is able to move freely within the space; The spatial model creation unit includes: The gaze position analysis system according to claim 1 , wherein the first-person video and the first-person video gaze position are acquired from the gaze measurement device.

12. The gaze measurement device includes: A photographing device is provided. The spatial model creation unit includes: The gaze position analysis system according to claim 11, wherein the surrounding spatial model is created from a plurality of images captured by the imaging device.

13. A gaze position analysis system that automatically associates a gaze position of a user on an object model, which is a three-dimensional model of an object existing in a space, a spatial model creation unit that creates a spatial model, which is a three-dimensional model of the surrounding space, from a plurality of captured images; an object position and orientation estimation unit that matches the spatial model with the object model and places the object model on the spatial model based on a position and orientation obtained by the matching; a gaze position calculation unit that calculates the gaze position on the object model based on the arranged object model and a gaze direction in the space model; A gaze position analysis system comprising:

14. The gaze position calculation unit The gaze position analysis system according to claim 13, wherein the gaze position on the object model is calculated by determining an intersection between the gaze direction in the spatial model and the object model.

15. A gaze position analysis method for automatically associating an object model gaze position, which is a gaze position of a user on an object model, with an object model that is a three-dimensional model of an object existing in a space, comprising: acquiring a first-person view similar to the user's field of view; acquiring a first-person image gaze position, which is a gaze position of the user on the first-person image; creating a spatial model, which is a three-dimensional model of a space in a range in which the user is looking, from the first-person image; calculating a shooting position and orientation of the first-person video in the spatial model; estimating a position and orientation of the object in the spatial model by matching the object model with the spatial model; placing the object model in the spatial model using a position and orientation of the object in the spatial model; calculating a gaze direction in the spatial model using the shooting position and orientation of the first-person image and the first-person image gaze position in the spatial model; calculating an intersection of the gaze direction and the object model in the spatial model to calculate the object model gaze position; A gaze position analysis method comprising the steps of:

Images