Line-of-sight prediction device, line-of-sight prediction method, and

Abstract

To accurately predict a line of sight of a user in a real space.SOLUTION: A line-of-sight prediction device 10 includes a first acquisition unit 101 that acquires first line-of-sight information including a three dimensional direction of a line of sight obtained by measuring the line of sight of a user in a real space, position information of the user, and attribute information of the user, a second acquisition unit 102 that expresses the first line-of-sight information as a line segment, a straight line, or a region having a predetermined width on a three dimensional model representing the real space and acquires second line-of-sight information including a distance until the expressed line segment, straight line, or region having a predetermined width intersects with a line-of-sight object in the three dimensional model and attribute information of the line-of-sight object, and a generation unit 103 that receives the three dimensional model including the attribute information of the line-of-sight object, the position information of the user, and the attribute information of the user and generates a line-of-sight prediction model 141 that predicts the line of sight of the user.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a gaze prediction device, a gaze prediction method, and a gaze prediction program. [Background technology]

[0002] Conventionally, there is known a technique for predicting where a user's gaze is directed in a two-dimensional image (see, for example, Patent Document 1). There is also known a technique for representing a user's field of view as an image based on their posture in a three-dimensional space (see, for example, Patent Document 2). There is also known a technique for evaluating advertisements for information presentations by analyzing and accumulating not only gaze information but also behavioral pattern information (see, for example, Patent Document 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4396430 [Patent Document 2] Japanese Patent Publication No. 2023-053631 [Patent Document 3] Japanese Patent Application Laid-Open No. 2003-216938 Summary of the Invention [Problem to be solved by the invention]

[0004] However, to accurately predict a user's gaze in real space, attribute information of the user and the object should also be taken into consideration, but the two-dimensional image described in Patent Document 1 does not take this attribute information into consideration and is represented only by color, which is insufficient for gaze prediction in real space. Furthermore, the technologies described in Patent Documents 2 and 3 detect the user's gaze, but do not predict the user's gaze.

[0005] An object of the present disclosure is to provide a gaze prediction device, a gaze prediction method, and a gaze prediction program that are capable of accurately predicting a user's gaze in real space. [Means for solving the problem]

[0006] In order to achieve the above object, the gaze prediction device according to the present disclosure includes a first acquisition unit that acquires first gaze information obtained by measuring a user's gaze in real space, the first gaze information including a three-dimensional direction of the gaze, user position information, and user attribute information; a second acquisition unit that represents the first gaze information as a line segment, a straight line, or an area of ​​a predetermined width on a three-dimensional model representing the real space, and acquires second gaze information including a distance until the represented line segment, straight line, or area of ​​a predetermined width intersects with a gaze object in the three-dimensional model, and attribute information of the gaze object; and a generation unit that generates the gaze prediction model for predicting the user's gaze by machine learning the acquired user position information, user attribute information, the distance, and attribute information of the gaze object using as training data the three-dimensional model including the attribute information of the gaze object, the user position information, and the user attribute information as input. [Effects of the Invention]

[0007] According to the present disclosure, an effect is obtained in which the user's line of sight in real space can be predicted with high accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the line-of-sight prediction device according to the embodiment. [Figure 2] 1 is a block diagram illustrating an example of a functional configuration of a line-of-sight prediction device according to an embodiment. [Figure 3] 10A and 10B are diagrams illustrating a method for reproducing a line of sight on a three-dimensional model. [Figure 4] FIG. 10 is a diagram illustrating a gaze prediction model. [Figure 5]FIG. 10 is a diagram showing an example of an image of a line of sight that can be physically assumed from user position information. [Figure 6A] FIG. 1 is a diagram showing a three-dimensional model as seen from a first-person perspective. [Figure 6B] FIG. 10 is a diagram showing a heat map displayed on a three-dimensional model as viewed from a first-person perspective. [Figure 6C] FIG. 10 is a diagram showing a heat map displayed on a three-dimensional model when viewed from a bird's-eye view. [Figure 7] 10 is a flowchart illustrating an example of the flow of a model generation process by the gaze prediction program according to the embodiment. [Figure 8] 10 is a flowchart showing an example of the flow of a gaze prediction process by the gaze prediction program according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of the technology of the present disclosure will be described in detail below with reference to the drawings. Note that components and processes that perform the same operations, actions, and functions are given the same reference numerals throughout the drawings, and duplicated descriptions may be omitted as appropriate. Each drawing is merely a schematic illustration to allow a sufficient understanding of the technology of the present disclosure. Therefore, the technology of the present disclosure is not limited to the illustrated examples. Furthermore, in this embodiment, descriptions of configurations that are not directly related to the technology of the present disclosure or well-known configurations may be omitted.

[0010] 1 is a block diagram showing an example of a hardware configuration of a gaze prediction device 10 according to this embodiment. The gaze prediction device 10 according to this embodiment is implemented by a general-purpose computer such as a personal computer (PC) or a server computer.

[0011] 1, the line-of-sight prediction device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a monitor 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 18 so as to be able to communicate with each other.

[0012] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a gaze prediction program.

[0013] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0014] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various information to the device itself.

[0015] The monitor 16 is, for example, a liquid crystal display, and displays various information. The monitor 16 may be a touch panel type monitor, and may function as the input unit 15.

[0016] The communication interface 17 is an interface for the device itself to communicate with other external devices, and uses standards such as Ethernet (registered trademark), FDDI (Fiber Distributed Data Interface), and Wi-Fi (registered trademark).

[0017] Next, the functional configuration of the line-of-sight prediction device 10 will be described with reference to FIG.

[0018] FIG. 2 is a block diagram showing an example of the functional configuration of the line-of-sight prediction device 10 according to this embodiment.

[0019] 2, the gaze prediction device 10 includes, as functional components, a first acquisition unit 101, a second acquisition unit 102, a generation unit 103, and an output unit 104. Each functional component is realized by the CPU 11 reading out a gaze prediction program stored in the ROM 12 or the storage 14, expanding the program in the RAM 13, and executing the program.

[0020] The first acquisition unit 101 acquires first gaze information. The first gaze information includes the three-dimensional direction of the user's gaze in real space, user position information, and user attribute information. The three-dimensional direction of the user's gaze in real space can be obtained by, for example, using an eye-tracking device to measure the direction in which the user is looking. The user position information can be obtained by, for example, continuously measuring the user's position in real space using a camera, a sensor, or the like. The user's position is expressed, for example, as coordinates representing the real space. The user attribute information includes, for example, at least one of the user's height, age, and gender.

[0021] The second acquisition unit 102 acquires second gaze information. The second gaze information is acquired by expressing the first gaze information as a line segment on a three-dimensional model representing real space. The second gaze information includes the distance from the line segment expressed on the three-dimensional model to the intersection with the gaze target in the three-dimensional model, and attribute information of the gaze target. Note that the first gaze information is not limited to a line segment, and may be a straight line or an area of ​​a predetermined width. The three-dimensional model can be created based on an actual plan using, for example, CAD (Computer Aided Design) or the like. The attribute information of the gaze target includes at least one of the material, color, component name, size, and brightness of the gaze target.

[0022] The generation unit 103 generates a gaze prediction model 141 using the acquired user position information, user attribute information, distance to the gaze target, and attribute information of the gaze target as training data. The gaze prediction model 141 is a trained model generated by machine learning using the acquired user position information, user attribute information, distance to the gaze target, and attribute information of the gaze target as training data. For example, a neural network (NN) is applied to the gaze prediction model 141. The gaze prediction model 141 is a model that predicts a user's gaze using a three-dimensional model, user position information, and user attribute information as input. This three-dimensional model includes attribute information of the gaze target. In other words, the gaze prediction model 141 classifies a large number of expected gaze information items into looking or not looking using the three-dimensional model, user position information, and user attribute information as input. Note that the prediction accuracy of the gaze prediction model 141 improves as the training data is expanded.

[0023] The output unit 104 inputs the three-dimensional model, the user's position information, and the user's attribute information into the gaze prediction model 141, and outputs information representing the user's gaze output by the gaze prediction model 141. The output destination may be, for example, the monitor 16 or the storage 14.

[0024] FIG. 3 is a diagram illustrating a method for reproducing a gaze on a three-dimensional model 110. As shown in FIG. 3, a real space is represented as a three-dimensional model 110 using, for example, CAD. The gaze of a user 111 measured using an eye-tracking device is represented on the three-dimensional model 110 as, for example, a line segment 113. The point where the represented line segment 113 intersects with a gaze target 112 in the three-dimensional model 110 is assumed to be the point where the gaze is directed, and the distance L to the intersection and attribute information of the intersecting gaze target 112 are acquired. In this case, the distance L to the gaze target 112 is important for understanding the positional relationship between the user 111 and the gaze target 112. For example, even if there is no obstacle and the user 111's gaze can reach an object, the user 111 may not direct his or her gaze at it due to a certain distance. Therefore, by using the distance L to the gaze target 112 as training data, the accuracy of gaze prediction can be improved.

[0025] 4 is a diagram illustrating the gaze prediction model 141. The gaze prediction model 141 is generated by machine learning using acquired user position information, user attribute information, distance to the gaze target, and attribute information of the gaze target as training data. An example of the training data is shown in Table 1. The training data shown in Table 1 includes, for example, the age of the user 111, position information (X coordinate, Y coordinate), component name, color, brightness of the gaze target 112, and distance L to the gaze target 112.

[0026] [Table 1]

[0027] The gaze prediction model 141 receives input of a three-dimensional model 110 including attribute information of a gaze target 112, position information of the user 111, and attribute information of the user 111. Then, as an example, as shown in Table 2 and FIG. 5, gazes that can be assumed in all directions are automatically generated from the input position information of the user 111. FIG. 5 is a diagram showing an example of an image of a gaze that can be physically assumed from the position information of the user 111. According to Table 2, the component name, color, brightness, and distance L to the gaze target 112 of the gaze target 112 are associated with each assumed gaze.

[0028] [Table 2]

[0029] 6A to 6C are diagrams showing examples of images of output results of gaze prediction model 141. Fig. 6A is a diagram showing 3D model 110 as viewed from a first-person perspective. Fig. 6B is a diagram showing a state in which a heat map is displayed on 3D model 110 as viewed from a first-person perspective. Fig. 6C is a diagram showing a state in which a heat map is displayed on 3D model 110 as viewed from a bird's-eye perspective.

[0030] Moreover, an example of input data input to gaze prediction model 141 is shown in Table 3, and an example of output data output from gaze prediction model 141 is shown in Table 4. According to Table 3, the age and location information (X coordinate, Y coordinate) of user A are input.

[0031] [Table 3]

[0032] [Table 4]

[0033] When the gaze prediction results are output as a table, for each input data shown in Table 3, the attribute information of the gaze target 112 and the probability that the gaze target 112 will be seen are output according to the expected gaze, as shown in Table 4.

[0034] Next, the operation of the line-of-sight prediction device 10 according to this embodiment will be described with reference to FIGS.

[0035] FIG. 7 is a flowchart showing an example of the flow of a model generation process by the gaze prediction program according to this embodiment.

[0036] The CPU 11 reads out the line-of-sight prediction program from the ROM 12 or the storage 14 and executes it, thereby executing each step shown in FIG.

[0037] First, in step S101, the CPU 11 acquires first line-of-sight information. This first line-of-sight information includes, for example, the three-dimensional direction of the user's line of sight in real space, user position information, and user attribute information.

[0038] In step S102, the CPU 11 acquires a three-dimensional model 110 representing the real space, as shown in the above-mentioned Fig. 3, for example. This three-dimensional model 110 is created based on an actual plan using, for example, CAD or the like.

[0039] In step S103, the CPU 11 expresses the first line of sight information acquired in step S101 as a line segment 113 on the three-dimensional model 110, as shown in FIG. 3 above, for example.

[0040] In step S104, CPU 11 acquires second line of sight information. As an example, as shown in Fig. 3 above, this second line of sight information includes a distance L from a line segment 113 expressed on three-dimensional model 110 to an intersection with line of sight object 112 in three-dimensional model 110, and attribute information of line of sight object 112. The attribute information of line of sight object 112 includes, for example, at least one of the material, color, component name, size, and brightness of line of sight object 112.

[0041] In step S105, CPU 11 generates gaze prediction model 141 by machine learning using the user's position information, user attribute information, distance L to the gaze target, and attribute information of the gaze target acquired in steps S101 and S104 as training data (see Table 1), stores the generated gaze prediction model 141 in storage 14, for example, and terminates the model generation process by this gaze prediction program.

[0042] FIG. 8 is a flowchart showing an example of the flow of the gaze prediction process by the gaze prediction program according to this embodiment.

[0043] The CPU 11 reads out the line-of-sight prediction program from the ROM 12 or the storage 14 and executes it, thereby executing each step shown in FIG.

[0044] First, in step S111, the CPU 11 acquires a three-dimensional model 110 representing real space, as shown in Fig. 3 above, for example. This three-dimensional model 110 includes attribute information of a line-of-sight object 112. Note that the three-dimensional model 110 and line-of-sight object 112 acquired here are the same as the three-dimensional model and line-of-sight object used in the training data, for example.

[0045] In step S112, the CPU 11 acquires the location information and attribute information of the user.

[0046] In step S113, the CPU 11 inputs the three-dimensional model 110 acquired in step S111 and the user's position information and attribute information acquired in step S112 into the gaze prediction model 141, as shown in FIG. 4 above, for example.

[0047] In step S114, the CPU 11 outputs the gaze prediction result from the gaze prediction model 141 to, for example, the monitor 16, and ends the gaze prediction process by the gaze prediction program. Here, the gaze prediction result is output as a probability that the gaze target 112 will be seen according to the expected gaze, for example, as shown in Table 4 above.

[0048] As described above, according to this embodiment, it is possible to represent a real space as a 3D model and learn attributes of gaze targets in the real space and distances to the gaze targets as training data. This makes it possible to improve the accuracy of gaze prediction compared to when gaze prediction is performed using 2D images.

[0049] In addition, since the system utilizes existing 3D models and takes into account the user's position and attributes in 3D to predict gaze, it can be used in conjunction with already established people flow simulation technology to easily predict gaze in real space.

[0050] Furthermore, the line-of-sight prediction technology according to this embodiment can be applied to the planning, design, and evaluation of architectural spaces. The following are assumed use cases (1) to (3).

[0051] (1) It is possible to accurately predict gaze directions when considering the layout of signboards in various spaces, product shelves in commercial facilities, exhibits in cultural facilities, and fixture placement in event spaces. This makes it possible to communicate the characteristics of spatial design to customers without relying on the organizer's intuition. (2) It is possible to predict the shift in line of sight in urban spaces, thereby obtaining information that can be used to attract customers to development plans currently in the planning stages. (3) It is possible to predict pedestrians' gaze at signs, traffic lights, etc. This can lead to the development of infrastructure for pedestrian safety management.

[0052] The present disclosure is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present disclosure.

[0053] The gaze prediction device according to the embodiment has been described above by way of example. The embodiment may be in the form of a program for causing a computer to execute the functions of each unit of the gaze prediction device, or in the form of a program product including the program. The embodiment may be in the form of a non-transitory storage medium that stores the program and is readable by a computer.

[0054] Furthermore, the configuration of the line-of-sight prediction device described in the above embodiment is merely an example, and may be changed depending on the situation without departing from the spirit of the invention.

[0055] Furthermore, the processing flow of the program described in the above embodiment is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the main idea.

[0056] In the above embodiment, the processing according to the embodiment is realized by a software configuration using a computer by executing a program, but the present invention is not limited to this. The embodiment may be realized by, for example, a hardware configuration or a combination of a hardware configuration and a software configuration. [Explanation of symbols]

[0057] 10. Gaze prediction device 11 CPU 12 ROM 13 RAM 14. Storage 15 Input section 16 monitors 17 Communication I / F 18 Bus 101 First acquisition part 102 Second acquisition part 103 Generation part 104 Output section 141 Gaze Prediction Model

Claims

1. a first acquisition unit that acquires first gaze information including a three-dimensional direction of the gaze obtained by measuring the gaze of a user in real space, position information of the user, and attribute information of the user; a second acquisition unit that expresses the first gaze information as a line segment, a straight line, or an area of ​​a predetermined width on a three-dimensional model that represents the real space, and acquires second gaze information including a distance to an intersection of the expressed line segment, straight line, or area of ​​a predetermined width with a gaze target in the three-dimensional model, and attribute information of the gaze target; a generation unit that generates a gaze prediction model by machine learning using the acquired user position information, user attribute information, the distance, and attribute information of the gaze object as training data, the generation unit receiving a three-dimensional model including the attribute information of the gaze object, the user position information, and the user attribute information as inputs, and generates the gaze prediction model that predicts the user's gaze; A gaze prediction device comprising:

2. An output unit that outputs information representing the user's gaze output by the gaze prediction model, The gaze prediction device according to claim 1 .

3. the user attribute information includes at least one of the user's height, age, and sex; The attribute information of the gaze object includes at least one of the material, color, component name, size, and brightness of the gaze object.

3. A gaze prediction device according to claim 1 or claim 2.

4. acquiring first gaze information including a three-dimensional direction of the gaze obtained by measuring the gaze of the user in real space, position information of the user, and attribute information of the user; expressing the first gaze information as a line segment, a straight line, or an area of ​​a predetermined width on a three-dimensional model representing the real space, and acquiring second gaze information including a distance until the expressed line segment, a straight line, or an area of ​​a predetermined width intersects with a gaze target in the three-dimensional model, and attribute information of the gaze target; A gaze prediction model is generated by machine learning using acquired position information of the user, attribute information of the user, the distance, and attribute information of the gaze object as training data, and the gaze prediction model is generated by using a three-dimensional model including attribute information of the gaze object, position information of the user, and attribute information of the user as input, and predicting the gaze of the user. A computer-implemented method for gaze prediction.

5. acquiring first gaze information including a three-dimensional direction of the gaze obtained by measuring the gaze of the user in real space, position information of the user, and attribute information of the user; expressing the first gaze information as a line segment, a straight line, or an area of ​​a predetermined width on a three-dimensional model representing the real space, and acquiring second gaze information including a distance until the expressed line segment, a straight line, or an area of ​​a predetermined width intersects with a gaze target in the three-dimensional model, and attribute information of the gaze target; A gaze prediction model is generated by machine learning using acquired position information of the user, attribute information of the user, the distance, and attribute information of the gaze object as training data, and the gaze prediction model is generated by using a three-dimensional model including attribute information of the gaze object, position information of the user, and attribute information of the user as input, and predicting the gaze of the user. A gaze prediction program to be executed by a computer.

Images