Learning model generation program, information processing device, and learning model generation method

The method enhances gaze estimation model generalization by using teacher data with rotation matrices from multiple cameras, addressing the challenge of head orientation variability and reducing the workload for model generation, resulting in accurate and efficient gaze direction estimation.

WO2024232398A1PCT designated stage expired Publication Date: 2024-11-14THE UNIV OF TOKYO
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017162
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-05-08
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing gaze estimation techniques face challenges in accurately estimating the gaze direction when the head orientation in image data is not learned during training, leading to poor generalization performance, and require significant workload to generate teacher data for multiple camera combinations.

Method used

A method that generates a learning model using teacher data including image data from multiple cameras, a rotation matrix representing the relative rotation angle between virtual cameras, and line-of-sight information, allowing the model to estimate gaze direction accurately regardless of head orientation and reducing the need for extensive teacher data preparation.

Benefits of technology

Improves the generalization performance of the learning model for gaze estimation while minimizing the workload required for model generation, enabling accurate gaze direction estimation across various head orientations and camera configurations without needing teacher data for each camera combination.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017162_14112024_PF_FP_ABST
    Figure JP2024017162_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided are a learning model generation program, an information processing device, and a learning model generation method making it possible to improve the generalization performance of a learning model used for gaze estimation. A first rotation matrix corresponding to a relative angle of rotation between a first virtual camera obtained by subjecting a first camera to normalization processing and a second virtual camera obtained by subjecting a second camera to normalization processing is identified from first subject image data obtained by imaging a person's head with the first camera and first reference image data obtained by imaging the person's head with the second camera. Training data including the first subject image data, the first reference image data, the first rotation matrix, and first gaze information indicating the person's gaze direction is generated, and the generated training data is learned to generate a learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Learning model generation program, information processing device, and learning model generation method

[0001] The present invention relates to a learning model generation program, an information processing device, and a learning model generation method.

[0002] For example, there is a technology (hereinafter simply referred to as a gaze estimation technology) that estimates the gaze direction of a target person (person) by using image data that shows the target person's head. In such a technology, for example, image data that shows the target person's eyes is input into a trained learning model (hereinafter simply referred to as a learning model) to estimate the gaze direction of the target person (see Patent Document 1, Patent Document 2, and Non-Patent Document 1).

[0003] JP 2022-187546 A JP 2022-187547 A

[0004] X. Zhang, Y. Sugano, and A. Bulling, “Revisiting data normalization for appearance-based gaze estimation,” in ETRA, 2018.

[0005] Here, the above-described learning model may not be able to accurately estimate the gaze direction of the subject depending on the orientation of the subject's head as seen in the image data. Specifically, for example, if the orientation of the subject's head as seen in the image data is an orientation that has not been learned during learning, the learning model may not be able to accurately estimate the subject's gaze direction. In other words, the above-described gaze estimation technology may not be able to generate a learning model with high generalization performance, for example.

[0006] In this regard, for example, by generating a learning model using multiple image data captured by multiple cameras (image capture devices) in various positions, it is possible to improve the generalization performance of the learning model.

[0007] However, in this case, for example, it becomes necessary to generate training data for each combination of cameras, which increases the workload required to generate a learning model. Therefore, in the gaze estimation technology described above, for example, a method is desired that can improve the generalization performance of the learning model while reducing the workload required to generate the learning model.

[0008] Therefore, an object of the present invention is to provide a learning model generation program, an information processing device, and a learning model generation method that enable improving the generalization performance of a learning model used for gaze estimation.

[0009] In order to achieve the above-mentioned object, the learning model generation program of the present invention causes a computer to execute the following process: from first target image data obtained by photographing a person's head with a first camera and first reference image data obtained by photographing the head with a second camera, identify a first rotation matrix corresponding to the relative rotation angle between a first virtual camera in which normalization processing has been performed for the first camera and a second virtual camera in which normalization processing has been performed for the second camera; generate teacher data including the first target image data, the first reference image data, the first rotation matrix, and first gaze information indicating the person's gaze direction; and generate a learning model by learning the generated teacher data.

[0010] According to the learning model generation program, information processing device, and learning model generation method of the present invention, it is possible to improve the generalization performance of the learning model used for gaze estimation.

[0011] FIG. 1 is a diagram illustrating a configuration example of an information processing system 10 according to a first embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing device 1 according to the first embodiment. FIG. 3 is a diagram illustrating an outline of the first embodiment. FIG. 4 is a diagram illustrating an outline of the first embodiment. FIG. 5 is a diagram illustrating an outline of the first embodiment. FIG. 6 is a diagram illustrating an outline of the first embodiment. FIG. 7 is a diagram illustrating an outline of the first embodiment. FIG. 8 is a flowchart illustrating a learning process according to the first embodiment. FIG. 9 is a flowchart illustrating a learning process according to the first embodiment. FIG. 10 is a flowchart illustrating a learning process according to the first embodiment. FIG. 11 is a flowchart illustrating a learning process according to the first embodiment. FIG. 12 is a flowchart illustrating an estimation process according to the first embodiment. FIG. 13 is a diagram illustrating a specific example of the configuration of a learning model MD. FIG. 14 is a diagram illustrating a specific example of the configuration of a learning model MD.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, such descriptions should not be interpreted in a limiting sense, and do not limit the subject matter described in the claims. Furthermore, various changes, substitutions, and modifications can be made without departing from the spirit and scope of the present disclosure. Furthermore, different embodiments can be combined as appropriate.

[0013] [Configuration Example of Information Processing System 10 According to First Embodiment] First, a configuration example of the information processing system 10 according to the first embodiment will be described. Fig. 1 is a diagram showing a configuration example of the information processing system 10 according to the first embodiment.

[0014] The information processing system 10 includes, for example, an information processing device 1, an operation terminal 2, and a storage unit 130. The storage unit 130 may be a storage unit disposed outside the information processing device 1, or may be a storage unit mounted within the information processing device 1.

[0015] The operation terminal 2 is, for example, a mobile terminal such as a PC (Personal Computer) or a smartphone, and is a terminal through which an operator of the information processing device 1 (hereinafter simply referred to as an operator) inputs necessary information.

[0016] The information processing device 1 is, for example, a physical machine or a virtual machine, and performs a process of generating a learning model that estimates the gaze direction of a subject (hereinafter simply referred to as the learning process), and a process of estimating the gaze direction of a subject by using the learning model (hereinafter also referred to as the estimation process).

[0017] Specifically, in the learning process, the information processing device 1 generates a learning model by, for example, performing machine learning on teacher data (plurality of teacher data) stored in advance in the storage unit 130, and stores the generated learning model in the storage unit 130. Furthermore, in the estimation process, the information processing device 1 estimates the gaze direction of the subject by, for example, using the learning model generated in the learning process.

[0018] The following description will be given assuming that the information processing system 10 has one information processing device 1, but the information processing system 10 may have, for example, multiple information processing devices 1. The following description will be given assuming that the information processing system 10 has one operation terminal 2, but the information processing system 10 may have, for example, multiple operation terminals 2.

[0019] Furthermore, the information processing device 1 and the operation terminal 2 may be, for example, a single device. Specifically, the information processing system 10 may not include the operation terminal 2 when, for example, a PC or a mobile terminal into which a worker can directly input information is used as the information processing device 1.

[0020] [Configuration Example of Information Processing Device 1 in First Embodiment] Next, a description will be given of a configuration example of the information processing device 1. Fig. 2 is a diagram showing a configuration example of the information processing device 1 in the first embodiment.

[0021] 2, the information processing device 1 includes, for example, a CPU 101 which is a processor, a memory 102, a communication interface 103, and a storage medium 104. The respective components are connected to each other via a bus 105.

[0022] The storage medium 104 has a program storage area (not shown) that stores a program 110 for performing, for example, learning processing and estimation processing (hereinafter, these are also collectively referred to as learning processing, etc.).

[0023] The storage medium 104 also has a storage unit 130 (hereinafter also referred to as an information storage area 130) that stores information used when performing, for example, a learning process, etc. The storage medium 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0024] The CPU 101 executes, for example, a program 110 loaded from the storage medium 104 into the memory 102 to perform learning processing and the like.

[0025] The communication interface 103 also communicates with, for example, the operation terminal 2 .

[0026] [Outline of the First Embodiment] Next, an outline of the first embodiment will be described. Figures 3 to 7 are diagrams for explaining the outline of the first embodiment.

[0027] As shown in FIG. 3, the information processing device 1 realizes various functions including, for example, an image acquisition unit 111, an information acquisition unit 112, a matrix identification unit 113, a data generation unit 114, a model generation unit 115, a data estimation unit 116, and a data output unit 117.

[0028] Specifically, the image acquisition unit 111, the information acquisition unit 112, the matrix identification unit 113, the data generation unit 114, the model generation unit 115, and the second model generation unit 116 are functions that realize, for example, a learning process. Also, the image acquisition unit 111, the matrix identification unit 113, the data estimation unit 116, and the data output unit 117 are functions that realize, for example, an estimation process.

[0029] Note that, although the following description will be given assuming that the learning process and the estimation process are executed in the information processing device 1, this is not limiting. Specifically, either the learning process or the estimation process may be executed, for example, in another information processing device (not shown) different from the information processing device 1. Furthermore, if the information processing system 10 has multiple information processing devices 1, each function that executes the learning process and the estimation process may be executed, for example, in one of the multiple information processing devices 1 or in a distributed state across each of the multiple information processing devices 1.

[0030] First, the functions in the learning process will be explained.

[0031] As shown in Figure 4, camera CA1 (hereinafter also referred to as the first camera CA1) acquires image data DT1 (hereinafter also referred to as the first target image data DT1) showing the head of subject OB, for example, by photographing the head of subject OB from a position where it can photograph the head of subject OB.

[0032] The camera CA2 (hereinafter also referred to as the second camera CA2) acquires image data DT2 (hereinafter also referred to as the first reference image data DT2) showing the head of the subject OB, for example, by capturing an image of the head of the subject OB from a position where the head of the subject OB can be captured (a position different from the installation position of the camera CA1). Note that the camera CA2 captures the image data DT2 at the same time as the camera CA1 captures the image data DT1, for example.

[0033] The image acquisition unit 111 acquires, for example, image data DT1 captured by the camera CA1 from the camera CA1. Then, the image acquisition unit 111 stores, for example, the acquired image data DT1 in the information storage area 130, as shown in FIG.

[0034] Furthermore, the image acquisition unit 111 acquires, for example, image data DT2 captured by the camera CA2 from the camera CA2, and stores the acquired image data DT2 in the information storage area 130, for example.

[0035] The image data DT1 may be, for example, data obtained by an operator from the camera CA1 and stored in the information storage area 130. The image data DT2 may be, for example, data obtained by an operator from the camera CA2 and stored in the information storage area 130. Hereinafter, the image data DT1 and the image data DT2 may be collectively referred to simply as image data DT.

[0036] The information acquisition unit 112 acquires, for example, gaze information DT4 (hereinafter also referred to as first gaze information DT4) indicating the gaze direction SD of the subject OB at the time when the camera CA1 captured the image data DT1 (at the time when the camera CA2 captured the image data DT2). Specifically, the information acquisition unit 112 acquires, for example, the gaze information DT4 input by the worker to the information processing device 1. Then, the information acquisition unit 112 stores the acquired gaze information DT4 in the information storage area 130, for example.

[0037] For example, the matrix determination unit 113 determines, from the image data DT1 and DT2 acquired by the image acquisition unit 111, a rotation matrix DT3 (hereinafter also referred to as a first rotation matrix DT3) corresponding to the relative rotation angle between a virtual camera CA1 (hereinafter also referred to as a virtual camera CA1a or a first virtual camera CA1a) after normalization processing has been performed on the camera CA1 and a virtual camera CA2a (hereinafter also referred to as a virtual camera CA2a or a second virtual camera CA2a) after normalization processing has been performed on the camera CA2. The normalization processing for each camera will be described later. Then, the matrix determination unit 113 stores the determined rotation matrix DT3 in the information storage area 130, for example.

[0038] 5 , the data generation unit 114 generates teacher data DT5 including at least image data DT1 acquired by the image acquisition unit 111, image data DT2 acquired by the image acquisition unit 111, a rotation matrix DT3 identified by the matrix identification unit 113, and line-of-sight information DT4 acquired by the information acquisition unit 112. Then, the data generation unit 114 stores the generated teacher data DT5 in the information storage area 130, for example.

[0039] 5 , the model generation unit 115 generates the learning model MD by performing machine learning using, for example, the teacher data DT5 (plurality of teacher data DT5) generated by the data generation unit 114. Then, the model generation unit 115 stores the generated learning model MD in, for example, the information storage area 130.

[0040] Note that the following describes the case where a learning model MD is generated using image data DT (image data DT1 and image data DT2) captured by two cameras (camera CA1 and camera CA2), but the information processing device 1 may also generate a learning model MD using image data captured by, for example, three or more cameras.

[0041] Next, the functions in the estimation process will be described.

[0042] 6, the image acquisition unit 111 acquires image data DT11 (hereinafter also referred to as second target image data DT11) captured by a camera CA11 (hereinafter also referred to as third camera CA11) from the camera CA11. Note that the camera CA11 may be, for example, the camera CA1 or a camera different from the cameras CA1 and CA2.

[0043] Furthermore, the image acquisition unit 111 acquires, for example, image data DT12 (hereinafter also referred to as second reference image data DT12) captured by camera CA12 (hereinafter also referred to as fourth camera CA12) from camera CA12. Camera CA12 captures the image data DT12 at the same timing as camera CA11 captures the image data DT11, for example. Note that camera CA12 may be, for example, camera CA2, or may be a camera different from cameras CA1 and CA2.

[0044] The matrix determination unit 113, for example, determines, from the image data DT11 and DT12 acquired by the image acquisition unit 111, a rotation matrix DT13 (hereinafter also referred to as the second rotation matrix DT13) corresponding to the relative rotation angle between a virtual camera CA11 (hereinafter also referred to as the virtual camera CA11a or the third virtual camera CA11a) after normalization processing has been performed on the camera CA11, and a virtual camera CA12 (hereinafter also referred to as the virtual camera CA12a or the fourth virtual camera CA12a) after normalization processing has been performed on the camera CA12.

[0045] 7, the data estimation unit 116 inputs, for example, image data DT11 acquired by the image acquisition unit 111, image data DT12 acquired by the image acquisition unit 111, and a rotation matrix DT13 identified by the matrix identification unit 113 into the learning model MD. Then, the data estimation unit 116 acquires, for example, gaze information DT14 (hereinafter also referred to as second gaze information DT14) output from the learning model MD.

[0046] The data output unit 117 outputs, for example, the line-of-sight information DT14 acquired by the data estimation unit 116 to the operation terminal 2 .

[0047] That is, the information processing device 1 in this embodiment generates a learning model MD that estimates the gaze direction SD of the subject OB, for example, by using training data DT5 that includes a rotation matrix DT3 corresponding to the relative rotation angle between the virtual camera CA1a and the virtual camera CA2a.

[0048] As a result, the information processing device 1 in this embodiment can accurately estimate the gaze direction SD of the subject OB even if, for example, the cameras CA11 and CA12 used in the estimation process (the cameras that captured the image data DT11 and DT12 input to the learning model MD) are different from the cameras CA1 and CA2 used in the learning process (the cameras that captured the image data DT1 and DT2 used to generate the learning model MD).

[0049] In other words, the information processing device 1 according to the present embodiment can generate, for example, a learning model MD with high generalization performance in the learning process, and therefore, can accurately estimate the gaze direction SD of the subject OB in the estimation process, regardless of, for example, the orientation of the head of the subject OB captured in the image data DT11 or DT12 (the shooting direction of the camera CA11 or CA12).

[0050] Furthermore, the information processing device 1 in this embodiment generates the learning model MD using, for example, teacher data DT5 including a rotation matrix DT3, thereby eliminating the need to prepare teacher data DT5 for each camera combination. Therefore, the information processing device 1 can reduce the number of teacher data DT5 required to generate the learning model MD (the number of teacher data DT5 required to improve the determination accuracy of the learning model MD to a desired level) compared to, for example, generating another learning model (not shown) using other teacher data (not shown) that does not include the rotation matrix DT3.

[0051] [Flowchart of Learning Process in First Embodiment] Next, the learning process in the first embodiment will be described. Figures 8 to 11 are flow charts for explaining the learning process in the first embodiment.

[0052] 8, the image acquisition unit 111 waits until it receives input of image data DT1 captured by the camera CA1 (NO in S11). Specifically, the image acquisition unit 111 waits until the image data DT1 captured by the camera CA1 is transmitted from the camera CA1.

[0053] Then, when the image data DT1 is received as input (YES in S11), the image acquisition unit 111 stores the received image data DT1 in the information storage area 130 (S12).

[0054] Similarly, the image acquisition unit 111 waits until it receives input of image data DT2 captured by the camera CA2 (NO in S11).

[0055] Then, when the image data DT2 is received as input (YES in S11), the image acquisition unit 111 stores the received image data DT2 in the information storage area 130 (S12).

[0056] 9, the information acquisition unit 112 waits until it receives input of line-of-sight information DT4 indicating the line-of-sight direction SD of the subject OB at the time when the camera CA1 captured the image data DT1 (the time when the camera CA2 captured the image data DT1) (NO in S21). Specifically, the information acquisition unit 112 waits until the worker inputs line-of-sight information DT4 indicating the line-of-sight direction SD of the subject OB at the time when the camera CA1 captured the image data DT1.

[0057] Then, when the input of the line-of-sight information DT4 is accepted (YES in S21), the information acquisition unit 112 stores the accepted input line-of-sight information DT4 in the information storage area 130 (S22), for example.

[0058] 10, the matrix identification unit 113 waits until the learning timing arrives (NO in S31). The learning timing may be, for example, the timing when the operator inputs information to the information processing device 1 indicating that generation of the learning model MD is to be started.

[0059] Then, when the learning timing arrives (YES in S31), the matrix identification unit 113 identifies a rotation matrix DT3 corresponding to the relative rotation angle between the virtual cameras CA1a and CA2a, for example, from the image data DT1 and DT2 stored in the information storage area 130 (S32). Then, the matrix identification unit 113 stores the identified rotation matrix DT3 in the information storage area 130. The processing of S32 will be described in detail below.

[0060] 11 is a flowchart illustrating the details of the process of S32. The following describes the case where the rotation matrix DT3 is calculated using the method described in Non-Patent Document 1. Note that the rotation matrix DT3 will also be referred to as the rotation matrix R below.

[0061] First, as shown in FIG. 11, the matrix specifying unit 113 performs, for example, normalization processing on each of the image data DT1 and the image data DT2 (S101).

[0062] Specifically, the matrix specification unit 113 calculates, for each image data, a projective transformation matrix N (hereinafter simply referred to as transformation matrix N) that realizes an image transformation that satisfies the following conditions, for example. First, the transformed image data is made equivalent to the state in which the image data is captured by a virtual camera positioned a predetermined distance from the center of the subject OB's face. Next, the line connecting the subject OB's eyes in the transformed image data is made parallel to the horizontal direction (Y-axis direction) in a three-dimensional coordinate system (hereinafter also referred to as a normalized camera coordinate system) relative to the shooting direction of the virtual camera. Furthermore, the center of the subject OB's face in the transformed image data is made to be located on the optical axis of the virtual camera. Note that, hereinafter, the three-dimensional coordinate system relative to the shooting direction of the camera that captured each image data DT is also referred to as a pre-normalized camera coordinate system.

[0063] Specifically, the transformation matrix N is calculated using, for example, the following equations (1) to (4).

[0064]

[0065]

[0066]

[0067]

[0068] In the above formulas (1) to (4), x represents an element in the X-axis direction in the transformation matrix N, c represents the center position of the face of the subject OB, y represents an element in the Y-axis direction in the transformation matrix N, and e left indicates the position of the left eye of the subject OB, and e right indicates the position of the right eye of the subject OB, and x indicates the cross product of the vector. left and e right The position may be the average position of the center position of the subject and the position of the subject's OB mouth.

[0069] Then, in the processing of S101, the matrix identification unit 113 converts the gaze direction SD of the subject OB reflected in each of the image data DT1 and the image data DT2 from the gaze direction SD in the camera coordinate system before normalization to the gaze direction SD in the camera coordinate system after normalization, for example, by using each transformation matrix N calculated from each of the image data DT1 and the image data DT2.

[0070] Next, the matrix specification unit 113 calculates the rotation matrix R by using, for example, the following equation (5) (S102).

[0071]

[0072] In the above formula (5), R 0 indicates the rotation matrix between the camera CA1 and the camera CA2. 0 may be obtained by stereo calibration of the camera CA1 and the camera CA2. tgt denotes the transformation matrix N calculated from the image data DT1, and N ref denotes the transformation matrix N calculated from the image data DT2.

[0073] That is, the rotation matrix R is, for example, a rotation matrix between a virtual camera CA1a obtained by normalizing the camera CA1 and a virtual camera CA2a obtained by normalizing the camera CA2.

[0074] Furthermore, in the process of S103, the matrix specification unit 113 may calculate the rotation matrix R by using, for example, the following equation (6).

[0075]

[0076] In the above formula (6), H indicates a rotation matrix between a three-dimensional coordinate system (hereinafter also referred to as a head coordinate system) centered on the head of the subject OB and the coordinate system of the normalized camera, and is calculated for each image data in the same way as the transformation matrix N. tgt indicates the rotation matrix H calculated from the image data DT1, and H ref indicates the rotation matrix H calculated from the image data DT2.

[0077] This enables the information processing device 1 to calculate, for example, the rotation matrix DT3 to be included in the training data DT5.

[0078] 10 , the data generation unit 114 generates teacher data DT5 (S33) including at least the image data DT1 stored in the information storage area 130, the image data DT2 stored in the information storage area 130, the rotation matrix DT3 identified in the processing of S32, and the line-of-sight information DT4 stored in the information storage area 130. Then, the data generation unit 114 stores the generated teacher data DT5 in the information storage area 130, for example.

[0079] In addition, in the processes of S32 and S33, the information processing device 1 may generate multiple pieces of training data DT5 by repeatedly performing each process, for example.

[0080] Alternatively, instead of the processes of S32 and S33, the data generation unit 114 may generate image data DT1 and image data DT2 that satisfy a pre-specified rotation matrix DT3 by using a technique such as CG (computer graphics).The data generation unit 114 may also generate teacher data DT5 that includes at least the generated image data DT1 and image data DT2, a rotation matrix DT3 pre-specified by the worker, and pre-specified line-of-sight information DT4.

[0081] Next, the model generation unit 115 generates a learning model MD by performing machine learning using, for example, the teacher data DT5 (plurality of teacher data DT5) generated in the processing of S33 (S34).

[0082] Thereafter, the model generation unit 115 stores, for example, the learning model MD generated in the processing of S34 (S35).

[0083] [Flowchart of Estimation Process in First Embodiment] Next, a flowchart of the estimation process in the first embodiment will be described. Fig. 12 is a flowchart for explaining the estimation process in the first embodiment.

[0084] As shown in FIG. 12, the matrix specification unit 113 waits until it receives input of, for example, image data DT11 and image data DT12 (NO in S41).

[0085] Then, when the input of image data DT11 and image data DT12 is accepted (YES in S41), the matrix identification unit 113, for example, identifies a rotation matrix DT13 corresponding to the relative rotation angle between the virtual camera CA11a and the virtual camera CA12a from the accepted input image data DT11 and image data DT12 (S42).

[0086] Next, the data estimation unit 116 inputs, for example, the image data DT11 received in the processing of S41, the image data DT12 received in the processing of S41, and the rotation matrix DT13 identified in the processing of S42 into the learning model MD (S43).

[0087] Then, the data estimation unit 116 acquires, for example, the gaze information DT14 output from the learning model MD (S44).

[0088] That is, the data estimation unit 116 acquires, for example, the gaze information DT14 output from the learning model MD as an estimation result of the gaze direction SD of the subject OB appearing in the image data DT11 and DT12.

[0089] Thereafter, the data output unit 117 outputs, for example, the line-of-sight information DT14 acquired in the process of S44 to the operation terminal 2 (S45). A specific example of the configuration of the learning model MD will be described below.

[0090] 13 and 14 are diagrams illustrating a specific example of the configuration of the learning model MD. Hereinafter, the image data DT11 will be referred to as image data I tgt and the image data DT12 is expressed as image data I ref and the line-of-sight information DT14 estimated using the image data DT11 is expressed as g tgt or target gaze information g tgt The gaze information DT14 estimated using the image data DT12 is expressed as gaze information g ref Or reference gaze information g ref It is written as Tomo.

[0091] As shown in FIG. 13, the learning model MD is tgt The feature value f corresponding to the head of the subject OB is tgt (hereinafter, the first target feature f tgt Specifically, the module M11 extracts, for example, image data I tgt is used as input and the feature value f tgt Output.

[0092] The learning model MD is, for example, ref The feature value f corresponding to the head of the subject OB is ref (Hereinafter, the first reference feature f ref Specifically, the module M21 extracts, for example, image data I ref is used as input and the feature value f ref The module M21 may share the weights trained in the learning process with the module M11. In other words, the module M21 may be the same module as the module M11.

[0093] The learning model MD is, for example, tgt The rotatable feature F used in the process of fusing each feature from tgt (0) Specifically, the module M12 extracts, for example, the feature quantity f tgt is used as input and the feature F tgt (0) Output.

[0094] The learning model MD is, for example, ref The rotatable feature F used in the feature fusion process is ref (0) Specifically, the module M22 extracts, for example, the feature quantity fref is used as input and the feature F ref (0) The module M22 may share the weights trained in the learning process with the module M12. In other words, the module M22 may be the same module as the module M12.

[0095] The learning model MD is, for example, tgt and the feature F multiplied by the rotation matrix R ref (0) By performing feature fusion processing using tgt (1) Generate (feature F tgt (0) Specifically, the module M13 updates the feature quantity f multiplied by the rotation matrix R. tgt and feature F ref (0) and the feature F tgt (1) Output.

[0096] The learning model MD is, for example, ref and the feature F multiplied by the inverse matrix of the rotation matrix R. tgt (0) By performing feature fusion processing using ref (1) Generate (feature F ref (0) Specifically, the module M23 updates the feature quantity f multiplied by the rotation matrix R. ref and feature F tgt (0) and the feature F ref (1) The module M23 may share the weights trained in the learning process with the module M13. In other words, the module M23 may be the same module as the module M13.

[0097] That is, the learning model MD is, for example, tgt (1) When calculating the image data I tgt The feature f extracted from tgt In addition, the other image data I ref Feature F extracted from and multiplied by rotation matrix R ref (0) The learning model MD also uses, for example, the feature F ref (1) When calculating the image data I ref The feature f extracted from ref In addition, the other image data I ref The feature vector F is extracted from the vector and then multiplied by the inverse of the rotation matrix R. tgt (0) The calculation is also performed using

[0098] The learning model MD is, for example, tgt and feature F tgt (1) By using the image data I tgt The gaze information of the target OB reflected in tgt (1) Specifically, the module M14 estimates, for example, the feature quantity f tgt and feature F tgt (1) and the gaze information g tgt (1) Output.

[0099] The learning model MD is, for example, ref and feature F ref (1) By using the image data I ref The gaze information of the target OB reflected in ref (1) Specifically, the module M24 estimates, for example, the feature quantity f ref and feature F ref (1) and the gaze information g ref(1) The module M24 may, for example, share the weights trained in the learning process with the module M14. In other words, the module M24 may, for example, be the same module as the module M14. Hereinafter, the modules M11, M12, M13, M14, M21, M22, M23, and M24 will also be collectively referred to simply as modules M.

[0100] Next, the learning model MD uses, for example, the gaze information g output from the module M14. tgt (1) After that, in the process of S44, the data estimation unit 116 outputs, for example, the gaze information g output from the learning model MD. tgt (1) Get.

[0101] Then, in the learning process, each module M is trained to minimize the estimation error of the gaze direction in the training data DT5.

[0102] As a result, the learning model MD can acquire feature representations to which a rotation operation by multiplication of the rotation matrix R can be applied, for example. Furthermore, by performing adaptive feature fusion according to the positional relationship between the cameras capturing the image data DT, the gaze information g tgt This makes it possible to improve the estimation accuracy of

[0103] 14, the learning model MD may have n (n is an integer of 2 or more) sets of modules M13, M14, M23, and M24. In this case, the learning model MD may have, for example, a feature quantity F output from the (n-1)th set of module M13 (hereinafter also referred to as the fifth target module M13). tgt (n-1) is input to the n-th module M23 (hereinafter also referred to as the sixth reference module M23), and the feature F output from the (n-1)-th module M23 (hereinafter also referred to as the fifth reference module M23) is ref (n-1)to the n-th module M13 (hereinafter also referred to as the sixth target module M13), the line of sight information g tgt (n) and gaze information g ref (n) It may be assumed that:

[0104] Next, the learning model MD uses, for example, the gaze information g output from the n-th module M14. tgt (n) Thereafter, in the process of S44, the data estimation unit 116 may output, for example, the gaze information g output from the learning model MD. tgt (n) The above-mentioned information may be acquired.

[0105] Then, each module M (each module M included in each of the first to nth sets) is trained in the learning process to minimize the estimation error of the gaze direction in the training data DT5.

[0106] This enables the learning model MD to further improve the estimation accuracy of the gaze information DT14, for example.

[0107] In this way, the information processing device 1 in this embodiment determines a rotation matrix DT3 corresponding to the relative rotation angle between virtual camera CA1a and virtual camera CA2a, for example, from image data DT1 of the head of the subject OB photographed by camera CA1 and image data DT2 of the head of the subject OB photographed by camera CA2.

[0108] The information processing device 1 then generates training data DT5 including, for example, image data DT1, image data DT2, a rotation matrix DT3, and gaze information DT4 indicating the gaze direction SD of the subject OB. Thereafter, the information processing device 1 generates a learning model MD by learning the generated training data DT5, for example.

[0109] That is, the information processing device 1 in this embodiment generates a learning model MD that estimates the gaze direction SD of the subject OB, for example, by using training data DT5 that includes a rotation matrix DT3 corresponding to the relative rotation angle between the virtual camera CA1a and the virtual camera CA2a.

[0110] As a result, the information processing device 1 in this embodiment can accurately estimate the gaze direction SD of the subject OB even if, for example, the cameras CA11 and CA12 used in the estimation process (the cameras that captured the image data DT11 and DT12 input to the learning model MD) are different from the cameras CA1 and CA2 used in the learning process (the cameras that captured the image data DT1 and DT2 used to generate the learning model MD).

[0111] In other words, the information processing device 1 according to the present embodiment can generate, for example, a learning model MD with high generalization performance in the learning process, and therefore can accurately estimate, for example, the gaze direction SD of the subject OB in the estimation process, regardless of the orientation of the head of the subject OB reflected in the image data DT11 and the image data DT12.

[0112] Furthermore, the information processing device 1 in this embodiment generates the learning model MD using, for example, training data DT5 including a rotation matrix DT3, thereby eliminating the need to prepare training data DT5 for each camera combination. Therefore, the information processing device 1 can reduce the amount of training data DT5 required to generate the learning model MD compared to, for example, generating another learning model (not shown) using other training data (not shown) that does not include a rotation matrix DT3.

[0113] 1: Information processing device 2: Operation terminal 10: Information processing system 101: CPU 102: Memory 103: Communication interface 104: Storage medium 105: Bus 110: Program 111: Image acquisition unit 112: Information acquisition unit 113: Matrix identification unit 114: Data generation unit 115: Model generation unit 116: Data estimation unit 117: Data output unit 130: Information processing device CA1: Camera CA2: Camera CA11: Camera CA12: Camera DT1: Image data DT2: Image data DT3: Rotation matrix DT4: Gaze information DT5: Teacher data DT11: Image data DT12: Image data DT13: Rotation matrix DT14: Gaze information MD: Learning model OB: Subject SD: Gaze direction

Claims

1. A learning model generation program that causes a computer to execute the following processes: determining a first rotation matrix corresponding to the relative rotation angle between a first virtual camera in which normalization processing has been performed for the first camera and a second virtual camera in which normalization processing has been performed for the second camera, from first object image data obtained by photographing a person's head with a first camera and first reference image data obtained by photographing the head with a second camera; generating teacher data including the first object image data, the first reference image data, the first rotation matrix, and first gaze information indicating the gaze direction of the person; and generating a learning model by studying the generated teacher data.

2. A learning model generation program as claimed in claim 1, comprising the steps of: calculating a second rotation matrix corresponding to the relative rotation angle between a third virtual camera in which normalization processing has been performed for the third camera and a fourth virtual camera in which normalization processing has been performed for the fourth camera, from second target image data obtained by photographing the head with a third camera and second reference image data obtained by photographing the head with a fourth camera; inputting the second target image data, the second reference image data and the second rotation matrix into the learning model; obtaining second gaze information output from the learning model; and outputting the obtained second gaze information.

3. In claim 2, the learning model has a first object module, a second object module, a third object module, a fourth object module, a first reference module, a second reference module, a third reference module, and a fourth reference module, the first object module extracts a first object feature corresponding to the head shown in the second object image data, the first reference module extracts a first reference feature corresponding to the head shown in the second reference image data, the second object module extracts a rotatable second object feature used in the third reference module from the first object feature extracted by the first object module, the second reference module extracts a rotatable second reference feature used in the third object module from the first reference feature extracted by the first reference module, and the third object module updates the second object feature using the first object feature, the first rotation matrix, and the second reference feature extracted by the second reference module, the third reference module updates the second reference features using the first reference features, the first rotation matrix, and the second target features extracted by the second target module; the fourth target module estimates target gaze information indicating the gaze direction of the person appearing in the second target image data using the first target features and the second target features updated by the third target module; and the fourth reference module estimates reference gaze information indicating the gaze direction of the person appearing in the second reference image data using the second reference features and the second reference features updated by the third reference module.

4. A learning model generation program according to claim 3, characterized in that in the acquisition process, the target gaze information estimated by the fourth target module is acquired as the second gaze information.

5. The learning model generation program according to claim 3, characterized in that the learning model has: a fifth object module that further updates the second object features using the first object features, the first rotation matrix, and the second reference features extracted by the third reference module; a fifth reference module that further updates the second reference features using the first reference features, the first rotation matrix, and the second object features extracted by the third object module; a sixth object module that updates the object gaze information using the first object features and the second object features updated by the fifth object module; and a sixth reference module that estimates the reference gaze information using the first reference features and the second reference features updated by the fifth reference module.

6. A learning model generation program according to claim 5, characterized in that in the acquisition process, the target gaze information updated by the sixth target module is acquired as the second gaze information.

7. An information processing device comprising: a matrix identification unit that identifies a first rotation matrix corresponding to the relative rotation angle between a first virtual camera in which normalization processing has been performed for the first camera and a second virtual camera in which normalization processing has been performed for the second camera, from first object image data obtained by photographing a person's head with a first camera and first reference image data obtained by photographing the head with a second camera; a data generation unit that generates teacher data including the first object image data, the first reference image data, the first rotation matrix, and first gaze information indicating the gaze direction of the person; and a model generation unit that generates a learning model by learning the generated teacher data.

8. A learning model generation method characterized by being executed by a computer to perform the following processes: determining a first rotation matrix corresponding to the relative rotation angle between a first virtual camera in which normalization processing has been performed for the first camera and a second virtual camera in which normalization processing has been performed for the second camera, from first object image data obtained by photographing a person's head with a first camera and first reference image data obtained by photographing the head with a second camera; generating teacher data including the first object image data, the first reference image data, the first rotation matrix, and first gaze information indicating the gaze direction of the person; and generating a learning model by learning the generated teacher data.

Citation Information

Patent Citations

  • Information processing apparatus, information processing method, learning method, and program

    JP2022161690A

  • Systems and methods for capturing training data for a gaze estimation model

    US20200089315A1

  • Sight line estimation device, sight line estimation method, model generation device, and model generation method

    WO2021140642A1