Learning model generation program, information processing device, and learning model generation method

By using multiple cameras and normalization processing with specified rotation matrices, the method enhances gaze estimation accuracy and reduces workload, addressing the challenge of head orientation variations in existing gaze estimation techniques.

JP2024162431APending Publication Date: 2024-11-21THE UNIV OF TOKYO
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023077924
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing gaze estimation techniques struggle with accurately estimating gaze direction due to variations in head orientation, leading to suboptimal generalization performance and increased workload in generating learning models.

Method used

A method that utilizes multiple cameras at different positions to capture image data, applying normalization processing and specifying a rotation matrix to generate teacher data, which is used to create a learning model that enhances generalization performance while reducing the workload.

Benefits of technology

The approach improves the accuracy and generalization performance of gaze estimation by enabling accurate gaze direction estimation regardless of head orientation, without the need for extensive training data preparation for each camera combination.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162431000001_ABST
    Figure 2024162431000001_ABST
Patent Text Reader

Abstract

To provide a learning model generation program, an information processing device, and a learning model generation method that can improve the performance of generating a learning model to be used for gaze estimation.SOLUTION: A learning model generation program includes: identifying a first rotation matrix corresponding to a relative angle of rotation between a first virtual camera obtained by subjecting a first camera to normalization processing and a second virtual camera obtained by subjecting a second camera to normalization processing, from first subject image data obtained by imaging a person's head with the first camera and first reference image data obtained by imaging the person's head with the second camera; generating training data including the first subject image data, the first reference image data, the first rotation matrix, and first gaze information indicating the person's gaze direction; and learning the generated training data to generate a learning model.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a learning model generation program, an information processing device, and a learning model generation method. [Background technology]

[0002] For example, there is a technology (hereinafter also simply referred to as gaze estimation technology) that estimates the gaze direction of a target person (person) by using image data showing the target person's head. In such a technology, for example, the gaze direction of a target person is estimated by inputting image data showing the target person's eyes into a trained learning model (hereinafter also simply referred to as a learning model) (see Patent Document 1, Patent Document 2, and Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-187546 A [Patent Document 2] JP 2022-187547 A [Non-patent literature]

[0004] [Non-Patent Document 1] X. Zhang, Y. Sugano, and A. Bulling, “Revisiting data normalization for appearance-based gaze estimation,” in ETRA, 2018. Summary of the Invention [Problem to be solved by the invention]

[0005] Here, the above-mentioned learning model may not be able to accurately estimate the gaze direction of the subject depending on the orientation of the subject's head shown in the image data. Specifically, for example, if the orientation of the subject's head shown in the image data is an orientation that has not been learned during learning, the learning model may not be able to accurately estimate the gaze direction of the subject. That is, the above-mentioned gaze estimation technology may not be able to generate a learning model with high generalization performance, for example.

[0006] In this regard, for example, by generating a learning model using multiple image data captured by multiple cameras (imaging devices) placed in various positions, it is possible to improve the generalization performance of the learning model.

[0007] However, in this case, for example, it becomes necessary to generate teacher data for each combination of cameras, and the workload required for generating the learning model increases. Therefore, in the above-mentioned gaze estimation technology, for example, a method capable of improving the generalization performance of the learning model while suppressing the workload required for generating the learning model is desired.

[0008] Therefore, an object of the present invention is to provide a learning model generation program, an information processing device, and a learning model generation method that enable improving the generalization performance of a learning model used for gaze estimation. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the learning model generation program of the present invention causes a computer to execute the following process: identify a first rotation matrix corresponding to the relative rotation angle between a first virtual camera in which normalization processing has been performed for the first camera and a second virtual camera in which normalization processing has been performed for the second camera, from first target image data obtained by photographing a person's head with a first camera and first reference image data obtained by photographing the head with a second camera; generate teacher data including the first target image data, the first reference image data, the first rotation matrix, and first gaze information indicating the gaze direction of the person; and generate a learning model by learning the generated teacher data. Effect of the Invention

[0010] According to the learning model generation program, information processing device, and learning model generation method of the present invention, it is possible to improve the generalization performance of a learning model used for gaze estimation. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of a configuration of an information processing system 10 according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a configuration of the information processing device 1 according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining an outline of the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining an outline of the first embodiment. [Diagram 5] FIG. 5 is a diagram for explaining an outline of the first embodiment. [Figure 6] FIG. 6 is a diagram for explaining an outline of the first embodiment. [Figure 7] FIG. 7 is a diagram for explaining an outline of the first embodiment. [Figure 8] FIG. 8 is a flowchart illustrating the learning process in the first embodiment. [Figure 9]FIG. 9 is a flowchart illustrating the learning process in the first embodiment. [Figure 10] FIG. 10 is a flowchart illustrating the learning process in the first embodiment. [Figure 11] FIG. 11 is a flowchart illustrating the learning process in the first embodiment. [Figure 12] FIG. 12 is a flowchart illustrating the estimation process according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating a specific example of the configuration of the learning model MD. [Figure 14] FIG. 14 is a diagram illustrating a specific example of the configuration of the learning model MD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. However, such description should not be interpreted in a limiting sense, and does not limit the subject matter described in the claims. In addition, various changes, substitutions, and modifications can be made without departing from the spirit and scope of the present disclosure. In addition, different embodiments can be appropriately combined.

[0013] [Configuration Example of Information Processing System 10 in the First Embodiment] First, a configuration example of an information processing system 10 according to the first embodiment will be described. Fig. 1 is a diagram showing a configuration example of an information processing system 10 according to the first embodiment.

[0014] The information processing system 10 includes, for example, an information processing device 1, an operation terminal 2, and a storage unit 130. Note that the storage unit 130 may be a storage unit disposed outside the information processing device 1, or may be a storage unit mounted within the information processing device 1.

[0015] The operation terminal 2 is, for example, a mobile terminal such as a PC (Personal Computer) or a smartphone, and is a terminal through which an operator of the information processing device 1 (hereinafter also simply referred to as an operator) inputs necessary information.

[0016] The information processing device 1 is, for example, a physical machine or a virtual machine, and performs a process of generating a learning model for estimating the gaze direction of a subject (hereinafter also referred to as a learning process) and a process of estimating the gaze direction of a subject by using the learning model (hereinafter also referred to as an estimation process).

[0017] Specifically, in the learning process, the information processing device 1 generates a learning model by performing machine learning of teacher data (a plurality of teacher data) stored in advance in the storage unit 130, and stores the generated learning model in the storage unit 130. In addition, in the estimation process, the information processing device 1 estimates the gaze direction of the subject by using the learning model generated in the learning process, for example.

[0018] The following description will be given of a case where the information processing system 10 has one information processing device 1, but the information processing system 10 may have, for example, a plurality of information processing devices 1. The following description will be given of a case where the information processing system 10 has one operation terminal 2, but the information processing system 10 may have, for example, a plurality of operation terminals 2.

[0019] Furthermore, the information processing device 1 and the operation terminal 2 may be, for example, a single device. Specifically, the information processing system 10 may not include the operation terminal 2 when, for example, a PC or a mobile terminal into which a worker can directly input information is used as the information processing device 1.

[0020] [Configuration Example of Information Processing Device 1 in First Embodiment] Next, a description will be given of a configuration example of the information processing device 1. Fig. 2 is a diagram showing a configuration example of the information processing device 1 in the first embodiment.

[0021] 2, the information processing device 1 includes, for example, a CPU 101 which is a processor, a memory 102, a communication interface 103, and a storage medium 104. Each unit is connected to one another via a bus 105.

[0022] The storage medium 104 has a program storage area (not shown) that stores a program 110 for performing, for example, a learning process and an estimation process (hereinafter, these are also collectively referred to as a learning process, etc.).

[0023] The storage medium 104 also includes a storage unit 130 (hereinafter also referred to as an information storage area 130) that stores information used when performing a learning process, etc. The storage medium 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0024] The CPU 101 executes, for example, a program 110 loaded from the storage medium 104 to the memory 102 to perform learning processing and the like.

[0025] Furthermore, the communication interface 103 communicates with, for example, the operation terminal 2.

[0026] [Outline of the first embodiment] Next, an outline of the first embodiment will be described with reference to Figures 3 to 7. The first embodiment will be described with reference to Figures 3 to 7.

[0027] As shown in FIG. 3, the information processing device 1 realizes various functions including, for example, an image acquisition unit 111, an information acquisition unit 112, a matrix identification unit 113, a data generation unit 114, a model generation unit 115, a data estimation unit 116, and a data output unit 117.

[0028] Specifically, each of the image acquisition unit 111, the information acquisition unit 112, the matrix specification unit 113, the data generation unit 114, the model generation unit 115, and the second model generation unit 115 is a function that realizes, for example, a learning process. Also, each of the image acquisition unit 111, the matrix specification unit 113, the data estimation unit 116, and the data output unit 117 is a function that realizes, for example, an estimation process.

[0029] In the following, the learning process and the estimation process will be described as being executed in the information processing device 1, but the present invention is not limited to this. Specifically, either the learning process or the estimation process may be executed in, for example, another information processing device (not shown) different from the information processing device 1. In addition, when the information processing system 10 has a plurality of information processing devices 1, each function that executes each of the learning process and the estimation process may be executed in, for example, one of the plurality of information processing devices 1, or in a distributed state in each of the plurality of information processing devices 1.

[0030] First, the functions in the learning process will be described.

[0031] As shown in FIG. 4, camera CA1 (hereinafter also referred to as the first camera CA1) acquires image data DT1 (hereinafter also referred to as the first target image data DT1) showing the head of subject OB, for example, by photographing the head of subject OB from a position where the head of subject OB can be photographed.

[0032] The camera CA2 (hereinafter also referred to as the second camera CA2) captures the head of the subject OB from a position where the head of the subject OB can be captured (a position different from the installation position of the camera CA1) to obtain image data DT2 (hereinafter also referred to as the first reference image data DT2) in which the head of the subject OB is captured. Note that the camera CA2 captures the image data DT2 at the same timing as the camera CA1 captures the image data DT1, for example.

[0033] The image acquisition unit 111 acquires, for example, image data DT1 captured by the camera CA1 from the camera CA1. Then, the image acquisition unit 111 stores, for example, the acquired image data DT1 in the information storage area 130 as shown in FIG.

[0034] Furthermore, the image acquiring unit 111 acquires, for example, image data DT2 captured by the camera CA2 from the camera CA2. Then, the image acquiring unit 111 stores the acquired image data DT2 in the information storage area 130, for example.

[0035] The image data DT1 may be, for example, acquired by an operator from the camera CA1 and stored in the information storage area 130. The image data DT2 may be, for example, acquired by an operator from the camera CA2 and stored in the information storage area 130. Hereinafter, the image data DT1 and the image data DT2 are collectively referred to simply as image data DT.

[0036] The information acquiring unit 112 acquires, for example, gaze information DT4 (hereinafter also referred to as first gaze information DT4) indicating the gaze direction SD of the subject OB at the timing when the camera CA1 captured the image data DT1 (the timing when the camera CA2 captured the image data DT2). Specifically, the information acquiring unit 112 acquires, for example, gaze information DT4 input by the worker to the information processing device 1. Then, the information acquiring unit 112 stores, for example, the acquired gaze information DT4 in the information storage area 130.

[0037] The matrix determination unit 113, for example, from the image data DT1 and image data DT2 acquired by the image acquisition unit 111, determines a rotation matrix DT3 (hereinafter also referred to as the first rotation matrix DT3) corresponding to a relative rotation angle between a virtual camera CA1 (hereinafter also referred to as the virtual camera CA1a or the first virtual camera CA1a) after normalization processing is performed on the camera CA1 and a virtual camera CA2a (hereinafter also referred to as the virtual camera CA2a or the second virtual camera CA2a) after normalization processing is performed on the camera CA2. The normalization processing of each camera will be described later. Then, the matrix determination unit 113 stores the determined rotation matrix DT3 in the information storage area 130, for example.

[0038] 5, the data generation unit 114 generates teacher data DT5 including at least image data DT1 acquired by the image acquisition unit 111, image data DT2 acquired by the image acquisition unit 111, a rotation matrix DT3 identified by the matrix identification unit 113, and line of sight information DT4 acquired by the information acquisition unit 112. Then, the data generation unit 114 stores the generated teacher data DT5 in the information storage area 130, for example.

[0039] 5, the model generation unit 115 generates a learning model MD by performing machine learning using, for example, the teacher data DT5 (plurality of teacher data DT5) generated by the data generation unit 114. Then, the model generation unit 115 stores the generated learning model MD in the information storage area 130, for example.

[0040] Note that, below, we will explain the case where a learning model MD is generated using image data DT (image data DT1 and image data DT2) captured by two cameras (camera CA1 and camera CA2), but the information processing device 1 may also generate a learning model MD using image data captured by three or more cameras, for example.

[0041] Next, the functions in the estimation process will be described.

[0042] 6, the image acquisition unit 111 acquires image data DT11 (hereinafter also referred to as second target image data DT11) captured by a camera CA11 (hereinafter also referred to as a third camera CA11) from the camera CA11. Note that the camera CA11 may be, for example, the camera CA1 or may be a camera different from the cameras CA1 and CA2.

[0043] Furthermore, the image acquisition unit 111 acquires image data DT12 (hereinafter also referred to as second reference image data DT12) captured by, for example, a camera CA12 (hereinafter also referred to as a fourth camera CA12) from the camera CA12. The camera CA12 captures the image data DT12 at the same timing as the camera CA11 captures the image data DT11. The camera CA12 may be, for example, the camera CA2, or may be a camera different from the cameras CA1 and CA2.

[0044] The matrix determination unit 113, for example, determines, from the image data DT11 and image data DT12 acquired by the image acquisition unit 111, a rotation matrix DT13 (hereinafter also referred to as the second rotation matrix DT13) corresponding to the relative rotation angle between a virtual camera CA11 (hereinafter also referred to as the virtual camera CA11a or the third virtual camera CA11a) after normalization processing has been performed on the camera CA11, and a virtual camera CA12 (hereinafter also referred to as the virtual camera CA12a or the fourth virtual camera CA12a) after normalization processing has been performed on the camera CA12.

[0045] 7, the data estimation unit 116 inputs, for example, image data DT11 acquired by the image acquisition unit 111, image data DT12 acquired by the image acquisition unit 111, and a rotation matrix DT13 identified by the matrix identification unit 113 to the learning model MD. Then, the data estimation unit 116 acquires, for example, gaze information DT14 (hereinafter also referred to as second gaze information DT14) output from the learning model MD.

[0046] The data output unit 117 outputs, for example, the line-of-sight information DT14 acquired by the data estimation unit 116 to the operation terminal 2.

[0047] That is, the information processing device 1 in this embodiment generates a learning model MD that estimates the gaze direction SD of the subject OB, for example, by using teacher data DT5 including a rotation matrix DT3 corresponding to the relative rotation angle between the virtual camera CA1a and the virtual camera CA2a.

[0048] As a result, the information processing device 1 in this embodiment is able to accurately estimate the gaze direction SD of the subject OB even when, for example, the cameras CA11 and CA12 used in the estimation process (the cameras that captured the image data DT11 and image data DT12 input to the learning model MD) are different from the cameras CA1 and CA2 used in the learning process (the cameras that captured the image data DT1 and image data DT2 used to generate the learning model MD).

[0049] In other words, the information processing device 1 in this embodiment can generate a learning model MD with high generalization performance in the learning process. Therefore, the information processing device 1 can accurately estimate the gaze direction SD of the subject OB in the estimation process, regardless of the direction of the head of the subject OB captured in the image data DT11 and image data DT12 (the shooting direction of the camera CA11 and the camera CA12).

[0050] In addition, the information processing device 1 in this embodiment generates the learning model MD using the teacher data DT5 including the rotation matrix DT3, so that it is not necessarily necessary to prepare teacher data DT5 for each combination of cameras. Therefore, the information processing device 1 can reduce the number of teacher data DT5 required to generate the learning model MD (the number of teacher data DT5 required to improve the judgment accuracy of the learning model MD to a desired level) compared to the case where another learning model (not shown) is generated using other teacher data (not shown) not including the rotation matrix DT3.

[0051] [Flowchart of learning process in the first embodiment] Next, a description will be given of the learning process in the first embodiment. Figures 8 to 11 are flow charts illustrating the learning process in the first embodiment.

[0052] 8, the image acquisition unit 111 waits until it receives input of image data DT1 captured by the camera CA1 (NO in S11). Specifically, the image acquisition unit 111 waits until the image data DT1 captured by the camera CA1 is transmitted from the camera CA1.

[0053] Then, when the input of the image data DT1 is accepted (YES in S11), the image acquisition unit 111 stores, for example, the accepted input of the image data DT1 in the information storage area 130 (S12).

[0054] Similarly, the image acquisition unit 111 waits until it receives input of image data DT2 captured by the camera CA2 (NO in S11).

[0055] Then, when the input of the image data DT2 is accepted (YES in S11), the image acquisition unit 111 stores, for example, the accepted input of the image data DT2 in the information storage area 130 (S12).

[0056] 9, the information acquiring unit 112 waits until it receives input of line-of-sight information DT4 indicating the line-of-sight direction SD of the subject OB at the time when the camera CA1 captured the image data DT1 (the time when the camera CA2 captured the image data DT1) (NO in S21). Specifically, the information acquiring unit 112 waits until the operator inputs line-of-sight information DT4 indicating the line-of-sight direction SD of the subject OB at the time when the camera CA1 captured the image data DT1.

[0057] Then, when the input of the line-of-sight information DT4 is accepted (YES in S21), the information acquiring section 112 stores the inputted line-of-sight information DT4 in the information storage area 130 (S22), for example.

[0058] 10, the matrix identification unit 113 waits until the learning timing comes (NO in S31), for example. The learning timing may be the timing when the operator inputs information to the information processing device 1 to start generating the learning model MD, for example.

[0059] Then, when the learning timing arrives (YES in S31), the matrix specification unit 113 specifies a rotation matrix DT3 corresponding to the relative rotation angle between the virtual cameras CA1a and CA2a from, for example, the image data DT1 and image data DT2 stored in the information storage area 130 (S32). Then, the matrix specification unit 113 stores, for example, the specified rotation matrix DT3 in the information storage area 130. Details of the process of S32 will be described below.

[0060] [Details of S32 processing] 11 is a flowchart illustrating the details of the process of S32. Hereinafter, a case will be described in which the rotation matrix DT3 is calculated using the method described in Non-Patent Document 1. In addition, hereinafter, the rotation matrix DT3 is also referred to as the rotation matrix R.

[0061] First, as shown in FIG. 11, the matrix specifying unit 113 performs, for example, normalization processing on each of the image data DT1 and the image data DT2 (S101).

[0062] Specifically, the matrix specification unit 113 calculates, for each image data, a projective transformation matrix N (hereinafter also simply referred to as transformation matrix N) that realizes an image transformation that satisfies the following conditions, for example. First, the transformed image data is made equal to a state photographed by a virtual camera placed at a predetermined distance from the center of the face of the subject OB. Next, the straight line connecting the eyes of the subject OB reflected in the transformed image data is made parallel to the horizontal direction (Y-axis direction) in a three-dimensional coordinate system (hereinafter also referred to as a normalized camera coordinate system) for the shooting direction of the virtual camera. Furthermore, the center of the face of the subject OB reflected in the transformed image data is made to be located on the optical axis of the virtual camera. Note that, hereinafter, the three-dimensional coordinate system for the shooting direction of the camera that photographed each image data DT is also referred to as a pre-normalized camera coordinate system.

[0063] Specifically, the transformation matrix N is calculated, for example, by using the following formulas (1) to (4).

[0064]

number

[0065]

number

[0066]

number

[0067]

number

[0068] In the above formulas (1) to (4), x indicates an element in the X-axis direction in the transformation matrix N, c indicates the center position of the face of the subject OB, y indicates an element in the Y-axis direction in the transformation matrix N, and e left indicates the position of the left eye of subject OB, and e rightindicates the position of the right eye of the subject OB, and × indicates the cross product of vectors. Note that c is, for example, left and e right It may be the average position between the center position of the subject and the position of the subject's OB mouth.

[0069] Then, in the processing of S101, the matrix determination unit 113, for example, by using each transformation matrix N calculated from each of the image data DT1 and the image data DT2, converts the gaze direction SD of the subject OB reflected in each of the image data DT1 and the image data DT2 from the gaze direction SD in the camera coordinate system before normalization to the gaze direction SD in the camera coordinate system after normalization.

[0070] Next, the matrix specification unit 113 calculates the rotation matrix R by using, for example, the following equation (5) (S102).

[0071]

number

[0072] In the above formula (5), R0 indicates a rotation matrix between the camera CA1 and the camera CA2. Therefore, R0 may be obtained by, for example, stereo calibration between the camera CA1 and the camera CA2. tgt indicates the transformation matrix N calculated from the image data DT1, and N ref indicates the transformation matrix N calculated from the image data DT2.

[0073] That is, the rotation matrix R is, for example, a rotation matrix between a virtual camera CA1a obtained by normalizing the camera CA1 and a virtual camera CA2a obtained by normalizing the camera CA2.

[0074] Furthermore, in the process of S103, the matrix specification unit 113 may calculate the rotation matrix R by using, for example, the following equation (6).

[0075]

number

[0076] In the above formula (6), H indicates a rotation matrix between a three-dimensional coordinate system centered on the head of the subject OB (hereinafter also referred to as the head coordinate system) and the normalized coordinate system of the camera, and is calculated for each image data in the same way as the transformation matrix N. tgt indicates the rotation matrix H calculated from the image data DT1, and H ref indicates the rotation matrix H calculated from the image data DT2.

[0077] This enables the information processing device 1 to calculate, for example, the rotation matrix DT3 to be included in the teacher data DT5.

[0078] 10, the data generation unit 114 generates teacher data DT5 including at least image data DT1 stored in the information storage area 130, image data DT2 stored in the information storage area 130, the rotation matrix DT3 identified in the processing of S32, and line of sight information DT4 stored in the information storage area 130 (S33). Then, the data generation unit 114 stores the generated teacher data DT5 in the information storage area 130, for example.

[0079] Note that the information processing device 1 may generate a plurality of pieces of teacher data DT5 by repeatedly performing the processes in S32 and S33, for example.

[0080] Furthermore, the data generating unit 114 may generate image data DT1 and image data DT2 that satisfy a pre-specified rotation matrix DT3 by using a technology such as CG (Computer Graphics) instead of the processes of S32 and S33. The data generating unit 114 may generate teacher data DT5 that includes at least the generated image data DT1 and image data DT2, a rotation matrix DT3 specified in advance by an operator, and gaze information DT4 specified in advance.

[0081] Next, the model generation unit 115 generates a learning model MD by performing machine learning using, for example, the teacher data DT5 (the plurality of teacher data DT5) generated in the process of S33 (S34).

[0082] After that, the model generation unit 115 stores the learning model MD generated in the process of S34, for example (S35).

[0083] [Flowchart of Estimation Processing in the First Embodiment] Next, a description will be given of a flowchart of the estimation process in the first embodiment. Fig. 12 is a flowchart illustrating the estimation process in the first embodiment.

[0084] As shown in FIG. 12, the matrix specification unit 113 waits until it receives input of, for example, image data DT11 and image data DT12 (NO in S41).

[0085] Then, when input of image data DT11 and image data DT12 is accepted (YES in S41), the matrix determination unit 113, for example, determines a rotation matrix DT13 corresponding to the relative rotation angle between the virtual camera CA11a and the virtual camera CA12a from the accepted input image data DT11 and image data DT12 (S42).

[0086] Next, the data estimation unit 116 inputs, for example, image data DT11 received in the processing of S41, image data DT12 received in the processing of S41, and the rotation matrix DT13 identified in the processing of S42 into the learning model MD (S43).

[0087] Then, the data estimation unit 116 acquires, for example, the line-of-sight information DT14 output from the learning model MD (S44).

[0088] That is, the data estimation unit 116 acquires, for example, the gaze information DT14 output from the learning model MD as an estimation result of the gaze direction SD of the subject OB reflected in the image data DT11 and the image data DT12.

[0089] After that, the data output unit 117 outputs, for example, the line-of-sight information DT14 acquired in the process of S44 to the operation terminal 2 (S45). A specific example of the configuration of the learning model MD will be described below.

[0090] [Specific example of the configuration of the learning model MD] 13 and 14 are diagrams for explaining a specific example of the configuration of the learning model MD. Hereinafter, the image data DT11 is referred to as image data I tgt and image data DT12 is represented as image data I ref and the gaze information DT14 estimated using the image data DT11 is expressed as gaze information g tgt Or target gaze information g tgt The gaze information DT14 estimated using the image data DT12 is denoted as gaze information g ref Or reference gaze information g ref It is written as Tomo.

[0091] As shown in FIG. 13, the learning model MD is, for example, tgt The feature value f corresponding to the head of the subject OB is tgt (Hereinafter, the first target feature f tgt Specifically, the module M11 extracts image data I tgt With input, feature value f tgt Output.

[0092] In addition, the learning model MD is, for example, ref The feature value f corresponding to the head of the subject OB is ref (Hereinafter, the first reference feature f ref Specifically, the module M21 extracts image data I ref With input, feature value f refThe module M21 may share the weight trained in the learning process with the module M11. In other words, the module M21 may be the same module as the module M11.

[0093] In addition, the learning model MD is, for example, tgt The rotatable feature F used in the process of fusing each feature from tgt (0) Specifically, the module M12 extracts, for example, a feature quantity f tgt The input is the feature F tgt (0) Output.

[0094] In addition, the learning model MD is, for example, ref The rotatable feature F used in the feature fusion process ref (0) Specifically, the module M22 extracts, for example, a feature quantity f ref The input is the feature F ref (0) The module M22 may share the weights trained in the learning process with the module M12. In other words, the module M22 may be the same module as the module M12.

[0095] In addition, the learning model MD is, for example, tgt and the feature vector F multiplied by the rotation matrix R ref (0) By performing feature fusion processing using tgt (1) Generate (feature F tgt (0) Specifically, the module M13 updates the feature quantity f multiplied by the rotation matrix R. tgt and feature Fref (0) Enter the feature F tgt (1) Output.

[0096] In addition, the learning model MD is, for example, ref and the feature vector F multiplied by the inverse of the rotation matrix R. tgt (0) By performing feature fusion processing using ref (1) Generate (feature F ref (0) Specifically, the module M23 updates the feature quantity f multiplied by the rotation matrix R. ref and feature F tgt (0) Enter the feature F ref (1) The module M23 may share the weight trained in the learning process with the module M13. In other words, the module M23 may be the same module as the module M13.

[0097] That is, the learning model MD is, for example, tgt (1) When calculating the image data I tgt The feature f extracted from tgt In addition, the other image data I ref The feature vector F is extracted from the vector and multiplied by the rotation matrix R. ref (0) The learning model MD also uses, for example, the feature amount F ref (1) When calculating the image data I ref The feature f extracted from ref In addition, the other image data I ref The feature vector F is extracted from the matrix and then multiplied by the inverse of the rotation matrix R. tgt (0) The calculation is also performed using

[0098] In addition, the learning model MD is, for example, tgt and feature F tgt (1) By using the image data I tgt The gaze information of the subject OB reflected in g tgt (1) Specifically, the module M14 estimates, for example, a feature quantity f tgt and feature F tgt (1) With the input of and, the gaze information g tgt (1) Output.

[0099] In addition, the learning model MD is, for example, ref and feature F ref (1) By using the image data I ref The gaze information of the subject OB reflected in g ref (1) Specifically, the module M24 estimates, for example, a feature quantity f ref and feature F ref (1) With the input of and, the gaze information g ref (1) The module M24 may share the weight trained in the learning process with the module M14. In other words, the module M24 may be the same module as the module M14. Hereinafter, the modules M11, M12, M13, M14, M21, M22, M23, and M24 will be collectively referred to simply as the module M.

[0100] Next, the learning model MD, for example, uses the gaze information g output from the module M14. tgt (1) After that, in the process of S44, the data estimation unit 116 outputs, for example, the gaze information g tgt (1) Get the.

[0101] Then, in the learning process, each module M is trained to minimize the estimation error of the gaze direction in the teacher data DT5.

[0102] As a result, the learning model MD can acquire feature representations to which rotation operations can be applied, for example, by multiplying the rotation matrix R. Furthermore, by performing adaptive feature fusion according to the positional relationship between the cameras that capture the image data DT, the gaze information g tgt It is possible to improve the estimation accuracy.

[0103] As shown in Fig. 14, the learning model MD may have n (n is an integer of 2 or more) pairs of modules M13, M14, M23, and M24. In this case, the learning model MD may have a feature quantity F output from the (n-1)th pair of modules M13 (hereinafter, also referred to as the fifth target module M13). tgt (n-1) is input to the n-th module M23 (hereinafter also referred to as the sixth reference module M23), and the feature quantity F output from the (n-1)-th module M23 (hereinafter also referred to as the fifth reference module M23) is ref (n-1) The process of inputting the line of sight information g to the n-th module M13 (hereinafter also referred to as the sixth target module M13) is repeated. tgt (n) and gaze information g ref (n) It may be an estimate of the above.

[0104] Next, the learning model MD, in this case, for example, uses the gaze information g output from the n-th module M14. tgt (n) After that, in the process of S44, the data estimation unit 116 may output, for example, the gaze information g output from the learning model MD. tgt (n) The above-mentioned information may be acquired.

[0105] Then, each module M (each module M included in each of the first to nth sets) is trained in the learning process so as to minimize the estimation error of the gaze direction in the teacher data DT5.

[0106] This makes it possible for the learning model MD to further improve the estimation accuracy of the gaze information DT14, for example.

[0107] In this way, the information processing device 1 in this embodiment determines a rotation matrix DT3 corresponding to the relative rotation angle between virtual cameras CA1a and CA2a, for example, from image data DT1 in which the head of the subject OB is photographed by camera CA1 and image data DT2 in which the head of the subject OB is photographed by camera CA2.

[0108] Then, the information processing device 1 generates teacher data DT5 including, for example, image data DT1, image data DT2, a rotation matrix DT3, and gaze information DT4 indicating the gaze direction SD of the subject OB. After that, the information processing device 1 generates a learning model MD by learning, for example, the generated teacher data DT5.

[0109] That is, the information processing device 1 in this embodiment generates a learning model MD that estimates the gaze direction SD of the subject OB, for example, by using teacher data DT5 including a rotation matrix DT3 corresponding to the relative rotation angle between the virtual camera CA1a and the virtual camera CA2a.

[0110] As a result, the information processing device 1 in this embodiment is able to accurately estimate the gaze direction SD of the subject OB even when, for example, the cameras CA11 and CA12 used in the estimation process (the cameras that captured the image data DT11 and image data DT12 input to the learning model MD) are different from the cameras CA1 and CA2 used in the learning process (the cameras that captured the image data DT1 and image data DT2 used to generate the learning model MD).

[0111] In other words, the information processing device 1 in the present embodiment can generate, for example, a learning model MD with high generalization performance in the learning process. Therefore, the information processing device 1 can accurately estimate the gaze direction SD of the subject OB in the estimation process, for example, regardless of the direction of the head of the subject OB reflected in the image data DT11 and the image data DT12.

[0112] In addition, the information processing device 1 in this embodiment generates the learning model MD using the teacher data DT5 including the rotation matrix DT3, so that it is not necessarily necessary to prepare teacher data DT5 for each combination of cameras. Therefore, the information processing device 1 can reduce the amount of teacher data DT5 required to generate the learning model MD, compared to the case where another learning model (not shown) is generated using other teacher data (not shown) that does not include the rotation matrix DT3. [Explanation of symbols]

[0113] 1: Information processing device 2: Operation terminal 10: Information processing systems 101:CPU 102: Memory 103: Communication interface 104:Storage medium 105: Bus 110: Program 111: Image acquisition unit 112: Information acquisition department 113: Matrix identification part 114: Data generation unit 115: Model generation unit 116: Data Estimation Unit 117: Data output section 130: Information processing device CA1: Camera CA2: Camera CA11: Camera CA12: Camera DT1: Image data DT2: Image data DT3: Rotation matrix DT4: Line of Sight DT5: Teacher data DT11:Image data DT12: Image data DT13: Rotation matrix DT14: Line of Sight MD: Learning model OB: Target SD: Gaze direction

Claims

1. determining a first rotation matrix corresponding to a relative rotation angle between a first virtual camera in which a normalization process has been performed for the first camera and a second virtual camera in which a normalization process has been performed for the second camera, from first target image data in which a head of a person is photographed by a first camera and first reference image data in which the head is photographed by a second camera; generating teacher data including the first target image data, the first reference image data, the first rotation matrix, and first line-of-sight information indicating a line-of-sight direction of the person; A learning model is generated by learning the generated training data. A learning model generation program that causes a computer to execute a process.

2. In claim 1, calculating a second rotation matrix corresponding to a relative rotation angle between a third virtual camera obtained by performing normalization processing for the third camera and a fourth virtual camera obtained by performing normalization processing for the fourth camera, from second target image data obtained by photographing the head with a third camera and second reference image data obtained by photographing the head with a fourth camera; inputting the second target image data, the second reference image data, and the second rotation matrix into the learning model; Acquire second gaze information output from the learning model; outputting the acquired second gaze information; A learning model generation program comprising:

3. In claim 2, The learning model includes a first object module, a second object module, a third object module, a fourth object module, a first reference module, a second reference module, a third reference module, and a fourth reference module; The first object module extracts a first object feature corresponding to the head shown in the second object image data, The first reference module extracts a first reference feature corresponding to the head shown in the second reference image data, The second object module extracts rotatable second object features from the first object features extracted by the first object module, the second object features being used in the third reference module; The second reference module extracts a rotatable second reference feature to be used in the third reference module from the first reference feature extracted by the first reference module; The third object module updates the second object features using the first object features, the first rotation matrix, and the second reference features extracted by the second reference module; the third reference module updates the second reference feature using the first reference feature, the first rotation matrix, and the second target feature extracted by the second target module; and the fourth target module estimates target gaze information indicating a gaze direction of the person appearing in the second target image data using the first target feature and the second target feature updated by the third target module; the fourth reference module estimates reference gaze information indicating a gaze direction of the person appearing in the second reference image data, using the second reference feature amount and the second reference feature amount updated by the third reference module; A learning model generation program comprising:

4. In claim 3, In the acquiring process, the target gaze information estimated by the fourth target module is acquired as the second gaze information. A learning model generation program comprising:

5. In claim 3, The learning model is a fifth object module that further updates the second object features using the first object features, the first rotation matrix, and the second reference features extracted by the third reference module; a fifth reference module that further updates the second reference feature using the first reference feature, the first rotation matrix, and the second target feature extracted by the third target module; a sixth object module that updates the target gaze information by using the first object feature amount and the second object feature amount updated by the fifth object module; a sixth reference module that estimates the reference gaze information by using the first reference feature amount and the second reference feature amount updated by the fifth reference module; A learning model generation program comprising:

6. In claim 5, In the acquiring process, the target gaze information updated by the sixth target module is acquired as the second gaze information. A learning model generation program comprising:

7. a matrix determination unit that determines a first rotation matrix corresponding to a relative rotation angle between a first virtual camera in which normalization processing has been performed for the first camera and a second virtual camera in which normalization processing has been performed for the second camera, based on first target image data in which a head of a person is photographed by a first camera and first reference image data in which the head is photographed by a second camera; a data generating unit that generates teacher data including the first target image data, the first reference image data, the first rotation matrix, and first line-of-sight information indicating a line-of-sight direction of the person; A model generation unit that generates a learning model by learning the generated teacher data.

23. An information processing apparatus comprising:

8. determining a first rotation matrix corresponding to a relative rotation angle between a first virtual camera in which a normalization process has been performed for the first camera and a second virtual camera in which a normalization process has been performed for the second camera, from first target image data in which a head of a person is photographed by a first camera and first reference image data in which the head is photographed by a second camera; generating teacher data including the first target image data, the first reference image data, the first rotation matrix, and first line-of-sight information indicating a line-of-sight direction of the person; A learning model is generated by learning the generated training data. A learning model generating method characterized in that the processing is executed by a computer.

Citation Information

Patent Citations

  • Gaze estimation system

    JP2022187546A

  • Gaze estimation system

    JP2022187547A