Gaze estimation method based on multi-modal three-dimensional feature fusion

Abstract

The invention discloses a staring estimation method based on multi-modal three-dimensional feature fusion. The method comprises the following steps: firstly, extracting multi-modal three-dimensional features such as human body three-dimensional posture features, view direction priori and scene object three-dimensional position features from an RGB image and a depth image; then the extracted three-dimensional features are converted from a camera coordinate system to a self-center space with the head as the original point; then, direction and distance separation coding is carried out on the transformed three-dimensional features; and finally, interactive modeling is carried out on the coded three-dimensional features of the human body and the object through a Transform network, so that a final three-dimensional staring direction is predicted. According to the method, multi-modal three-dimensional information is fused and normalized interactive modeling is carried out, so that the dependence of a traditional method on a clear and close-range facial image is effectively overcome, and the robustness and accuracy of three-dimensional staring estimation in unconstrained scenes such as long-distance and shielding scenes are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a gaze estimation method based on multimodal 3D feature fusion. Background Technology

[0002] Gaze direction estimation, which determines what a person is looking at, is a key technology for understanding human intentions and behaviors. Traditional gaze estimation methods typically rely on high-resolution images of the eyes or face, predicting the gaze point on a two-dimensional screen by analyzing eye morphology (such as pupil position and iris features) or head posture. These methods perform well in controlled environments, such as when the user is close to the screen.

[0003] However, in unconstrained real-world scenarios, such as surveillance environments like shopping malls, exhibition halls, and streets, traditional gaze estimation methods face significant challenges, primarily due to the following issues: Scene and distance limitations: Traditional methods typically require the subject to be close to the camera to ensure clear facial and eye details are captured. In long-distance or large-scale surveillance scenarios, insufficient image resolution makes it difficult to extract facial and eye features, causing the method to fail.

[0004] Poor robustness to viewpoint and occlusion: When the camera viewpoint changes significantly (e.g., side shot, overhead shot), or when the subject's face is partially occluded (e.g., by hair, glasses, mask, or merchandise on a shelf), the performance of methods relying on specific facial feature points deteriorates sharply.

[0005] Insufficient utilization of 3D spatial information: Most methods treat gaze estimation as a 2D problem or simply use head pose as a 3D cue. They rarely effectively integrate the 3D spatial location information of other objects in the scene with the complete 3D pose information of the human body itself. This makes it difficult for the model to understand the true 3D spatial relationship between the person and objects in the scene, such as whether a person is "looking" at a distant object or a nearby object, which is difficult to distinguish with only 2D information, thus limiting the accuracy of gaze estimation.

[0006] Weak generalization ability: Existing methods are usually trained for specific scenarios or tasks, and are sensitive to environmental changes (such as lighting and background complexity). The models have weak generalization ability and are difficult to adapt to diverse and dynamically changing real-world application scenarios.

[0007] Therefore, how to overcome the dependence of traditional methods on clear, close-up facial images, effectively integrate multimodal 3D information of people and objects in a scene, and achieve robust and accurate 3D gaze direction prediction in unconstrained complex scenes has become a technical problem that urgently needs to be solved in this field.

[0008] This invention aims to achieve fast and accurate 3D gaze direction prediction in an unconstrained environment by integrating human posture and 3D information of scene objects, combined with geometric transformation and deep learning networks. This overcomes the dependence of traditional methods on clear close-up facial images and improves the robustness and generalization ability of the system in complex scenes. Summary of the Invention

[0009] This invention primarily addresses the technical problems of existing technologies, such as limited scene and distance constraints, poor robustness, insufficient utilization of 3D spatial information, and weak generalization ability. It provides a gaze estimation method based on multimodal 3D feature fusion. By fusing 3D information of human posture and scene objects, combined with geometric transformation and deep learning networks, it achieves fast and accurate 3D gaze direction prediction in unconstrained environments. This overcomes the dependence of traditional methods on clear close-up facial images and improves the robustness and generalization ability of the system in complex scenes.

[0010] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a gaze estimation method based on multimodal 3D feature fusion, comprising the following steps: S1: Multimodal 3D feature extraction: Extracting human 3D pose features, view orientation priors, and scene object 3D position features from RGB and depth images; RGB and depth images can be acquired by devices such as TOF cameras or binocular stereo vision cameras; S2: Egocentric Space Geometric Transformation: Transforms the extracted 3D features from the camera coordinate system to the egocentric space with the head as the origin, eliminating the influence of camera viewpoint differences; S3: Direction-Distance Separation Encoding: Separate encoding of 3D features by direction and distance to enhance the ability to express spatial relationships; S4: Interactive Modeling Based on Transformer: Interactive modeling of the 3D features of the human body and objects is performed using a Transformer network to predict the 3D gaze direction.

[0011] Preferably, in step S1, the extraction process of the human body's three-dimensional pose features is as follows: S111: Obtain the human body region I in an RGB image using an object detection algorithm (such as the YOLO series). body ; S112: Transfer human body region I body The input is fed into the pre-trained 3D human pose estimation model f pose (e.g., a network based on the SMPL model) outputs the coordinates P of several 3D key points of the human body. pose These coordinates are located in the camera coordinate system; the 3D keypoint coordinates include the head keypoint coordinates, which include the coordinates of the outer corners of the eyes. S113: The average coordinates of all head key points are used to obtain the head center position t. pose The head dimension s is determined by the reciprocal of the distance between the outer corners of the eyes (or other suitable head landmarks), i.e.: s = 1 / ||P pose,eye1 - P pose,eye2 ||; Among them, P pose,eye1 and P pose,eye2 These are the three-dimensional coordinates of the outer corners of the eyes.

[0012] Preferably, in step S1, the prior extraction process of the view direction is specifically as follows: Cropping the head region from an RGB image (I) head Then head region I head Input into a lightweight convolutional neural network f view (Such as the MobileNet series) Predict the head view direction v, represented by a unit vector, which is the approximate orientation of the head in the camera coordinate system.

[0013] Preferably, in step S1, the process of extracting the three-dimensional position features of the scene objects is as follows: S121: Object Detection and Mask Generation: A lightweight instance segmentation model (such as MobileSAM) is used to process RGB images to generate two-dimensional masks {M} of objects in the scene. i}; S122: 3D coordinate calculation: For each object mask M i Combining the depth image D and the camera intrinsic parameter matrix K, the 3D point cloud is calculated by back-projection from the pinhole camera model. The specific formula is as follows: P mask,i =K -1 ·(u,v,1) T ·d; Where (u,v) are the pixel coordinates within the mask, and d is the depth value of the corresponding pixel; S123: Take the median of the point cloud as the three-dimensional center coordinate P of the object. object,i The coordinates of all objects constitute a set P. object .

[0014] Preferably, step S2 specifically includes: S21: Coordinate system transformation objective: Transform the human body's 3D pose features, view orientation priors, and scene object 3D position features from the camera coordinate system to an egocentric space with the head as the origin, so that the model is not affected by changes in camera viewpoint; S22: Head Position Alignment: Calculate the head position t corresponding to the object. object Specifically, it is through the center of the head bounding box (u head ,vhead Combined with depth value d head It is obtained by back-projection using the camera intrinsic parameter matrix, i.e., t object =K -1 ·(u head ,v head ,1) T ·d head ; S23: Calculate the rotation matrix R, which is determined by the Euler angles (θ,φ,0). Optimize the objective function. The solution process is as follows: φ=arctan2(-v x ,v z ); θ=arctan2(v y ,v z cosφ- v x cosφ); In the formula, v x v y and v z These are the coordinate values ​​in the x, y, and z axes of the corresponding three-dimensional space; S24: Rotation Alignment: Using a rotation method based on Euler angles, the view direction v is aligned to the positive z-axis direction in self-space, thus obtaining the normalized view direction v′; S25: Human posture normalization, calculated using the following formula: P pose =s·R·(P pose -t pose ); Where s is the head size and R is the rotation matrix, the above formula converts the human posture into a coordinate representation centered on the head and in units of head size; S26: Object position normalization, the calculation formula is: P object =R·(P) object -t object ); The above formula transforms the object's position to an egocentric spatial coordinate system, which is the same as the human body's posture.

[0015] Preferably, step S3 specifically includes: View orientation code: E view =γ dir (v′), where v′ is the normalized view direction [0,0,1]; Human posture coding: E pose =γ′(P pose ′), encoding the normalized human body key points; Object position code: E object =γ′(P object ′), to encode the normalized object position; γ′ is a hybrid coding formula, obtained by concatenating directional coding and distance coding. The formula is: γ′=γ dir (p^)⊕ γ dist (d); The unit direction vector p^ and the distance scalar d are obtained by decomposing the three-dimensional point p into the unit direction vector p^=p / ||p|| and the distance scalar d=||p||. γ dir (p^) is the direction encoding function, which is a sine encoding of the direction angle (α,β) of the unit direction vector p^, and the formula is: γ dir (p^)=[sin(2 k α),cos(2 k α),sin(2 k β),cos(2 k β)] k=0 K-1 ;γ dir (v′) and so on; Where K is the number of coding layers, which is determined through experimental optimization or experience; γ dist (d) is the distance coding formula, which uses logarithmic sine coding. The formula is: γ dist =[sin((log(d+1)·2 k ) / δ),cos((log(d+1)·2 k ) / δ)] k=0 K ; Where δ is the distance scale adjustment parameter, which is determined through experimental optimization or experience.

[0016] Preferably, step S4 specifically involves: S41: Encode the object's position E object Input to N enc An object encoder composed of layers of Transformer encoders learns the spatial relationships between objects through a self-attention mechanism and outputs object features F. object ; S42: E encoded by human posture pose and view orientation encoding E view The main code E after splicing subject =E view ⊕E pose As a query, object feature F object As keys and values, input into Nenc The gaze decoder, composed of layered Transformer decoders, outputs the gaze direction residual g′ in the egocentric space through a cross-attention mechanism. S43: Transform the gaze direction residual g′ in egocentric space back to the camera coordinate system, using the following formula: g=R T ·g′+v; Where R T The final gaze direction is obtained by transposing the rotation matrix R, where v is the original view direction, and g is the 3D gaze direction in the camera coordinate system.

[0017] Preferably, the loss function of the Transformer network in the encoder-decoder structure adopts the angle error loss function, as shown in the formula: ; Where N is the sample size, g i For the predicted gaze direction, g GT,i To determine the true gaze direction, the network is trained by minimizing this loss function. This loss function is the loss function of the Transformer network with an encoder-decoder structure; it can also be considered the loss function of the entire gaze estimation system, as there is only this one loss function throughout the entire process.

[0018] The substantial effects of this invention are: 1. Strong robustness and wide applicability: By directly utilizing information such as the human body's 3D pose, head orientation, and the 3D position of objects in the scene, this invention eliminates the heavy reliance on high-definition, close-up facial or eye images found in traditional methods. Therefore, even in unconstrained scenarios where the subject is far away, the face is partially obscured (e.g., side profile, wearing a mask), lighting conditions are poor, or camera angles are variable, this invention can still perform stable and reliable gaze direction estimation, demonstrating strong robustness and greatly expanding the applicability of the technology. 2. High accuracy and deep utilization of 3D spatial relationships: This invention innovatively introduces egocentric spatial geometric transformation, unifying all 3D features into a coordinate system with the human head as the origin, effectively eliminating the influence of camera pose changes. More importantly, through Transformer-based interactive modeling, this invention can deeply learn and infer the 3D spatial relationships between the human body (subject) and various objects (objects) in the scene, thereby more accurately determining the gaze target. This has a significant accuracy advantage over 2D or quasi-3D methods that rely solely on head pose. 3. Excellent Generalization Ability and End-to-End Prediction: This invention proposes an end-to-end gaze estimation framework, where a unified deep learning network handles everything from multimodal feature extraction to final gaze direction prediction. This approach avoids the complex, staged post-processing steps of traditional methods, enabling the model to automatically learn complex intrinsic relationships from the data and achieving better generalization ability. Simultaneously, the innovative direction-distance separation encoding method enhances the model's ability to express spatial relationships, further improving its adaptability and predictive performance in different scenarios. 4. Computationally Efficient and Easy to Deploy: This invention employs lightweight neural network models in the feature extraction stage (such as view direction prediction and object segmentation), effectively controlling computational overhead while ensuring accuracy. The entire framework is simple in design and easy to deploy and implement on hardware platforms equipped with standard RGB and depth sensors, providing possibilities for real-time applications of the technology in fields such as intelligent security, human-computer interaction, and business analytics. Attached Figure Description

[0019] Figure 1 This is a flowchart of a gaze estimation method based on multimodal 3D feature fusion according to the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0021] Example: This example illustrates a gaze estimation method based on multimodal 3D feature fusion, such as... Figure 1 As shown, it includes the following steps: S1: Multimodal 3D feature extraction: Extracting human 3D pose features, view orientation priors, and scene object 3D position features from RGB and depth images; RGB and depth images can be acquired by devices such as TOF cameras or binocular stereo vision cameras; S2: Egocentric Space Geometric Transformation: Transforms the extracted 3D features from the camera coordinate system to the egocentric space with the head as the origin, eliminating the influence of camera viewpoint differences; S3: Direction-Distance Separation Encoding: Separate encoding of 3D features by direction and distance to enhance the ability to express spatial relationships; S4: Interactive Modeling Based on Transformer: Interactive modeling of the 3D features of the human body and objects is performed using a Transformer network to predict the 3D gaze direction.

[0022] In step S1, the extraction process of human three-dimensional pose features is specifically as follows: S111: Obtain the human body region I in an RGB image using an object detection algorithm (such as the YOLO series). body ; S112: Transfer human body region I body The input is fed into the pre-trained 3D human pose estimation model f pose (e.g., a network based on the SMPL model) outputs the coordinates P of several 3D key points of the human body. pose These coordinates are located in the camera coordinate system; the 3D keypoint coordinates include the head keypoint coordinates, which include the coordinates of the outer corners of the eyes. S113: The average coordinates of all head key points are used to obtain the head center position t. pose The head dimension s is determined by the reciprocal of the distance between the outer corners of the eyes (or other suitable head landmarks), i.e.: s = 1 / ||P pose,eye1 - P pose,eye2 ||; Among them, P pose,eye1 and P pose,eye2 These are the three-dimensional coordinates of the outer corners of the eyes.

[0023] In step S1, the prior extraction process of view direction is specifically as follows: Cropping the head region from an RGB image (I) head Then head region I head Input into a lightweight convolutional neural network f view (Such as the MobileNet series) Predict the head view direction v, represented by a unit vector, which is the approximate orientation of the head in the camera coordinate system.

[0024] In step S1, the process of extracting the three-dimensional position features of objects in the scene is as follows: S121: Object Detection and Mask Generation: A lightweight instance segmentation model (such as MobileSAM) is used to process RGB images to generate two-dimensional masks {M} of objects in the scene. i}; S122: 3D coordinate calculation: For each object mask M i Combining the depth image D and the camera intrinsic parameter matrix K, the 3D point cloud is calculated by back-projection from the pinhole camera model. The specific formula is as follows: P mask,i =K -1 ·(u,v,1) T ·d; Where (u,v) are the pixel coordinates within the mask, and d is the depth value of the corresponding pixel; S123: Take the median of the point cloud as the three-dimensional center coordinate P of the object. object,i The coordinates of all objects constitute a set P. object .

[0025] Step S2 specifically involves: S21: Coordinate system transformation objective: Transform the human body's 3D pose features, view orientation priors, and scene object 3D position features from the camera coordinate system to an egocentric space with the head as the origin, so that the model is not affected by changes in camera viewpoint; S22: Head Position Alignment: Calculate the head position t corresponding to the object. object Specifically, it is through the center of the head bounding box (u head ,v head Combined with depth value d head It is obtained by back-projection using the camera intrinsic parameter matrix, i.e., t object =K -1 ·(u head ,v head ,1) T ·d head ; S23: Calculate the rotation matrix R, which is determined by the Euler angles (θ,φ,0). Optimize the objective function. The solution process is as follows: φ=arctan2(-v x ,v z ); θ=arctan2(v y ,v z cosφ- v x cosφ); In the formula, v x v y and v zThese are the coordinate values ​​in the x, y, and z axes of the corresponding three-dimensional space; S24: Rotation Alignment: Using a rotation method based on Euler angles, the view direction v is aligned to the positive z-axis direction in self-space, thus obtaining the normalized view direction v′; S25: Human posture normalization, calculated using the following formula: P pose =s·R·(P pose -t pose ); Where s is the head size and R is the rotation matrix, the above formula converts the human posture into a coordinate representation centered on the head and in units of head size; S26: Object position normalization, the calculation formula is: P object =R·(P) object -t object ); The above formula transforms the object's position to an egocentric spatial coordinate system, which is the same as the human body's posture.

[0026] Step S3 specifically involves: View orientation code: E view =γ dir (v′), where v′ is the normalized view direction [0,0,1]; Human posture coding: E pose =γ′(P pose ′), encoding the normalized human body key points; Object position code: E object =γ′(P object ′), to encode the normalized object position; γ′ is a hybrid coding formula, obtained by concatenating directional coding and distance coding. The formula is: γ′=γ dir (p^)⊕ γ dist (d); The unit direction vector p^ and the distance scalar d are obtained by decomposing the three-dimensional point p into the unit direction vector p^=p / ||p|| and the distance scalar d=||p||. γ dir (p^) is the direction encoding function, which is a sine encoding of the direction angle (α,β) of the unit direction vector p^, and the formula is: γ dir (p^)=[sin(2 k α),cos(2 k α),sin(2 k β),cos(2 k β)] k=0 K-1 ;γ dir(v′) and so on; Where K is the number of coding layers, which is determined through experimental optimization or experience; γ dist (d) is the distance coding formula, which uses logarithmic sine coding. The formula is: γ dist =[sin((log(d+1)·2 k ) / δ),cos((log(d+1)·2 k ) / δ)] k=0 K ; Where δ is the distance scale adjustment parameter, which is determined through experimental optimization or experience.

[0027] Step S4 specifically involves: S41: Encode the object's position E object Input to N enc An object encoder composed of layers of Transformer encoders learns the spatial relationships between objects through a self-attention mechanism and outputs object features F. object ; S42: E encoded by human posture pose and view orientation encoding E view The main code E after splicing subject =E view ⊕E pose As a query, object feature F object As keys and values, input into N enc The gaze decoder, composed of layered Transformer decoders, outputs the gaze direction residual g′ in the egocentric space through a cross-attention mechanism. S43: Transform the gaze direction residual g′ in egocentric space back to the camera coordinate system, using the following formula: g=R T ·g′+v; Where R T The final gaze direction is obtained by transposing the rotation matrix R, where v is the original view direction, and g is the 3D gaze direction in the camera coordinate system.

[0028] The loss function of the Transformer network with an encoder-decoder structure adopts the angle error loss function, and the formula is: ; Where N is the sample size, g i For the predicted gaze direction, g GT,i To determine the true gaze direction, the network is trained by minimizing this loss function.

[0029] The following section uses customer product attention analysis in a smart retail scenario to illustrate this solution.

[0030] Input data: RGB image: resolution 1280*720, frame rate 30fps. Depth image: acquired by a ToF depth sensor, resolution matched to the RGB image. Camera intrinsic parameter matrix. .

[0031] Processing flow: Data Acquisition and Preprocessing: Using YOLOv8 to detect human bodies and heads, MobileSAM segments the products on the shelf to obtain masks.

[0032] 3D feature extraction: f pose Estimate 15 3D key points of the customer's body and calculate the head position t. pose = (1.5, 1.2, 2.0), head scale s = 0.12. All coordinate units are meters.

[0033] f view Predict the view direction v=[0.2,0.1,0.97].

[0034] The three-dimensional position is obtained by back-projecting the product mask, such as the coordinates of product A (2.0, 1.3, 2.5).

[0035] Geometric Transformations and Coding: Calculate the rotation matrix R to align the view orientation to the z-axis, and obtain the coordinates P of item A in egocentric space. object，A = (0.5, 0.1, 0.8).

[0036] D-factor analysis of view orientation, human posture, and object position. 2 E-encoding yields the encoded vector.

[0037] Gaze prediction: The Transformer network outputs a gaze direction residual g′=[0,0,1]. After transformation to the camera coordinate system, g=[0.18,0.08,0.98], indicating that the customer is looking at product A.

[0038] By analyzing how long and where customers gaze at different products, we can optimize product display layout and improve sales conversion rates.

[0039] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0040] Although this paper makes extensive use of terms such as human 3D pose features, view orientation prior, and rotation matrix, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A gaze estimation method based on multimodal 3D feature fusion, characterized in that, Includes the following steps: S1: Multimodal 3D feature extraction: Extracting human 3D pose features, view orientation priors, and scene object 3D position features from RGB and depth images; S2: Egocentric Space Geometric Transformation: Transforms the extracted 3D features from the camera coordinate system to the egocentric space with the head as the origin; S3: Orientation-Distance Separation Encoding: Separate encoding of 3D features using orientation and distance; S4: Interactive Modeling Based on Transformer: Interactive modeling of the 3D features of the human body and objects is performed using a Transformer network to predict the 3D gaze direction.

2. The gaze estimation method based on multimodal 3D feature fusion according to claim 1, characterized in that, In step S1, the extraction process of human three-dimensional pose features is specifically as follows: S111: Obtain the human body region I from the RGB image using a target detection algorithm. body ; S112: Transfer human body region I body The input is fed into the pre-trained 3D human pose estimation model f pose Output the coordinates P of several 3D key points of the human body. pose The three key point coordinates include the head key point coordinates, which in turn include the coordinates of the outer corners of the eyes. S113: The average coordinates of all head key points are used to obtain the head center position t. pose The head size s is determined by the reciprocal of the distance between the outer corners of the eyes, i.e.: s = 1 / ||P pose,eye1 - P pose,eye2 ||; Among them, P pose,eye1 and P pose,eye2 These are the three-dimensional coordinates of the outer corners of the eyes.

3. The gaze estimation method based on multimodal 3D feature fusion according to claim 2, characterized in that, In step S1, the prior extraction process of view direction is specifically as follows: Cropping the head region from an RGB image (I) head Then head region I head Input into a lightweight convolutional neural network f view Predict the head view orientation v, represented as a unit vector.

4. The gaze estimation method based on multimodal 3D feature fusion according to claim 3, characterized in that, In step S1, the process of extracting the three-dimensional position features of objects in the scene is as follows: S121: Object Detection and Mask Generation: A lightweight instance segmentation model is used to process RGB images and generate two-dimensional masks {M} of objects in the scene. i }; S122: 3D coordinate calculation: For each object mask M i Combining the depth image D and the camera intrinsic parameter matrix K, the 3D point cloud is calculated by back-projection from the pinhole camera model. The specific formula is as follows: P mask,i =K -1 ·(u,v,1) T ·d; Where (u,v) are the pixel coordinates within the mask, and d is the depth value of the corresponding pixel; S123: Take the median of the point cloud as the three-dimensional center coordinate P of the object. object,i The coordinates of all objects constitute a set P. object .

5. The gaze estimation method based on multimodal 3D feature fusion according to claim 4, characterized in that, Step S2 specifically involves: S21: Coordinate system transformation objective: Transform the human body's 3D pose features, view orientation priors, and scene object 3D position features from the camera coordinate system to an egocentric space with the head as the origin; S22: Head Position Alignment: Calculate the head position t corresponding to the object. object Specifically, it is through the center of the head bounding box (u head ,v head Combined with depth value d head It is obtained by back-projection using the camera intrinsic parameter matrix, i.e., t object =K -1 ·(u head ,v head ,1) T ·d head ; S23: Calculate the rotation matrix R, which is determined by the Euler angles (θ,φ,0). Optimize the objective function. The solution process is as follows: φ=arctan2(-v x ,in z ); θ=arctan2(v y ,in z cosφ-v x cosφ); In the formula, v x v y and v z These are the coordinate values ​​in the x, y, and z axes of the corresponding three-dimensional space; S24: Rotation Alignment: Using a rotation method based on Euler angles, the view direction v is aligned to the positive z-axis direction in self-space, thus obtaining the normalized view direction v′; S25: Human posture normalization, calculated using the following formula: P pose ′=s·R·(P pose -t pose ); Where s is the head size and R is the rotation matrix, the above formula converts the human posture into a coordinate representation centered on the head and in units of head size; S26: Object position normalization, the calculation formula is: P object ′=R·(P object -t object ); The above formula transforms the object's position to an egocentric spatial coordinate system, which is the same as the human body's posture.

6. The gaze estimation method based on multimodal 3D feature fusion according to claim 5, characterized in that, Step S3 specifically involves: View orientation code: E view =γ dir (v′), where v′ is the normalized view direction [0,0,1]; Human posture coding: E pose =γ′(P pose ′), encoding the normalized human body key points; Object position code: E object =γ′(P object ′), to encode the normalized object position; γ′ is a hybrid coding formula, obtained by concatenating directional coding and distance coding. The formula is: γ′=γ dir (p^) ⊕γ dist (d); The unit direction vector p^ and the distance scalar d are obtained by decomposing the three-dimensional point p into the unit direction vector p^=p / ||p|| and the distance scalar d=||p||. γ dir (p^) is the direction encoding function, which is a sine encoding of the direction angle (α,β) of the unit direction vector p^, and the formula is: γ dir (p^)=[sin(2 k α),cos(2 k α),sin(2 k β),cos(2 k β)] k=0 K-1 ;γ dir (v′) and so on; Where K is the number of coding layers; γ dist (d) is the distance coding formula, which uses logarithmic sine coding. The formula is: c dist =[sin((log(d+1)·2 k ) / δ),cos((log(d+1)·2 k ) / d)] k=0 K ; Where δ is the distance scale adjustment parameter.

7. The gaze estimation method based on multimodal 3D feature fusion according to claim 6, characterized in that, Step S4 specifically involves: S41: Encode the object's position E object Input to N enc An object encoder composed of layers of Transformer encoders learns the spatial relationships between objects through a self-attention mechanism and outputs object features F. object ; S42: E encoded by human posture pose and view orientation encoding E view The main code E after splicing subject =E view ⊕E pose As a query, object feature F object As keys and values, input into N enc The gaze decoder, composed of layered Transformer decoders, outputs the gaze direction residual g′ in the egocentric space through a cross-attention mechanism. S43: Transform the gaze direction residual g′ in egocentric space back to the camera coordinate system, using the following formula: g=R T ·g′+v; Where R T Let v be the transpose of the rotation matrix R, v be the original view direction, and g be the 3D gaze direction in the camera coordinate system.

8. The gaze estimation method based on multimodal 3D feature fusion according to claim 7, characterized in that, The loss function of the Transformer network with an encoder-decoder structure adopts the angle error loss function, and the formula is: ； Where N is the sample size, g i For the predicted gaze direction, g GT,i To determine the true gaze direction, the network is trained by minimizing this loss function.

Images