An attention mechanism-based multi-modal gaze target estimation method
By using a multimodal gaze target estimation method based on an attention mechanism, the problems of insufficient deep channel encoding and insufficient modal feature interaction in traditional methods are solved, and higher gaze target estimation accuracy and behavior understanding ability are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI ZHONGJUYUAN INTELLIGENT TECH CO LTD
- Filing Date
- 2023-06-19
- Publication Date
- 2026-07-14
AI Technical Summary
Traditional methods fail to effectively encode the depth channel in gaze direction estimation and lack modal feature interaction, leading to errors in gaze target estimation and out-of-scene prediction, thus affecting accuracy.
A multimodal gaze target estimation method based on attention mechanism is adopted. Features are extracted by sharing a backbone network, and feature enhancement module and cross-attention module are used for feature interaction. Text feature encoding is performed by combining BERT-base model, and a loss function is calculated to improve estimation accuracy.
It improves the accuracy of gaze target estimation, enables a better understanding of human behavior and intentions in a scene, and enhances the predictive capabilities of computer vision systems.
Smart Images
Figure CN116682049B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, and specifically relates to a multimodal gaze target estimation method based on an attention mechanism. Background Technology
[0002] Traditional methods take cropped head images and scene images as input, utilizing head pose features and saliency maps of potential gazing targets for prediction. These methods have the following drawbacks:
[0003] 1. Traditional methods explore gaze direction in 2D representations, with almost no encoding of the depth channel. However, there may be multiple candidate objects at different depths in the gaze direction of a subject, which can easily lead to incorrect estimates.
[0004] 2. Although some methods take depth images into account, they only perform simple feature stitching and do not enable the features of the two modalities to interact to a certain extent, thus preventing the modalities from learning from each other.
[0005] 3. When the object being gazed at is outside the scene, the model often gives an incorrect prediction that the object is inside the scene. The lack of a judgment on whether the object is inside or outside the scene affects the accuracy of the object estimation.
[0006] Therefore, it is necessary to invent a multimodal gaze target estimation method based on attention mechanism to solve the above problems. Summary of the Invention
[0007] To address the aforementioned problems, this invention provides a multimodal gaze target estimation method based on an attention mechanism, thereby resolving the issues raised in the background section.
[0008] To achieve the above objectives, the present invention provides the following technical solution: a multimodal gaze target estimation method based on an attention mechanism, comprising the following steps:
[0009] S1. Scene images are captured through a camera to obtain a two-dimensional scene image S and a depth image D, and the head image H and head position of the target person are acquired simultaneously.
[0010] S2. Features of the two-dimensional scene image S, depth image D, and head image H are extracted respectively through a shared backbone network to obtain scene features f. s Depth features f d and head features f h ;
[0011] S3. Obtain the positional features p of the head location through multiple embedding layers. h , location feature p hScene features f s and head features f h The enhanced scene features F are obtained by feeding them into the feature enhancement module. s , location feature p h Depth features f d and head features f h The enhanced deep features F are obtained by feeding them into the feature enhancement module. d ;
[0012] S4, Enhance scene features F s and enhanced deep features F d The input is fed into the cross-attention module to perform feature interaction, resulting in enhanced features after interaction.
[0013] S5. After processing the enhanced features through the interaction of the three output heads, calculate the design loss function to obtain the target prediction model. The video captured by the camera is preprocessed and input into the target prediction model to obtain the prediction result.
[0014] Furthermore, this also includes encoding text features by selecting the BERT-base model as the backbone network, where the text features f t Text features f t With enhanced scene features F s Feature interaction is performed through a cross-attention module to obtain the fused feature F. ts and will fuse feature F ts The enhanced features after interaction are fused together to obtain the fused enhanced features after interaction.
[0015] Furthermore, S4 specifically includes:
[0016] Enhance scene features F s and enhanced deep features F d After each image is fed into a Transformer Encoder, the image feature keys and values of the two images are swapped and then processed by a Transformer Encoder again. Finally, the resulting features are concatenated to output the enhanced features after interaction.
[0017] Furthermore, the calculation process for the enhanced features after the interaction is as follows:
[0018] Enhance scene features F s and enhanced deep features F d Each is fed into the Transformer Encoder:
[0019] Equation (1) is obtained:
[0020] ;
[0021] Swap the key and value of both and then perform another Transformer Encoder operation:
[0022] Equation (2) is obtained:
[0023] ;
[0024] The obtained features are then concatenated:
[0025] Equation (3) is obtained:
[0026] .
[0027] Furthermore, the enhanced features processed through the three output heads after interaction include:
[0028] The first output head is set as the target detection module, used to detect objects within the scene;
[0029] The second output head is set as the gaze target estimation module, which is used to output gaze heatmaps through the codec;
[0030] The third output head is set as an encoder, used to output binary classification judgments (In:0; Out:1) for attention falling on the scene's inward / outward focus.
[0031] Furthermore, the calculation of the design loss function includes:
[0032] Equation (4): ;
[0033] in, This represents the total model loss of the target prediction model. This represents the loss detected by the target detection module. This represents the mean squared error loss of the gaze heatmap. This represents the weighted binary cross-entropy loss of the In / Out output. This represents the cross-loss between the gaze heatmap and the ground truth of the target bounding box. , , and All of these are learnable weights.
[0034] Furthermore, the calculation of the design loss function also includes:
[0035] Equation (5):
[0036] ;
[0037] in, This indicates the number of valid pixels in the gaze heatmap that fall within the target bounding box. This indicates the total number of pixels within the target bounding box.
[0038] Furthermore, equation (4) includes:
[0039] Equation (6):
[0040] ;
[0041] and ;
[0042] Among them, category weight The training set consists of two categories of labels: In and Out. proportion It is calculated that For the label truth value, For tags The predicted probability.
[0043] The technical effects and advantages of this invention are as follows:
[0044] This invention guides model training through text features, guides gaze estimation through the positional information of the target bounding box, and enables full interaction between the depth information and two-dimensional texture information of the scene, thereby improving the accuracy of gaze target estimation. It can help computer vision systems better understand what people do and their intentions in a scene and predict human behavior in various scenarios. Attached Figure Description
[0045] Figure 1 This is a flowchart of the multimodal gaze target estimation method based on the attention mechanism according to an embodiment of the present invention;
[0046] Figure 2 This is a block diagram of the multimodal gaze target estimation method based on the attention mechanism according to an embodiment of the present invention;
[0047] Figure 3 This is the enhanced feature block diagram obtained by the multimodal feature interaction enhancement module in this embodiment of the invention after interaction;
[0048] Figure 4 This is a diagram illustrating the characters, line of sight, and target in an embodiment of the present invention. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments.
[0050] This invention provides a multimodal gaze target estimation method based on an attention mechanism, comprising the following steps: mainly including five modules: a multimodal data acquisition module, a feature extraction module, a single-modal feature enhancement module, a multimodal feature interaction enhancement module, and a gaze target prediction module.
[0051] I. Multimodal Data Acquisition Module:
[0052] like Figure 1 As shown:
[0053] S1. Scene images are captured using a camera to obtain a two-dimensional scene image S and a depth image D, and the head image H and head position of the target person are acquired simultaneously.
[0054] For example, an RGB-D camera is used to acquire scene images, resulting in a two-dimensional scene image S and a depth image D. The open-source face detection model ArcFace is then used to obtain the head image H of the target person and the head position.
[0055] After acquiring the images, add corresponding text annotations and corresponding box labels (X1, Y1, X2, Y2) to all images, where (x1, y1) and (x2, y2) are the coordinates of the top left and bottom right corners, respectively.
[0056] II. Feature Extraction Module:
[0057] like Figure 1 and Figure 2 As shown:
[0058] S2. Features of the two-dimensional scene image S, depth image D, and head image H are extracted respectively through a shared backbone network to obtain scene features f. s Depth features f d and head features f h .
[0059] When extracting features, two extraction methods are used to extract features from the image and text respectively, mainly including:
[0060] For images: ResNet50 is used as the shared backbone network, and ResNet50 has been pre-trained on the large image classification dataset ImageNet. Utilizing the shared backbone network facilitates feature extraction from multiple modalities, thereby reducing the number of parameters in the overall model.
[0061] Feature extraction of images is performed using a shared backbone network to obtain scene features f. s Depth features f d and head features f h .
[0062] For text: The text features are encoded using the BERT-base model as the backbone network, and the text features f t Text features f t With enhanced scene features F sFeature interaction is performed through a cross-attention module to obtain the fused feature F. ts and will fuse feature F ts The enhanced features after interaction are fused together to obtain the fused enhanced features after interaction.
[0063] III. Single-modal feature enhancement module:
[0064] like Figure 1 and Figure 2 As shown:
[0065] S3. Obtain the positional features p of the head location through multiple embedding layers. h , location feature p h Scene features f s and head features f h The enhanced scene features F are obtained by feeding them into the feature enhancement module. s , location feature p h Depth features f d and head features f h The enhanced deep features F are obtained by feeding them into the feature enhancement module. d By using a feature enhancement module to enhance depth features, interference from depth-mismatched objects can be reduced, thereby improving the accuracy of gaze prediction.
[0066] For example, the feature enhancement module adopts the Gaze Prediction module from the "GaTector: A Unified Framework for GazeObject Prediction" model, and uses the Gaze Prediction module to obtain the enhanced scene features F. s and enhanced deep features F d .
[0067] IV. Multimodal Feature Interaction Enhancement Module:
[0068] like Figures 1 to 3 As shown:
[0069] S4. Input the enhanced scene feature Fs and the enhanced depth feature Fd into the cross-attention module to perform feature interaction and obtain the enhanced features after interaction.
[0070] During the training phase, text features are used to guide the model to focus on salient regions. Specifically, an early fusion approach is employed, using text features as the query and image features as the key. A self-attention mechanism is used to calculate the correlation between text and image, obtaining attention weights (values), and then weighted fusion is applied to the image features to improve the representational ability of visual features. The use of a cross-attention module helps the target prediction model better understand the relationships between different modalities, enabling scene and depth modalities to learn from each other, thereby improving the accuracy of gaze prediction.
[0071] The enhanced scene features Fs and enhanced depth features Fd are each fed into the Transformer Encoder.
[0072] Then, the key and value of the two are swapped and another Transformer Encoder is performed, which enhances the features by exchanging scene cues and depth cues.
[0073] Equation (1) is obtained:
[0074] ;
[0075] Swap the key and value of the two and then perform a Transformer Encoder again;
[0076] Equation (2) is obtained:
[0077] ;
[0078] The obtained features are then concatenated:
[0079] Equation (3) is obtained:
[0080] .
[0081] V. Fixation Target Prediction Module:
[0082] like Figures 1 to 4 As shown:
[0083] S5. After processing the enhanced features through the interaction of the three output heads, calculate the design loss function to obtain the target prediction model. The video captured by the camera is preprocessed and input into the target prediction model to obtain the prediction result.
[0084] When text features need to be input into the target prediction model, the fused and enhanced features after interaction must first be processed through three output heads. The target prediction model is then trained using these text features. After the target prediction model is trained, no more text input is needed. The video captured by the RGB-D camera is preprocessed and input into the target prediction model to obtain the prediction results, which are then displayed on the screen in real time, showing the person, gaze, and target. At this point, the enhanced features after interaction are processed through the three output heads, and the loss function is calculated to obtain the target prediction model.
[0085] Furthermore, employing text-image fusion during the training phase can promote visual modal feature learning, improve the representational ability of visual modal features, enhance model training performance, and thus improve the accuracy of gaze prediction. Using an object detection module to assist the object prediction model can further improve the accuracy of gaze prediction.
[0086] Specifically, the three output heads are the first output head, the second output head, and the third output head.
[0087] The first output head is configured as an object detection module, using a YOLOv4 detector to detect objects within the scene. The second output head is configured as a gaze target estimation module, used to output a gaze heatmap via an encoder and decoder. The third output head is configured as an encoder, used to output a binary classification judgment (In:0; Out:1) indicating whether attention falls on objects in or out of the scene.
[0088] The calculation of the design loss function includes:
[0089] Equation (4): ;
[0090] in, This represents the total model loss of the target prediction model. This represents the loss detected by the target detection module. This represents the mean squared error loss of the gaze heatmap. This represents the weighted binary cross-entropy loss of the In / Out output. This represents the cross-loss between the gaze heatmap and the ground truth of the target bounding box. , , and All of these are learnable weights.
[0091] Total model loss YOLOv4 object detection loss Add the mean squared error (MSE) loss of the gaze heatmap In addition, the weighted binary cross-entropy (WBCE) loss of the In / Out outputs is also included. Furthermore, to guide gaze estimation using the ground truth of the target bounding map, a cross-loss between the gaze heatmap and the ground truth of the target bounding map was designed. .
[0092] The calculation of the design loss function also includes:
[0093] Equation (5):
[0094] ;
[0095] in, This indicates the number of valid pixels in the gaze heatmap that fall within the target bounding box. This indicates the total number of pixels within the target bounding box.
[0096] Furthermore, equation (4) includes:
[0097] Equation (6):
[0098] ;
[0099] and ;
[0100] Among them, category weight The training set consists of two categories of labels: In and Out. proportion It is calculated that For the label truth value, For tags The predicted probability.
[0101] The loss function fully utilizes two types of outputs to guide model training, including class weights. This solves the problem of uneven distribution of in / out labels in samples, resulting in higher accuracy in fixation prediction.
[0102] This invention proposes a multimodal gaze target estimation method based on an attention mechanism. It uses text features to guide model training and the positional information of the target bounding box to guide gaze estimation. Furthermore, it enables full interaction between the depth information and two-dimensional texture information of the scene, thereby improving the accuracy of gaze target estimation. This method can help computer vision systems better understand what people are doing and their intentions in a scene, and predict human behavior in various scenarios.
[0103] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it.
Claims
1. A multimodal gaze target estimation method based on attention mechanism, characterized in that: Includes the following steps: S1. Scene images are captured through a camera to obtain a two-dimensional scene image S and a depth image D, and the head image H and head position of the target person are acquired simultaneously. S2. Features of the two-dimensional scene image S, depth image D, and head image H are extracted respectively through a shared backbone network to obtain scene features f. s Depth features f d and head features f h ; S3. Obtain the positional features p of the head location through multiple embedding layers. h , location feature p h Scene features f s and head features f h The enhanced scene features F are obtained by feeding them into the feature enhancement module. s , location feature p h Depth features f d and head features f h The enhanced deep features F are obtained by feeding them into the feature enhancement module. d ; S4, Enhance scene features F s and enhanced deep features F d The input is fed into the cross-attention module to perform feature interaction, resulting in enhanced features after interaction. S5. The enhanced features after interaction are processed through three output heads, the design loss function is calculated, and the target prediction model is obtained. The video captured by the camera is preprocessed and input into the target prediction model to obtain the prediction result. The calculation of the design loss function includes: Equation (4): ; in, This represents the total model loss of the target prediction model. This represents the loss detected by the target detection module. This represents the mean squared error loss of the gaze heatmap. This represents the weighted binary cross-entropy loss of the In / Out output. This represents the cross-loss between the gaze heatmap and the ground truth of the target bounding box. , , and All are learnable weights; The calculation of the design loss function also includes: Equation (5): ； in, This indicates the number of valid pixels in the gaze heatmap that fall within the target bounding box. This indicates the total number of pixels within the target bounding box.
2. The multimodal gaze target estimation method based on attention mechanism according to claim 1, characterized in that: This also includes encoding text features by selecting the BERT-base model as the backbone network to obtain text features f. t Text features f t With enhanced scene features F s Feature interaction is performed through a cross-attention module to obtain the fused feature F. ts and will fuse feature F ts The enhanced features after interaction are fused together to obtain the fused enhanced features after interaction.
3. The multimodal gaze target estimation method based on attention mechanism according to claim 1, characterized in that: In S4, specifically including: Enhance scene features F s and enhanced deep features F d After each image is fed into a Transformer Encoder, the image feature keys and values of the two images are swapped and then processed by a Transformer Encoder again. Finally, the resulting features are concatenated to output the enhanced features after interaction.
4. The multimodal gaze target estimation method based on attention mechanism according to claim 1, characterized in that: The enhanced features processed through the interaction via three output heads include: The first output head is set as the target detection module, used to detect objects within the scene; The second output head is set as the gaze target estimation module, which is used to output gaze heatmaps through the codec; The third output head is set as an encoder, used to output binary classification judgments (In:0; Out:1) for attention falling on the scene's inward / outward focus.
5. The multimodal gaze target estimation method based on attention mechanism according to claim 4, characterized in that: Equation (4) includes: Equation (6): ； and ; Among them, category weight The training set consists of two categories of labels: In and Out. proportion It is calculated that For the label truth value, For tags The predicted probability.