Hierarchical supervision based cross-modal gaze target estimation method, system and medium

Abstract

The application discloses a cross-modal gaze target estimation method and system based on hierarchical supervision and a medium, relates to the technical field of computer vision, and inputs a target person in an acquired RGB image into a gaze target model after adding face frame coordinate labeling to the target person, so as to generate a corresponding gaze target estimation heat map; the training process of the gaze target model is as follows: a training set is constructed, original visual features and original text features are obtained through feature extraction, a face mask and reinforced visual features are generated at the same time, the original text features and the reinforced visual features are encoded and spliced, the spliced feature sequence obtained through splicing is input into a cross-modal Transformer, the reinforced visual features are connected with visual feature residuals output by the cross-modal Transformer, and a gaze target estimation heat map is generated; hierarchical supervision loss is set to optimize model parameters; and the method improves target estimation precision.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system and medium for cross-modal gaze target estimation based on hierarchical supervision. Background Technology

[0002] Gaze target estimation is a crucial research topic in computer vision and human-computer interaction. Its core objective is to accurately infer the focal point of human gaze by analyzing scene images or video frames, providing key technical support for applications such as intelligent interaction systems, behavior analysis, and attention computation. For example, in retail store monitoring scenarios, it's necessary to determine the specific product area a customer is looking at; in psychological diagnostic aid scenarios, it's necessary to identify the toy or person a child is currently focusing on; and in multi-person attention analysis scenarios, it's also necessary to distinguish the true gaze target of different individuals in complex scenes. All of these scenarios place higher demands on models in terms of complex backgrounds, multi-target interference, and semantic understanding capabilities. With the continuous evolution of deep learning, gaze target estimation methods have gradually developed from early designs relying on single visual cues to complex architectures that integrate multiple branches and modalities. Current mainstream solutions can be divided into three categories:

[0003] The dual-stream architecture-based approach processes scene understanding and head features through two independent data streams. One stream encodes global scene information, while the other extracts facial features from the detected head region, fundamentally fulfilling the core requirements of "localizing individuals" and "learning scene features." However, this type of method relies solely on visual appearance information and lacks the utilization of multi-dimensional auxiliary cues, making it difficult to achieve optimal performance in complex scenes.

[0004] Multi-branch fusion methods extend the two-stream architecture by capturing diverse auxiliary information through dedicated branches to enhance model performance and interpretability. Common branches include head branches focusing on facial features, scene understanding branches encoding background information, depth estimation branches modeling 3D spatial structures, and pose branches utilizing human pose information. As the number of modal branches increases, multi-modal fusion strategies become crucial for performance improvement, but they also face challenges such as diminishing marginal returns on performance and increased data acquisition costs.

[0005] A streamlined approach based on a visual base model addresses the complexity of multi-branch architectures by constructing a lightweight architecture using a pre-trained base visual model, simplifying model design while maintaining strong task performance. However, such streamlined designs still face challenges such as insufficient modeling of visual information and inadequate focus on task semantics, limiting their generalization ability in complex scenarios.

[0006] Existing simplified methods based on visual base models (represented by the GazeLLE model, source: Ryan F, Bati A, Lee S, et al. Gaze-lle: Gaze target estimation via large-scale learned encoders[C] / / Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 28874-28884.) have the following core technical path: extracting visual features from data images using a base visual model, processing the features and inputting them into a Transformer decoder; capturing the global dependencies between visual features and strengthening the feature weights of key regions; and using a regression head or heatmap head to output the coordinates of the gaze target or a probability heatmap to complete the gaze estimation.

[0007] Disadvantages of existing gaze methods:

[0008] Multi-branch architecture suffers from an imbalance in semantic compensation costs: it requires additional dedicated supervision, increases annotation costs and deployment complexity, has weak domain migration capabilities, can only perform geometric correction, and cannot solve the problem of insufficient semantic representation.

[0009] Pure appearance model-driven semantic defocus: lacking semantic and intent priors, relying on low-level visual shortcuts, resulting in a disconnect between prediction and true semantics, and misjudging the gaze target.

[0010] Low feature fusion efficiency: The integration of auxiliary visual features often adopts simple splicing at the decision layer or shallow feature fusion, without achieving deep interaction between modalities.

[0011] Lack of targeted supervision mechanisms: Existing technologies only optimize the model through the final output loss and do not design supervision strategies for the feature fusion process, which easily leads to modal information imbalance and uncontrollable fusion results. Summary of the Invention

[0012] Based on the technical problems existing in the background technology, this invention proposes a cross-modal staring target estimation method, system and medium based on hierarchical supervision, which improves the accuracy of target estimation.

[0013] The proposed method for cross-modal gaze target estimation based on hierarchical supervision adds face bounding box coordinate annotations to the target person in the acquired RGB image and inputs them into the gaze target model to generate a corresponding gaze target estimation heatmap.

[0014] The training process for the gaze target model is as follows:

[0015] Construct a training set containing face gaze images and text descriptions;

[0016] Feature extraction is performed on the training set to obtain the original visual features and original text features, and a face mask is generated.

[0017] The enhanced visual features are obtained by multiplying the face mask element-wise with the original visual features. The original text features and the enhanced visual features are then encoded and concatenated.

[0018] The stitched feature sequence is input into the cross-modal Transformer, and the enhanced visual features are concatenated with the visual feature residuals output by the cross-modal Transformer to generate a gaze target estimation heatmap.

[0019] The mean squared error between the text tokens output by each layer of the cross-modal Transformer and the original text features is calculated. The mean squared errors of all layers are weighted and summed as the hierarchical supervision loss to optimize the model parameters.

[0020] Furthermore, a training set containing facial gaze images and text descriptions is constructed, specifically as follows:

[0021] Add face bounding box coordinates to the target person in the acquired RGB image to obtain a staring face image;

[0022] The facial gaze image is used as input to the visual large language model, and combined with the set general prompt words, a text description is generated based on the predicted appearance, actions, and gaze direction of the target person.

[0023] Furthermore, the step of extracting features from the training set to obtain the original visual features and original text features specifically involves:

[0024] The size of the staring face image is adjusted and normalized to obtain the preprocessed image features;

[0025] Text exceeding the upper length threshold in the text description is truncated, and text below the lower length threshold is padded. At the same time, a text mask is generated for the padded text to obtain the preprocessed text features.

[0026] After extracting features from the preprocessed image using a pre-trained model with replaceable visual self-supervised representations of frozen parameters, the original visual features are obtained through convolutional projection.

[0027] After extracting features from the preprocessed text using a parameter-frozen, replaceable pre-trained language model, the original text features are obtained by projection through a linear layer.

[0028] Furthermore, the specific steps of encoding and concatenating the original text features and enhanced visual features are as follows:

[0029] Learnable positional encoding is applied to the original text features;

[0030] The enhanced visual features are encoded using fixed 2D sinusoidal position coding.

[0031] Furthermore, during the generation of the spliced ​​feature sequence, a learnable field-of-view token is set, which is used to determine whether the gaze target is within the image's field of view.

[0032] By sequentially concatenating the in-field and out-of-field tokens, the encoded original text features, and the encoded enhanced visual features, a concatenated feature sequence is obtained.

[0033] Furthermore, the spliced ​​feature sequence obtained is input into the cross-modal Transformer, and the features output by the cross-modal Transformer for the inside and outside of the field of view tokens are used as the input of the field of view prediction head, so as to output the judgment result of whether the staring target is inside or outside the field of view.

[0034] Furthermore, the process for generating the determination result of whether the view is inside or outside the field of vision is as follows:

[0035] The features of the tokens inside and outside the field of vision are passed through the cross-modal Transformer and then through a linear layer, a ReLU activation layer, a Dropout layer, a linear layer, and a Sigmoid activation layer. The output is a probability value of 0-1. When the probability value is greater than a preset probability threshold, it means that the gaze target is inside the field of vision; otherwise, the gaze target is outside the field of vision.

[0036] Furthermore, the process of generating the gaze target estimation heatmap is as follows:

[0037] The enhanced visual features are concatenated with the visual feature residuals output by the cross-modal Transformer and then passed through a transposed convolutional layer, a convolutional layer, and an activation layer in sequence to output a probability heatmap, which serves as the gaze target estimation heatmap.

[0038] A computer system includes a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the method described above.

[0039] A computer-readable storage medium storing a plurality of classification programs, the plurality of classification programs being invoked by a processor to execute the method described above.

[0040] The advantages of the hierarchical supervision-based cross-modal gaze target estimation method, system, and medium provided by this invention are as follows: Firstly, text description is introduced, and based on large language model technology, text description can be easily obtained, thereby constructing a "visual-text" bimodal gaze target estimation framework, fully utilizing the semantics of text description to supplement visual information. Secondly, a deep fusion mechanism is designed based on the cross-modal Transformer architecture to achieve fine-grained interaction between text and vision, overcoming the limitations of shallow multimodal fusion in gaze target estimation tasks. Thirdly, a hierarchical supervision strategy based on a large model is proposed, using a BERT-base-uncased model as a pre-trained language model to generate original text features, constraining the effectiveness of text and vision during the fusion process, avoiding modal information imbalance, and ensuring estimation accuracy. Furthermore, this method only requires ordinary RGB camera data (i.e., face gaze images) and the target person's face coordinates as input, while the text description based on the visual large language model can be personalized according to specific scene needs, improving the generalization and applicability of the technology in diverse scenarios. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the module flow of the present invention;

[0042] Figure 2 This is a schematic diagram of the training process for the gaze target model. Detailed Implementation

[0043] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] like Figure 1 and Figure 2 As shown, the cross-modal gaze target estimation method based on hierarchical supervision proposed in this invention adds face bounding box coordinate annotations to the target person in the acquired RGB image and inputs them into the gaze target model to generate the corresponding gaze target estimation heatmap.

[0045] The training process for the gaze target model is as follows:

[0046] Step 1: Construct a training set containing face gaze images and text descriptions; where the text description cues are the text descriptions that the visual big language model understands under RGB images, and the face gaze images are the images of the target person in the acquired RGB images after adding face bounding box coordinate annotations;

[0047] Step 2: Extract features from the training set to obtain the original visual features and original text features, and generate face masks and enhanced visual features;

[0048] Step 3: Multiply the face mask element-wise with the original visual features to obtain the enhanced visual features. Then, encode the original text features and the enhanced visual features and concatenate them.

[0049] Step 4: Input the stitched feature sequence into the cross-modal Transformer, connect the enhanced visual features with the visual feature residuals output by the cross-modal Transformer, and generate a gaze target estimation heatmap.

[0050] Step 5: Calculate the mean squared error between the text tokens output by each layer of the cross-modal Transformer and the original text features. The mean squared errors of all layers are weighted and summed as the hierarchical supervision loss to optimize the model parameters.

[0051] This embodiment uses large language model technology to make descriptions easy to obtain, and uses this to construct a "visual-text" bimodal gaze target estimation framework, making full use of text semantics to supplement visual information;

[0052] Furthermore, a deep fusion mechanism is designed based on the cross-modal Transformer architecture to achieve fine-grained interaction between text and vision, overcoming the limitations of shallow multimodal fusion in gaze target estimation tasks. Finally, a hierarchical supervision strategy based on a large model is proposed to use original text features to constrain the effectiveness of text and vision during the fusion process, avoid modal information imbalance, and ensure estimation accuracy.

[0053] This embodiment only requires ordinary RGB image input and the face bounding box coordinates of the target person. At the same time, the text description based on the visual large language model can be customized according to the specific needs of the scene, improving the generalization and feasibility of the technology in diverse scenarios.

[0054] In one embodiment, step one, constructing a training set containing facial gaze images and text descriptions, specifically involves:

[0055] The image data reading module reads RGB images, adds face bounding box coordinates to the target person in the RGB images, and then uses this to create a face gaze image. A visual large language model is used to understand the RGB images and obtain corresponding text descriptions. Combined with the ground truth labels of the gaze target, a training set is constructed; that is, the training set includes the face gaze image, text descriptions, and ground truth labels of the gaze target.

[0056] Specifically, the large model processing module uses Qwen2.5VL-7B as the visual large language model for image understanding and text description. It designs general and appropriate prompts to generate text descriptions of the scene.

[0057] The complete Chinese version of the prompt is as follows:

[0058] The bounding box coordinates of the target person's head in the RGB image are: The RGB image shows only one person. Please use "his / her" or "her" for precise reference, rather than "their / them".

[0059] Describe the physical features and pose of the target person in the RGB image, including their orientation relative to themselves. Ensure your answer uses only the target person as the subject. Do not repeat the target person's head coordinates in your answer. Only describe the direction the target person is looking; do not specify the object / person they are looking at. Example: The target person is wearing a white sweatshirt, standing on the grass, with their head tilted back, looking towards their upper right.

[0060] In one embodiment, step two involves extracting features from the training set to obtain original visual features and original text features, and generating a face mask and enhanced visual features, specifically as follows:

[0061] (1) Preprocessing module;

[0062] Image preprocessing: For the input face staring image, the size is first adjusted to fit the input requirements of the visual self-supervised representation pre-trained model (e.g., DinoV2 model), and then normalized according to its pre-training configuration to eliminate the influence of pixel value range differences on feature extraction, resulting in the preprocessed image.

[0063] Text preprocessing: Text exceeding the upper length threshold is truncated, and text below the lower length threshold is padded. At the same time, a text mask is generated for the padded text to distinguish between valid tokens and padded tokens, thereby obtaining the preprocessed text.

[0064] Face mask generation: Based on the input face gaze image, generate a binary mask with the same size as the face gaze image. Pixels inside the face gaze image are set to 1 to preserve face region features, and pixels outside the face gaze image are set to 0 to suppress background interference, thereby generating the face mask.

[0065] (2) Dual-modal feature extraction module;

[0066] Visual branch: After extracting features from the preprocessed image using a pre-trained model with frozen parameters, the original visual features are obtained through convolutional projection.

[0067] For example, the pre-trained DinoV2 (optional configuration) is selected as the pre-trained model for visual self-supervised representation, and all its parameters are frozen to reduce training costs and utilize the pre-trained visual representation capabilities; the visual features output by the network are projected through convolutional layers to achieve dimensional alignment with the text features.

[0068] Text branch: After extracting the preprocessed text features using a pre-trained language model with frozen parameters (such as a pre-trained BERT model), the original text features are obtained by projection through a linear layer.

[0069] For example, a pre-trained BERT-base-uncased model is used as a pre-trained language model, and its parameters are frozen to retain general semantic understanding capabilities; the text features output by the pre-trained language model are projected through a linear layer to lay the foundation for subsequent cross-modal fusion.

[0070] In one embodiment, step three involves multiplying the face mask element-wise with the original visual features to obtain enhanced visual features, and then encoding and concatenating the original text features and enhanced visual features. Specifically:

[0071] Step 3 is achieved through the feature fusion preparation module.

[0072] (b1) Enhanced visual features: The face mask is multiplied element by element with the original visual features to obtain enhanced visual features, thereby increasing the feature weight of the face region and reducing the interference of the background region on gaze estimation.

[0073] (b2) Special Token Design: Add learnable in-field and out-of-field tokens to determine whether the gaze target is within the image field of view, enrich the model output information, and adapt to the needs of multiple scenarios.

[0074] Understandably, the In-and-Out-of-View Token is optional. It serves as a pre-defined special input marker in the model (no manual specification is required) to assist the model in learning the discriminative features of in-and-out-of-view locations. When the In-and-Out-of-View Token is enabled, the model outputs the judgment result of whether the gaze target is in or out of the view through the corresponding view prediction head; if the In-and-Out-of-View Token is not enabled, there is no need to set the view prediction head. Regardless of whether the In-and-Out-of-View Token is enabled or not, it does not affect the generation of the final gaze target estimation heatmap.

[0075] (b3) Location coding;

[0076] The enhanced visual features are encoded with fixed 2D sinusoidal position codes to accurately encode their spatial position information. The original text features are encoded with learnable position codes to capture temporal semantic relationships. If there are tokens inside or outside the field of vision, they are not encoded with positions and are placed directly at the beginning of the concatenated feature sequence (to make them global identifier tokens and avoid position information from interfering with global attribute judgment).

[0077] During cross-modal fusion, the query (Q) and key (K) of the visual-language Transformer are fused with the above-mentioned hybrid positional encoding (visual fixed sine encoding + text learnable encoding), while the value (V) retains the original features without additional positional encoding. The core reason for this is that Q and K need to model the relative positional relationship between tokens through positional information (such as visual spatial neighborhood and text temporal association) to ensure that attention calculation accurately captures cross-modal positional related features. As the "value" vector of feature aggregation, V does not encode positional information, which can avoid repeatedly introducing redundant positional noise and make the output features more focused on cross-modal semantic fusion, which is in line with the core requirement of "textual semantics guiding visual region attention" in the gaze prediction task (target gaze prediction).

[0078] (b4) Feature concatenation: The in-visual-field tokens (optional), the encoded original text features, and the encoded enhanced visual features are concatenated sequentially to form a concatenated feature sequence, which serves as the input to the cross-modal Transformer.

[0079] In one embodiment, step four involves inputting the stitched feature sequence into a cross-modal Transformer, concatenating the enhanced visual features with the visual feature residuals output by the cross-modal Transformer, and generating a gaze target estimation heatmap; specifically:

[0080] Cross-modal Transformer module configuration: It contains 3 layers of encoders. Each encoder consists of multi-head self-attention (e.g., 8 heads, attention head dimension 32), layer normalization, and feedforward network (Linear (linear layer) → ReLU activation layer → Linearr (linear layer)). The dropout (regularization layer) parameter is set to 0.1 to balance the complexity and generalization of the cross-modal Transformer.

[0081] It should be noted that in this embodiment, the enhanced visual features are connected with the visual feature residuals output by the cross-modal Transformer to form a visual feature sequence in order to preserve visual detail information and avoid local feature degradation caused by global modeling of the cross-modal Transformer.

[0082] Heatmap header: It adopts a "transposed convolution + convolution + activation" structure. The transposed convolution layer amplifies the visual feature sequence, and then the convolution layer reduces the dimensionality. Finally, the Sigmoid activation layer outputs a probability heatmap as the gaze target estimation heatmap. The higher the heatmap value, the greater the probability that the position is the gaze target.

[0083] Field of view determination head: The features of the tokens inside and outside the field of view output by the cross-modal Transformer are used as input, and then passed through the linear layer, ReLU activation layer, dropout layer (regularization layer), linear layer, and Sigmoid activation layer in sequence. The output is a probability value of 0-1. If the probability is greater than the preset probability threshold (e.g., 0.5), it means that the gaze target is inside the field of view; otherwise, it means that the gaze target is outside the field of view.

[0084] In one embodiment, step five involves calculating the mean squared error between the text tokens output by each layer of the cross-modal Transformer and the original text features. The weighted sum of the mean squared errors of all layers is used as the hierarchical supervision loss to optimize the model parameters. Specifically:

[0085] In the cross-modal Transformer module, the mean squared error between the text token and the original text features is used as the hierarchical supervision loss. The core role of this hierarchical supervision loss is twofold: firstly, it constrains the encoding process of the original text feature parts in the concatenated feature sequence by each layer of the cross-modal Transformer encoder, avoiding the loss or shift of text semantics in multi-layer feature fusion, and ensuring that the semantic information of the text modality can be stably transmitted to subsequent layers of the model; secondly, through multi-level loss supervision, it strengthens the model's deep understanding and retention of the original text feature parts in the concatenated feature sequence, making up for the text semantic weakening problem that easily occurs when relying solely on the final layer supervision, thereby improving the effectiveness of cross-modal fusion, and ultimately enabling text semantics to more accurately guide visual features to focus on the gaze target-related region, improving the accuracy of gaze target estimation.

[0086] The main loss of the heatmap header: In the gaze target model, the BCE (Binary Cross-Entropy) loss serves as the main loss function for the heatmap prediction task. Its core role is to measure the pixel-level difference between the predicted gaze target estimation heatmap output by the model and the real gaze target estimation heatmap, thus driving the model to accurately learn the distribution characteristics of the gaze point. The model normalizes the output predicted gaze target estimation heatmap to the [0,1] interval through the Sigmoid activation function, where each pixel value corresponds to the probability that the position is a gaze point. The real gaze target estimation heatmap is usually a Gaussian distribution or a binary mask around the actual gaze point. The BCE loss penalizes the deviation between the predicted probability of each pixel and the real label of the gaze target, especially focusing on optimizing the prediction accuracy of the core region of the gaze point, ultimately making the model output a gaze target estimation heatmap that fits the real gaze point. Its calculation formula is: For a single gaze target estimation heatmap containing N pixels, let the i-th pixel be the first pixel. The predicted probability of each pixel is The true label for the gaze target is ( If ∈{0, 1}), then the total BCE loss is... This is the average loss across all pixels, i.e.:

[0087] .

[0088] To verify the effectiveness of this embodiment in cross-modal gaze target estimation, the following comparative experiments were conducted: Existing methods include Recasens, Chong, Bao, Gupta, Ryan (ViT-B), and Ryan (ViT-L). In this embodiment, for replaceable visual self-supervised representation pre-trained models, following the configuration of existing best methods, two versions of the DinoV2 model, ViT-B and ViT-L, were used to investigate the impact of the visual pedestal's performance on the performance of this method.

[0089] The source of Recasens's work is: Recasens, Adria, et al. "Following gaze in video." Proceedings of the IEEE International Conference on Computer Vision. 2017.

[0090] Chong's reference is: Chong, Eunji, et al. "Detecting attended visual targets in video." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020.

[0091] Bao's literature sources are: Bao, Jun, Buyu Liu, and Jun Yu. "Escnet: Gaze target detection with the understanding of 3d scenes." Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2022.

[0092] Gupta's literature sources are: Gupta, Anshul, Samy Tafasca, and Jean-Marc Odobez. "A modular multimodal architecture for gaze target prediction: Application to privacy-sensitive settings." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022.

[0093] The source of Ryan (ViT-B) and Ryan (ViT-L)'s work is: Ryan, Fiona, et al. "Gaze-lle: Gaze target estimation via large-scale learned encoders." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[0094] Experimental results on the GazeFollow dataset, VideoAttentionTarget dataset, GOO-Real dataset, and ChildPlay dataset show that the method of this embodiment reaches or surpasses the current best level in many key metrics.

[0095] For the GazeFollow dataset, which contains approximately 10 gaze point annotations per image, the metrics used are the average L2 distance (Avg L2) from the predicted point to all annotated points and the distance (MinL2) from the predicted point to the nearest annotated point in the test set samples. The mean of all samples in the test set is used as the final result. For the VideoAttentionTarget, ChildPlay, and GOO-Real datasets, which contain only one gaze point annotation per image, the average AUC and average L2 distance metrics from the test set are used. For VideoAttentionTarget and ChildPlay, which have intra-frame / out-frame annotations, the average accuracy (AP_in / out) of the intra-frame / out-frame prediction task is additionally introduced as an evaluation metric.

[0096] Key metrics include: Area under the ROC curve (AUC), mean squared error (L2), and average precision (AP_in / out).

[0097] Among them, the GazeFollow dataset is a large-scale dataset focused on gaze target estimation; the VideoAttentionTarget dataset is a dataset focused on capturing complex dynamic gaze behavior patterns; GOO-Real is a gaze behavior dataset in real retail scenarios; and the ChildPlay dataset is a dataset focusing on gaze target estimation for children in natural play scenarios.

[0098] Specifically, as shown in Table 1, without introducing additional hardware modalities such as depth or pose, and based solely on the "image + text" bimodal input, the DINOv2 ViT-L-based model in this embodiment reduces the average L2 distance on the GazeFollow dataset to 0.099. Compared to methods such as Gupta that rely on depth and pose information (0.114), this represents a relative error reduction of over 13%, while maintaining a higher AUC level (0.956 vs. 0.943). This result fully demonstrates that the hierarchical supervision loss introduced in this embodiment can effectively replace expensive multimodal perception components without increasing system complexity.

[0099] Further comparison with current pure vision methods such as Ryan et al. shows that this embodiment achieves consistent advantages in both core tasks: on the GazeFollow dataset, the average L2 distance is further reduced to 0.099, achieving a stable improvement compared to Ryan's (0.101); on the more challenging VideoAttentionTarget dataset, this embodiment surpasses the comparison method in AUC, L2, and AP_in / out metrics simultaneously, with AP_in / out reaching 0.911, an improvement of nearly 1 percentage point from 0.903, indicating that the proposed solution has stronger discriminative ability in the highly semantically complex task of "inside / outside the field of view" discrimination. These results validate the significant effect of the hierarchical supervision loss introduced in this embodiment in alleviating the semantic defocusing problem of pure vision methods.

[0100] like Figure 2 The cross-dataset performance comparison shown in the GOO-Real dataset demonstrates that, in the untuned test on the GOO-Real retail real-world scenario dataset, the DINOv2 ViT-L model in this embodiment achieves an AUC of 0.918 and an L2 distance reduction to 0.159. This not only outperforms the others in AUC metrics but also achieves a relative error reduction of over 9% (0.175→0.159) compared to the current best method, Ryan et al.'s ViT-L version, in terms of positioning accuracy. These results indicate that the hierarchical supervised loss introduced in this embodiment significantly improves the model's ability to understand complex target relationships in unfamiliar retail scenarios, thus maintaining high-precision positioning performance even in cross-domain migration scenarios.

[0101] On the ChildPlay interactive dataset, this embodiment achieves an AP_in / out metric of 0.996, surpassing all existing comparative methods. It also demonstrates a stable improvement over Ryan et al.'s ViT-L version, while maintaining comparable or even better performance in AUC and L2 metrics. These results clearly demonstrate that this embodiment retains highly reliable visual field discrimination and spatial positioning accuracy even in complex scenarios involving children's non-normative gaze behavior.

[0102] In Tables 1 to 3, GF corresponds to the GazeFollow dataset, VAT corresponds to the VideoAttentionTarget dataset, GR corresponds to the GOO-Real dataset, and CP corresponds to the ChildPlay children's interactive dataset. The up arrow (↑) indicates that the higher the value of the metric, the better the method performs; the down arrow (↓) indicates that the lower the value of the metric, the better the method performs.

[0103] Table 1. Performance Comparison of GazeFollow and VideoAttentionTarget Datasets

[0104]

[0105] Table 2. Cross-dataset performance comparison of the GOO-Real dataset.

[0106]

[0107] Table 3. Cross-dataset performance comparison of the ChildPlay dataset.

[0108]

[0109] Based on the above description of the embodiments, those skilled in the art will understand that the hierarchical supervision-based cross-modal staring target estimation method, system, and medium described in this embodiment can be implemented in pure software or deployed and run on general-purpose or dedicated computing hardware platforms. Based on this essence, the technical solution of this embodiment can be specifically implemented in the form of a software product containing program instructions. This software product can be stored on various non-volatile storage media or directly deployed as a local or cloud service. The program instructions are used to cause computer devices with processing capabilities—including but not limited to personal computers, server clusters, mobile terminals, or other network devices—to execute the steps described in this embodiment.

[0110] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A cross-modal gaze target estimation method based on hierarchical supervision, characterized in that, After adding face bounding box coordinates to the target person in the acquired RGB image, the coordinates are input into the gaze target model to generate the corresponding gaze target estimation heatmap. The training process for the gaze target model is as follows: Construct a training set containing face gaze images and text descriptions; Feature extraction is performed on the training set to obtain the original visual features and original text features, and a face mask is generated. The enhanced visual features are obtained by multiplying the face mask element-wise with the original visual features. The original text features and the enhanced visual features are then encoded and concatenated. The stitched feature sequence is input into the cross-modal Transformer, and the enhanced visual features are concatenated with the visual feature residuals output by the cross-modal Transformer to generate a gaze target estimation heatmap. The mean squared error between the text tokens output by each layer of the cross-modal Transformer and the original text features is calculated. The mean squared errors of all layers are weighted and summed as the hierarchical supervision loss to optimize the model parameters.

2. The method according to claim 1, characterized in that, Construct a training set containing face gaze images and text descriptions, specifically as follows: Add face bounding box coordinates to the target person in the acquired RGB image to obtain a staring face image; The facial gaze image is used as input to the visual large language model, and combined with the set general prompt words, a text description is generated based on the predicted appearance, actions, and gaze direction of the target person.

3. The method according to claim 2, characterized in that, The process of extracting features from the training set to obtain the original visual features and original text features is as follows: The size of the staring face image is adjusted and normalized to obtain the preprocessed image features; Text exceeding the upper length threshold in the text description is truncated, and text below the lower length threshold is padded. At the same time, a text mask is generated for the padded text to obtain the preprocessed text features. After extracting features from the preprocessed image using a pre-trained model with replaceable visual self-supervised representations of frozen parameters, the original visual features are obtained through convolutional projection. After extracting features from the preprocessed text using a parameter-frozen, replaceable pre-trained language model, the original text features are obtained by projection through a linear layer.

4. The method according to claim 1, characterized in that, The process of encoding and concatenating the original text features and enhanced visual features specifically involves: Learnable positional encoding is applied to the original text features; The enhanced visual features are encoded using fixed 2D sinusoidal position coding.

5. The method according to claim 1, characterized in that, During the generation of spliced ​​feature sequences, learnable field-of-view tokens are also set, which are used to determine whether the gaze target is within the image field of view. By sequentially concatenating the in-field and out-of-field tokens, the encoded original text features, and the encoded enhanced visual features, a concatenated feature sequence is obtained.

6. The method according to claim 5, characterized in that, The spliced ​​feature sequence is input into the cross-modal Transformer, and the features output by the cross-modal Transformer for the inside and outside of the field of view tokens are used as the input of the field of view prediction head, so as to output the judgment result of whether the staring target is inside or outside the field of view.

7. The method according to claim 6, characterized in that, The process for generating the determination result of whether the view is inside or outside the field of vision is as follows: The features of the tokens inside and outside the field of vision are passed through the cross-modal Transformer and then through a linear layer, a ReLU activation layer, a Dropout layer, a linear layer, and a Sigmoid activation layer. The output is a probability value of 0-1. When the probability value is greater than a preset probability threshold, it means that the gaze target is inside the field of vision; otherwise, the gaze target is outside the field of vision.

8. The method according to claim 1, characterized in that, The process of generating the gaze target estimation heatmap is as follows: The enhanced visual features are concatenated with the visual feature residuals output by the cross-modal Transformer and then passed through a transposed convolutional layer, a convolutional layer, and an activation layer in sequence to output a probability heatmap, which serves as the gaze target estimation heatmap.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of classification programs, which are used by a processor to execute the method as described in any one of claims 1-8.

Images