A method and system for gaze localization based on error peak mining and cue migration

By constructing a bias cue token sequence through cross-inference and error peak mining, and performing two-stage training, the problems of viewpoint and geometric uncertainty bias and interference from non-target regions with high error response in gaze target localization are solved, achieving high-precision and efficient gaze target localization.

CN123118767APending Publication Date: 2026-11-17BEIJING INFORMATION SCI & TECH UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611386120.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-08
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing gaze target localization techniques suffer from prediction bias due to viewpoint and geometric uncertainties, as well as interference from non-target regions with high error response. Furthermore, existing methods cannot effectively distinguish between gaze cues and bias shortcuts used by the model.

Method used

By cross-inference and error peak mining, a learnable biased cue token sequence is constructed. Cross-modal alignment is performed using unbiased text templates and biased cue text templates. Two-stage training is conducted to perceive and suppress bias. Finally, text branches are removed, and only the pure visual path is retained.

Benefits of technology

It enables autonomous reduction of reliance on high-response non-target regions without the need for text branches, thereby improving the accuracy and efficiency of gaze target localization and adapting to various edge devices and real-time application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118767A_ABST
    Figure CN123118767A_ABST
Patent Text Reader

Abstract

This invention provides a method and system for gaze target localization based on error peak mining and cue transfer, comprising: acquiring a scene image and a target person's head bounding box; dividing training samples into mutually exclusive subsets, generating a first heatmap using a gaze model that does not include the current sample, occluding the image region corresponding to the first error peak and inferring again, and determining the interference region and its weight based on the distance improvement of the second predicted point relative to the real gaze point; constructing a clean cue containing head, scene, and gaze target conditions, and a bias cue with interference region conditions, and learning cue parameters through global and local image-text alignment; removing the bias cue, and training a conditional visual model of the person with fixed clean text features, gaze heatmap supervision, and interference region suppression constraints. During inference, only the image and head bounding box are input, and the gaze heatmap and in-image or out-of-image probabilities are output.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to a method and system for gaze target localization based on error peak mining and cue transfer. Background Technology

[0002] Gaze target localization is a core task in computer vision and human-computer interaction. It aims to predict the actual gaze location of a person in a current scene, or determine whether the gaze target is outside the image, based on the person's head position, pose orientation, scene structure, and the distribution of surrounding objects in a natural scene image or video. This technology can be widely applied in fields such as intelligent surveillance, human-computer interaction, assisted driving, virtual reality, behavior understanding, social scene analysis, and robot perception. Unlike traditional eye-tracking methods that rely on dedicated eye-tracking acquisition equipment, image-based gaze target localization infers the gaze area of ​​a person from ordinary RGB images or videos, offering greater application flexibility and ease of deployment.

[0003] Existing technologies typically utilize convolutional neural networks or visual Transformers to extract scene and head features, then predict the gaze location using attention mechanisms, person-specific tokens, or heatmap decoders. With the development of large-scale self-supervised visual models, using frozen pre-trained visual encoders (such as DINOv2 and DINOv3) to extract general scene features, followed by training a lightweight task decoder, has gradually become an effective approach for gaze localization. In the Gaze-LLE framework, the model uses a frozen image encoder to extract scene features, injects the person's position condition based on the head bounding box, and then predicts the gaze location using a lightweight Transformer decoder and a heatmap head.

[0004] However, existing technical solutions have the following technical drawbacks. First, prediction bias caused by viewpoint and geometric uncertainties: relying solely on the head bounding box and rough head orientation often cannot uniquely determine the gaze target; changes in shooting angle, person scale, head occlusion, and target distance can all cause systematic prediction bias. Second, interference from non-target regions with high false responses: High-contrast regions, strong textures, large objects, or semantically prominent non-target persons in the image may receive abnormally high model responses, causing the first peak of the prediction heatmap to deviate from the true gaze target. While traditional end-to-end heatmap supervision can improve average localization accuracy, it cannot clearly distinguish whether the model utilizes stable gaze cues or bias shortcuts—the model may learn during training to rely on saliency biases rather than genuine gaze cues to make predictions.

[0005] In terms of cue learning, visual language pre-trained models can map images and text to a shared semantic space, while cue learning methods allow the replacement of entirely manually written category names with continuous vectors. However, gaze points are essentially continuous spatial variables and cannot be simply equated to discrete category labels. By combining fixed natural language fragments with coordinate conditions, image conditions, and learnable tokens, the structural semantics of the natural language template can be preserved, while continuous tokens can be used to express head position, scene conditions, and gaze target coordinates. However, if only a free learnable vector named "error peak interference bias" is set, without sample-level observable data sources and supervision, this vector may not be able to learn real interference information, and its representation may degenerate into meaningless noise or be highly coupled with viewpoint bias.

[0006] Therefore, a technical solution is needed to achieve the transition from "deviation perception" to "deviation suppression". Summary of the Invention

[0007] To address the technical problems in gaze target localization caused by biases in viewpoint and geometry, as well as interference from non-target regions with high error responses, leading to prediction offsets, this invention provides a gaze target localization method and system based on error peak mining and cue migration, comprising the following steps: The first prediction heatmap is obtained by cross-inference on the training samples using an out-of-sample gaze model. The image region corresponding to the first erroneous peak in the first predicted heatmap is suppressed, and the second predicted heatmap is obtained by reasoning again. Based on the improvement of the distance between the second predicted heatmap and the real gaze point, the interference region of the erroneous peak is determined. Construct an unbiased text template containing a learnable head cue token sequence, a learnable scene cue token sequence, and a learnable gaze target cue token sequence, and a biased cue text template obtained by inserting a learnable bias token sequence into the unbiased text template, wherein the learnable bias token sequence is used to characterize bias information that interferes with gaze prediction. In the first training phase, unbiased text templates and biased cue text templates are used to perform cross-modal alignment with the image features of the training samples to learn the representation of the learnable biased token sequence. In the second training phase, the learnable bias token sequence is removed, and the text features corresponding to the unbiased text template are used as fixed supervision signals to constrain the gaze prediction output of the pure vision branch and reduce the pure vision branch's response dependence on error peak interference regions. After training is completed, the text branch is removed, and the pure vision branch is used to receive the image to be tested and the target person's head bounding box, and the gaze heatmap is output.

[0008] The technical advantages of the above method are as follows: Through cross-inference and error peak mining, the error peak interference region is not an abstract concept subjectively set by humans, but rather sample-level bias information automatically mined from the model's actual error response behavior. This provides a clear and observable source of supervision for the bias cue token, solving the problem of free learnable vectors lacking real interference learning basis. By constructing a dual-template structure of unbiased text template and biased cue text template, the interference information originally implicit in visual features can be clearly carried in the text semantic space, while the clean cue maintains stable task semantics between the person's head, scene, and gaze target. Through a two-stage training strategy—the first stage aligns visual features with bias cue to perceive bias, and the second stage removes bias and uses clean cue transfer to guide the pure visual branch—the final inference model can autonomously reduce its dependence on high-response non-target regions without the need for text branches, achieving an effective transfer from "bias perception" to "bias suppression." After completing the cue transfer during the training stage, the entire method retains only the pure visual path during the inference stage, without relying on any text encoder or additional modal input, balancing high performance and inference efficiency.

[0009] In some preferred embodiments, the learnable bias token sequence in the biased cue text template includes a learnable viewpoint bias token sequence and a learnable error peak interference bias token sequence. The first training phase also includes applying a bias token separation loss to promote the formation of mutually distinguishable representation directions between the learnable viewpoint bias token sequence and the learnable error peak interference bias token sequence. An additional effect of this design is to prevent the two types of bias tokens from converging to the same representation direction, ensuring that the two different types of bias information—viewpoint uncertainty and error peak interference—can occupy independent information channels in the text semantic space, thereby enabling more complete stripping of bias information during the second-stage transfer learning.

[0010] In some preferred embodiments, the first training phase further includes: locally aligning the text features corresponding to the learnable error peak interference bias token sequence with the region image features corresponding to the error peak interference region. This feature provides precise spatial localization supervision for the bias token, enabling the error peak interference representation in the text branch to accurately correspond to the specific interference region in the image, rather than merely obtaining coarse-grained image-level semantics. This helps improve the accuracy of the second phase in suppressing this region.

[0011] In some preferred embodiments, the training samples are cross-inferenced using an out-of-sample gaze model to obtain a first prediction heatmap. This includes: dividing the training sample set into at least two mutually exclusive subsets; training a gaze model using each subset, and using the gaze model trained on one subset to infer the other subset, thereby obtaining out-of-sample prediction results. Obtaining prediction results through out-of-sample inference avoids the overfitting bias of the model to its own training samples, making the discovered error peak interference regions more objective and having greater generalization reference value.

[0012] In some preferred embodiments, determining the error peak interference region based on the improvement in distance between the second predicted heatmap and the actual gaze point includes: calculating a first prediction error and a second prediction error, where the first prediction error is the distance between the predicted position of the first error peak and the actual gaze point, and the second prediction error is the distance between the predicted position of the second predicted heatmap and the actual gaze point; when the first prediction error is greater than a first threshold and the improvement of the second prediction error relative to the first prediction error is greater than a second threshold, the image neighborhood corresponding to the first error peak is determined as the error peak interference region. This dual-threshold determination mechanism can effectively filter out truly difficult samples affected by error peak interference, exclude samples with small prediction errors or insignificant improvement after occlusion, and make the annotation of the error peak interference region have a higher signal-to-noise ratio. Furthermore, a continuous difficulty weight can be constructed based on the first prediction error and the improvement amount to differentiate the weighting of samples with different degrees of interference.

[0013] In some preferred embodiments, the cross-modal alignment in the first training phase includes global image-text alignment and local region alignment. Global image-text alignment establishes a correspondence between the features of the entire image and the features of the entire cue text. Local region alignment includes at least aligning the text features corresponding to the learnable head cue token sequence with the head region features in the image, and aligning the text features corresponding to the learnable gaze target cue token sequence with the gaze target region features in the image. Global alignment establishes a match between each image and its corresponding cue text at the overall semantic level, while local region alignment ensures that the head information and gaze target information in the structured cue accurately correspond to their respective spatial locations in the image, avoiding the model only learning the coarse-grained semantics of the entire text while ignoring the fine-grained spatial correspondence.

[0014] In some preferred embodiments, constraining the gaze prediction output of the pure visual branch in the second training phase includes: using the text features corresponding to the unbiased text template as the training target; transferring the task semantics between the person's head, scene context, and gaze target contained in the unbiased text template to the pure visual branch through text-visual feature alignment loss; and reducing the prediction response of the pure visual branch in error peak interference regions through error peak region suppression loss. These two loss functions form a complementary mechanism: the alignment loss provides positive task semantic guidance to the visual branch, and the suppression loss provides negative interference suppression signals to the visual branch. Their synergistic effect enables the final model to correctly predict the gaze position while reducing erroneous responses to interference regions. In the early stages of training, the weight of the text-visual feature alignment loss can be gradually increased from zero to a preset value to improve training stability.

[0015] In some preferred embodiments, the method further includes: outputting the in-image / out-of-image state judgment result; wherein, the person-specific visual representation output by the pure visual branch is obtained, and the probability that the gaze target is located inside the image is output through the in-image / out-of-image classification head; when the probability is lower than a preset threshold, the out-of-image gaze state is output. This design enables a single model to handle both scenarios where the gaze target is located inside and outside the image simultaneously. For out-of-image samples, no heatmap supervision is constructed; the in-image / out-of-image classification branches are used for independent judgment, avoiding the erroneous compression of out-of-image gaze behavior to the image edge.

[0016] Compared with the prior art, the present invention has the following beneficial effects: By employing cross-inference and error peak mining mechanisms, this invention establishes the determination of interference regions based on the actual error response behavior of the model, rather than subjective human presupposition. Two-fold cross-inference ensures out-of-sample objectivity in interference region identification, while dual-threshold judgment and continuous difficulty weight design further improve the signal-to-noise ratio of interference sample screening, providing a reliable data foundation for subsequent bias modeling. This enables the model to specifically focus on and learn those high-response, non-target regions that truly cause prediction deviations.

[0017] By introducing independent, learnable bias token sequences into the structured cue template, this invention achieves textual semantic space representation of bias information in the gaze target localization task. The viewpoint bias token and the error peak interference bias token correspond to two different sources of bias: geometric uncertainty and visual interference, respectively, and are combined with token separation loss to ensure their respective functions. A local alignment mechanism between the bias tokens and image interference regions further guarantees the spatial accuracy of bias modeling. This design transforms bias information from an unobservable state originally implicit in visual features into an explicit representation that can be clearly expressed and manipulated in the textual semantic space.

[0018] By employing a two-stage training framework of "bias perception and bias suppression," this invention achieves effective transfer of bias information rather than simple removal. The first stage allows the visual model to fully perceive the existence and characteristics of biases. The second stage, after removing bias information, uses clean cues as a bridge to write pure task semantics into the pure visual branch. This strategy of perception followed by suppression yields better training results than directly ignoring biases, enabling the final visual model to autonomously reduce its dependence on interfering regions even without text branches or additional modal input.

[0019] Through a purely visual design in the final inference stage, the text encoder, prompt module, and text feature cache of this invention are all removed after training. Inference only requires image and header box inputs, and the model size and inference speed are consistent with end-to-end pure vision models, requiring no additional computational overhead. This enables this invention to adapt to various edge devices and real-time application scenarios, maintaining advanced positioning accuracy while also considering engineering practicality and deployment convenience. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating the overall architecture of the gaze target localization method based on error peak mining and cue migration of the present invention. Figure 2 This is a schematic diagram illustrating the occlusion effect of the error peak interference region in this invention. Figure 3 This is a schematic diagram of the two-stage training strategy of the present invention; Figure 4 This is a visualization of the model's effect in this invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The purpose of this invention is to provide a method and system for gaze target localization based on error peak mining and cue transfer. This method retains head cues, scene cues, and gaze target cues from the structured cue learning framework, and adds viewpoint bias cues and error high-response interference cues. The error high-response interference cues are jointly supervised by two-fold cross-inference, occlusion or suppression of the first error peak, and distance improvement. The overall structure is as follows: Figure 1 As shown, the model consists of an image encoder, a text encoder, a prompting module, and a cross-modal alignment module, and learns the features of perceptual bias and suppression bias in two stages.

[0023] Cue-based gaze task and error peak interference bias modeling mechanism This invention constructs a "structured, learnable cue template for gaze task perception," guiding image features to adaptively align with the position of the person's head, scene context, gaze target position, and potential bias factors. Unlike methods that simply concatenate several tokens, this invention inserts coordinate condition tokens, image condition tokens, and learnable bias tokens between fixed natural language segments in the text encoder. This ensures the cue sequence possesses both a clear linguistic and grammatical structure and the ability to express continuous spatial locations and sample-related interference. Specifically, the following two types of text cue templates are constructed in the first training phase: (1) Offset hint text template: The structure is "Target person [x1]...[x n [a1]...[a] n Error peak interference [b1]...[b n Located in the scene [y1]...[y n [Observing target [z1]...[z] n The text prompt is "[x1]...[x]". n ] is used to represent the head information of the person who needs to be focused on in the current image, [y1]...[y n [z1] describes the current image scene information. n [A] is used to describe the target area of ​​a person's gaze. Bias information includes [A1]...[A2] n [b1] is a learnable perspective bias cue used to provide information on head posture and shooting angle. n This template is used to express the attraction of regions with high erroneous responses but not true gaze targets to the model's predictions. It explicitly includes bias-related text fragments, allowing the perspective uncertainty and error peak interference information, originally implicit in visual features, to be clearly expressed and aligned in the text semantic space.

[0024] (2) Unbiased text template: using the structure "target person [x1]...[x n Located in the scene [y1]...[y n [Observing target [z1]...[z] nThe text prompts retain task-related information such as head position, scene context, and gaze target, but do not include bias prompts. This aims to extract bias information from the image, thereby guiding the model to focus on learning the main character, i.e., the relationship between the head features of the target character and the image information and gaze target point. It is suitable for the later stage of model training. The text features generated by the unbiased text template are used as training supervision, prompting the image branch to pay more attention to the relationship between the target character's head, scene structure, and the real gaze target, and reducing the dependence of visual representation on viewpoint uncertainty and error-prone high-response non-target regions.

[0025] Data source and judgment mechanism for error peak interference bias Unlike viewpoint bias, error peak interference bias is not manually labeled information directly provided by the dataset, but rather obtained through reverse mining of the model's erroneous prediction behavior during training. Specifically, in this invention, in the gaze target localization task, if the model's first prediction peak does not fall near the actual gaze target, but instead falls on a region in the image that generates a high response in the prediction heatmap but is not the actual gaze target, then this region may constitute an error high-response non-target interference. Error high-response non-target interference refers to regions in the image with strong visual appeal, such as regions with high brightness, large area, complex texture, prominent color, or semantically more likely to elicit a model response (e.g., people and objects). Although such regions are not the actual gaze targets, they may generate error high responses during model inference, thereby pulling the prediction result away from the actual gaze point.

[0026] To ensure a clear data source for the erroneous peak interference bias, this invention employs a two-fold cross-inference approach to construct erroneous peak interference samples. Specifically, the training data is divided into two mutually exclusive subsets, and two intermediate-cycle models are trained on each subset. Then, the model trained on the first subset is used to infer the second subset, and the model trained on the second subset is used to infer the first subset, thereby obtaining out-of-sample prediction results. For each sample, the predicted heatmap and its peak position obtained from the first inference are recorded. Then, the erroneous high-response region corresponding to the peak is suppressed or occluded, and a second inference is performed to obtain the second predicted peak position. If the second prediction is closer to the true gaze point than the first prediction, it indicates that the first high-response region is not the true gaze target, but rather an erroneous peak interference region that misleads the model. Based on this determination, the neighborhood of the first predicted erroneous peak can be considered as the erroneous peak interference region and further used to construct erroneous peak interference bias cues.

[0027] In this invention, the error peak interference bias is not an abstract concept subjectively set by humans, but rather bias information automatically mined from the model's actual error response behavior. Therefore, it can provide sample-level supervision for the error peak interference cue token, and also provide more targeted cross-modal alignment for the first-stage bias cue learning. Its error peak interference region occlusion effect is as follows: Figure 2 As shown. Figure 2 The left side displays the original sample and annotation information, where the green box represents the head bounding box of the target person and the red dot represents the true gaze point; Figure 2 The middle section shows the model's first prediction results under unoccluded conditions. The blue dots and heatmap peaks represent the gaze regions predicted by the model. It can be seen that the prediction peaks are biased towards regions in the image with high false responses but not the actual gaze targets. Figure 2 The right side shows the re-prediction results after occluding or suppressing the neighborhood of the first erroneous peak. The second predicted point is closer to the true gaze point than the first predicted point. Therefore, the neighborhood of the first erroneous peak can be used as the erroneous peak interference region, which can then be used to generate erroneous peak interference bias cues.

[0028] Two-stage training strategy design To improve training stability and feature generalization ability, this invention proposes a two-stage optimization strategy, which gradually achieves the transition from "bias awareness" to "bias suppression," as follows: Figure 3 As shown, an additional in-image / out-of-image task branch is added to the gaze heatmap prediction task branch. If the dataset annotations include information on whether the gaze target point is located inside the image, this optional task can be enabled. The target outputs a gaze heatmap and comprehensively judges whether the gaze target of the target person is located inside the image.

[0029] In the first stage, the visual and text branches encode the input image and the two types of cue templates, respectively, and establish global and local correspondences in a shared feature space. Bias cues explicitly include viewpoint bias cues and error peak interference bias cues, providing a learnable information-carrying space for geometric uncertainties and the attraction of high-response non-target regions in the training samples. Clean cues do not include the above bias cues and are used to maintain stable task semantics between the person's head, scene context, and the real gaze target. After the first stage, bias cues are removed, and only the text features corresponding to the clean cues are retained.

[0030] In the second stage, the clean text features derived in the first stage serve as a fixed supervision target and are not directly input into the gaze decoder. The input image and the target person's head bounding box still generate gaze heatmaps and in-image / out-of-image probabilities through a pure visual path. Clean text supervision updates the image encoder, the final normalization layer, and the gaze decoder inversely through alignment loss, enabling the visual model to retain gaze task information while reducing its dependence on viewpoint bias and error peak interference regions. After training, the text encoder, cue module, and text features are all removed, and the final model retains only the pure visual inference path, outputting in-image gaze points or out-of-image states based on in-image probabilities.

[0031] Suppose a training batch contains There are 10 samples, with the sample index being 1. . No. The input images are denoted as The head of the target person is marked as The actual fixation point during the training period is recorded as Output There are 1 visual patch, and the feature dimension of each patch is 1. The text feature dimension is The dimensions of the image and text sharing space are: The output heatmap size is Use the same format throughout. Represents the visual feature matrix. Represents the text feature vector. Represents the shared space vector after projection and normalization. Represents the loss function. and These represent the sets of learnable parameters for the two stages, respectively.

[0032] The following is a detailed description of the two phases of training: Phase 1: Learning from Cueed Awareness of Bias and Aligning Text and Images The primary goal of the first phase is to establish a stable correspondence between image features and structured cue text, and to obtain representations of the viewpoint bias token and error peak interference bias token that are relevant to the actual image content. This phase freezes the image encoder, text encoder, and gaze decoder, updating only the scene cue generator, viewpoint bias token, error peak interference bias token, error peak interference condition generator, and visual / text projection layer.

[0033] (1) Image features and two types of text features For the The image encoder outputs a visual patch feature matrix from 1 training sample. Average pooling is performed on all patches to obtain global visual features: . In the formula, B is the training batch size; i is the sample index; n is the number of visual patches; n is the patch index. The nth patch feature of the i-th sample; Dimensions of a single visual feature; To The global visual features are obtained by arithmetically averaging the features of each patch. Clean cues and bias cues are then processed by a frozen but differentiable text encoder to obtain: . In the formula, Indicates a text encoder; It consists of fixed text segments, HEAD_TOKENS, SCENE_TOKENS, and GAZE_TOKENS; exist Based on this, VIEW_TOKENS and SALIENCY_TOKENS are further added. Although the CLIP parameter is frozen, the forward propagation does not use the stopping gradient operation, allowing the loss gradient to be propagated back to the soft hint token and hint generation module.

[0034] Because the original feature dimensions and distributions of the selected image encoder and text encoder are different, a visual projector is used. and text projector Map them to the same shared space and perform sampling. Normalization, all three types of projection features belong to After normalization, the inner product of two features equals the cosine similarity, thus avoiding the excessive influence of feature magnitude differences on the image-text alignment results.

[0035] ; In the formula , , These represent the shared space vectors of the normalized image, unbiased text, and biased text, respectively.

[0036] (2) Global text and image alignment for clean and offset tooltips To match each image with its corresponding prompt text, a modality is defined. Next The image features and the first Temperature scaling similarity between text features: . In the formula These are the image sample index and the text sample index, respectively; T represents the vector transpose. Let be the scalar similarity between the i-th image and the j-th class a cue. This represents the image feature vector that the i-th image sample is mapped into the image-text shared space after passing through the visual encoder and projector. The text feature vector τ>0 represents the text feature vector of class a prompt text in the j-th text sample that is mapped to the same shared space after passing through the text encoder and projector. τ>0 is a temperature parameter, which is preferably selected from the candidate range of the logarithmic scale through the validation set. The smaller τ is, the sharper the similarity distribution.

[0037] . In the formula, B is the batch size; and These are the exponential function and the natural logarithm, respectively; the denominator is the sum of all B candidate texts or images in the same batch; i=j indicates a positive match, and the rest i≠j indicate a negative match within the batch. and represents the average cross-entropy contrast loss in the image-to-text and text-to-image directions, respectively.

[0038] Combining the two directions, we obtain the mode. Symmetric global alignment loss: . In the formula, To indicate the symmetric global image-text alignment loss for modality a; a coefficient of 1 / 2 is used for equal-weighted averaging of the two retrieval directions. When a=clean, the main semantics of the task are preserved; when a=bias, both viewpoint bias and error peak interference information are simultaneously included. This establishes a correspondence between the image and a complete cue containing viewpoint bias and error peak interference bias, providing text-side carrying space for potential bias information; Preserve the task-specific semantics between the character's head, the scene context, and the gaze target to prevent the model from learning only bias descriptions while ignoring gaze localization itself.

[0039] (3) Local alignment of the head region and the gaze target region Simply performing global alignment between the entire image and the entire text cannot guarantee that HEAD_TOKENS and GAZE_TOKENS correspond to the head region and the actual gaze region in the image, respectively. The region visual features are calculated using weighted pooling: . In the formula These represent the head region, the true gaze region, and the erroneous peak interference region, respectively. Let be the weight of the nth patch of the i-th sample relative to region r. This represents the total number of patches obtained after the image is segmented by the visual encoder. Region-weighted visual features; To prevent the division by zero constant, it is preferable to choose a sufficiently small positive number so as not to significantly change the normalization result of the effective region.

[0040] Let the average text features corresponding to HEAD_TOKENS and GAZE_TOKENS in the hidden state of the text encoder be respectively... and Not a visual feature and Regional visual features Corresponding text features After being mapped to the shared space by their respective projectors and L2 normalized according to the samples, to avoid introducing redundant symbols, they are still denoted as follows: and ,in Therefore, the head area uses and Pairing, gaze area use and pair.

[0041] The head region alignment loss and the gaze region alignment loss are defined as follows: , . In the formula, B represents the cosine similarity; B is the batch size. , and , All are unit vectors in the shared space; Indicates the first Does each sample have a valid in-figure fixation point? To prevent division by zero constants, when there are no valid samples within the batch, the denominator... The loss is guaranteed to be defined. Two local alignments together constitute... This allows the information about a person's head and the gaze target in the structured prompts to be associated with corresponding spatial regions in the image, thus avoiding the model only learning the coarse-grained semantics of the entire text.

[0042] (4) Alignment of the region of error peak interference bias This invention employs two-fold cross-inference to obtain erroneous peak interference samples. Let the first predicted peak, the second peak after suppressing the first peak, and the true fixation point be respectively... , and The prediction errors and improvements for the two predictions are as follows: . In the formula, , , These are the first predicted peak value, the second predicted peak value after suppressing the first peak value, and the true gaze point in the normalized image coordinate system, respectively. , and represent the errors of the two normalized Euclidean distances, respectively; The amount of error improvement after occlusion or suppression. This indicates that the second prediction is closer to the true fixation point. When the error of the first prediction exceeds a threshold... Furthermore, the improvement of the second peak compared to the first peak is greater than the threshold. When this occurs, the sample is identified as an erroneous peak interference sample, and its continuous difficulty weight is defined: . It is an indicator function; The first prediction error threshold; The minimum effective improvement threshold; To improve the scale of quantity saturation; The three thresholds are determined by the coordinate normalization method of the training set and by a combination of location metrics and the proportion of selected samples on the validation set. Excluding samples that were already close to the true value, Excluding samples where occlusion did not significantly improve the situation. Control the speed at which the weights reach 1. Let the normalized projections of the first error peak region features and the corresponding text features of SALIENCY_TOKENS be respectively... and The alignment loss for the error peak interference region is: . In the formula, B represents the sample difficulty weight; B represents the batch size. To prevent division by zero constant. Only The confirmed interference samples participate in this loss. To reduce the risk that VIEW_TOKENS and SALIENCY_TOKENS converge to the same representation direction, the two sets of tokens are arranged row by row. After normalization, they are respectively and The biased token separation loss is defined as: . In the formula, and Each by A perspective biased token and The unit vectors of the error peak interference bias tokens are formed by stacking them row by row; T represents matrix transpose; It follows the Frobenius norm. This loss suppresses the overall magnitude of the pairwise inner product of the two sets of tokens to reduce overlap in representation directions. It can facilitate the formation of mutually distinguishable potential representation directions for the two types of bias cues, but it does not mean that an absolute and unique semantic decomposition can be achieved.

[0043] (5) Overall goal of the first phase Taking into account global image-text alignment, local region alignment, error peak interference region alignment, and token separation constraints, the total loss for the first stage is: . In the formula, , , and The relative contributions of unbiased global alignment, local region alignment, error peak region alignment, and biased token separation constraints are controlled separately. Each weight can be enabled sequentially on the validation set, starting from 0, and determined based on the main task localization metric, loss magnitude, and training stability. The first stage learnable parameter set... This includes a scene cue generator, a viewpoint offset token, an error peak interference offset token and its condition generator, a visual projector, and a text projector. Its optimization objective is: . In the formula, This is the set of learnable parameters for the first stage. Parameter estimation to minimize the total loss in the first stage; arg min indicates returning the parameters that minimize the objective function, not the minimum value of the objective function. The frozen image encoder, text encoder body, and gaze decoder are not included. After the first phase of training is completed, VIEW_TOKENS and SALIENCY_TOKENS are removed, and the fixed clean text features are re-exported using clean hints and saved with the sample index as the key for the second phase of training.

[0044] Phase 2: Joint Optimization of Visual Branches Without Bias Cueing The goal of the second stage is to incorporate the task-related semantics contained in the clean prompts from the first stage into the pure visual model. This stage freezes the text encoder, all prompt parameters, and the text features from the first stage, updating only the b-part visual encoder, the final normalization layer, the character head prompt module, the decoder, the heatmap head, the in-image / out-of-image classification head, and the second-stage visual projector. The clean text features are only used as training targets in loss calculations and are not used as direct input to the heatmap decoder.

[0045] (1) Specific visual representation of people, heat map prediction and in-map / out-of-map determination Suppose the visual encoder consists of L Transformer Blocks. For the i-th sample, the input image is first fed into the visual encoder, and a person-specific visual representation is constructed using the head bounding box information. Then, the i-th... The prediction process for each sample is as follows: . For the i-th RGB input image, , These represent the height and width of the input image, respectively. The normalized head bounding box coordinates of the target person; The parameter is The last two Transformer Blocks and the final normalization layer of the visual encoder; For parameters The character condition feature construction module is used to highlight the current target character and its related scene information from the scene visual features based on the head bounding box; A specific visual representation of a character.

[0046] . In the above formula, For parameters gaze heatmap decoder, This represents the predicted heatmap after Sigmoid normalization. , These represent the height and width of the output heatmap, respectively. The prediction point can be the location of the maximum response (Hi) or obtained by weighting the neighborhood of high-response points. The head bounding box is used to specify the target person to be predicted; therefore, different head bounding boxes in the same scene can yield different gaze results. Meanwhile, the in-image / out-of-image classification head outputs the in-image probability based on the person's specific visual representation. This is used to determine whether the gaze target of a person is within the input image range. Let the in-image / out-of-image threshold be... Then when The system determines that the gaze target is located inside the image and extracts the in-image gaze location from the predicted heatmap. Conversely, if the gaze target is not located inside the image, it determines that the gaze target is located outside the image, and the response location in the predicted heatmap is not considered as a valid in-image gaze point. The determination threshold can be preset to 0.5, or it can be adjusted according to the average accuracy, F1 score, or in-image / out-of-image classification accuracy on the validation set to adapt to different datasets and application scenarios.

[0047] (2) Monitoring by observing heat maps For samples with valid in-map fixation points, the true fixation points are mapped to the output heatmap coordinates, denoted as . Construct a two-dimensional Gaussian supervised heatmap: . In the formula, For row index, For column indexes, The width of the Gaussian kernel is determined by the validation set based on the output resolution and the desired supervision radius. When the supervised response prediction heatmap for pixel (u,v) has already passed through the Sigmoid function, first define the first... The binary cross-entropy heatmap loss for each sample is: . For the i-th sample in all Average binary cross-entropy at each output position; To predict probabilities; This is a soft supervision value; It is the natural logarithm. In implementation, the predicted probability can be truncated to... This is to avoid logarithmic value overflow.

[0048] The heatmap supervision loss after using an effective sample mask is: . This is the most important task loss in the second stage, used to ensure that the final pure vision branch can directly output the correct gaze heatmap. B is the batch size; ε>0 is the zero-prevention constant. To obtain an effective intra-image fixation indicator, the out-of-image fixation sample is obtained through... Excluded from heatmap loss, avoiding the incorrect compression of off-map targets to the image edges.

[0049] (3) Text features guide the visual branch Spatial pooling of specific visual features of the character is performed, and then the second-stage visual projector is used. Mapped to shared space: . Pooling operations on the spatial or token dimension; For the second stage of learnable visual projectors; Let be the unit vector representing the person's specific visual representation in the shared space; the denominator is the L2 norm of its projection vector. The clean text feature derived in the first stage is denoted as . In the second stage, a stopping gradient operation is used. The clean text-visual feature alignment loss is defined as: . In the formula, This indicates that the gradient operation has been stopped. This is a fixed, unbiased text unit vector exported in the first stage. This represents the second-stage visual unit vector; T denotes transpose. The loss is the in-batch average cosine distance. This formula only generates gradients for the visual branch and the second-stage visual projector. Here, the clean text features are only supervised through the loss function, whose gradients back-update the image encoder, the final normalization layer, and the Gazelle person-specific decoder, thus writing the person, scene, and gaze target semantics from the clean cues into the pure visual branch.

[0050] (4) Error peak region suppression The first phase of two-fold cross-inference preserves the mask of the first error peak region for difficult samples. and error peak interference weight To reduce the probability that the final visual model will continue to produce a high response in this region, we first define the first... The average response of each sample within the error peak interference region is: . In the formula, For the error peak interference region mask; To predict the response of the heat map; Let be the average predicted response of the i-th sample within the masked region.

[0051] Error peak region suppression loss is defined as: . In the formula, To weight the disturbance by the error peak Weighted intra-batch average regional suppression loss; B is the batch size; ε is the zero-prevention constant. Ordinary samples are not included in this constraint to avoid suppressing normal regions without justification.

[0052] This can minimize the predicted response in non-target regions identified through error peak mining. (Phase 1) Used to establish a correspondence between SALIENCY_TOKENS and the error peak region, in the second stage. Both are used to reduce the final visual model's reliance on that region; their functions are different but interconnected.

[0053] (5) Update of overall goals and parameters for the second phase This invention adds an in-image / out-of-image classification head after the specific visual representation of a person, outputting the probability that the target person's gaze is located inside the image. This branch shares the front-end conditional visual features of the person with the heatmap branch, but has an independent classification output layer. During training, binary cross-entropy of in-image / out-of-image is used for supervision; in-image samples participate in both heatmap supervision and in-image / out-of-image supervision, while out-of-image samples do not construct an effective in-image gaze region heatmap. Training is mainly carried out through the in-image / out-of-image branch.

[0054] Finally, combining heatmap supervision, in-image / out-of-image state supervision, clean text guidance, and error peak region suppression, the total loss for the second stage is defined as: . In the formula, The in-graph / out-of-graph binary cross-entropy loss; These are the weights for the unbiased text-guided loss and the error peak region suppression loss, respectively. The two weights are jointly selected based on the validation set AUC, distance metrics, and in-plot / out-plot AP. The text constraint can be linearly increased from 0 to the target value within the first 10% to 30% of the total training rounds to avoid overly strict text constraints in the early stages of training. The setting is based on the proportion of interfering samples and the magnitude of the loss. This is the loss function for determining whether the gaze target point is located inside or outside the image. It is used to supervise the model to determine whether the gaze target of the i-th sample is within the range of the input image. Its formula is: ; in, For real labels inside and outside the image; when the gaze target is inside the image When the object being gazed upon is outside the image , The loss is calculated using binary cross-entropy, representing the probability that the gaze target is located within the image as predicted by the model.

[0055] The second stage learnable parameter set The optimization objective, which includes only the last two Transformer Blocks of the visual encoder, the final normalization layer, the head cue module, the gaze decoder, and the second-stage visual projector, is as follows: . In the second stage, the text encoder, clean cue, bias cue, and the front network layers of the image encoder are not updated. For datasets containing in-image / out-image gaze annotations, the in-image / out-image classification head and the heatmap branch are trained synchronously, and the in-image / out-image binary cross-entropy is used as part of the total loss in the second stage; for datasets containing only in-image gaze samples, the loss weight of this branch can be reset to 0.

[0056] Reasoning stage After completing the second phase of training, the text encoder, scene cue generator, VIEW_TOKENS, SALIENCY_TOKENS, GAZE_TOKENS, and the text feature cache from the first phase are deleted. During the inference phase, only the image to be tested and the target person's head bounding box are input. The trained image encoder, gaze decoder, and in-image / out-of-image classification head output a predicted heatmap and in-image probabilities. If the in-image probability is lower than a preset threshold, the out-of-image gaze state is output; if the in-image probability is not lower than the preset threshold, the in-image gaze point location is determined from the predicted heatmap.

[0057] . Here The image to be tested is an RGB image. Normalized header bounding box for the target person; , and These represent the visual encoder, the person conditional feature construction module, and the gaze heatmap decoder, respectively, after training. This is the gaze heatmap output during the inference phase. The text encoder, cue module, or text feature cache are not invoked during inference.

[0058] Experimental results To verify the effectiveness of this invention in the gaze target localization task, experiments were conducted on two public datasets, GazeFollow and VideoAttentionTarget, and compared with existing gaze target localization methods. The method of this invention is named DPGaze. Experiments were performed using the PyTorch deep learning framework, trained on an NVIDIA GPU. In one implementation, the input image was uniformly scaled to 448×448 pixels. Since the DINOv3 patch size is 16, the input image was divided into 28×28 visual patches. The gaze decoder outputs a predicted heatmap with a size of 64×64.

[0059] Depending on the size of the visual encoder, this invention provides two versions: DPGaze-B and DPGaze-L. DPGaze-B uses DINOv3 ViT-B / 16, featuring a smaller model size and higher training efficiency; DPGaze-L uses DINOv3ViT-L / 16, offering stronger scene semantic and spatial feature extraction capabilities. Both versions employ the same cue template, error peak interference bias mining mechanism, and two-stage training strategy, differing only in the size of the visual encoder. The official ViT-B and ViT-L models both have a patch size of 16.

[0060] The input information in the table is used to indicate whether different methods rely on additional auxiliary modalities. Here, I represents the RGB image and the target human head bounding box required for the task, D represents depth information, P represents human pose information, O represents the target detection result, and E represents eye features. Although DPGaze uses CLIP and structured text hints during training, the text branch does not participate in the final inference; therefore, the input for both DPGaze-B and DPGaze-L is denoted as I.

[0061] The GazeFollow dataset is primarily used for static image gaze target localization. Each test image typically contains multiple ground truth gaze point annotations. Evaluation metrics include AUC, Avg L2, and Min L2. AUC represents the area under the ROC curve of the predicted heatmap; a higher value indicates more accurate ranking of the response of the ground truth gaze region. Avg L2 represents the normalized Euclidean distance between the predicted point and the average position of multiple annotation points. Min L2 represents the distance between the predicted point and the nearest annotation among all annotation points; for both distance metrics, lower values ​​are better.

[0062] On the GazeFollow dataset, DPGaze-B achieved an AUC of 0.958, an Avg L2 of 0.101, and a Min L2 of 0.043; DPGaze-L, using a larger-scale visual encoder, achieved an AUC of 0.961, an Avg L2 of 0.096, and a Min L2 of 0.040. Compared to DPGaze-B, DPGaze-L improved the AUC by 0.003, while reducing Avg L2 and Min L2 by 0.005 and 0.003, respectively, indicating that ViT-L / 16 can extract more comprehensive visual information in scenes with complex backgrounds, multiple people, and small targets. Both versions used only RGB images and head bounding boxes, without relying on additional inputs such as depth, pose, or object detection. Detailed results are compared in Table 1.

[0063]

[0064] The VideoAttentionTarget dataset contains both in-image and out-of-image gaze samples. Therefore, in addition to AUC and L2 distance, the AP_in / out evaluation model is used to determine whether the gaze target is located inside or outside the image. The relevant metric definitions are consistent with the evaluation methods of existing gaze target localization benchmarks. On the VideoAttentionTarget dataset, DPGaze-B achieves an AUC of 0.936, an L2 distance of 0.104, and an AP_in / out distance of 0.901; DPGaze-L achieves an AUC of 0.941, an L2 distance of 0.099, and an AP_in / out distance of 0.908. DPGaze-L improves on all three metrics compared to DPGaze-B, indicating that stronger visual encoder visual features not only help improve the accuracy of in-image gaze point localization but also help determine whether a person is gazing outside the image. Detailed results are compared in Table 2.

[0065]

[0066] The results from both datasets show that this invention, through bias cueing modeling in the first stage and clean cueing guidance in the second stage, reduces the model's dependence on viewpoint bias and high-response non-target regions. DPGaze-B is suitable for applications prioritizing computational efficiency, while DPGaze-L is suitable for applications requiring higher positioning accuracy. The final model visualization results are shown below. Figure 4 .

[0067] It should be noted that the models / frameworks / hardware mentioned in the embodiments are only one of the options and do not mean that they are limited to the fixed models / frameworks / hardware mentioned.

[0068] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any equivalent substitutions or improvements made based on the technical concept of the present invention should be considered as falling within the scope of protection of the present invention.

Claims

1. A gaze target positioning method based on error peak mining and prompt migration, characterized in that, The method includes: The first prediction heatmap is obtained by cross-inference on the training samples using an out-of-sample gaze model. The image region corresponding to the first erroneous peak in the first predicted heatmap is suppressed, and the second predicted heatmap is obtained by reasoning again. Based on the improvement of the distance between the second predicted heatmap and the real gaze point, the interference region of the erroneous peak is determined. Construct an unbiased text template containing a learnable head cue token sequence, a learnable scene cue token sequence, and a learnable gaze target cue token sequence, and a biased cue text template obtained by inserting a learnable bias token sequence into the unbiased text template, wherein the learnable bias token sequence is used to characterize bias information that interferes with gaze prediction. In the first training phase, the unbiased text template and the biased cue text template are respectively aligned with the image features of the training samples across modalities to learn the representation of the learnable biased token sequence; In the second training phase, the learnable bias token sequence is removed, and the text features corresponding to the unbiased text template are used as fixed supervision signals to constrain the gaze prediction output of the pure vision branch and reduce the pure vision branch's response dependence on the error peak interference region. After training, the text branch is removed, and the test image and the target person's head bounding box are received only through the pure vision branch, and a gaze heatmap is output.

2. The method of claim 1, wherein, The learnable bias token sequence in the bias prompt text template includes a learnable viewpoint bias token sequence and a learnable error peak interference bias token sequence. The first training phase also includes applying a bias token separation loss to promote the formation of mutually distinguishable representation directions between the learnable viewpoint bias token sequence and the learnable error peak interference bias token sequence.

3. The method of claim 2, wherein, The first training phase further includes: locally aligning the text features corresponding to the learnable error peak interference bias token sequence with the region image features corresponding to the error peak interference region.

4. The method of claim 1, wherein, The step of using an out-of-sample gaze model to perform cross-inference on the training samples to obtain the first prediction heatmap includes: Divide the training sample set into at least two mutually exclusive subsets; The fixation model is trained on each subset, and the fixation model trained on one subset is used to infer the other subset to obtain out-of-sample prediction results.

5. The method of claim 1, wherein, The step of determining the error peak interference region based on the improvement in distance between the second predicted heatmap and the actual gaze point includes: Calculate a first prediction error and a second prediction error, where the first prediction error is the distance between the predicted position of the first error peak and the actual gaze point, and the second prediction error is the distance between the predicted position of the second prediction heatmap and the actual gaze point. When the first prediction error is greater than the first threshold and the improvement of the second prediction error relative to the first prediction error is greater than the second threshold, the image neighborhood corresponding to the first error peak is determined as the error peak interference region.

6. The method of claim 1, wherein, The cross-modal alignment in the first training phase includes global image-text alignment and local region alignment; the global image-text alignment establishes a correspondence between the features of the entire image and the features of the entire prompt text. The local region alignment includes at least aligning the text features corresponding to the learnable head cue token sequence with the head region features in the image, and aligning the text features corresponding to the learnable gaze target cue token sequence with the gaze target region features in the image.

7. The method of claim 1, wherein, The gaze prediction output constrained in the second training phase for the pure visual branch includes: Using the text features corresponding to the unbiased text template as the training target, and through text-visual feature alignment loss, the task semantics between the person's head, scene context and gaze target contained in the unbiased text template are transferred to the pure visual branch. The loss is suppressed by the error peak region, which reduces the prediction response of the pure vision branch within the error peak interference region.

8. The method of claim 1, wherein, The method further includes: outputting the in-image / out-of-image state judgment result; wherein, the person-specific visual representation output by the pure visual branch is obtained, and the probability that the gaze target is located inside the image is output by the in-image / out-of-image classification head. When the probability is lower than a preset threshold, the out-of-image gaze state is output.

9. A gaze target positioning system based on error peak mining and hinting of migrated fixation, characterized in that, The system includes: The error peak mining module is used to perform cross-inference on training samples using an out-of-sample gaze model to obtain a first predicted heatmap. After suppressing the response of the image region corresponding to the first error peak in the first predicted heatmap, it infers again to obtain a second predicted heatmap. Based on the improvement of the distance between the second predicted heatmap and the real gaze point, the error peak interference region is determined. The prompt construction module is used to construct an unbiased text template containing a learnable head prompt token sequence, a learnable scene prompt token sequence, and a learnable gaze target prompt token sequence, and a biased prompt text template obtained by inserting a learnable bias token sequence into the unbiased text template, wherein the learnable bias token sequence is used to characterize bias information that interferes with gaze prediction. The first-stage training module is used to perform cross-modal alignment between the unbiased text template and the biased prompt text template and the image features of the training samples, respectively, to learn the representation of the learnable biased token sequence; The second-stage training module is used to use the text features corresponding to the unbiased text template as a fixed supervision signal after removing the learnable bias token sequence, to constrain the gaze prediction output of the pure vision branch and reduce the response dependence of the pure vision branch on the error peak interference region. The inference module is used to remove the text branch after training and receive the test image and the target person's head bounding box only through the pure vision branch, and output a gaze heatmap.

10. The system of claim 9, wherein, The learnable bias token sequence in the biased prompt text template includes a learnable viewpoint bias token sequence and a learnable error peak interference bias token sequence; the first-stage training module is also used to apply bias token separation loss to promote the learnable viewpoint bias token sequence and the learnable error peak interference bias token sequence to form mutually distinguishable representation directions.