A gaze target prediction method based on a diffusion model

Abstract

The application belongs to the technical field of computer vision, artificial intelligence and human-computer interaction, and discloses a gaze target prediction method based on a diffusion model, which comprises the following steps: S1: extracting multi-scale visual features; S2: constructing a gaze target prediction model, which comprises: a frequency perception feature enhancement module, which is used to obtain scene features and head features; a human-centered condition generation module, which is used to obtain a multi-scale condition feature set; a diffusion gaze heat map regressor, which is used to generate a gaze heat map according to the multi-scale condition feature set; and a diffusion target detector, which is used to obtain a target bounding box, a target category and a target level context feature; S3: inputting training samples into the gaze target prediction model for training; and S4: extracting multi-scale visual features from a to-be-predicted image and inputting the trained gaze target prediction model for prediction. The application can relieve the one-step prediction ambiguity problem in a complex scene and a multi-candidate target scene, and improve the gaze target prediction accuracy.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, artificial intelligence and human-computer interaction technology, and specifically relates to a gaze target prediction method based on a diffusion model. This method can be used to jointly predict the observer's gaze heatmap, target bounding box and target category based on scene image, observer's head image and head position map. Background Technology

[0002] Gaze target prediction is a crucial task in visual attention understanding. Its goal is not only to generate gaze heatmaps reflecting the distribution of an observer's visual attention, but also to identify the specific target the observer is actually gazing at, including the target's location and semantic category. This task can be applied to scenarios such as human-computer interaction, autonomous driving, service robots, retail behavior analysis, assisted medical observation, intelligent monitoring, and virtual reality.

[0003] Existing staring target prediction methods mostly employ discriminative models such as convolutional neural networks (CNNs) or Transformers, directly generating staring heatmaps, target bounding boxes, and target categories through a single forward propagation. These methods achieve good results when dealing with a single target or relatively simple scenes. However, in real-world scenarios, when multiple candidate objects exist in the observer's gaze direction or the background is cluttered, one-step deterministic prediction easily overlooks the uncertain relationships between the observer's head, the scene, and the staring target, leading to problems such as inaccurate staring heatmap distribution, unstable staring target localization, or incorrect category prediction.

[0004] Diffusion models, capable of progressively recovering the target state from a noisy state, have been used in structured vision tasks such as image generation and object detection. Existing diffusion model detection methods primarily target ordinary object detection, failing to explicitly model the relationship between the observer, the scene, and the staring target. Therefore, an end-to-end method is urgently needed to solve the staring target prediction task, improving the accuracy, stability, and interpretability of staring target prediction in complex scenes. Summary of the Invention

[0005] The purpose of this invention is to provide a gaze target prediction method based on a diffusion model, in order to solve the problems existing in the prior art:

[0006] (1) Existing methods usually employ one-step discriminative prediction, which makes it difficult to express the relationship between the observer, the scene, and the gaze target when faced with multiple candidate targets.

[0007] (2) The information coupling between the gaze heatmap regression branch and the target detection branch is insufficient, making it difficult to output accurate gaze heatmap, target bounding box and target category at the same time.

[0008] (3) Ordinary spatial domain convolution features are prone to losing target boundaries and local structural details, making it difficult to distinguish adjacent candidate targets in complex scenarios such as shelves, desktops, and densely arranged multiple objects.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A staring target prediction method based on a diffusion model, characterized by the following steps: S1: Extract multi-scale visual features from scene images and observer head images to obtain scene spatial features, head spatial features and object detection branch features; S2: Construct a gaze target prediction model, which includes the following modules: The frequency-aware feature enhancement module is used to perform wavelet frequency decomposition on the scene spatial features and head spatial features obtained by S1 to obtain low-frequency sub-bands and high-frequency sub-bands, and reconstruct the low-frequency sub-bands and high-frequency sub-bands into frequency-aware features, which are then fused with the original spatial domain features to obtain scene features and head features that fuse spatial semantic information and frequency domain detail information. A human-centered conditional generation module is used to generate a head position map. The head position map is then fused with scene features and head features obtained by the frequency-aware feature enhancement module after fusing spatial semantic information and frequency domain detail information. The fusion result is then fused with the target-level context features obtained by the diffusion target detector to obtain a multi-scale conditional feature set. The diffused gaze heatmap regressor is used to generate gaze heatmaps by diffusion denoising based on the multi-scale conditional feature set obtained from the human-centered conditional generation module. The diffusion target detector is used to obtain the target detection branch features obtained from S1, and the target bounding box is used as the diffusion modeling object. Through a stepwise denoising process from noisy candidate boxes to target bounding boxes, the target bounding box and target category are obtained, and the target-level contextual features are obtained at the same time. S3: Input the training samples into the gaze target prediction model to train it, and obtain the trained gaze target prediction model. S4: Extract multi-scale visual features from the image to be predicted and input them into the trained gaze target prediction model, outputting the observer's predicted gaze heatmap, target bounding box, and target category.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention unifies the staring heatmap prediction and target detection into a diffusion denoising process, which can start from the noise heatmap and the candidate state of the noise target bounding box and gradually refine the staring distribution and target candidate results in an iterative manner, thereby alleviating the one-step prediction ambiguity problem in complex scenes and multi-candidate target scenes.

[0011] (2) This invention integrates scene image features, head image features, head position prior and target-level context features through a frequency-aware feature enhancement module and a human-centered condition generation module, so that the staring target prediction model has clear interaction constraints of observer, scene and staring target in the diffusion denoising process, thereby improving the accuracy of staring target prediction.

[0012] (3) This invention extracts low-frequency global information and high-frequency boundary structure information through the fusion operation of frequency domain and spatial domain, which can enhance the model’s ability to perceive target contours, adjacent target boundaries and fine-grained local structures, and is beneficial to distinguish the real staring target among multiple adjacent candidate targets. Detailed Implementation

[0013] The specific embodiments of the present invention will be described in detail below.

[0014] 1. Input and Output Definitions This invention provides a gaze target prediction method based on a diffusion model. The gaze target prediction refers to a joint visual prediction task that simultaneously performs gaze heatmap prediction and detects the target bounding box and target category. The input to this method includes a scene image, denoted as... And the head image obtained by cropping the observer's head region from the scene image is denoted as... It also includes a head position map generated from the observer's head bounding box position, denoted as... .

[0015] The output of the method of the present invention includes: a gaze thermogram denoted as The target bounding box is denoted as And the target category is denoted as .

[0016] 2. Specific technical steps The staring target prediction method based on a diffusion model provided in this invention includes the following steps: S1: Image Input and Multi-Scale Feature Extraction: Extracting multi-scale visual features from scene images and observer head images to obtain scene spatial features. Head spatial features and object detection branch features .

[0017] S1 specifically includes the following sub-steps: S1.1, Input scene images acquired in the same scene. and observer head image The scene image described and observer head image The input is a shared backbone network. In this shared backbone network, ResNet is used as the backbone, and a feature pyramid network is combined to fuse visual information from different levels to extract multi-scale visual features. Among these features, scene images... Used to provide global environment information, candidate target information, and scene spatial layout information; observer head image. This provides local appearance information related to the observer's head pose, head orientation, and gaze direction. After processing by ResNet-FPN, it outputs three types of multi-scale features, namely scene multi-scale features. Head multi-scale features } and object detection branch features The process is represented as: { }, { }, = ResNet-FPN( , (1) in, The multi-scale visual features extracted from scene images are denoted as scene multi-scale features; } represents the multi-scale visual features extracted from the observer's head image, denoted as head multi-scale features; This represents the multi-scale detection features used in subsequent object detection branches, denoted as object detection branch features.

[0018] S1.2, Scene multi-scale features obtained from S1.1 { } and head multi-scale features { The features from the layer with the strongest semantics are selected and further refined using residual blocks to obtain scene space features. and head spatial features ,in, , ∈ .

[0019] S2: Construct a gaze target prediction model, which includes the following: a frequency-aware feature enhancement module, a human-centered condition generation module, a diffuse gaze heatmap regressor, and a diffuse target detector, wherein: The frequency-aware feature enhancement module is used to enhance the scene spatial features obtained by S1. and head spatial features Wavelet frequency decomposition is performed to obtain low-frequency and high-frequency subbands. These subbands are then reconstructed into frequency-aware features, which are subsequently fused with the original spatial domain features to obtain scene features that integrate spatial semantic information and frequency domain detail information. and head features .

[0020] The specific process of the frequency-aware feature enhancement module is as follows: S2.1, the scene space features obtained in S1 and head spatial features Input a three-stage wavelet convolutional cascade structure. Since multi-stage wavelet decomposition requires the feature map space size to be even, the scene space features can be decomposed first. and head spatial features The 7×7 features are zero-paddinged to 8×8 before the first stage of wavelet convolution is performed.

[0021] S2.2: Spatial-Frequency Feature Fusion S2.2.1, in the first stage of wavelet convolution, the zero-padding scene space features obtained in S2.1 are... and head spatial features After wavelet transform, each frequency band is decomposed into a low-frequency subband. With three high-frequency sub-bands , , , respectively representing global layout information, horizontal structure information, vertical structure information, and diagonal structure information.

[0022] { } = WTConv( (2) { } = WTConv( (3) Where, the subscript 's' represents 'scene', corresponding to the scene image branch; the subscript 'h' represents 'head', corresponding to the head image branch; wavelet convolution WTConv is performed on these two features respectively to obtain... This represents the low-frequency subband, used to preserve coarse-grained scene layout and long-range contextual information; , , These are the high-frequency subbands, used to preserve object contours, boundary responses, and local structural details. In subsequent stages, the low-frequency subbands obtained in the previous stage are used... The input is used to continue the recursive decomposition to obtain deeper frequency subbands. Through recursive frequency decomposition, the scene branch and the head branch can simultaneously retain low-frequency global semantic information and high-frequency local structural information, thereby enhancing the spatial perception and detail expression capabilities in subsequent line-of-sight condition modeling.

[0023] S2.2.2, remap the frequency sub-bands obtained in S2.2.1 back to the spatial domain to obtain frequency-aware features. Specifically, the low-frequency sub-bands of the scene image branch... and high-frequency subband , , Inputting the transposed convolutional module TConv yields scene frequency reconstruction features. ; the low-frequency subband of the head branch and high-frequency subband , , Inputting the transposed convolutional module TConv yields the head frequency reconstruction features. The process is represented as: = TConv( (4) = TConv( (5) Here, TConv represents the transpose convolution operation, which is used to restore the frequency subband features to a spatial domain representation that is compatible with the original spatial features.

[0024] S2.2.3, the reconstructed spatial domain frequency-aware features are fused with the original spatial domain features. Specifically, the original spatial features are first projected using convolution (Conv), and then concatenated with the corresponding frequency-aware features along the channel dimension.

[0025] = Conv( ) ⊕ (6) = Conv( ) ⊕ (7) Where ⊕ indicates splicing along the channel dimension; This represents the scene features after fusing spatial semantic information and frequency domain detail information; This represents the head features after fusing spatial semantic information and frequency domain detail information. Through this step, the present invention enhances object boundaries, local textures, and candidate object structural information while preserving semantic layout.

[0026] The human-centered conditional generation module generates a head location map. This head location map is then fused with scene features and head features obtained from the frequency-aware feature enhancement module (which integrates spatial semantic information and frequency domain detail information). The fused result is then fused with target-level contextual features obtained from the diffusion target detector to obtain a multi-scale conditional feature set. .

[0027] The specific process of the human-centered condition generation module is as follows: (1) Generate a head position map based on the observer's head bounding box position, denoted as Head position diagram Scene features after encoding and fusion of spatial semantic information and frequency domain detail information The scene features are obtained by fusing them together and incorporating the observer's prior position. .

[0028] = Conv( ) ⊕ (8) (2) Head position diagram After max pooling, flattening, and feature mapping, the head features are combined with spatial semantic information and frequency domain detail information. The head features are fused together to obtain head features with observer attention weights. .

[0029] = Conv(MaxPool( ), (9) (3) Through element-wise product pairs and Interaction modeling is performed to explicitly characterize the relationship between scene context and head-related gaze cues. Simultaneously, target-level contextual features are introduced. This is added as a supplementary condition to the fusion representation to enhance the semantic information of the candidate gaze target region. During the training phase, The target-level contextual features obtained during the training phase are used by the diffusion target detector; during the inference phase, The target-level context features obtained during the inference phase are obtained using a diffusion target detector. Used to enhance multi-scale conditional feature sets The ability to perceive candidate gaze target areas.

[0030] = Upsample( ⊙ ⊙ (10) Where ⊙ represents element-wise product; Upsample represents upsampling operation; It is a multi-scale conditional feature set.

[0031] By using a human-centered conditional generation module, a multi-scale conditional feature set is obtained that simultaneously includes spatial-frequency gaze cues, human-centered attention priors, and contextual features. It is used to guide the diffusion gaze heatmap regressor in denoising gaze heatmaps.

[0032] Diffusion gaze heatmap regressor: Based on the multi-scale conditional feature set obtained from the human-centered conditional generation module. A gaze heatmap is generated using a diffusion denoising method.

[0033] The specific process of the diffusion gaze heatmap regressor is as follows: ① During the training phase, input real gaze heatmaps. And randomly sample the diffusion time step t and Gaussian noise. Based on the forward noise addition process, the true gaze heatmap is... Noise is added to obtain a noise staring heatmap at time step t. ,in The process is represented as: (11) Where ε represents standard Gaussian noise, Let represent the cumulative noise scheduling coefficient corresponding to the t-th time step, and , This represents the noise heatmap at time step t. As t increases, It gradually approaches random Gaussian noise.

[0034] ② Noise staring heatmap The diffusion time step t and the multi-scale conditional feature set obtained by the human-centered conditional generation module. In the input conditional U-Net, in the multi-scale conditional feature set Guided by the algorithm, the noise-generating gaze heatmap is denoised and predicted, and the predicted gaze heatmap is output. .

[0035] (12) in, The conditional U-Net with parameter θ; t is injected into the conditional U-Net via sinusoidal time encoding; multi-scale conditional feature set. Involves feature modulation at multiple scale levels in U-Net to guide heatmap recovery to attention-related regions.

[0036] ③Diffusion gaze heatmap regression loss Using predictive gaze heatmaps Compared to real gaze heatmap L2 loss between: (13) ④ During the inference phase, the diffusion staring heatmap regressor starts with a random Gaussian noise heatmap and then applies it to a multi-scale conditional feature set. Guided by the DDIM sampling strategy, multi-step iterative denoising was performed to obtain the observer's gaze heatmap.

[0037] The diffusion target detector is used to obtain the target detection branch features obtained from S1, with the target bounding box as the diffusion modeling object. Through a stepwise denoising process from noisy candidate boxes to target bounding boxes, the target bounding box and target category are obtained, and the target-level context features are obtained at the same time.

[0038] The specific process of the diffusion target detector is as follows: A. During the training phase, use the real target bounding box. As the diffusion starting point, Gaussian noise is added to it according to a preset noise schedule to obtain the diffusion time step. Noise candidate boxes below Then, the noise candidate boxes diffusion time step and object detection branch features Inputting a diffusion target detector based on the DiffusionDet structure enables the diffusion target detector to learn from noisy candidate boxes. Inverse recovery of predicted target bounding box The denoising process yields target-level contextual features. And simultaneously predict the corresponding target category. This process can be represented as: (14) in, This indicates a diffusion target detector based on DiffusionDet. This represents the predicted target bounding box. Indicates the predicted target category, This represents target-level contextual features.

[0039] B. During the inference phase, from randomly generated noisy candidate boxes Starting with object detection branch features The candidate boxes are gradually refined through multi-step iterative denoising, and the final target bounding box is obtained. and target category During the multi-step iterative denoising process, the diffusion target detector based on DiffusionDet simultaneously generates target-level contextual features. .

[0040] C. Construct a diffusion target detection loss, denoted as... The detection loss is used to constrain the prediction results of the target bounding box and the target category, and includes the target category prediction loss and the target bounding box regression loss. Its form can be expressed as: (15) in, Indicates the target category prediction loss. This represents the regression loss of the target bounding box.

[0041] S3: Joint Optimization Objective during Training Phase: Input training samples into the gaze target prediction model for training, resulting in a well-trained gaze target prediction model. The training samples include scene images. Observer head image Head position diagram Real gaze heatmap True target bounding box And the true target category y. Specifically, the training samples are input into the gaze target prediction model to obtain predicted gaze heatmaps. Predicted target bounding box and predict target category According to the predicted gaze heatmap Compared to real gaze heatmap Calculate the regression loss of the diffusion gaze heatmap Based on the predicted target bounding box Predict target category bounding box of the real target Calculate the diffusion target detection loss based on the true target category y. The overall objective function during training is as follows: (16) in, To reduce the regression loss of the gaze heatmap, Loss for target detection; These are the loss weighting coefficients, used to balance the proportions of gaze heatmap regression loss and object detection loss in the overall training objective. The value is 1000.

[0042] S4: Model Inference and Gaze Target Output: Extract multi-scale visual features from the image to be predicted and input them into the trained gaze target prediction model. Output the predicted observer gaze heatmap, target bounding box, and target category.

[0043] S4 specifically includes the following operations: Obtain scene images from the scene to be predicted and observer head image The scene image and the observer's head image are then normalized in size and pixels, and the head region is cropped. Simultaneously, a head position map is generated based on the observer's head bounding box position. ; Processed scene image Observer head image and head position diagram Input the trained gaze target prediction model and output the predicted gaze heatmap. Predicted target bounding box and predict target category Thus, the end-to-end inference process from the input scene image and the observer's head image to the prediction result of the staring target is completed.

[0044] 3. Example: To verify the effectiveness of the method of this invention, experiments were conducted on the publicly available gaze target prediction datasets GOO-Synth and GOO-Real. Experimental samples included scene images, observer head images, gaze point annotations, target bounding box annotations, and target category annotations. The training set was input into the gaze target prediction model of this invention for joint optimization training, and the gaze heatmap prediction results and target detection results were evaluated on the test set, respectively.

[0045] In the experiment, mSoC and wUoC were used to evaluate the staring target prediction performance, AP was used to evaluate the target detection performance, and AUC, Dist., and Ang. were used to evaluate the staring heatmap prediction performance. The experimental results are shown in Tables 1 to 3. In the following experiments, the method of this invention will be referred to as DiffGOP.

[0046] Table 1. Prediction Results of Gazing Targets

[0047] Table 2 Target Detection Results

[0048] Table 3. Results of gaze point estimation

[0049] As shown in Table 1, in the gaze target prediction task, the method of this invention achieved the best results on both the GOO-Synth and GOO-Real datasets. On the GOO-Synth dataset, the mSoC of the method of this invention reached 93.5, higher than GaTector's 67.9 and TransGOP's 92.8; wUoC reached 76.6, significantly higher than GaTector's 28.5. This indicates that the method of this invention can not only accurately determine the target object being gazed at by the observer, but also has better stability in evaluating the degree of overlap of the target region. On the GOO-Real dataset, the mSoC of the method of this invention reached 97.4, an improvement of 35.0 compared to GaTector and 14.8 compared to TransGOP; wUoC reached 85.9, also significantly higher than GaTector's 3.31. The above results show that the method of this invention still has strong gaze target localization capabilities in real complex scenes.

[0050] As shown in Table 2, the method of this invention also achieved good detection performance in the object detection task. On the GOO-Synth dataset, the AP of the method of this invention reached 88.4, which is higher than GaTector, DINO, and TransGOP, indicating that the progressive denoising and optimization of the target candidate region by the diffusion target detector can improve the detection accuracy of staring targets. On the GOO-Real dataset, the AP of the method of this invention reached 94.6, which is a significant improvement compared to GaTector's 52.2, DINO's 82.8, and TransGOP's 84.1. These results show that the method of this invention can more accurately detect stared targets in real images and has better adaptability to complex backgrounds, occlusion, and multi-target interference scenes.

[0051] As shown in Table 3, the method of this invention maintains good overall performance on both datasets in the gaze point estimation task. On the GOO-Synth dataset, the method of this invention achieves an AUC of 0.964, a distance error (Dist.) of 0.075, and an angle error (Ang.) of 13.7, outperforming GaTector and approaching the level of TransGOP. On the GOO-Real dataset, the method of this invention has a Dist. of 0.090, lower than GaTector's 0.111 and TransGOP's 0.097, indicating that the method of this invention can predict the gaze point location more accurately. Although the AUC and Ang. metrics are close to TransGOP, the method of this invention achieves more significant improvements in object detection and gaze target prediction tasks, indicating that it is more suitable for joint modeling in gaze target prediction scenarios.

[0052] The experimental results summarized in Tables 1 to 3 show that the method of this invention, by introducing frequency-aware feature enhancement, human-centered condition generation, and a progressive denoising prediction mechanism for the diffusion target detector, can effectively improve the accuracy of staring target prediction and target detection in complex scenes. Compared with methods based on convolutional neural networks (CNN) and Transformer-based methods, the method of this invention demonstrates superior staring target prediction and target detection capabilities on both the GOO-Synth and GOO-Real datasets, verifying the effectiveness and practical value of the method of this invention.

Claims

1. A staring target prediction method based on a diffusion model, characterized in that, Includes the following steps: S1: Extract multi-scale visual features from scene images and observer head images to obtain scene spatial features, head spatial features and object detection branch features; S2: Construct a gaze target prediction model, which includes the following modules: The frequency-aware feature enhancement module is used to perform wavelet frequency decomposition on the scene spatial features and head spatial features obtained by S1 to obtain low-frequency sub-bands and high-frequency sub-bands, and reconstruct the low-frequency sub-bands and high-frequency sub-bands into frequency-aware features, which are then fused with the original spatial domain features to obtain scene features and head features that fuse spatial semantic information and frequency domain detail information. A human-centered conditional generation module is used to generate a head position map. The head position map is then fused with scene features and head features obtained by the frequency-aware feature enhancement module after fusing spatial semantic information and frequency domain detail information. The fusion result is then fused with the target-level context features obtained by the diffusion target detector to obtain a multi-scale conditional feature set. The diffused gaze heatmap regressor is used to generate gaze heatmaps by diffusion denoising based on the multi-scale conditional feature set obtained from the human-centered conditional generation module. The diffusion target detector is used to obtain the target detection branch features obtained from S1, with the target bounding box as the diffusion modeling object. Through a stepwise denoising process from noisy candidate boxes to target bounding boxes, the target bounding box and target category are obtained, and the target-level context features are obtained at the same time. S3: Input the training samples into the gaze target prediction model to train it, and obtain the trained gaze target prediction model. S4: Extract multi-scale visual features from the image to be predicted and input them into the trained gaze target prediction model, outputting the observer's predicted gaze heatmap, target bounding box, and target category.

2. The staring target prediction method based on a diffusion model as described in claim 1, characterized in that, In S2, the specific process of the frequency-aware feature enhancement module is as follows: S2.1, the scene space features obtained in S1 and head spatial features Input a three-stage wavelet convolutional cascade structure; S2.2: Spatial-frequency feature fusion, including the following process: S2.2.1, in the first stage of wavelet convolution, the zero-padding scene space features obtained in S2.1 are... and head spatial features After wavelet transform, each frequency band is decomposed into a low-frequency subband. With three high-frequency sub-bands , , These represent global layout information, horizontal structure information, vertical structure information, and diagonal structure information, respectively. { } = WTConv( ) { } = WTConv( ) Wherein, the subscript s represents scene, corresponding to the scene image branch; the subscript h represents head, corresponding to the head image branch; For low-frequency sub-band; , , These represent the high-frequency subbands; WTConv represents wavelet convolution; in subsequent stages, the low-frequency subbands obtained in the previous stage are... The input is used to continue the recursive decomposition to obtain deeper frequency sub-bands; S2.2.2, remap the frequency sub-bands obtained in S2.2.1 back to the spatial domain to obtain frequency-aware features; specifically, this involves remapping the low-frequency sub-bands of the scene image branch. and high-frequency subband , , Inputting the transposed convolutional module TConv yields scene frequency reconstruction features. ; the low-frequency subband of the head branch and high-frequency subband , , Inputting the transposed convolutional module TConv yields the head frequency reconstruction features. ; S2.2.3, the reconstructed spatial domain frequency-aware features are fused with the original spatial domain features; specifically, the original spatial features are first projected through convolution Conv, and then spliced ​​with the corresponding frequency-aware features along the channel dimension; = Conv( ) ⊕ = Conv( ) ⊕ Where ⊕ indicates splicing along the channel dimension; This represents the scene features after fusing spatial semantic information and frequency domain detail information; This represents the head features after fusing spatial semantic information and frequency domain detail information.

3. The staring target prediction method based on a diffusion model as described in claim 2, characterized in that, In S2, the specific process of the human-centered condition generation module is as follows: (1) Generate a head position map based on the observer's head bounding box position, denoted as Head position diagram Scene features after encoding and fusion of spatial semantic information and frequency domain detail information The fusion yields scene features incorporating prior observer positions. : = Conv( ) ⊕ ; (2) Head position diagram After max pooling, flattening, and feature mapping, the head features are combined with spatial semantic information and frequency domain detail information. The head features are fused together to obtain head features with observer attention weights. : = Conv(MaxPool( ), ); (3) Through element-wise product pairs and Interaction modeling is performed; simultaneously, target-level contextual features are introduced. As a supplementary condition added to the fusion representation, a multi-scale conditional feature set is obtained. During the training phase, The target-level contextual features obtained during the training phase of the diffusion target detector are used; During the reasoning stage, The target-level context features obtained during the inference phase are employed using a diffusion target detector. = Upsample( ⊙ ⊙ ) Where ⊙ represents element-wise product; Upsample represents upsampling operation.

4. The staring target prediction method based on the diffusion model as described in claim 3, characterized in that, In S2, the specific process of the diffusion gaze heatmap regressor is as follows: ① During the training phase, input real gaze heatmaps. And randomly sample the diffusion time step t and Gaussian noise. Based on the forward noise addition process, the true gaze heatmap is... Noise is added to obtain a noise staring heatmap at time step t. : Where ε represents standard Gaussian noise, Let represent the cumulative noise scheduling coefficient corresponding to the t-th time step, and , This represents the noise heatmap at time step t; as t increases, Gradually approaching random Gaussian noise; ② Noise staring heatmap The diffusion time step t and the multi-scale conditional feature set obtained by the human-centered conditional generation module. In the input conditional U-Net, through multi-scale conditional feature sets Guided by the algorithm, the noise-generating gaze heatmap is denoised and predicted, and the predicted gaze heatmap is output. : in, Represents a conditional U-Net with parameter θ; ③Diffusion gaze heatmap regression loss Using predictive gaze heatmaps Compared to real gaze heatmap L2 loss between: ④ In the inference phase, the diffusion stare heatmap regressor starts with a random Gaussian noise heatmap, and then... Guided by the DDIM sampling strategy, multi-step iterative denoising was performed to obtain the observer's gaze heatmap.

5. The staring target prediction method based on the diffusion model as described in claim 4, characterized in that, In S2, the specific process of the diffusion target detector is as follows: A. During the training phase, use the real target bounding box. As the diffusion starting point, Gaussian noise is added to it according to a preset noise schedule to obtain the diffusion time step. Noise candidate boxes below Then, the noise candidate boxes are... diffusion time step and object detection branch features Inputting a diffusion target detector based on the DiffusionDet structure enables the detector to learn from noisy candidate boxes. Inverse recovery of predicted target bounding box The denoising process yields target-level contextual features. And simultaneously predict the corresponding target category. The process is represented as: in, This indicates a diffusion target detector based on DiffusionDet. This represents the predicted target bounding box. Indicates the predicted target category, Represents target-level contextual features; B. During the inference phase, the actual target bounding box is not used. Instead, it is from randomly generated noisy candidate boxes. Starting with object detection branch features The candidate boxes are gradually refined through multi-step iterative denoising, and finally the predicted target bounding box is obtained. and target category During the multi-step iterative denoising process, the diffusion target detector based on DiffusionDet simultaneously generates target-level contextual features. ; C. Construct a diffusion target detection loss, denoted as... The detection loss is used to constrain the prediction results of the target bounding box and the target category, and includes the target category prediction loss and the target bounding box regression loss, which are expressed in the following form: in, Indicates the target category prediction loss. This represents the regression loss of the target bounding box.

Images