Gaze target detection method and system based on double-branch diffusion model

Abstract

The invention provides a fixation target detection method and system based on a double-branch diffusion model, and belongs to the technical field of target detection, and the method comprises the steps: obtaining a to-be-detected scene image containing a detected person; using a pre-trained fixation target detection model to process the acquired to-be-detected scene image containing the detected person to obtain a fixation target detection result; the fixation target detection model comprises a fixation target detection backbone network, a scene perception diffusion branch network and a semantic perception diffusion branch network. According to the method, process constraints are applied to the trunk network, the accuracy of extracting the related scene features of the fixation target is enhanced, and the prediction result is difficult to be interfered by noise factors possibly existing in a complex scene. On the basis of a semantic perception diffusion branch of a pre-training diffusion model, prior knowledge of the pre-training diffusion model is utilized to supplement semantic information related to a gaze target for a trunk network, and the robustness of gaze target detection of the model on a detected person in a non-visible scene is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of gaze target detection, and in particular to a gaze target detection method and system based on a double-branch diffusion model. Background Art

[0002] Gaze target detection aims to infer the area or object that the subject is looking at in a scene and is one of the important means to understand human behavior. Early gaze target detection methods were all invasive methods, requiring the use of physical equipment such as eye trackers to detect the direction of gaze and then detect the gaze target based on the gaze direction. In recent years, with the development of deep learning, non-invasive deep learning-based gaze target detection technology has gradually become mainstream. Its core idea is to design an effective deep learning model for gaze target detection and detect the location of the subject's gaze target based on the appearance of the subject in a large number of annotated image or video samples. Depending on the core information source relied on when modeling gaze behavior, existing methods can be divided into the following categories: gaze target detection methods based on scene saliency, gaze target detection methods based on multimodal information, and gaze target detection methods based on human-scene interaction.

[0003] Existing gaze target detection models have poor scene information perception capabilities, difficulty in accurately extracting gaze target-related features, ignoring implicit semantic information in the scene, and weak generalization capabilities.

[0004] (1) The model has poor perception of scene information and is difficult to accurately extract features related to the gaze target. The existing technical solutions essentially regard gaze target detection as a regression problem, that is, the input scene image is mapped into a gaze target position heat map. Subsequently, the model is mainly optimized by pixel loss between the gaze target position heat maps. This loss function has weak constraints on the model and is difficult to fully encourage the model to mine useful gaze target related information from the scene image. Therefore, the features they extract are easily interfered by noise factors that may be brought by multimodal information, and perform poorly in complex scenes, such as when the target is surrounded by interfering objects or there are multiple salient targets in the field of view.

[0005] (2) Ignoring the implicit semantic information in the scene, the generalization ability is weak. The existing technical solutions only use the explicit visual information of the scene image (such as gaze direction, human posture, etc.), but ignore the implicit semantic information (such as role identity, event context or activity clues, etc.). The subject's attention can reflect the role he plays in the event, and his gaze target can be determined by understanding the subject's activities. For example, players (roles) in a football game (event) usually focus on the football (target). Therefore, the existing solutions are prone to overfitting on specific training scenes that only contain fixed scenes, and cannot generalize to other unseen scene images, and there is serious overfitting in the detection of gaze targets of subjects with different identities in the same scene. Summary of the Invention

[0006] The object of the present invention is to provide a gaze target detection method and system based on a dual-branch diffusion model to solve at least one technical problem existing in the above-mentioned background technology.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for detecting a gaze target based on a dual-branch diffusion model, comprising:

[0009] Acquire a scene image to be detected containing a subject;

[0010] A pre-trained gaze target detection model is used to process the acquired scene image containing the subject to be detected to obtain a gaze target detection result; the gaze target detection model includes a gaze target detection backbone network, a scene-aware diffusion branch network and a semantic-aware diffusion branch network; the gaze target detection backbone network is used to extract gaze target-related scene features from the scene image to be detected, aggregate them and regress them into a gaze target position heat map; the scene-aware diffusion branch network is used to impose interpretable constraints on the target-related feature extraction process of the gaze target detection backbone network, divide the given gaze target position heat map into grid heat maps, and gradually add noise in the Markov diffusion process. At the same time, the gaze target-related features extracted by the backbone network are used as denoising conditions and injected into the branch network through the modulation unit to capture factors affecting the gaze position at different granularities; the semantic-aware diffusion branch network is used to provide semantic priors to enhance the primary scene features extracted by the gaze target detection backbone network.

[0011] As a further limitation of the first aspect of the present invention, the gaze target detection backbone network extracts gaze target related scene features from the input image, aggregates and regresses them into a final prediction result; wherein, gaze target detection related features are extracted by a feature extraction module, and the features are mapped into a gaze target position heat map by a regression module; the regression module includes a feature aggregation module and two prediction heads, and the two prediction heads are used to generate a gaze target position heat map and a classification result of whether the gaze target is located inside or outside the image, respectively.

[0012] As a further limitation of the first aspect of the present invention, the data input to the gaze target detection backbone network includes: the original scene image, which is used to extract global scene information; the cropped image of the subject's head, which is used to extract the subject's gaze direction information; the subject's binary head position mask, which is used to extract the positional relationship information between the subject and the scene; the normalized depth map corresponding to the scene image, which is used to extract the three-dimensional geometric information of the scene; all input images are stacked along the channel dimension and input to obtain the main scene features; after being processed by the feature aggregation layer of the self-attention mechanism, the main scene features are refined by two regression heads into a gaze target position heat map and internal and external classification results, respectively.

[0013] As a further limitation of the first aspect of the present invention, for a given gaze target position heat map, it is evenly divided into m×m grids, and the probability score of each grid is determined by the maximum value of the heat map in the grid; noise is gradually added to the grid heat map in a Markov diffusion process; during training, a UNet decoder is used to predict the noise added to a given noise image, and the main scene feature f sce ∈R c×h′×w′ After flattening and linear layer adjustment, we get

[0014] As a further limitation of the first aspect of the present invention, the scene-aware diffusion branch network further includes a potential embedding modulation unit that aligns the potential embedding of the UNet decoder with the main network scene features. Splitting into scaling factors via linear projection and bias Adaptive group normalization is used to modulate the potential embeddings of adjacent layers. Through multi-layer modulation, factors affecting gaze position at different granularities are captured, thereby improving generation quality.

[0015] As a further limitation of the first aspect of the present invention, the semantic perception diffusion branch network includes a variational autoencoder and a UNet module, with the original scene image I sce ∈R 3×h×w As input, the variational autoencoder first maps the scene image to the latent space and obtains the latent representation f vae ∈R c×n×n ; UNet module with f vae As input, and with the adjusted main scene features As a condition, the feature maps of each layer of UNet are uniformly scaled to 1 / 4 resolution of the latent space to generate fused semantic features; the adaptive fusion module consists of cascaded attention units, namely two self-attention units and a class cross-attention unit; first, the main scene feature f sce and semantic features f sem Two self-attention units are applied respectively; then, the attention map and f are tagged using learnable weights ω.sec application class cross attention unit; finally, the enhanced target-related features are mapped to the final result by a regression module.

[0016] In a second aspect, the present application provides a gaze target detection system based on a double-branch diffusion model, comprising:

[0017] An acquisition module is configured to acquire a scene image containing a subject to be detected.

[0018] A processing module is configured to process the acquired scene image containing the subject to be detected using a pre-trained gaze target detection model to obtain a gaze target detection result. The gaze target detection model comprises a gaze target detection backbone network, a scene-aware diffusion branch network, and a semantic-aware diffusion branch network. The gaze target detection backbone network is configured to extract gaze target-related scene features from the scene image to be detected, aggregate and regress the gaze target-related scene features into a gaze target location heat map. The scene-aware diffusion branch network is configured to impose an interpretable constraint on the target-related feature extraction process of the gaze target detection backbone network, divide the given gaze target location heat map into a grid heat map, and gradually add noise in a Markov diffusion process, while taking the gaze target-related features extracted by the backbone network as a denoising condition, injecting the branch network through a modulation unit, and capturing factors affecting the gaze position at different granularities. The semantic-aware diffusion branch network is configured to provide semantic priors to enhance the primary scene features extracted by the gaze target detection backbone network.

[0019] In a third aspect, the present application provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the gaze target detection method based on the double-branch diffusion model as described in the first aspect.

[0020] In a fourth aspect, the present application provides a computer device comprising a memory and a processor, wherein the processor and the memory are in communication with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the gaze target detection method based on the double-branch diffusion model as described in the first aspect.

[0021] In a fifth aspect, the present application provides an electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute instructions for implementing the gaze target detection method based on the double-branch diffusion model as described in the first aspect.

[0022] Terminology:

[0023] Gaze target detection: Gaze target prediction aims to determine whether the subject's gaze target is located within the scene image and predict the precise location of the gaze target. It is one of the hot research areas in computer vision.

[0024] Diffusion model: The diffusion model is a generative model that simulates data by gradually adding noise (forward process) and reversely learning denoising (reverse process). It can ultimately generate high-quality images, audio, and other content based on input conditions. The core idea is to allow the model to learn underlying patterns through data destruction and reconstruction.

[0025] The present invention has the following beneficial effects: The scene perception diffusion branch based on the diffusion model leverages the diffusion model's powerful generation and representation capabilities to impose process constraints on the backbone network, enhancing its accuracy in extracting scene features related to the gaze target and making the prediction results less susceptible to noise interference in complex scenes. The semantic perception diffusion branch based on the pre-trained diffusion model utilizes the prior knowledge of the pre-trained diffusion model to supplement the backbone network with semantic information related to the gaze target, enhancing the model's robustness in detecting the subject's gaze target in unseen scenes.

[0026] Additional advantages of the present invention will be more clearly given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 This is a framework diagram of a gaze target detection model based on a dual-branch diffusion model according to an embodiment of the present invention.

[0029] Figure 2 This is a grid heat map obtained by gradually denoising a noisy image using the noise predicted by the scene-aware diffusion branch according to an embodiment of the present invention.

[0030] Figure 3 This is a qualitative comparison result diagram of the gaze target position heat map described in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Embodiments of the present application will be described in detail below with reference to the attached drawings, which are given by way of illustration and thus do not limit the present application. In addition, the present application can have various modifications and various examples, and they can be implemented in other specific forms without departing from the spirit of the present application. The purpose of the attached drawings is merely to provide a clearer understanding of the application and the concept of the application can be derived from the description below.

[0032] Those skilled in the art can understand that unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0033] It should also be understood that terms such as those defined in a general dictionary should be interpreted in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0034] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms unless specifically stated otherwise. It should be further understood that the use of the term "include" in the specification of the present application means that the stated features, integers, steps, operations, elements, and / or components are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0035] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Those skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0036] In order to facilitate understanding of the present application, the present application will be further explained and described below with reference to the attached drawings in specific embodiments, and the specific embodiments do not constitute a limitation on the embodiments of the present application.

[0037] Those skilled in the art should understand that the drawings are only schematic diagrams of the embodiments, and the components in the drawings are not necessarily necessary for the implementation of the present application.

[0038] The application provides a gaze target detection algorithm based on a double-branch diffusion model, and the main content of the application is as follows: in order to improve the perception ability of the model to scene information, the application designs a scene perception diffusion branch, which is a latent diffusion model, in order to improve the calculation efficiency, the application constructs a gaze target position grid heat map and makes the diffusion model learn how to reconstruct the heat map. The core idea of this branch is: if the model can reconstruct the heat map according to the primary scene features given by the image feature extractor, the conditions given to it must be strongly related to the gaze target, so the reconstruction loss can be used to constrain the scene feature enhancement process for the image feature extractor. In addition, in order to strengthen the control of the input conditions on the reconstruction result, the application designs a latent embedding modulation unit. This unit dynamically adjusts the latent embedding inside the diffusion model through feature disassembly, so that it maintains feature alignment with the primary scene features and captures factors affecting the gaze position at different granularity levels. In order to improve the perception ability of the model to the semantic information implicitly contained in the scene, the application designs a semantic perception diffusion branch. This branch is a diffusion model (Stable Diffusion) pre-trained with a large amount of data, which is used to provide semantic prior information to the model. Integrating the multi-scale feature maps of each layer of the decoder in the diffusion model will obtain unified semantic features. In addition, in order to better balance and integrate the scene features extracted by the image feature extractor and the semantic information provided by the pre-trained model, the application designs a latent embedding modulation unit, which gives the two kinds of features learnable weights respectively and uses the attention mechanism for feature fusion, so that the synergistic parts in the two are strengthened.

[0039] Embodiment 1

[0040] In this embodiment 1, first, a gaze target detection system based on a double-branch diffusion model is provided, which includes: an acquisition module for acquiring a scene image containing a measured person to be detected. A processing module is used to process the acquired scene image containing the measured person to be detected by using a pre-trained gaze target detection model to obtain a gaze target detection result; the gaze target detection model includes a gaze target detection backbone network, a scene perception diffusion branch network and a semantic perception diffusion branch network; the gaze target detection backbone network is used to extract gaze target related scene features from the scene image to be detected, aggregate and regress the gaze target position heat map; the scene perception diffusion branch network is used to impose an interpretable constraint on the target related feature extraction process of the gaze target detection backbone network, divide the given gaze target position heat map into a grid heat map, and gradually add noise in the Markov diffusion process, and at the same time, the gaze target related features extracted by the backbone network are injected into the branch network through the modulation unit to capture the factors affecting the gaze position at different granularity levels; the semantic perception diffusion branch network is used to provide semantic prior to enhance the primary scene features extracted by the gaze target detection backbone network.

[0041] In this embodiment, the above-mentioned system is used to implement a gaze target detection method based on a dual-branch diffusion model, including: using an acquisition module to acquire a scene image containing a subject to be detected. Using a processing module to process the acquired scene image containing a subject to be detected using a pre-trained gaze target detection model to obtain a gaze target detection result; the gaze target detection model includes a gaze target detection backbone network, a scene-aware diffusion branch network, and a semantic-aware diffusion branch network; the gaze target detection backbone network is used to extract gaze target-related scene features from the scene image to be detected, aggregate them, and regress them into a gaze target position heat map; the scene-aware diffusion branch network is used to impose interpretable constraints on the target-related feature extraction process of the gaze target detection backbone network, divide the given gaze target position heat map into grid heat maps, and gradually add noise in the Markov diffusion process. At the same time, the gaze target-related features extracted by the backbone network are used as denoising conditions and injected into the branch network through the modulation unit to capture factors affecting the gaze position at different granularities; the semantic-aware diffusion branch network is used to provide semantic priors to enhance the primary scene features extracted by the gaze target detection backbone network.

[0042] Among them, the gaze target detection backbone network extracts gaze target-related scene features from the input image, aggregates them and regresses them into the final prediction results; among them, the feature extraction module extracts gaze target detection-related features, and the features are mapped into a gaze target position heat map through the regression module; the regression module includes a feature aggregation module and two prediction heads, which are used to generate the gaze target position heat map and the classification results of whether the gaze target is inside or outside the image, respectively.

[0043] The input data of the gaze target detection backbone network include: the original scene image, which is used to extract global scene information; the cropped image of the subject's head, which is used to extract the subject's gaze direction information; the subject's binary head position mask, which is used to extract the positional relationship information between the subject and the scene; the normalized depth map corresponding to the scene image, which is used to extract the three-dimensional geometric information of the scene; all input images are stacked along the channel dimension and input to obtain the main scene features; after being processed by the feature aggregation layer of the self-attention mechanism, the main scene features are refined by two regression heads into the gaze target position heat map and the internal and external classification results respectively.

[0044] The scene-aware diffusion branch network is used to generate a target-guided grid heat map from a noise image. For a given gaze target position heat map, it is evenly divided into m×m grids, and the probability score of each grid is determined by the maximum value of the heat map within the grid. Noise is gradually added to the grid heat map in a Markov diffusion process. During training, the UNet decoder is used to predict the noise added to the given noise image. The main scene feature f sce ∈Rc×h′×w′ After flattening and linear layer adjustment, we get

[0045] The scene-aware diffusion branch network also includes a latent embedding modulation unit to align the latent embedding of the UNet decoder with the main network scene features. Splitting into scaling factors via linear projection and bias Adaptive group normalization is used to modulate the potential embeddings of adjacent layers. Through multi-layer modulation, factors affecting gaze position at different granularities are captured, thereby improving generation quality.

[0046] The semantic perception diffusion branch network includes a variational autoencoder and a UNet module, with the original scene image I sce ∈R 3 ×h×w As input, the variational autoencoder first maps the scene image to the latent space and obtains the latent representation f vae ∈R c×n×n ; UNet module with f vae As input, and with the adjusted main scene features As a condition, the feature maps of each layer of UNet are uniformly scaled to 1 / 4 resolution of the latent space to generate fused semantic features; the adaptive fusion module consists of cascaded attention units, namely two self-attention units and a class cross-attention unit; first, the main scene feature f sce and semantic features f sem Two self-attention units are applied respectively; then, the attention map and f are tagged using learnable weights ω. sce A cross-attention unit is applied; finally, the enhanced target-related features are mapped to the final result through a regression module.

[0047] Example 2

[0048] The gaze target detection method based on the dual-branch diffusion model is proposed in this embodiment 2, which is realized based on a pre-trained gaze target detection model based on the dual-branch diffusion model (Dual Diffusion Kindled Gaze Following Network, DualDiff). The model includes a gaze target detection backbone network, a scene-aware diffusion branch and a semantic-aware diffusion branch. The backbone network is used to extract gaze target related scene features from the input image, aggregate and regress them as the final prediction result. Then, the scene-aware diffusion branch is designed to impose process constraints on the backbone network by using the powerful generation and representation capabilities of the diffusion model, and to enhance the accuracy of the extraction of gaze target related scene features. At the same time, the semantic-aware diffusion branch is designed to supplement the gaze target related semantic information for the backbone network by using the prior knowledge of the pre-trained diffusion model. The overall process of the scheme is as shown in Figure 1 .

[0049] The total loss function of the gaze target detection model based on the dual-branch diffusion model proposed in this embodiment is

[0050] L = λ1L cls + λ2L sce + λ3L sem ,

[0051] wherein λ1, λ2 and λ3 are coefficients for adjusting the contribution of each constraint term. In the training stage, the semantic-aware diffusion branch is initialized using the published pre-training weights, and the loss function L is used to optimize the parameters of the entire network (the variational autoencoder weights of the semantic-aware diffusion branch are always kept fixed). In the testing stage, the scene-aware diffusion branch will be removed, and the optimized main network and the semantic-aware diffusion branch will jointly receive the scene image and related images (head position mask, head cropped image, depth map) to generate the final result.

[0052] The gaze target detection backbone network (Primary Gaze Following Network, PGFNet) aims to extract gaze target related scene features from the input image, aggregate and regress them as the final prediction result. The network extracts gaze target detection related features through a feature extraction module M feat , and the features are mapped to gaze target location heat maps through a regression module M reg . The regression module includes a feature aggregation module M agg , and two prediction heads H map and H out , which are respectively used to generate gaze target location heat maps and classification results of whether the gaze target is inside or outside the image.

[0053] To fully utilize scene information (such as depth information, appearance information, etc.) and personal information (such as gaze direction, human posture, etc.) that are crucial for locating the gaze target, this embodiment uses a classic four-way architecture to build a feature extraction module. This module simultaneously receives the following four types of input images:

[0054] (1) Original scene image (I sce ∈R 3×h×w ), used to extract global scene information.

[0055] (2) Cropped image of the subject's head (I head ∈R 3×h×w ), used to extract the subject’s gaze direction information.

[0056] (3) The subject’s binary head position mask (I pos ∈{0,1} h×w ), used to extract the positional relationship information between the subject and the scene.

[0057] (4) Normalized depth map corresponding to the scene image (I depth ∈[0,1] h×w ), used to extract the 3D geometric information of the scene.

[0058] All input images are stacked along the channel dimension and input into the feature extraction module, and the main scene feature f is obtained by formula (1) sce ∈R c×h′×w′ .

[0059] f sce =M feat (I sce ,I head ,I pos ,I depth ), (1)

[0060] Among them, M feat Represents the feature extraction module. After being processed by the feature aggregation layer of the self-attention mechanism, f sce It is refined by the two regression heads into the gaze target position heat map h and the internal and external classification results cls (Equations (2)-(3)):

[0061] h=H map (M agg (f sce )), (2)

[0062] cls=H out (M agg (f sce )), (3)

[0063] Among them, M agg is the feature aggregation layer, H mapand H out They are the gaze target location heat map prediction head and the internal and external classification head respectively. In order to optimize the performance of the main network, the predicted heat map h and the real heat map h of formula (4) are used. * The mean square error loss between , and the classification loss of formula (5):

[0064]

[0065] L cls =-(cls * log(cls)+(1-cls * )log(1-cls)), (5)

[0066] where cls * is the true internal and external classification label (cls * ∈{0,1}), cls * =1 means the gaze target is located in the current video frame, cls * =0 means the gaze target is outside the current video frame.

[0067] As mentioned above, the feature extraction process of the backbone network is mainly limited by the mean square error loss of formula (4). This constraint is actually weak and it is difficult to ensure that the obtained features are strongly correlated with the target. At the same time, once the diffusion model successfully synthesizes any object, it can fully understand the relevant knowledge of the object. In other words, if the diffusion model can roughly generate a heat map of the gaze target position from the noise map, it means that it has mastered the target-related clues implicit in the scene image. Therefore, this embodiment designs a scene-aware diffusion branch (ScDBrh) sce Imposing interpretable constraints on target-related feature extraction of the main network.

[0068] To improve efficiency, a target-gaze grid heatmap generation task is constructed, whose goal is to generate a target-guided grid heatmap g∈R from a noisy image. m×m , rather than directly synthesizing the original heat map. For a given gaze target position heat map h∈R n×n , which is evenly divided into m×m (m<<n) grids. The probability score of each grid is determined by the maximum value of the heat map in the grid (Equation (6)):

[0069]

[0070] Where N(i) represents the set of pixels in the i-th grid.

[0071] Scene perception diffusion branch B sce The input is a grid heat map, and noise is gradually added in the Markov diffusion process (Equation (7)).

[0072]

[0073] Where ∈ is random Gaussian noise, g0 is the original grid heat map, g t is the noise image after t-step diffusion. During training, the UNet decoder is used to predict the given noise image g t To enhance the controllability of the generation process, the main scene feature f sce ∈R c×h′×w′ After flattening and linear layer adjustment, we get The latent embedding modulation unit is injected into the diffusion branch of the UNet (Equation (8)).

[0074]

[0075] The latent embedding modulation unit is introduced into the scene-aware diffusion branch to align the latent embedding of the UNet decoder with the main network scene features. Inspired by group normalization, Split into R d / 2 (scaling factor) and (bias term), modulates the potential embedding of adjacent layers through adaptive group normalization (Equation (9)):

[0076]

[0077] in and denote the potential embeddings of adjacent layers of the UNet. Through multi-layer modulation, the model can capture factors that affect gaze position at different granularities, thereby improving generation quality. Ultimately, the prediction results of the scene-aware diffusion branch are used to construct the noise reconstruction loss (Equation (10)), jointly optimizing the main network and the diffusion branch, imposing strict process constraints on the main network.

[0078]

[0079] In addition to explicit visual information, scene images also contain rich implicit semantic information, which is also crucial for locating the gaze target. Diffusion models pre-trained based on large-scale datasets are good at capturing such semantic information. The embedded representations learned by such pre-trained models can be decoded and show excellent generalization and semantic perception capabilities in multiple visual tasks. Therefore, this embodiment designs a semantic-aware diffusion branch (SeDBrh) sem , semantic priors are provided by pre-training diffusion models to enhance the primary scene features extracted by the backbone network.

[0080] In this embodiment, the Stable Diffusion pre-trained version is selected as the basic architecture of the semantic perception diffusion branch, which includes a variational autoencoder (VAE) and a UNet module. To avoid losing detailed information, this branch uses the original scene image I sce ∈R 3×h×w As input, the VAE encoder first maps the scene image to the latent space and obtains the latent representation f vae ∈R c×n×n The UNet module is based on f vae As input, and with the adjusted main scene features As a condition, the feature maps of each layer of UNet are uniformly scaled to 1 / 4 resolution of the latent space to generate the fused semantic feature f sem ∈R c′×n / 32×n / 32 , the process can be expressed as formula (11).

[0081]

[0082] In order to extract target-related semantics from the Stable Diffusion prior to further enhance the main scene features, this embodiment designs an adaptive fusion module M af The module consists of cascaded attention units, namely two self-attention units and one cross-class attention unit. First, the main scene feature f sce and semantic features f sem Two self-attention units are applied respectively (Equation (12) and Equation (13)).

[0083]

[0084] Q1=f sce W Q ,K1=f sce W K ,Q2=f sem W Q ,K2=f sem W K (13)

[0085] Among them, M Q and W K Is a learnable weight matrix, Q1, K1 is determined by f sce Generated, Q2, K2 by f sem Then, the attention map and f are combined using the learnable weight ω sce Application-class cross attention unit (Formula (14)):

[0086] f fus =ωf sce+ (1 - ω) (attn x V1) T , (14)

[0087] where V1 is generated by f sce in the same way as Q1 and K1. Finally, the enhanced target-related features are mapped to the final results by the regression module (Equations (15) and (16)).

[0088] h = H map (M agg (f fus )), (15)

[0089] cls = H out (M agg (f fus )), (16)

[0090] In this embodiment, based on the performance of the gaze target detection model based on the double-branch diffusion model on the commonly used data set, in order to verify the effectiveness of the method, related experiments are carried out on two public data sets of GazeFollow and VideoAttentionTarget, and the performance of the model is evaluated by using the commonly used indicators of this task.

[0091] GazeFollow is a large-scale gaze target detection picture data set, which contains more than 122,000 images and more than 130,000 person instances. Among them, the test set contains 4,782 labeled samples, and the gaze target of each sample is independently labeled by 10 annotators to improve the reliability of the label. VideoAttentionTarget is a gaze target detection video data set collected from 50 programs on YouTube platform, containing 1,331 video segments (1-80 seconds in length). The data set provides 109,574 in-frame gaze point labels and 54,967 cross-frame gaze target labels, and also contains head bounding box information to support multi-modal analysis.

[0092] All experiments carried out in this embodiment use three types of standard indicators for performance evaluation:

[0093] (1) Area under the curve (AUC): by calculating the confidence distribution of the predicted gaze target heat map and the real gaze target heat map, the overall capture ability of the model to the gaze area is quantified. The closer the AUC is to 1, the higher the matching degree of the heat map confidence distribution and the real gaze target.

[0094] (2) Distance Error (Dist): Calculates the distance between the predicted gaze point and the true annotation point. Since each gaze target in the test set of the GazeFollow dataset corresponds to 10 annotations, the experiments on the GazeFollow dataset will record the minimum distance error (Min Dist) and the average distance error (AvgDist) between the predicted gaze point and the 10 annotations.

[0095] (3) Average Precision (AP): It is specifically used to evaluate the model’s binary classification prediction accuracy for gaze targets inside / outside the frame. Since the test set of the GazeFollow dataset does not provide samples where the gaze target is outside the frame, this metric is only recorded on the VideoAttentionTarget dataset.

[0096] Table 1 shows the performance comparisons of our proposed method with state-of-the-art methods on the GazeFollow and VideoAttentionTarget datasets. All comparison data is based on the published results in the respective method papers. The experimental results show that our proposed method achieves superior results on both datasets, achieving the best results across all metrics on the VideoAttentionTarget dataset.

[0097] Table 1. Comparative experimental results of cutting-edge methods

[0098]

[0099] Experiments show that DualDiff surpasses the current best methods in all indicators on the VideoAttentionTarget dataset and reaches a level comparable to the existing best methods on the GazeFollow dataset, demonstrating that the present invention can effectively extract target-related features in complex scene images. In addition, to verify the generalization ability of DualDiff to unseen scenes, this section also designs a cross-domain evaluation experiment. After training the model on GazeFollow, it is directly applied to the VideoAttentionTarget test set (without any fine-tuning, corresponding to the Ours* results in the table). As shown in Table 1, even without using target domain training data, the present method still outperforms existing supervised learning methods, demonstrating that the target-related features containing semantic information learned by the model through the double diffusion branch have strong generalization and robustness.

[0100] To verify the effectiveness of the dual-branch diffusion structure, ablation experiments were conducted by gradually removing two branches to construct degradation models for comparison. The results are shown in Table 2 (where 'Baseline' represents the baseline model with all newly added branches completely removed, i.e., the fixation object detection backbone network). The experimental results show that the full DualDiff model performs best among all degradation models, and each degradation model significantly outperforms the baseline, fully demonstrating the necessity and rationality of the dual-branch diffusion structure.

[0101] Table 2. Double-branch ablation experimental results

[0102]

[0103] In addition, in order to verify the effectiveness of the adaptive fusion module proposed in this invention, this experiment adopts three different semantic feature-scene feature combination methods to observe the final results. The experimental results are shown in Table 3. In the 'Concat' method, we directly splice the scene features and semantic features along the channel dimension; in the 'CA' method, we use the scene features as queries and the semantic features as keys and values, and perform feature fusion through cross-attention operations. As shown in Table 3, the performance of the adaptive fusion module proposed in this method is better than the direct splicing and cross-attention methods, and all indicators are improved. The experimental results prove that through the adaptive fusion module, the synergistic components of the semantic features and backbone features in the prior information provided by Stable Diffusion will be strengthened, thereby effectively representing the information related to the gaze target.

[0104] Table 3. Ablation experiment results of adaptive fusion module

[0105]

[0106]

[0107] Figure 2 Shown are grid heatmaps obtained by progressively denoising a noisy image using the noise predicted by the scene-aware diffusion branch. Figure 2 (a) is the original image, Figure 2 (b) is the denoised gaze target grid heat map. Figure 2 (c) shows the true gaze target mesh heatmap. As can be seen from the figure, the denoised mesh heatmap is highly consistent with the true mesh heatmap, demonstrating that this branch can accurately predict image noise based on the conditions. Therefore, this branch can impose strong constraints on the feature extraction module of the backbone network, thereby enhancing its ability to extract features related to the gaze target.

[0108] Figure 3The qualitative comparison results of the gaze target position heat map are shown. The red box marked row is the prediction output of this model. The first three rows correspond to: scene graph, true gaze target position heat map and predicted heat map of other methods. It can be observed from the figure that compared with other methods, this method is better in complex scenes (such as Figure 3 In (c), there are multiple salient objects in the target user’s sight path; Figure 3 (a), where the target object is surrounded by multiple similar objects, demonstrates stronger prediction accuracy. These visualizations demonstrate that the dual-branch diffusion model-based gaze object detection model can effectively extract scene semantic information, thereby improving model performance.

[0109] Example 3

[0110] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method for detecting a fixated target based on the dual-branch diffusion model described above is implemented. The method includes:

[0111] Acquire a scene image to be detected containing a subject;

[0112] A pre-trained gaze target detection model is used to process the acquired scene image containing the subject to be detected to obtain a gaze target detection result; the gaze target detection model includes a gaze target detection backbone network, a scene-aware diffusion branch network and a semantic-aware diffusion branch network; the gaze target detection backbone network is used to extract gaze target-related scene features from the scene image to be detected, aggregate them and regress them into a gaze target position heat map; the scene-aware diffusion branch network is used to impose interpretable constraints on the target-related feature extraction process of the gaze target detection backbone network, divide the given gaze target position heat map into grid heat maps, and gradually add noise in the Markov diffusion process. At the same time, the gaze target-related features extracted by the backbone network are used as denoising conditions and injected into the branch network through the modulation unit to capture factors affecting the gaze position at different granularities; the semantic-aware diffusion branch network is used to provide semantic priors to enhance the primary scene features extracted by the gaze target detection backbone network.

[0113] Example 4

[0114] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the above-mentioned method for detecting a fixated target based on a dual-branch diffusion model, the method comprising:

[0115] Acquire a scene image to be detected containing a subject;

[0116] processing the obtained scene image containing the measured person to be detected by using a pre-trained gaze target detection model to obtain a gaze target detection result; the gaze target detection model comprises a gaze target detection backbone network, a scene-aware diffusion branch network and a semantic-aware diffusion branch network; the gaze target detection backbone network is used to extract gaze target related scene features from the scene image to be detected, aggregate and regress the gaze target related scene features into a gaze target position heat map; the scene-aware diffusion branch network is used to impose an interpretable constraint on the target related feature extraction process of the gaze target detection backbone network, divide the given gaze target position heat map into a grid heat map, and gradually add noise in the Markov diffusion process, and at the same time, the gaze target related features extracted by the backbone network are taken as a denoising condition, which is injected into the branch network through a modulation unit to capture factors affecting the gaze position at different granularities; the semantic-aware diffusion branch network is used to provide semantic prior to enhance the primary scene features extracted by the gaze target detection backbone network.

[0117] Embodiment 5

[0118] The embodiment 5 provides an electronic device, comprising a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing a gaze target detection method based on a double-branch diffusion model as described above, the method comprising:

[0119] obtaining a scene image containing a measured person to be detected;

[0120] processing the obtained scene image containing the measured person to be detected by using a pre-trained gaze target detection model to obtain a gaze target detection result; the gaze target detection model comprises a gaze target detection backbone network, a scene-aware diffusion branch network and a semantic-aware diffusion branch network; the gaze target detection backbone network is used to extract gaze target related scene features from the scene image to be detected, aggregate and regress the gaze target related scene features into a gaze target position heat map; the scene-aware diffusion branch network is used to impose an interpretable constraint on the target related feature extraction process of the gaze target detection backbone network, divide the given gaze target position heat map into a grid heat map, and gradually add noise in the Markov diffusion process, and at the same time, the gaze target related features extracted by the backbone network are taken as a denoising condition, which is injected into the branch network through a modulation unit to capture factors affecting the gaze position at different granularities; the semantic-aware diffusion branch network is used to provide semantic prior to enhance the primary scene features extracted by the gaze target detection backbone network.

[0121] In summary, the present invention proposes a gaze target detection algorithm based on a dual-branch diffusion model, which integrates the generation ability, scene perception ability and semantic prior information of the diffusion model into the backbone network without relying on complex multi-stage training strategies. The present invention constructs dual diffusion branches of scene perception and semantic perception, which strengthen the feature expression ability of the main network from the two dimensions of explicit process constraints and implicit semantic priors. First, the scene perception branch imposes interpretable process constraints on the main network through the grid heat map generation task, significantly improving the accuracy of extracting target-related features; secondly, the semantic perception branch uses the prior knowledge provided by the pre-trained diffusion model to inject the semantic prior obtained from large-scale image and text pre-training into the feature space of the backbone network, and dynamically coordinates the complementary information of the backbone network and the semantic perception diffusion branch through the adaptive fusion module, so that the synergistic part of the two is strengthened.

[0122] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A method for gaze target detection based on a dual-branch diffusion model, characterized in that: include: Acquire a scene image to be detected containing a subject; A pre-trained gaze target detection model is used to process the acquired scene image containing the subject to be detected to obtain a gaze target detection result; the gaze target detection model includes a gaze target detection backbone network, a scene perception diffusion branch network, and a semantic perception diffusion branch network; the gaze target detection backbone network is used to extract gaze target-related scene features from the scene image to be detected, aggregate them, and regress them into a gaze target position heat map; The scene-aware diffusion branch network is used to impose interpretable constraints on the target-related feature extraction process of the gaze target detection backbone network, divide the given gaze target position heat map into a grid heat map, and gradually add noise in the Markov diffusion process. At the same time, the gaze target-related features extracted by the backbone network are used as denoising conditions and injected into the branch network through the modulation unit to capture factors affecting the gaze position at different granularities; the semantic-aware diffusion branch network is used to provide semantic priors to enhance the primary scene features extracted by the gaze target detection backbone network.

2. The method for detecting gaze targets based on a dual-branch diffusion model according to claim 1, wherein: The gaze target detection backbone network extracts gaze target-related scene features from the input image, aggregates them, and regresses them into the final prediction result. Among them, the feature extraction module extracts gaze target detection-related features, and the features are mapped into a gaze target position heat map through the regression module. The regression module contains a feature aggregation module and two prediction heads, which are used to generate the gaze target position heat map and the classification result of whether the gaze target is inside or outside the image, respectively.

3. The method for detecting gaze targets based on a dual-branch diffusion model according to claim 2, wherein: The input data of the gaze target detection backbone network include: the original scene image, which is used to extract global scene information; the cropped image of the subject's head, which is used to extract the subject's gaze direction information; the subject's binary head position mask, which is used to extract the positional relationship information between the subject and the scene; the normalized depth map corresponding to the scene image, which is used to extract the three-dimensional geometric information of the scene; all input images are stacked along the channel dimension and input to obtain the main scene features; after being processed by the feature aggregation layer of the self-attention mechanism, the main scene features are refined by two regression heads into the gaze target position heat map and the internal and external classification results respectively.

4. The method for detecting gaze targets based on a dual-branch diffusion model according to claim 1, wherein: The scene-aware diffusion branch network includes a gaze target grid heat map generation unit for generating a target-guided grid heat map from a noise image; for a given gaze target position heat map, the heat map is evenly divided into m×m grids, and the probability score of each grid is determined by the maximum value of the heat map within the grid; noise is gradually added to the grid heat map in a Markov diffusion process; During training, the UNet decoder is used to predict the noise added to a given noise image, the main scene feature f sce ∈R c ×h′×w′ After flattening and linear layer adjustment, we get 5. The method for detecting gaze targets based on a dual-branch diffusion model according to claim 4, wherein: The scene-aware diffusion branch network also includes a latent embedding modulation unit to align the latent embedding of the UNet decoder with the main network scene features. Splitting into scaling factors via linear projection and bias Adaptive group normalization is used to modulate the potential embeddings of adjacent layers. Through multi-layer modulation, factors affecting gaze position at different granularities are captured, thereby improving generation quality.

6. The method for detecting gaze targets based on a dual-branch diffusion model according to claim 1, wherein: The semantic perception diffusion branch network includes a variational autoencoder and a UNet module, with the original scene image I sce ∈R 3×h×w As input, the variational autoencoder first maps the scene image to the latent space and obtains the latent representation f vae ∈R c×n×n ; UNet module with f vae As input, and with the adjusted main scene features As a condition, the feature maps of each layer of UNet are uniformly scaled to 1 / 4 resolution of the latent space to generate fused semantic features; The adaptive fusion module consists of cascaded attention units, namely two self-attention units and one cross-attention unit; first, the main scene feature f sce and semantic features f sem Two self-attention units are applied respectively; then, the attention map and f are tagged using learnable weights ω. sce A cross-attention unit is applied; finally, the enhanced object-related features are mapped to the final result through a regression module.

7. A gaze target detection system based on a dual-branch diffusion model, characterized in that: include: An acquisition module, used for acquiring a scene image to be detected containing a subject; A processing module is configured to process the acquired scene image containing the subject to be detected using a pre-trained gaze target detection model to obtain a gaze target detection result; the gaze target detection model includes a gaze target detection backbone network, a scene perception diffusion branch network, and a semantic perception diffusion branch network; the gaze target detection backbone network is configured to extract gaze target-related scene features from the scene image to be detected, aggregate them, and regress them into a gaze target position heat map; The scene-aware diffusion branch network is used to impose interpretable constraints on the target-related feature extraction process of the gaze target detection backbone network. It divides a given gaze target position heat map into a grid heat map and gradually adds noise in a Markov diffusion process. At the same time, the gaze target-related features extracted by the backbone network are used as denoising conditions and injected into the branch network through the modulation unit to capture factors affecting the gaze position at different granularities. The semantic-aware diffusion branch network is used to provide semantic priors to enhance the primary scene features extracted by the fixation target detection backbone network.

8. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the gaze target detection method based on the dual-branch diffusion model according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the gaze target detection method based on the dual-branch diffusion model according to any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the gaze target detection method based on the dual-branch diffusion model as described in any one of claims 1 to 6.

Images