Gaze point prediction method and system based on multi-task reciprocal converter

Abstract

The invention discloses a staring point prediction method and system based on a multi-task reciprocal converter, and belongs to the field of computer vision and artificial intelligence. The method comprises the following steps: acquiring a scene image containing human beings, and generating global memory features; generating a first human query set and a first gaze query set from the global memory features through a self-attention mechanism and a cross-attention mechanism respectively, and performing cross-branch information interaction processing through a multi-task reciprocal converter to obtain a second human query set and a second gaze query set; performing position information coding through global-local position coding to obtain a third human query set and a third gaze query set; and obtaining a prediction result of a human head bounding box and a staring target from the third human query set and the third staring query set by adopting a multi-layer perception mechanism. According to the method, the problem of information asymmetry is relieved, and the accuracy and reliability of gaze analysis in a complex scene are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of prediction methods and systems, and relates to a gaze point prediction method and system based on a multi-task reciprocal transformer. BACKGROUND

[0002] Gaze prediction is a specific branch of computer vision or artificial intelligence. Predicting people's gaze position is the key to understanding human intentions. As a research hotspot, gaze point prediction has developed from predicting the gaze of a single person to simultaneously predicting the positions of all people and their corresponding gaze targets, and is widely used in many disciplines such as human-computer interaction, predictive learning, virtual reality, behavior understanding, and diagnosis of psychological disorders such as autism.

[0003] The development of gaze prediction mainly experienced three stages, namely, strong constraint stage, weak constraint stage, and unconstrained stage. It has changed from predicting the gaze target of one person at a time in the scene to simultaneously predicting the head position and gaze target of multiple people.

[0004] The existing one-stage gaze prediction model, the end-to-end one-stage framework, has become the mainstream paradigm for group gaze tracking or gaze prediction because it can detect the positions and gaze objects of all people simultaneously without using any additional components.

[0005] Defects and deficiencies of the prior art: there is an information asymmetry problem in the existing one-stage gaze prediction model. The existing one-stage model only focuses on context modeling and relationship reasoning on a single task branch, ignores any correlation between tasks, and finally only uses a matching algorithm to constrain the pairwise output results of the gaze target, resulting in a misalignment between head detection and gaze prediction performance. Head detection and gaze prediction are obviously two tasks with close correlation, but in the existing method they do not promote each other, which is referred to as the information asymmetry problem. The information asymmetry problem greatly affects the overall accuracy of group gaze tracking or gaze prediction results. SUMMARY

[0006] To solve the problems in the background art, the application provides a gaze point prediction method and system based on a multi-task reciprocal transformer.

[0007] To achieve the above purpose, the technical scheme adopted by the application is as follows: A gaze point prediction method based on a multi-task reciprocal transformer, comprising: obtaining a scene image containing humans, performing image feature extraction on the scene image, and generating global memory features; The global memory features are used to generate a first human query set and a first gaze query set through a self-attention mechanism and a cross-attention mechanism, respectively. Then, a second human query set and a second gaze query set are obtained by cross-branch information interaction processing through a multi-task reciprocal transformer. The second human query set and the second gaze query set are encoded with location information through global-local location coding to obtain the third human query set and the third gaze query set; The prediction results of human head bounding boxes and gaze targets are obtained by using a multi-layer perception mechanism with the third human query set and the third gaze query set. The multi-layer perception mechanism performs joint optimization constraints based on the binary pairing dependency relationship formed by human head position and gaze target.

[0008] Specifically, the scene image is processed by extracting image features through a convolutional neural network; the image features are then transformed into global memory features through self-attention and cross-attention mechanisms.

[0009] Specifically, all human heads and all gaze targets in the scene image are detected using a third human query set and a third gaze query set.

[0010] Specifically, the multi-task reciprocal transformer calculates cross-gating weights based on the first gaze query set and performs feature recalibration on the first human query set, using privileged information in the recalibrated human head features to guide the generation of discriminative features.

[0011] Specifically, the cross-gating weight calculation process includes: Input gaze query set and human query sets ; Through the formula: Calculate cross-gating weights ,in For learnable weight matrix, For bias; Through the formula: human query set Perform feature recalibration, where ⊙ represents element-wise multiplication.

[0012] Specifically, the global-local location encoding uses smooth dilated one-dimensional convolution SC to capture multi-scale location information; The smooth dilated one-dimensional convolution SC contains 3 sets of convolution pairs. Each set of convolution pairs is a dilated convolution with a kernel size of 3 and a stride of 1. The dilation rates of the 3 sets of convolution pairs are r=1, 2 and 4, respectively, corresponding to a shared convolution with a kernel size of 2r-1.

[0013] Specifically, the pairing dependency is determined through a loss function. accomplish; wherein is the number of samples, is the matching loss function, is the prediction result, is the target function.

[0014] Specifically, the multi-layer perception mechanism predicts the human head bounding box through a 3-layer multi-layer perception machine (MLP); The multi-layer perception mechanism predicts the gaze heat map, the gaze target position and the gaze inside and outside confidence through a 5-layer multi-layer perception machine, a 2-layer multi-layer perception machine (MLP) and a 1-layer multi-layer perception machine (MLP), respectively; The outputs of all multi-layer perception machines (MLPs) are processed by a softmax function and a sigmoid function to obtain the prediction results.

[0015] The application also provides a gaze point prediction system based on a multi-task reciprocal transformer, comprising: An image acquisition module is configured to acquire a scene image containing a human; A visual coding module is configured to perform image feature extraction on the scene image to generate global memory features; A double-branch decoding module is configured to generate a first human query set and a first gaze query set through a self-attention mechanism and a cross-attention mechanism, respectively; A multi-task reciprocal transformer is configured to perform cross-branch information interaction processing on the first human query set and the first gaze query set to obtain a second human query set and a second gaze query set; A global-local position encoding module is configured to encode the position information of the second human query set and the second gaze query set; A prediction output module is configured to obtain the prediction results of the human head bounding box and the gaze target by using a multi-layer perception mechanism.

[0016] Specifically, the multi-task reciprocal transformer comprises a cross-gating module and an information guiding module, the cross-gating module calculates cross-gating weights and completes feature re-scaling, and the information guiding module realizes cross-task feature guidance based on privileged information in the re-scaled features.

[0017] Compared with the prior art, the application has the following beneficial effects: in view of the one-stage information asymmetry problem, the multi-task reciprocal transformer captures multi-task information interaction through the multi-task reciprocal transformer, and the information guiding module dynamically aggregates the information most relevant to the task to prevent the prediction performance on different tasks from being misaligned. Each branch thereof builds and embeds a new type of learnable position encoding, i.e., global-local position encoding, which can adaptively learn the task-specific pixel locality and long-range semantic continuity.

[0018] In addition, the multi-task reciprocal transformer outperforms the state-of-the-art methods in gaze following and video attention target to a large extent, and all indicators on the gaze following and video attention target datasets are significantly improved.

[0019] The multi-task reciprocal transformer improves various indicators of gaze target prediction and human head position detection to a certain extent, and the average precision has made great progress. This verifies the effectiveness and rationality of the multi-task reciprocal transformer in solving the problem of ambiguous frames. When the scene is more complex and the prediction task is more difficult, the multi-task reciprocal transformer has better prediction performance and stronger robustness.

[0020] The multi-task reciprocal transformer guides the communication of the two branches through privileged information, which can effectively alleviate the information asymmetry between human head detection and gaze prediction, and the performance on the two standard benchmark datasets gaze following and video attention target is significantly better than the existing state-of-the-art methods. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a gaze point prediction method based on a multi-task reciprocal transformer of the present application; Figure 2 is a quantitative comparison diagram of the present application and mainstream algorithms based on the GazeFollow dataset; Figure 3 is a quantitative comparison diagram of the present application and mainstream algorithms based on the VideoAttentionTarget dataset; Figure 4 is a prediction result visualization diagram of the present application randomly selected for a single individual; Figure 5 is an ablation study diagram of the present application about MAM and GLPE; Figure 6 is a prediction effect comparison diagram of the present application with or without a multi-task reciprocal transformer; Figure 7 is an ablation study diagram of the present application about CGM and IGM; Figure 8 is a different model calculation cost comparison diagram of the present application; Figure 9 is a visualization diagram of the privileged information, output and final prediction result of the multi-task reciprocal transformer of the present application; Figure 10 is a loss function weight influence diagram of the present application. DETAILED DESCRIPTION

[0022] Clearly, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0023] As shown in Figures 1-10 The technical solutions adopted by the present application are as follows: A gaze point prediction method based on a multi-task reciprocal transformer, characterized in that it comprises the following steps: An image containing a human is acquired, and image feature extraction is performed on the image to generate global memory features.

[0024] The global memory features are respectively processed by a self-attention mechanism and a cross-attention mechanism to generate a first human query set and a first gaze query set, and then a multi-task reciprocal transformer is used to perform cross-branch information interaction processing to obtain a second human query set and a second gaze query set.

[0025] The second human query set and the second gaze query set are encoded by global-local position encoding to obtain a third human query set and a third gaze query set.

[0026] The third human query set and the third gaze query set are processed by a multi-layer perception mechanism to obtain a prediction result of a human head bounding box and a gaze target, and the multi-layer perception mechanism is based on a binary pair dependency relationship formed by the human head position and the gaze target to perform joint optimization constraint.

[0027] A gaze point prediction method based on a multi-task reciprocal transformer is implemented based on a gaze point prediction system based on a multi-task reciprocal transformer.

[0028] The gaze point prediction system based on a multi-task reciprocal transformer comprises: An image acquisition module is configured to acquire an image containing a human.

[0029] A visual encoding module is configured to perform image feature extraction on the image to generate global memory features.

[0030] A double-branch decoding module is configured to respectively generate a first human query set and a first gaze query set by a self-attention mechanism and a cross-attention mechanism.

[0031] A multi-task reciprocal transformer is configured to perform cross-branch information interaction processing on the first human query set and the first gaze query set to obtain a second human query set and a second gaze query set.

[0032] A global-local position encoding module is configured to encode position information of the second human query set and the second gaze query set.

[0033] A prediction output module is configured to obtain a prediction result of the human head bounding box and the gaze target by using a multi-layer perception mechanism.

[0034] Specifically, the multi-task mutual transformation device (MAM) comprises a cross-gating module and an information guiding module. The cross-gating module calculates cross-gating weights and completes feature re-labeling. The information guiding module realizes cross-task feature guidance based on privileged information in the re-labeled features. The scene image is used to extract image features by a convolutional neural network.

[0035] The visual encoding module comprises a visual encoder.

[0036] The visual encoder comprises a ResNet-50 backbone network, which is used to preliminarily extract image features.

[0037] The Transformer encoder comprises 6 layers, each layer comprising 8 heads of self-attention mechanisms, which are used to enhance global context information and convert the output feature of the backbone network into the global memory feature.

[0038] The backbone network adopts a ResNet-50 convolutional neural network, and its core function is to preliminarily extract features of an input scene image containing a human. The ResNet-50 realizes step-by-step capturing of low-level to high-level features of the image by stacking multiple residual blocks, including basic visual information such as edges, textures and local contours, and finally outputs a high-dimensional image feature matrix. This process provides original visual feature support for subsequent global context enhancement of the features, and is a basic link for the visual encoder to realize feature extraction.

[0039] The image features are converted into the global memory feature by the self-attention mechanism and the cross-attention mechanism.

[0040] The Transformer encoder receives the preliminary features output by the backbone network, and enhances the global context information of the features by using a multi-layer self-attention mechanism. The specific structure and function are as follows: The Transformer encoder comprises 6 layers of encoding structure, and each layer is configured with 8 heads of self-attention mechanisms. The multi-head self-attention mechanism can simultaneously model the correlation of features from different dimensions, and capture the semantic correlation of distant pixels in the image, such as the spatial position relationship between a person and a potential gaze target in the scene, by calculating the attention weights between the features.

[0041] After 6-layer Transformer encoding processing, the high-dimensional features output by the backbone network are converted into global memory features with global context information. This feature not only contains local detail information of the image, but also integrates the global correlation of each element in the scene, providing a unified feature input basis for the human head detection and gaze prediction tasks in the double-branch decoder.

[0042] First, the input scene image is convolved by the ResNet-50 backbone network to extract preliminary features containing local visual information; then the preliminary features are input into a 6-layer Transformer encoder, which models the global context of the features through 8 self-attention mechanisms in each layer, and finally generates global memory features that integrate global semantic information.

[0043] The double-branch decoding module includes a double-branch decoder, which includes a human detection decoder and a gaze prediction decoder, each containing 3 Transformer decoding blocks, each decoding block containing 6 layers.

[0044] The number of queries in the first human query set and the first gaze query set is N_q=20.

[0045] The overall architecture of the double-branch decoder: The double-branch decoder contains two parallel and symmetric task branches, namely the human detection decoder and the gaze prediction decoder. Both branches use the Transformer architecture as the core processing unit, and maintain consistency in network layers and module configuration to ensure the symmetry and compatibility of task feature processing. The purpose is to focus on the feature requirements of human head detection and gaze prediction tasks through dedicated decoders, while laying a structural foundation for cross-task information interaction.

[0046] Each branch decoder, including the human detection decoder block and the gaze prediction decoder block, is composed of 3 stacked Transformer decoding blocks, and each Transformer decoding block contains 6 layers of processing units. Each layer of processing units integrates self-attention subunits and cross-attention subunits, where: The self-attention subunit is used to capture the internal association of the query set within the same task branch, and to strengthen the context modeling within the task.

[0047] The cross-attention subunit is used to establish the association between the query set and the global memory features output by the visual encoding module, and to realize the guidance of global features to local tasks.

[0048] Through the stacking of multiple levels of Transformer decoding blocks, the decoder can gradually refine the feature representation and improve the feature discrimination ability for human heads and gaze targets.

[0049] The human detection decoder implements human head detection through the first human query set and the specific human position embedding, and the gaze prediction decoder implements gaze target inference through the first gaze query set and the specific gaze position embedding. The core parameters and functions of the two types of query sets are as follows: The number of queries of the first human query set and the first gaze query set is set to N_q=20. This number is configured based on the balance between the coverage demand for the maximum number of people in the scene and the model calculation efficiency, and can support simultaneous detection and prediction of complex scenes containing up to 20 people.

[0050] The query set, as a learnable parameter of the decoder, gradually focuses on the human head region or the gaze target region in the image through iterative learning. The first human query set highlights the feature representation of the human head instance through interaction with the global memory feature. The first gaze query set captures the feature information of the gaze target through a similar mechanism, providing a basic feature carrier for cross-branch information interaction of the subsequent multi-task reciprocal transformer.

[0051] After receiving the global memory feature output by the visual encoding module, the human detection decoder generates the first human query set containing human head position information through cross-attention calculation between the human query set and the global memory feature.

[0052] At the same time, the gaze prediction decoder generates the first gaze query set containing gaze target information through interaction between the gaze query set and the global memory feature.

[0053] After being processed by multiple layers of the respective Transformer decoding block, the two types of query sets become the input of the multi-task reciprocal transformer for cross-branch information interaction.

[0054] The global memory feature is used to generate the first human query set and the first gaze query set through self-attention mechanism and cross-attention mechanism, respectively, and then the multi-task reciprocal transformer is used for cross-branch information interaction processing to obtain the second human query set and the second gaze query set.

[0055] The multi-task reciprocal transformer includes a cross-gating module (CGM) and an information guiding module (IGM).

[0056] The multi-task reciprocal transformer calculates the cross-gating weight based on the first gaze query set and recalibrates the features of the first human query set, and uses the privileged information in the recalibrated human head features to guide the generation of discriminative features.

[0057] Multi-task mutual-aid transformer (MAM) is embedded in a two-branch decoder, connecting the human detection decoder and the gaze prediction decoder. The core function is to establish a dynamic information communication channel between the two task branches: through the cross-gating module (CGM), the feature calibration is realized, and through the information guiding module (IGM), the privileged information is effectively utilized, so that the two branches can mutually empower each other, alleviate the information asymmetry problem, and improve the overall prediction performance.

[0058] The cross-gating module (CGM) is a key unit for cross-task information interaction. Its core function is to dynamically calibrate the features of the human head detection branch based on the features of the gaze prediction branch. The specific process is as follows: The cross-gating module (CGM) receives the first gaze query set (containing the feature representation of the gaze target) from the gaze prediction decoder and the first human query set (containing the feature representation of the human head region) from the human detection decoder.

[0059] Through a learnable weight matrix Feature mapping is performed on the first gaze query set, and bias is added. Then, the cross-gating weight is calculated through the sigmoid activation function , the formula is: This weight quantifies the importance of gaze features to human head features, ranging from 0 to 1.

[0060] The cross-gating weight is multiplied element-wise with the first human query set to dynamically calibrate the first human query set. The recalibrated feature retains the effective information related to the gaze task, while suppressing irrelevant noise, making the human head feature more suitable for cross-task collaboration. Through the above process, the cross-gating module (CGM) not only establishes a communication link between the two branches, but also captures the potential correlation between tasks through feature calibration, laying the foundation for the transmission of privileged information.

[0061] The information guiding module (IGM) further excavates and utilizes privileged information based on the features calibrated by the cross-gating module (CGM), guiding the generation of discriminative features most relevant to the current task. The specific process is as follows:

[0062] The human head feature calibrated by the cross-gating module (CGM) contains privileged information from the gaze prediction task (i.e., gaze-related features that are meaningful to the human head detection task). These information exists in the feature in an implicit form.

[0063] ​The information guiding module (IGM) adopts an attention mechanism to calculate the calibrated human head features and the correlation coefficient between the first gaze query set quantifies the association strength of the two types of features.

[0064] Based on the correlation coefficient, the calibrated human head features are weighted and aggregated to dynamically filter out the human head feature components most relevant to the gaze task, generating discriminative features for the human head detection task. This feature enhances the task-critical information and improves the collaborative accuracy of human head detection and gaze prediction.

[0065] The cross-gating module (CGM) and the information guiding module (IGM) form a calibrated-guided collaborative link: the cross-gating module (CGM) realizes cross-task feature calibration through dynamic weights, providing the information guiding module (IGM) with basic features rich in privileged information; the information guiding module (IGM) then deeply mines privileged information through an attention mechanism to generate targeted discriminative features. The combined effect of the two enables the human detection decoder and the gaze prediction decoder to efficiently exchange useful information and enhance each other's feature expression capabilities, ultimately achieving accurate prediction of human head bounding boxes and gaze target positions.

[0066] The cross-gating weight calculation process of the cross-gating module includes inputting the first gaze query set and the first human query set .

[0067] The cross-gating weight is calculated by the formula: where is a learnable weight matrix, is a bias.

[0068] The first human query set is feature re-scaled by the formula: where ⊙ represents element-wise multiplication.

[0069] The cross-gating weight calculation and feature re-scaling process is as follows: The inputs of the cross-gating module are two types of task query sets, respectively from different branches of the dual-branch decoder: the first gaze query set is output by the gaze prediction decoder and contains the feature representation of the gaze target in the scene, denoted as . The first human query set is output by the human detection decoder and contains the feature representation of the human head region in the scene, denoted as . The two types of query sets serve as the basis for cross-task information interaction, and their feature quality directly affects the accuracy of the cross-gating weight.

[0070] The cross-gating weight is used to quantify the guidance strength of the first gaze query set to the first human query set, and is calculated by the following formula: ; Wherein: is a learnable weight matrix, used to map the features of the first gaze query set to match the weight calculation requirements in terms of feature dimensions; is a bias parameter, used to compensate for the offset in the feature mapping process, and improve the flexibility of weight calculation; is an activation function, which compresses the calculation result to the interval [0, 1], so that the cross-gating weight has a probability meaning, quantifying the importance of gaze features to human head features.

[0071] This process generates dynamic weights by learning the privileged information related to human head detection in the first gaze query set, providing a basis for subsequent feature re-labeling.

[0072] Based on the calculated cross-gating weight , the feature of the first human query set is re-labeled, and the formula is: ; Wherein: represents element-wise multiplication, that is, the cross-gating weight is multiplied by the corresponding elements of the first human query set one by one.

[0073] The re-labeled feature retains the effective features in the first human query set that are strongly related to the gaze task, while suppressing the noise features that are unrelated to the task. This operation dynamically calibrates the gaze features to the human head features, strengthens the relevance between the two task branches, and makes the human head features more suitable for the gaze prediction task, laying a foundation for the discriminative feature generation of the subsequent information guiding module.

[0074] Through the above weight calculation and feature re-labeling process, the cross-gating module realizes the information feedback from the gaze prediction branch to the human head detection branch: on the one hand, it optimizes the feature representation of the first human query set using the privileged information in the first gaze query set, and obtains the second human query set and the second gaze query set through the cross-branch information interaction processing by the multi-task reciprocal transformer. On the other hand, a bidirectional communication channel is established between the two branches, effectively capturing the internal relationship between human head detection and gaze prediction tasks, and alleviating the information asymmetry problem existing in the existing model.

[0075] The global-local position encoding uses smooth dilated one-dimensional convolution SC to capture multi-scale position information.

[0076] The smooth dilated one-dimensional convolution SC includes three groups of convolution pairs, each group of convolution pairs is a dilated convolution with a kernel size of 3 and a step size of 1, and the dilation rates of the three groups of convolution pairs are r=1, 2 and 4 respectively, corresponding to shared convolutions with a kernel size of 2r-1.

[0077] The global-local position encoding (GLPE) is a key component for enhancing position information modeling in the gaze point prediction method based on a multi-task reciprocal transformer, which captures multi-scale position features through smooth dilated one-dimensional convolution (SC) to improve the learning ability of the model to the local and long-range semantic continuity of pixels, and the specific implementation is as follows: The global-local position encoding (GLPE) encodes the position information of the second human query set and the second gaze query set to obtain a third human query set and a third gaze query set.

[0078] The smooth dilated one-dimensional convolution SC includes three groups of convolution pairs, each group of convolution pairs is a dilated convolution with a kernel size of 3 and a step size of 1, and the dilation rates of the three groups of convolution pairs are r=1, 2 and 4 respectively, corresponding to shared convolutions with a kernel size of 2r-1.

[0079] The smooth dilated one-dimensional convolution (SC) is the core unit of the global-local position encoding (GLPE) to capture multi-scale position information, and its structure design is as follows: It includes three groups of independent convolution pairs, each group of convolution pairs models position information of different scales; Each group of convolution pairs uses a dilated convolution with a kernel size of 3 and a step size of 1 to ensure that the receptive field range is expanded without compressing the feature space dimension; The dilation rates of the three groups of convolution pairs are set to r=1, r=2 and r=4 respectively. The dilation rate determines the spacing between the elements of the convolution kernel. When r=1, the convolution kernel continuously covers the input features, when r=2, the spacing is 1 element, and when r=4, the spacing is 3 elements. Through the combination of different dilation rates, the capture of position information of different distances such as near, medium and far is realized; Each group of convolution pairs corresponds to a shared convolution with a kernel size of 2r-1, where r is the dilation rate of the group. When r=1, the shared convolution kernel size is 1; when r=2, the kernel size is 3; and when r=4, the kernel size is 7. The shared convolution further strengthens the fusion of multi-scale position information by integrating the output features of the dilated convolution.

[0080] The position information encoding process of the global-local position encoding (GLPE) is based on a two-step strategy of fixed position encoding + smooth dilated convolution enhancement: First, the fixed position encoding is applied to the second human query set and the second gaze query set to learn the absolute position information of the features, providing a basic coordinate reference for position modeling; The fixed position encoded features are input into 3 groups of smooth dilated one-dimensional convolution (SC). Each group of convolution captures local fine-grained position features, medium-range spatial correlation features and long-range semantic dependency features, respectively, through corresponding dilation rates (r = 1, 2, 4) and shared convolution kernels. The output features of the 3 groups of convolution pairs are fused with the original fixed position encoded features through residual connection to generate encoded features containing multi-scale position information. This fusion method not only preserves the basic position information, but also enhances the position correlation of different scales, making the features more suitable for the position modeling needs of human head detection and gaze prediction tasks.

[0081] Global-local position encoding (GLPE) realizes hierarchical modeling of position information through multi-scale design of smooth dilated one-dimensional convolution (SC): the convolution pair with dilation rate r = 1 focuses on pixel-level local position relationship, improving the fine positioning ability of human head and gaze target. The convolution pairs with dilation rates r = 2 and r = 4 capture medium and long-range spatial correlation, respectively, and strengthen the semantic position dependency between different targets in the scene, such as the long-distance spatial correspondence between human head and potential gaze target. Through this multi-scale position encoding, the dual-branch decoder can more accurately understand the spatial distribution of features, providing accurate position feature reference for cross-branch information interaction of the multi-task reciprocal transformer, and finally improving the prediction accuracy of human head bounding box and gaze target position.

[0082] The third human query set and the third gaze query set obtain the prediction results of human head bounding box and gaze target through a multi-layer perception mechanism, which is based on the joint optimization constraint of the pair dependency relationship formed by human head position and gaze target.

[0083] The joint optimization constraint of the pair dependency relationship is realized through a loss function .

[0084] Wherein is the number of samples, is the matching loss function, is the prediction result, is the target function.

[0085] : represents the number of samples, i.e. the total number of human head position and gaze target pair in the scene image participating in training. This parameter ensures that the loss calculation covers enough samples, so that the model learns the generalization of the pair rule.

[0086] : represents the matching loss function, which is used to quantify the difference between the prediction result of a single pair and the true target.

[0087] : Represents the prediction result of the i-th sample, that is, the head bounding box prediction and gaze target position prediction tuple output by the model, which contains the output information of the head detection head and the gaze prediction head.

[0088] : represents the true objective function corresponding to the i-th sample, where This is a matching function used to match the prediction results. Correctly pair the data with the true labels in the dataset to ensure that the loss calculation targets the one-to-one corresponding predictions and true pairs.

[0089] Pairing dependency refers to the inherent association between the human head position and the gaze target in a scene. That is, each detected human head must correspond to a unique gaze target, and the spatial and semantic relationship between the two must be accurately captured by the model. The goal of joint optimization constraints is to globally optimize the prediction results of this pairing through a loss function, ensuring performance alignment between human head detection and gaze prediction tasks, alleviating information asymmetry, and improving the consistency and accuracy of overall predictions.

[0090] The calculation process of the loss function: For each training sample (scene image), firstly, through the matching function... All predicted pairs from the model output Pair the prediction with the labeled real tuples in the dataset to ensure that each prediction corresponds to a unique real target, thus solving the alignment problem between the prediction and the real target.

[0091] Prediction of each successful pairing ,reality binary pairs, through The error between the two is calculated to reflect the accuracy of the prediction result.

[0092] All samples Single loss Summing up yields the total loss. , which serves as the optimization objective for updating model parameters.

[0093] By combining the joint loss of human head detection and gaze prediction, the model can focus on the performance of both tasks during training, avoiding the information asymmetry problem of optimizing one task while degrading the performance of the other.

[0094] By matching function To ensure the correct alignment between the prediction and the actual binary pair, guide the model to learn the spatial correlation patterns of human head position and gaze target, such as the consistency between the direction of the human head and the direction of the gaze target, and the reasonableness of the distance.

[0095] Total loss The overall prediction error of all samples is reflected, the model optimization is more global, and the modeling ability of the pairing relationship in complex scenes such as multiple people and occlusion scenes is improved. Through the constraint of the loss function, the model can gradually learn the internal dependence between the human head and the gaze target, so that the output human head bounding box and gaze target position are not only accurate, but also the pairing relationship of the binary tuple conforms to the scene semantics, finally improving the overall reliability and explainability of the gaze point prediction.

[0096] The multi-layer perception mechanism predicts the human head bounding box through a 3-layer multi-layer perception machine (MLP); The multi-layer perception mechanism predicts the gaze heat map, gaze target position and gaze inside-out confidence through a 5-layer multi-layer perception machine, a 2-layer multi-layer perception machine (MLP) and a 1-layer multi-layer perception machine (MLP), respectively; The outputs of all multi-layer perception machines (MLPs) are processed by the softmax function and the sigmoid function to obtain the prediction results.

[0097] The prediction head is the core module for outputting the final result in the gaze point prediction method based on the multi-task reciprocal transformer, including a human detection head and a gaze prediction head, which respectively perform feature mapping and result output for human head detection and gaze prediction tasks. The structure design and function implementation are as follows: Structure and function of the human detection head: The human detection head takes the third human query set processed by global-local position encoding as input, and realizes the prediction of the human head bounding box through a multi-layer perception machine (MLP). The specific details are as follows: A 3-layer multi-layer perception machine (MLP) is used, each layer of which maps the input features step by step through linear transformation and nonlinear activation function, and converts high-dimensional features into parameter representation of the human head bounding box.

[0098] The design of the 3-layer MLP aims to accurately predict the coordinate information of the human head bounding box such as the upper left corner and the lower right corner through step-by-step abstract feature processing. The first layer is responsible for feature dimension reduction and preliminary conversion, the second layer strengthens the discriminative features related to the bounding box, and the third layer outputs the specific parameters of the bounding box to realize accurate positioning of the human head region in the image.

[0099] Structure and function of the gaze prediction head: The gaze prediction head takes the third gaze query set processed by global-local position encoding as input, and outputs multiple dimensions of gaze-related results through multiple MLPs with different numbers of layers. The specific structure and function are as follows: 5-layered multi-layer perceptron (MLP): for predicting gaze heatmaps. Heatmaps represent the region where the gaze target is likely to exist in a pixel-level probability distribution, and the 5-layered MLP captures the spatial distribution pattern of the gaze target through deep feature learning, outputting a heatmap matrix matching the input image size, where each pixel value represents the probability of the location being the gaze target.

[0100] 2-layered multi-layer perceptron (MLP): for predicting gaze target position. Through two layers of feature transformation, the coordinate information of the gaze target, such as two-dimensional pixel coordinates, is directly outputted, realizing accurate coordinate positioning of the gaze target, and complementing the position representation of the heatmaps.

[0101] 1-layered multi-layer perceptron (MLP): for predicting gaze inside-outside confidence. This confidence quantifies the probability of the gaze target being inside or outside the image, such as output values close to 1 indicating the target is inside the image and close to 0 indicating it is outside the image. The 1-layered MLP quickly outputs this binary classification result through a simple linear transformation and activation function, assisting in determining the spatial range of the gaze target.

[0102] The outputs of all multi-layer perceptrons (MLPs) need to be processed by activation functions to adapt to the output requirements of different tasks. Among them, the softmax function: is mainly used for probability normalization of gaze heatmaps, converting the output values of each pixel in the heatmap to a probability in the [0, 1] interval, and the sum of all pixel probabilities is 1, clearly representing the spatial distribution probability of the gaze target in the image. The sigmoid function: is used for coordinate normalization of human head bounding boxes and binary classification output of gaze inside-outside confidence. For bounding box coordinates, the sigmoid function compresses the coordinate values to the [0, 1] interval to adapt to the uniform processing of different size images. For confidence, the sigmoid function directly outputs a probability value in the [0, 1] interval, quantifying the possibility of the gaze target being inside or outside the image.

[0103] Through the above processing, the human detection head finally outputs the normalized human head bounding box, and the gaze prediction head outputs the gaze heatmap, gaze target coordinates, and gaze inside-outside confidence, together forming the complete prediction result of the human head and gaze target.

[0104] The human detection head and the gaze prediction head achieve task differentiation output through their respective multi-layer perceptron (MLP) structures, while forming a synergy through the paired dependency relationship of the human head position and the gaze target pair. The bounding box result of the human detection head provides a spatial reference for gaze prediction, such as relative position calculation with the human head as the origin, and the result of the gaze prediction head provides semantic rationality verification for human head detection, such as consistency between the gaze direction and the human head orientation. After being processed by the activation function, the outputs of the two become inputs for subsequent joint optimization constraints, ultimately realizing precise synergy of human head detection and gaze prediction.

[0105] To verify the technical advantages of the above technical solutions, the following experiments are performed. The steps are as follows: S1, collect data sets and evaluation indexes: The proposed MAM multitask reciprocal transformer is evaluated using two large-scale gaze prediction datasets, namely GazeFollow and VideoAttentionTarget. GazeFollow is a large-scale image dataset containing more than 122,143 images and 130,339 individuals, of which 4,782 individuals are used for testing. The test images include annotations of gaze and head position of up to 10 individuals in the scene, while in the training set there is only one annotation for each individual indicating gaze and head position. VideoAttentionTarget consists of 50 YouTube video clips, each with a length of 1 to 80 seconds. It includes 1,331 head trajectories, 164,541 frame-level bounding boxes, 109,574 intra-frame gaze targets, and 54,967 out-of-frame gaze annotations. The training set and the test set only contain one gaze annotation for each individual. To avoid overfitting, in the training set, one image out of every 5 consecutive frames is randomly selected for training as they have almost the same gaze target. For testing, all images in the test set are used.

[0106] Evaluation indexes. Six criteria are used to evaluate the gaze target prediction and human head position detection performance of the model, including the confidence distribution quality of the gaze heat map (area under the curve), the average distance between the predicted gaze point and the true annotation, the minimum distance, the normalized distance (the Euclidean distance between the point with the highest confidence in the gaze heat map and the true annotation when the image size is normalized to 1), the average accuracy, and the comprehensive detection accuracy. For gaze target prediction, the results of gaze heat map quality (area under the curve), average distance, minimum distance, normalized distance, and average accuracy are reported. The gaze heat map quality is evaluated by the confidence distribution. In GazeFollow, the true annotation comes from ten annotators, so the average distance and the minimum distance are used to calculate the difference between the predicted position and all true annotations; while VideoAttentionTarget only uses a single annotation coordinate, so only the normalized distance is reported. In addition, the average accuracy is used to evaluate the model's ability to determine whether the gaze target is within the image.

[0107] For human head position detection, the comprehensive detection accuracy is used for evaluation, which requires that both conditions are met to be considered correct detection: the coverage ratio of the predicted human head region to the true region exceeds 50%, and the Euclidean distance between the predicted gaze point and the true annotation is less than 0.15.

[0108] In the visual encoder, a combination of convolutional neural network and Transformer encoder is adopted. ResNet-50 is used as the convolutional neural backbone network of the feature extractor, while the Transformer encoder consists of 6 layers, each of which contains self-attention mechanism and cross-attention mechanism with 8 attention heads.

[0109] In the dual-branch decoder, each branch decoder consists of 3 stacked attention mechanism-based decoder modules, each of which contains a decoding structure with 6 layers of multi-head self-attention and cross-attention, and the architecture of the two branch decoders is consistent. The number of human queries and gaze queries is set to 20.

[0110] MAM is implemented using PyTorch. The network is initialized using the parameters of the attention mechanism-based detection model (DETR) trained on the general object recognition dataset. The AdamW optimizer is used for training with a batch size of 16 for 100 epochs. Specifically, the initial learning rate of the backbone network is set to 10−5, and the initial learning rate of the other parts is set to 10−4, and the two learning rates are decayed after 80 epochs, and the weight decay is set to 10%. For the hyperparameters of the model, λ1 and λ2 are set to 1.0 and 2.5, and the weights of different cost functions in the loss function (θ1, θ2, θ3, θ4) are selected according to the dataset. All experiments are completed on a computing device equipped with 2 high-performance graphics processors.

[0111] Quantitative comparison results. The multi-task mutual transformer (MAM) of the present application is compared with the current existing method on two challenging benchmark datasets, GazeFollow and VideoAttentionTarget. The current existing method is a gaze prediction model that simultaneously detects the positions of all people and gaze objects in an end-to-end one-stage framework. The comparison results are shown in Figure 2 and Figure 3 , where bold numbers represent the best results, and underlined numbers represent suboptimal results. Reviewing previous work, most of them require head-cropped images and head positions as inputs to the model. Therefore, the present application reports their results under two different settings: “Head GT” and “Real”.

[0112] Under the "Head GT" setting, the present application directly utilizes the manual head crop provided by the dataset to perform model evaluation. Considering the practical situation, the present application refers to the Tu et al. model and adopts a ready-made head detector SSD to extract the head crop image in the "Real" setting and feed it into the existing model for practical application. It is worth noting that MAM does not require a head crop as input, but the present application still reports the results of MAM under the "Head GT" setting by inputting the scene image and the head crop into the visual encoder of the present application and disabling the human body decoder.

[0113] The Tu et al. model is the model disclosed in the literature Tu D, Min X, Duan H, et al. End-to-end human-gaze-target detection with transformers [C] / / 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022: 2192-2200.

[0114] From Figure 2 and Figure 3 the present application can draw the following conclusions: MAM is built on the basic network structure proposed by Tu et al., which is a gaze prediction method based on a single-branch multi-task learning network. Compared with the network structure of the Tu et al. model, the proposed MAM has achieved significant performance improvement on all indicators on the GazeFollow and VideoAttentionTarget datasets.

[0115] Specifically, when the model of the present application is compared with the Tu et al. model under the network of the "Real" setting of the GazeFollow dataset, the area under the curve (AUC) and the comprehensive detection precision (mAP) are increased by 1.9% and 8.4%, respectively, while the average distance (AvgDist) and the minimum distance (MinDist) are increased by 0.063 and 0.04, respectively. For the VideoAttentionTarget dataset, the area under the curve (AUC), the average accuracy (AP), and the comprehensive detection precision (mAP) are increased by 4.5%, 11.9%, and 14.3%, respectively, and the distance (Dist) is increased by 0.053. These improvements in gaze target prediction and head position detection tasks not only prove the effectiveness of the proposed MAM, but also prove its effectiveness in solving the problem of information asymmetry.

[0116] Overall, the MAM proposed by the present invention outperforms all other comparative methods on all datasets and evaluation metrics. For example, even with additional inputs or producing only one gaze result at a time, some other existing methods perform relatively worse compared to the MAM. It is worth noting that the MAM of the present invention is not the best in the “Real” setting on the VideoAttentionTarget dataset. Because there are a large number of instances of gaze targets outside the image on the VideoAttentionTarget dataset, and their positions are all labeled in the upper left corner, resulting in some instances outside the image do not have a pair-wise dependency relationship with real semantic information. The improvement made by the present invention is to solve the problem of information asymmetry.

[0117] Compared with the GazeFollow dataset, the VideoAttentionTarget dataset contains more HGT instances and stricter annotations. The MAM proposed by the present invention achieves more favorable results on VideoAttentionTarget than on GazeFollow, indicating that the MAM has strong robustness and performs better in more complex scenarios and more challenging prediction tasks.

[0118] The results of the MAM in the “Real” setting are better than in the “Head GT” setting. This finding indicates that predicting the head bounding box and the gaze target simultaneously helps the model achieve better gaze prediction, as it allows the model to explicitly focus on the relationship between the head features and the gaze target. Importantly, except for the Miao et al. model, all other SOTA comparative methods and the work of the present invention only use single-modality input (RGB images). Therefore, to ensure Figure 2 and Figure 3 For a fair comparison of performance, the present invention does not include the results of the Miao et al. model that uses depth information as an additional input. It is worth noting that even with single-modality input, the method of the present invention is superior to the multi-modal version proposed by the Miao et al. model.

[0119] The Miao et al. model is derived from the public literature Miao Q, Hoai M, Samaras D. Patch-level gaze distribution prediction for gaze following [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2023: 880-889.

[0120] Qualitative Comparison Results. To further demonstrate the performance of MAM, this invention visualizes the learned gaze heatmap and compares it with the method proposed by Tu et al. The Tu et al. model is a known method in the field of one-stage gaze prediction. The MAM framework of this invention can predict the head bounding box and gaze target location for each person in an image; however, for clarity, this invention randomly selects only one person and visualizes the corresponding prediction results, such as... Figure 4 As shown.

[0121] exist Figure 4 In the first row of images, the Tu et al. model accurately detects head position but poorly predicts gazing targets. The second row shows that the Tu et al. model accurately predicts gazing targets but has a large bias in head position detection. The third row shows that the Tu et al. model performs poorly. (This is compared with...) Figure 4 A point-by-point comparison of the results from the Tu et al. model shows that the MAM proposed in this invention can simultaneously predict accurate head position and gaze target, and its performance is significantly better than that of the Tu et al. model, indicating that the method of this invention effectively solves the problem of information asymmetry.

[0122] In this work, extensive ablation experiments were conducted on GazeFollow and VideoAttentionTarget to verify the effectiveness of each module in the proposed MAM from the perspectives of model design and loss design. All experiments used ResNet-50 as the backbone, and only results under the "Real" setting are reported. In particular, the basic components of the MAM are based on the framework proposed by Tu et al.; therefore, this invention uses the Tu et al. model as the baseline architecture and presents results of constructing different networks by gradually adding different modules to the baseline.

[0123] Model Design. This invention first tested MAM and GLPE under optimal configurations, and then tested the CGM and IGM submodules within MAM to demonstrate their importance to model design. Results are shown below. Figure 5 and Figure 7 .

[0124] like Figure 5 As shown, this invention conducted various ablation experiments on the proposed model using MAM and GLPE. Specifically, this invention added MAM and GLPE sequentially to the dual-branch decoder to verify the effectiveness of each module. When GLPE is not used, traditional positional coding is employed. Figure 5 In this context, "HDD" and "GPD" represent the human detection decoder and gaze prediction decoder in the dual-branch decoder, respectively. Figure 5 show: The integration of MAM into the human detection decoder significantly improves the model performance. This finding indicates that the privileged information learned from the gaze prediction branch effectively guides the feature learning of the human detection branch, thus improving the final results.

[0125] When MAM is also integrated into the gaze prediction decoder, the overall performance is further improved. This is because the human detection branch also guides the feature learning of the gaze prediction task.

[0126] After further introducing GLPE into the two branch decoders, the model performance is best. Notably, the MAM method of the present application achieves consistent results on the GazeFollow and VideoAttentionTarget datasets.

[0127] In summary, through MAM, privileged information can be learned from two different branches respectively, and the information learned from one branch can guide the other branch to complete its own task, effectively solving the problem of information asymmetry. In addition, by introducing GLPE into the MAM proposed by the present application, global and local context information can be captured to help the transformer decoder learn more consistent relative position information for specific tasks. Therefore, the subsequent experiments use the model setting that achieves the best performance.

[0128] The role of MAM. To further verify the importance of MAM to the model of the present application, the present application also analyzes the prediction accuracy with and without MAM. Specifically, the present application tests 3159 images from the VideoAttentionTarget dataset, which contain a total of 9207 individuals. The number of people in each image varies from 1 to 7, which enables the present application to evaluate the model's prediction of head position and gaze in the case of few or many individuals, as shown in Figure 6 .

[0129] In Figure 6 , each bar chart represents the head / gaze prediction accuracy of each image in different scenarios with different numbers of individuals. The blue bar represents the statistical results of the model without MAM, and the orange bar represents the statistical results of the model with MAM. As can be seen from Figure 6 , the introduction of MAM improves the accuracy of head and gaze prediction. In addition, comparing images containing 1 to 7 people, it can be found that as the number of people in the image increases, the performance improvement brought by MAM is more significant.

[0130] To more quantitatively understand how the use of privileged information improves the performance of the model of the present application, the present application conducts an ablation study on the sub-modules CGM and IGM of MAM, as shown in Figure 7 .

[0131] From Figure 7It can be seen that: After adding CGM, the performance is improved, for example, the mAP on the VideoAttentionTarget dataset is increased by 0.106, which shows that the privileged information learned by CGM enables the communication between the two branches to improve the model performance.

[0132] The introduction of IGM in the model can also improve the performance of the model. This is because IGM can select task-related semantic information from other branches and explore the relationship in the spatio-temporal context.

[0133] Compared with using only one sub-module of CGM or IGM, introducing CGM or IGM into the model in combination can further improve the performance of the model. This is because this setting not only promotes the exchange of information between branches through privileged information, but also effectively utilizes privileged information to select task-related discriminative features.

[0134] Model computational cost. The number of parameters (#Params.) and the number of floating-point operations (FLOPs) are key indicators for evaluating the complexity of the model. Figure 8 The changes in computational cost (#Params. and FLOPs) and model performance (AUC and mAP) on the VideoAttentionTarget dataset are shown, as MAM and GLPE are introduced into the baseline model in turn. The Tu et al. model first proposed a method for simultaneously detecting head position and predicting gaze target, which greatly inspired the work of the present application.

[0135] However, the method of the present application is different in structure; for example, the MAM proposed by the present application adopts a double-branch structure, while the Tu et al. model adopts a single-branch structure. In Figure 8 , “Baseline” refers to using only the traditional fixed position embedding without the double-branch structure of MAM and GLPE, “Baseline+MAM” refers to the baseline with the addition of MAM, and “Baseline+MAM+GLPE” refers to the baseline with the simultaneous addition of MAM and GLPE.

[0136] As shown in Figure 8 , with the introduction of MAM and GLPE, the model accuracy gradually improves. This finding shows that MAM and GLPE significantly improve the model accuracy with relatively small increases in computational cost. Although the parameters and FLOPs of Baseline+MAM+GLPE increase relatively greatly compared with the Tu et al. model, the increase in computational cost is mainly due to the transition from a single-branch structure to a double-branch structure, while the MAM and GLPE proposed by the present application are effective and efficient in a multi-branch architecture.

[0137] Visualization of learned privileged information. To further demonstrate that the MAM proposed in the present application can effectively learn privileged information, thereby improving the accuracy of head detection and gaze estimation, the present application visualizes the learned privileged information, outputs the gaze prediction heat map and head bounding box, and shows the final gaze prediction and head detection results, as shown in Figure 9 Visualization of multiple individuals' heat maps can cause significant interference and overlapping areas, which may hinder intuitive assessment of prediction accuracy. Therefore, the present application visualizes the heat map of a single individual in Figure 8

[0138] The images in the first column are the original images. The second and third columns show the privileged information learned by the two branch decoders. The fourth column shows the position of the head, and the corresponding gaze heat map output by the final model. The last column shows the gaze prediction results, where the red bounding box and line represent the ground truth, and the green bounding box and line represent the prediction results of the model of the present application.

[0139] In Figure 9 Privileged Information (HDD) and Privileged Information (GPD) represent the privileged information learned from the human detection decoder and the gaze prediction decoder, respectively. Output represents the paired output gaze heat map and head position of the model. Prediction represents the comparison between the results of the present application and the true situation. Figure 9 The visualization results show that: The human head detection branch contains privileged information, highlighting the prominent areas that the given person may focus on, which can guide the subsequent gaze prediction task.

[0140] The gaze prediction branch contains privileged information, highlighting the possible position of the given person, which guides the subsequent human head detection task.

[0141] The final output of the gaze target and the human head involves accurate positioning, which proves that learning privileged information from different tasks and guiding feature communication between the human head detection branch and the gaze prediction branch are effective, which can alleviate the problem of information asymmetry.

[0142] Loss design. Figure 10 The prediction results for different cost function weights θ of the pairwise matching loss (MPLoss) are shown in Figure 10 From Figure 10 the present application, the following conclusions can be drawn: The optimal results of AUC and mAP on the GazeFollow dataset are achieved when θ1=2.5, θ2=2, θ3=1 and θ4=1, which indicates that the model reaches the best overall performance at this time. However, when θ1=2.5, θ2=2, θ3=1 and θ4=1, the present application achieves the optimal performance of average distance (AvgDis) and minimum distance (MinDist), but AUC and mAP decrease significantly. This is because the GazeFollow dataset has multiple approximate gaze target annotations for a person, and increasing the gaze point position loss will cause the model to overfit to some of the gaze targets, thereby harming the overall performance of the model. Therefore, the present application adopts θ1, θ2, θ3 and θ4 as 2.5, 2, 1 and 2 respectively as the fixed parameters on the GazeFollow dataset, because the overall performance of the model reaches the optimal under this parameter setting.

[0143] Due to the strict and unique annotation of the VideoAttentionTarget dataset for each person, the model performance also improves significantly when gradually increasing the loss function weight about the head and gaze, until θ1=2.5, θ2=2, θ3=1 and θ4=1 reach the best performance, only suboptimal in AP. This is predictable because the location of the gaze target outside the image is not the actual location, but is artificially annotated to the top left corner of the picture, which has no semantic relationship with the head position. Therefore, θ1, θ2, θ3 and θ4 are adopted as 2.5, 2, 2 and 1 respectively as the fixed parameters on the VideoAttentionTarget dataset.

[0144] When comparing "1=1, 2=1, 3=2, 4=2" with "1=1, 2=1, 3=1, 4=2", there is a certain degree of performance decline in AUC, AvgDist, mAP on the GazeFollow dataset, and AUC and mAP on the VideoAttentionTarget dataset. This is because simply increasing the value of 3 will reduce the model's attention to head detection and gaze heat map, resulting in a decline in model performance.

[0145] Specifically, "3" corresponds to the attention weight parameter of the 5-layer MLP in the gaze prediction head, and its value directly affects the model's attention to the gaze heat map: When "3=2" is adjusted to "3=1", the attention weight of the 5-layer MLP decreases, resulting in a decrease in the model's feature extraction ability for the gaze heat map, and a decrease in the quality of the confidence distribution of the gaze heat map, which is manifested as a decrease in the AUC value.

[0146] The decrease in gaze heat map quality further affects the prediction accuracy of the gaze target position, resulting in an increase in AvgDist and an increase in normalized distance (reflected in the VideoAttentionTarget dataset).

[0147] At the same time, the pairing dependency relationship between the gaze heat map and the human head boundary box output by the human detection head is weakened due to the decrease in heat map quality, resulting in a decrease in mAP.

[0148] Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for gaze point prediction based on multi-task reciprocal transformer, characterized in that, The method comprises the steps of: obtaining a scene image containing a human, performing image feature extraction on the scene image, and generating global memory features; generating a first human query set and a first gaze query set from the global memory features through self-attention mechanism and cross-attention mechanism respectively, and then performing cross-branch information interaction processing on the first human query set and the first gaze query set through a multi-task reciprocal transformer to obtain a second human query set and a second gaze query set; performing position information coding on the second human query set and the second gaze query set through global-local position coding to obtain a third human query set and a third gaze query set; obtaining a human head bounding box and a gaze target prediction result through a multi-layer perception mechanism based on the binary pair dependency relationship formed by the human head position and the gaze target.

2. The gaze point prediction method based on the multi-task reciprocal transformer according to claim 1, wherein: the scene image is extracted through a convolutional neural network; the image features are converted into the global memory features through self-attention mechanism and cross-attention mechanism.

3. The gaze point prediction method based on the multi-task reciprocal transformer according to claim 1, wherein: all human heads and gaze targets in the scene image are detected through the third human query set and the third gaze query set.

4. The method of gaze point prediction based on multi-task reciprocal transformer of claim 1, wherein, The multi-task reciprocal transformer calculates a cross-gating weight based on the first gaze query set and recalibrates the features of the first human query set, and uses the privileged information in the recalibrated human head features to guide the generation of discriminative features.

5. The method of gaze point prediction based on multi-task reciprocal transformer of claim 4, wherein, The cross-gating weight calculation process comprises: Input gaze query set And human query set ; Through the formula: Calculate cross-gating weights ,in For learnable weight matrix, For bias; By the formula: for a set of human queries relabeling the features, where denotes element-wise multiplication.

6. The method of gaze point prediction based on multi-task reciprocal transformer of claim 1, wherein, The global-local position coding uses a smooth dilated one-dimensional convolution SC to capture multi-scale position information. The smooth dilated one-dimensional convolution SC comprises three groups of convolution pairs, each group of convolution pairs is a dilated convolution with a kernel size of 3 and a step size of 1, and the dilation rates of the three groups of convolution pairs are r=1, 2 and 4 respectively, corresponding to a shared convolution with a kernel size of 2r-1.

7. The gaze point prediction method based on multi-task reciprocal transformer of claim 1, wherein, The pairing dependency relationship is through a loss function implementation; wherein is the number of samples, is the matching loss function, is the prediction result, is the objective function.

8. The gaze point prediction method based on multi-task reciprocal transformer of claim 1, wherein, The multi-layer perception mechanism predicts the human head bounding box through a three-layer multi-layer perception MLP; The multi-layer perception mechanism predicts the gaze heat map, the gaze target position and the gaze inside-out confidence through a five-layer multi-layer perception MLP, a two-layer multi-layer perception MLP and a one-layer multi-layer perception MLP respectively; The outputs of all multi-layer perception MLPs are processed through a softmax function and a sigmoid function.

9. A gaze point prediction system based on a multi-task reciprocal transformer, characterized in that, The method comprises the steps of: an image acquisition module for acquiring a scene image containing a human; a visual coding module for performing image feature extraction on the scene image to generate global memory features; a double-branch decoding module for generating a first human query set and a first gaze query set through self-attention mechanism and cross-attention mechanism respectively; a multi-task reciprocal transformer for performing cross-branch information interaction processing on the first human query set and the first gaze query set to obtain a second human query set and a second gaze query set; a global-local position coding module for performing position information coding on the second human query set and the second gaze query set; The prediction output module adopts a multi-layer perception mechanism to obtain a prediction result of the human head boundary frame and the gaze target.

10. The system of claim 9, wherein, The multi-task mutual transformation device comprises a cross-gating module and an information guiding module, the cross-gating module calculates cross-gating weights and completes feature re-labeling, and the information guiding module realizes cross-task feature guiding based on privileged information in the re-labeled features.

Images