Complex scene indication expression understanding method based on cross-modal eye movement attention perception

Abstract

The invention discloses a complex scene indication expression understanding method based on cross-modal eye movement attention perception, and belongs to the field of computer vision, machine learning and multi-modal understanding. According to the method, a dynamic deformable attention mechanism perceived by a language is designed, a human eye gaze spectrum is used as supervision information, a corresponding visual area is adaptively captured according to language features, meanwhile, a Transform decoder driven by an eye movement spectrum is designed, and the position of a target area indicated by the language is deduced by gradually fusing visual feature representation, so that the target area can be recognized. Therefore, the human eye visual attention perception area and the transfer process are displayed and simulated, and the understanding precision of complex scene indication expression is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, machine learning and multimodal understanding, and proposes an algorithm for understanding complex scene indication expression based on cross-modal eye movement attention perception. Background Art

[0002] With the rapid development of information technology, images and videos are showing a trend of massive growth, and people's demand for computer vision technology is increasing. In addition to requiring computers to be able to perform simple and fixed recognition and positioning of targets in images, they are also required to be able to flexibly locate the indicated target objects of interest to users in the image according to the user input language, that is, the indicating expression understanding task. The present invention mainly focuses on the research of indicating expression understanding tasks to assist in building a bridge between human language and real-world visual content, and has broad application prospects in the fields of human-computer interaction, intelligent navigation, target search, etc.

[0003] However, due to the complexity and diversity of scenes and the flexibility of language, computers face huge challenges in understanding and interpreting these indicative expressions. Traditional indicative expression understanding models are usually divided into two categories: two-stage and single-stage methods. Among them, the two-stage method usually models the indicative expression understanding task as an indication-expression pair retrieval problem. First, a pre-trained deep convolutional network-based target detector is used to extract candidate regions, and then the box that best matches the user's language is selected from these candidate regions as the target. In contrast, the single-stage method removes the candidate region generation stage in the two-stage method, effectively avoiding the reliance on pre-trained detectors. This type of method directly embeds natural language features into the target detector, and predicts the region where the target is located through simple operations such as cascading or stacking convolutional layers. Although these methods have made significant progress, they usually need to rely on manually designed networks to associate language and visual features in order to locate the target area. These models are often limited by hyperparameters such as the size of the convolution kernel and the number of convolution layers, and only stay at the "black box" perception and fusion of language and vision. They have obvious limitations when processing these complex scenes, making it difficult to fully understand the semantic content of complex scenes and accurately perceive the key features of different image contents.

[0004] In response to the above problems, this project proposes an algorithm for understanding complex scene instructions based on eye movement attention perception. First, based on the public target detection data set, a human eye gaze spectrum database is constructed to collect the attention perception and transfer characteristics of the human eye under different language stimuli. Different from the traditional fixed feature fusion, this invention proposes an algorithm for understanding complex scene instructions based on cross-modal eye movement attention perception based on the human eye gaze spectrum. By designing a dynamic deformable attention mechanism for language perception, the human eye gaze spectrum is used as supervision information, and the corresponding visual area is adaptively captured according to the language features. At the same time, the eye movement spectrum-driven Transformer decoder is designed. By gradually fusing the visual feature representation, the target area position of the language instruction is inferred, thereby explicitly simulating the human eye visual attention perception area and transfer process, effectively improving the accuracy of understanding complex scene instructions. Summary of the invention

[0005] The present invention proposes a complex scene instruction expression understanding method based on cross-modal eye movement attention perception, so as to solve the problem that the existing instruction expression understanding algorithm only stays in the "black box" perception and fusion of language and vision, resulting in difficulty in effectively perceiving the visual areas corresponding to different languages ​​and inferring the target area when processing these complex scenes.

[0006] In order to solve the above problems, the present invention adopts the following specific technical solution: a method for understanding complex scene indication expression based on cross-modal eye movement attention perception, the method comprising:

[0007] Step 1: Build a human eye gaze spectrum database;

[0008] Use a desktop eye tracker to collect eye movement records of the subjects; record the subject's gaze position and dwell time changes on the image content; in order to ensure the quality of gaze data, each language-image pair will collect eye movement gaze information of multiple subjects;

[0009] Then the collected data are clustered and fused to reduce the impact of outliers and ensure the reliability of gaze information;

[0010] Step 2: Extract the language features of the deictic language;

[0011] First, the input instruction language is divided into word sequences, and then each word is encoded into a high-dimensional vector space using the Token Embedding layer. Finally, the encoded language sequence is input into the Bert model to extract the hidden state of the input language, that is, to obtain the language feature L of the input language.

[0012] Step 3: Extract the visual feature spectrum of the input image;

[0013] Firstly, a deep convolutional neural network ResNet is used to extract image features of different scales of the input image. Then, all scale feature spectra are aggregated, and stacked Transformer encoding layers are used to interact and associate features of different visual regions to generate a visual feature spectrum V.

[0014] Step 4: Predict cross-modal human eye attention spectrum;

[0015] First, a linear mapping layer and a nonlinear activation function are used to aggregate the current language features and the features of the previous gaze spectrum after average pooling to generate a position filter and a weight filter corresponding to the current attention perception area. Then, the position filter is used to convolve the visual features to generate the language-related visual perception area coordinate offset Δp t ; Use the weight filter to obtain the weight value w of each coordinate position in the visual perception area t Thus, for each spatial position p0 in the image, according to the coordinate offset and weight value, the perceptual region feature V associated with it is dynamically weighted and aggregated to obtain the language-related current moment visual attention perceptual region feature representation V t (p0):

[0016]

[0017]

[0018] In the formula, p0, p j ∈G represent the coordinate offset of any spatial position p0 in the image and its initial perception area, represents the attention perception coordinate offset at the jth position at the current time t, Δp t is the set of all coordinate position offsets in a fixed area; by adjusting Changing the visual perception area window at the previous moment can adaptively simulate the shift changes of the human eye's gaze spectrum; A t (p0) represents the predicted human eye attention spectrum at time t under the stimulation of the indicative language description:

[0019]

[0020] In the formula, represents the learnable weight matrix, A t (p0) represents the importance score of the p0th image region at the current time t. The higher the score, the higher the attention of the human eye to that position; conversely, it indicates that the attention is low; H and W represent the height and width of the current attention spectrum respectively; finally, all eye gaze spectra related to the current language are predicted To explicitly simulate the visual attention perception area and transfer process of the human eye; where τ represents the number of included fixation spectra;

[0021] Finally, in order to optimize the prediction quality of the gaze spectrum and the transfer process, the spectral distribution metric loss function is used Linear correlation measure loss function And spectral temporal metric loss function The gaze spectrum prediction model is trained together so that the network generates a visual gaze spectrum that is more consistent with human perception and understanding, namely:

[0022]

[0023] In the formula, represents the loss function of the overall eye movement spectrum, represents the number of recorded eye movement spectra moments, represents the KL divergence loss function, which is used to narrow the significant area distribution between the real gaze spectrum and the predicted spectrum at the current time t. To measure the linear correlation between the predicted gaze spectrum and the true spectrum; It is used to sort the predicted gaze spectra at all times to accurately describe the shift and change process of the human eye gaze spectra;

[0024] Step 5: Based on the predicted language-guided human eye attention spectrum, weighted aggregation is performed on the visual features to obtain cross-modal semantic features;

[0025] Based on the multi-head attention mechanism MATT(), a learnable feature vector is used as the target query, and the correlation is calculated with the cross-modal features at different times in turn to capture discriminative visual features to simulate the human vision and reasoning process for the target; for the cross-modal features at each moment, layer normalization LN() and a forward propagation network FFN() consisting of two fully connected layers are used to update the features; at the same time, based on the updated target query features, a multi-layer perceptron MLP() is used to predict the bounding box coordinates of the language-indicated target; that is:

[0026] Q t+1 =LN(Q t +FFN(Q t ))

[0027] B t (x,y,w,h)=MLP(Q t )

[0028] In the formula, B t (x, y, w, h) represents the coordinates of the target bounding box predicted at time t, x, y, w, h represent the center point coordinates, width and height of the bounding box respectively; Q t represents the target query vector at time t;

[0029] Step 6: Instruct the expression understanding network to optimize;

[0030] In the network training optimization stage, the positive and negative sample allocation process is removed, and the predicted target box B is directly calculated. t With the real annotation box The regression loss between , so the loss function of the entire network is:

[0031]

[0032] In the formula, It is the SmoothL1 loss function, which constrains the four coordinates of the predicted bounding box to be close to the true box value. It aims to comprehensively compare the intersection-over-union ratio between the predicted bounding box and the true bounding box; λ1 and λ2 are loss weight coefficients to control the balance between the two loss functions.

[0033] Furthermore, the specific method of step 1 is:

[0034] Firstly, according to the discrete fixation point map calculation method, the collected fixation data of all subjects are statistically analyzed to obtain the image fixation data corresponding to each input language; secondly, the Gaussian blur method is used to filter the Gaussian filter G σ Perform convolution operation on all gaze points to obtain the final gaze thermodynamic spectrum f(x);

[0035]

[0036] S(x)=f(x)*G σ (x)

[0037] In the formula, x is the image two-dimensional space coordinate vector, x i (k) represents the coordinates of the i-th subject at the k-th gaze point, M a and N a denote the number of fixations and the number of subjects of the ith subject in the current language-image pair; δ(·) denotes the Kronecker function; when xx i When (k) = 0, then δ(·) = 1, otherwise δ(·) = 0; σ is the standard deviation;

[0038] Since the collected gaze data is a continuous data, the gaze data with a continuous gaze time less than the set threshold is removed according to the time sequence of the gaze data. Finally, multiple gaze spectra are obtained to indicate the process of human attention transfer to language description.

[0039] Furthermore, in step 4, the fully connected layer is used to generate a position filter and a weight filter corresponding to the attention perception area at the current moment.

[0040] Further, in step 4

[0041] Further, in step 4 By calculating A t and Y t The covariance Cov(A t ,Y t ) and standard deviation Var(A t )Var(Y t ), measuring the linear correlation between the predicted gaze spectrum and the true spectrum.

[0042] Further, in step 4

[0043] The complex scene instruction expression understanding algorithm based on cross-modal eye movement attention perception proposed in the present invention collects the attention perception and transfer change characteristics of the human eye under different language stimuli based on the constructed human eye gaze spectrum database. Different from the traditional fixed feature fusion, the present invention designs a dynamic deformable attention mechanism for language perception, uses the human eye gaze spectrum as supervision information, and adaptively captures the corresponding visual area according to the language features. At the same time, it designs an eye movement spectrum-driven Transformer decoder, and infers the target area position of the language instruction by gradually fusing the visual feature representation, thereby explicitly simulating the human eye visual attention perception area and transfer process, and effectively improving the accuracy of complex scene instruction expression understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is an overall process framework diagram of the complex scene indication expression understanding algorithm based on cross-modal eye movement attention perception of the present invention.

[0045] Figure 2 It is a subjective effect diagram of the present invention. DETAILED DESCRIPTION

[0046] The present invention conducts experiments on a Pytorch-based deep learning platform, which specifically includes the following steps:

[0047] Step 1: Select a public database for pointing expression understanding, such as the RefCOCO database, the RefCOCO+ database, and the RefCOCOg database, which consists of images, pointing language-target region pairs. The RefCOCO database and the RefCOCO+ database consist of a training set train, a validation set val, a test set testA, and a test set testB, where testA mainly involves the target category "people" and testB is other categories. The pointing language of the RefCOCOg database is relatively longer, consisting of a training set train, a validation set val, and a test set test.

[0048] Step 2: Based on the public natural scene dataset, 10 experimenters with normal or corrected-to-normal vision and normal color vision were recruited to use a desktop eye tracker to record eye movement experiments. At the same time, each language-image pair was required to contain eye movement information of multiple subjects. These data were further clustered and fused to reduce the impact of outliers and ensure the reliability of gaze information. Finally, the gaze information was analyzed and processed to obtain the human eye gaze spectrum and attention transfer process information.

[0049] During the labeling process, the computer will randomly display the image and the corresponding indicative category vocabulary description. The subject needs to wear an eye tracker and find the corresponding target area of ​​the vocabulary according to the indicative vocabulary description. During the observation process, the eye tracker can record the subject's gaze position and dwell time changes on the image content by tracking eye movement information. In order to ensure the quality of gaze data, each language-image pair will collect eye movement gaze information of multiple subjects. The present invention needs to further cluster and fuse these data to reduce the impact of outliers and ensure the reliability of gaze information.

[0050] In order to obtain the degree of human eye's attention perception to different visual areas, the present invention firstly performs statistical analysis on all collected subject's gaze data according to the discrete gaze point map calculation method to obtain the image gaze data corresponding to each input language. Secondly, the Gaussian blur method is used to filter the Gaussian filter G σ Perform convolution operations on all gaze points to obtain the final gaze thermal spectrum of human eyes. That is:

[0051]

[0052] S(x)=f(x)*G σ (x)

[0053] In the formula, x is the image two-dimensional space coordinate vector, x i (k) represents the coordinates of the i-th subject at the k-th gaze point, M a and N a denote the number of fixations of the ith subject and the number of subjects in the current language-image pair. δ(·) denotes the Kronecker function. When xx i When (k) = 0, then δ(·) = 1; otherwise δ(·) = 0. σ is the standard deviation, which is generally related to the size of the computer screen and the viewing distance to reflect the area around the subject that is of high attention.

[0054] Since the collected gaze data is a continuous data, according to the time sequence of the gaze data, the present invention removes the gaze data whose continuous gaze time in a certain area is less than the set threshold, thereby ensuring the reliability of the gaze points finally obtained and avoiding the interference of abnormal data. Finally, multiple gaze spectra are obtained to show the process of human attention transfer to language description.

[0055] Step 3: Input the image in the database into the image feature encoder and extract the encoded visual features The image feature extractor uses the ResNet-50 or ResNet-101 model, the input image size is 640×640, C=512 represents the visual feature channel dimension, and the initial learning rate is set to 10e-5.

[0056] Step 4: Input the descriptive language description in the database into the language feature encoder to extract the corresponding descriptive language features The language feature encoder adopts the Bert model, C=768 represents the language feature channel dimension, the maximum number of words is set to 40, and the initial learning rate is set to 10e-5.

[0057] Step 5: Based on the extracted language and image features, in order to simulate the adaptive perception of the human eye and the shift changes in the visual area related to the language, the present invention designs a dynamic deformable attention mechanism. First, a linear mapping layer and a nonlinear activation function are used to aggregate the current language features and the features after the average pooling of the gaze spectrum at the previous moment to generate a position filter and a weight filter corresponding to the current attention perception area. Then, the position filter is used to convolve the visual features to generate the language-related visual perception area coordinate offset Δp t ; Use the weight filter to obtain the weight value w of each coordinate position in the visual perception area t Therefore, for each spatial position p0 in the image, the relevant perceptual region feature V can be dynamically weighted and aggregated according to the coordinate offset and weight value to obtain the language-related current moment visual attention perceptual region feature representation V t :

[0058]

[0059] In the formula, p0, p j ∈G represent the coordinate offset of any spatial position p0 in the image and its initial perception area, Represents the attention perception coordinate offset at the current time t. By adjusting The visual perception area window of the previous moment can be changed, so as to adaptively simulate the shift change of the human eye gaze spectrum. Here, A t It represents the predicted human eye attention spectrum at time t under the stimulation of the indicative language description:

[0060]

[0061] In the formula, A represents the learnable weight matrix. t (p0) represents the importance score of the p0th image region at the current time t. The higher the score, the higher the attention of the human eye to this position. On the contrary, it indicates that the attention is low. Finally, all eye gaze spectra related to the current language can be predicted. To explicitly simulate the visual attention perception area and transfer process of the human eye. Indicates the number of included fixation spectra.

[0062] Finally, in order to optimize the prediction quality of the gaze spectrum and the transfer process, the present invention adopts the spectral distribution metric loss function Linear correlation measure loss function And spectral temporal metric loss function The gaze spectrum prediction model is trained together so that the network generates a visual gaze spectrum that is more consistent with human perception and understanding, namely:

[0063]

[0064] In the formula, represents the KL divergence loss function, which is used to narrow the significant area distribution between the real gaze spectrum and the predicted spectrum at the current time t. By calculating A t and Y t The ratio of the covariance of and the product of the standard deviation of measures the linear correlation between the predicted and true gaze spectra. It is used to sort the predicted gaze spectra at all times to accurately describe the transfer and change process of the human eye gaze spectrum.

[0065] The visual features and language features extracted in steps 3 and 4 are input into the cross-modal human eye attention spectrum prediction module to capture the visual perception areas related to language at different times and obtain the human eye attention spectrum. Where G represents a 3×3 grid area, and the initial learning rate is set to 10e-4.

[0066] Step 6: Based on the predicted language-guided eye movement spectrum, the visual features are weighted and aggregated to obtain cross-modal semantic features. Based on the multi-head attention mechanism, a learnable feature vector is used as the target query, and the correlation is calculated with the cross-modal features at different times in turn to capture discriminative visual features to simulate the human vision and reasoning process for the target. For the cross-modal features at each moment, layer normalization LN and a forward propagation network FFN consisting of two fully connected layers are used to update the features. At the same time, based on the updated target query features, the multi-layer perceptron MLP is used to predict the bounding box coordinates of the language-indicated target. That is:

[0067] Q t+1 =LN(Q t +FFN(Q t ))

[0068] B t (x,y,w,h)=MLP(Q t )

[0069] In the formula, B t (x, y, w, h) represents the predicted target bounding box coordinates at time t, and x, y, w, h represent the center point coordinates, width, and height of the bounding box, respectively. t represents the target query vector at time t.

[0070] The language-guided eye movement spectrum and visual features predicted in step 5 are input into the target decoder driven by the eye movement spectrum to predict the bounding box coordinates of the language-indicated target at different times. The number of decoding layers is set to 6, and the initial learning rate is set to 10e-4.

[0071] Step 7: Send the bounding box coordinates predicted in step 6 and the true bounding box coordinates to the instruction expression understanding network optimization part to calculate the loss function of the entire network for network optimization. Therefore, the loss function of the entire network is:

[0072]

[0073] In the formula, It is the SmoothL1 loss function, which constrains the four coordinates of the predicted bounding box to be close to the true box value. The purpose is to comprehensively compare the intersection-over-union ratio between the predicted bounding box and the true bounding box. λ1 and λ2 are loss weight coefficients to control the balance between the two loss functions. Among them, λ1=5, λ2=2, and the AdamW optimizer is used for model training optimization. The training cycle is 90 rounds. After 60 rounds, the initial learning rate is reduced by 10 times.

[0074] The protection point of the present invention is mainly the algorithm model structure of complex scene indication expression understanding of cross-modal eye movement attention perception. A cross-modal human eye gaze spectrum database is constructed, and the attention perception and transfer change characteristics of the human eye under different language stimuli are collected. At the same time, a dynamic deformable attention mechanism for language perception is designed, which can adaptively capture the corresponding visual area according to the language characteristics and human eye gaze supervision information. At the same time, the Transformer decoder driven by the eye movement spectrum is used to gradually fuse the visual feature representation, and then explicitly infer the target area position of the language indication, thereby improving the accuracy of complex scene indication expression understanding. The present invention has demonstrated advanced performance in RefCOCO, RefCOCO+ and RefCOCOg.

Claims

1. A method for understanding complex scene instructions based on cross-modal eye movement attention perception, the method comprising: Step 1: Build a human eye gaze spectrum database; Use a desktop eye tracker to collect eye movement records of the subjects; Record the subject's gaze position and dwell time changes on the image content; in order to ensure the quality of gaze data, each language-image pair will collect eye movement gaze information from multiple subjects; Then the collected data are clustered and fused to reduce the impact of outliers and ensure the reliability of gaze information; Step 2: Extract the language features of the deictic language; First, the input instruction language is divided into word sequences, and then each word is encoded into a high-dimensional vector space using the Token Embedding layer. Finally, the encoded language sequence is input into the Bert model to extract the hidden state of the input language, that is, to obtain the language feature L of the input language. Step 3: Extract the visual feature spectrum of the input image; Firstly, a deep convolutional neural network ResNet is used to extract image features of different scales of the input image. Then, all scale feature spectra are aggregated, and stacked Transformer encoding layers are used to interact and associate features of different visual regions to generate a visual feature spectrum V. Step 4: Predict cross-modal human eye attention spectrum; First, a linear mapping layer and a nonlinear activation function are used to generate a position filter and a weight filter corresponding to the current attention perception area by aggregating the current language features and the features after the average pooling of the gaze spectrum at the previous moment. Then, the visual features are convolved with positional filters to generate language-dependent visual perception area coordinate shifts Δp t ; Use the weight filter to obtain the weight value w of each coordinate position in the visual perception area t ; Therefore, for each spatial position p0 in the image, according to the coordinate offset and weight value, the perceptual region feature V related to it is dynamically weighted and aggregated to obtain the language-related current moment visual attention perceptual region feature representation V t (p0): In the formula, p0, p j ∈G represent the coordinate offset of any spatial position p0 in the image and its initial perception area, represents the attention perception coordinate offset at the jth position at the current time t, Δp t is the set of all coordinate position offsets in a fixed area; by adjusting Changing the visual perception area window at the previous moment can adaptively simulate the shift changes of the human eye's gaze spectrum; A t (p0) represents the predicted human eye attention spectrum at time t under the stimulation of the indicative language description: In the formula, represents the learnable weight matrix, A t (p0) represents the importance score of the p0th image region at the current time t. The higher the score, the more attention the human eye pays to this position; otherwise, it indicates that the attention is low; H and W represent the height and width of the current attention spectrum respectively; finally, all eye gaze spectra related to the current language are predicted. To explicitly simulate the visual attention perception area and transfer process of the human eye; represents the number of included fixation spectra; Finally, in order to optimize the prediction quality of the gaze spectrum and the transfer process, the spectral distribution metric loss function is used Linear correlation measure loss function And spectral temporal metric loss function The gaze spectrum prediction model is trained together so that the network generates a visual gaze spectrum that is more consistent with human perception and understanding, namely: In the formula, represents the loss function of the overall eye movement spectrum, represents the number of recorded eye movement spectra moments, represents the KL divergence loss function, which is used to narrow the significant area distribution between the real gaze spectrum and the predicted spectrum at the current time t. To measure the linear correlation between the predicted gaze spectrum and the true spectrum; It is used to sort the predicted gaze spectra at all times to accurately describe the shift and change process of the human eye gaze spectra; Step 5: Based on the predicted language-guided human eye attention spectrum, weighted aggregation is performed on the visual features to obtain cross-modal semantic features; Based on the multi-head attention mechanism MATT(), a learnable feature vector is used as the target query, and the correlation is calculated with the cross-modal features at different times in turn to capture discriminative visual features to simulate the human vision and reasoning process for the target; for the cross-modal features at each moment, layer normalization LN() and a forward propagation network FFN() consisting of two fully connected layers are used to update the features; at the same time, based on the updated target query features, a multi-layer perceptron MLP() is used to predict the bounding box coordinates of the language-indicated target; that is: Q t+1 =LN(Q t +FFN(Q t )) B t (x,y,w,h)=MLP(Q t ) In the formula, B t (x, y, w, h) represents the coordinates of the target bounding box predicted at time t, x, y, w, h represent the center point coordinates, width and height of the bounding box respectively; Q t represents the target query vector at time t; Step 6: Instruct the expression understanding network to optimize; In the network training optimization stage, the positive and negative sample allocation process is removed, and the predicted target box B is directly calculated. t With the real annotation box The regression loss between , so the loss function of the entire network is: In the formula, It is the SmoothL1 loss function, which constrains the four coordinates of the predicted bounding box to be close to the true box value. It aims to comprehensively compare the intersection-over-union ratio between the predicted bounding box and the true bounding box; λ1 and λ2 are loss weight coefficients to control the balance between the two loss functions.

2. A complex scene instruction expression understanding method based on cross-modal eye movement attention perception as claimed in claim 1, characterized in that: The specific method of step 1 is: Firstly, according to the discrete fixation point map calculation method, the collected fixation data of all subjects are statistically analyzed to obtain the image fixation data corresponding to each input language; secondly, the Gaussian blur method is used to filter the Gaussian filter G σ Perform convolution operation on all gaze points to obtain the final gaze thermodynamic spectrum f(x); S(x)=f(x)*G σ (x) Where x is the image two-dimensional space coordinate vector, x i (k) represents the coordinates of the i-th subject at the k-th gaze point, M a and N a denote the number of fixations and the number of subjects of the ith subject in the current language-image pair; δ(·) denotes the Kronecker function; when xx i When (k) = 0, then δ(·) = 1, otherwise δ(·) = 0; σ is the standard deviation; Since the collected gaze data is a continuous data, the gaze data with a continuous gaze time less than the set threshold is removed according to the time sequence of the gaze data. Finally, multiple gaze spectra are obtained to indicate the process of human attention transfer to language description.

3. A complex scene instruction expression understanding method based on cross-modal eye movement attention perception as claimed in claim 1, characterized in that: In step 4, the fully connected layer is used to generate a position filter and a weight filter corresponding to the attention perception area at the current moment.

4. The method for understanding complex scene indication expression based on cross-modal eye movement attention perception as claimed in claim 1, characterized in that: Step 4 5. The method for understanding complex scene indication expression based on cross-modal eye movement attention perception as claimed in claim 1, characterized in that: Step 4 By calculating A t and Y t The covariance Cov(A t ,Y t ) and standard deviation Var(A t )Var(Y t ), measuring the linear correlation between the predicted gaze spectrum and the true spectrum.

6. A complex scene instruction expression understanding method based on cross-modal eye movement attention perception as claimed in claim 1, characterized in that: Step 4

Images