A complex scene instruction expression understanding method based on cross-modal eye movement attention perception

Abstract

The application discloses a complex scene instruction expression understanding method based on cross-modal eye movement attention perception, and belongs to the fields of computer vision, machine learning and multi-modal understanding. The application simulates the eye visual attention perception area and the transfer process by designing a dynamic deformable attention mechanism of language perception, using an eye gaze spectrum as supervision information, adaptively capturing the corresponding visual area according to language features, and designing an eye movement spectrum driven Transformer decoder to infer the target area position of the language instruction by gradually fusing the visual feature representation, thereby effectively improving the complex scene instruction expression understanding precision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, machine learning and multimodal understanding, and proposes a complex scene instruction expression understanding algorithm based on cross-modal eye movement attention perception. BACKGROUND

[0002] With the rapid development of information technology, images and videos are showing a trend of massive growth, and people's demand for computer vision technology is increasing. In addition to requiring computers to recognize and locate fixed targets in images, they also require computers to flexibly locate user-interested target objects in images according to user input language, i.e., instruction expression understanding tasks. The present application mainly focuses on instruction expression understanding tasks to assist in establishing a bridge between human language and real-world visual content, and has wide application prospects in human-computer interaction, intelligent navigation, target search, etc.

[0003] However, due to the complexity and diversity of scenes and the flexibility of language, computers face great challenges in understanding and interpreting these instruction expressions. Traditional instruction expression understanding models are generally divided into two-stage and single-stage methods. Among them, the two-stage method usually models the instruction expression understanding task as an instruction-expression pair retrieval problem, first extracts candidate regions using a pre-trained deep convolutional network-based target detector, and then selects the most matching frame from these candidate regions as the target. In contrast, the single-stage method eliminates the candidate region generation stage in the two-stage method, effectively avoiding the dependence on the pre-trained detector. This method directly embeds natural language features into the target detector, and predicts the target area through simple cascading or stacking convolution layers and other operations. Although these methods have made significant progress, they usually need to rely on hand-designed networks to associate language and visual features to locate the target area. These models are often limited by convolution kernel size, convolution layer number and other hyperparameters, and only stay in the "black box" perception and fusion of language and vision. There are obvious limitations in processing these complex scenes, making it difficult to fully understand the semantic content of complex scenes and accurately perceive the key features of different image content.

[0004] To solve the above problems, the project proposes a complex scene indication expression understanding algorithm based on eye movement attention perception. First, based on the public target detection data set, an eye fixation spectrum database is constructed to collect the attention perception and transfer change characteristics of the human eye under different language stimuli. Unlike traditional fixed feature fusion, the application proposes a complex scene indication expression understanding algorithm based on cross-modal eye movement attention perception based on the human eye fixation spectrum. By designing a dynamic deformable attention mechanism for language perception, the human eye fixation spectrum is used as supervision information to adaptively capture the corresponding visual area according to the language features, and an eye movement spectrum driven Transformer decoder is designed to gradually fuse the visual feature representation and infer the target area position indicated by the language, thereby effectively simulating the human eye visual attention perception area and the transfer process, and effectively improving the complex scene indication expression understanding precision. SUMMARY

[0005] The application proposes a complex scene indication expression understanding method based on cross-modal eye movement attention perception to solve the problem that the current indication expression understanding algorithm only stays in the "black box" perception and fusion of language and vision, making it difficult to effectively perceive the visual area corresponding to different languages and infer the target area when processing these complex scenes.

[0006] To solve the above problems, the application adopts the following specific technical solutions: a complex scene indication expression understanding method based on cross-modal eye movement attention perception, the method comprising:

[0007] Step 1: Construct an eye fixation spectrum database;

[0008] Use a desktop eye tracker to collect eye movement records of the subjects; record the fixation position and dwell time change information of the subjects on the image content; in order to ensure the quality of the fixation data, eye fixation information of multiple subjects will be collected for each language-image pair;

[0009] Then cluster and fuse the collected data to reduce the influence of outliers and ensure the reliability of the fixation information;

[0010] Step 2: Extract language features of the indication language;

[0011] First, the input indication language is segmented into a word sequence, then Token Embedding layer is used to encode each word into a high-dimensional vector space, and finally the encoded language sequence is input into the Bert model to extract the hidden state of the input language, i.e. the language features L of the input language are obtained;

[0012] Step 3: Extract the visual feature spectrum of the input image;

[0013] First, the deep convolutional neural network ResNet is used to extract image features of different scales of the input image, and then all scale feature spectra are aggregated, and stacked Transformer encoding layers are used to interact and correlate between different visual region features to generate visual feature spectrum V;

[0014] Step 4: Predicting cross-modal human eye attention spectrum;

[0015] First, a linear mapping layer and a nonlinear activation function are used to generate a position filter and a weight filter corresponding to the current attention perception region by aggregating the current language feature and the feature after average pooling of the last time gaze spectrum; Then, the position filter is used to convolve the visual feature to generate the language-related visual perception region coordinate offset Δp t ; The weight filter is used to obtain the weight value w t of each coordinate position in the visual perception region; Thus, for each spatial position p0 in the image, according to the coordinate offset and the weight value, the perception region feature V related to it is dynamically weighted and aggregated to obtain the language-related visual attention perception region feature representation V t (p0) at the current time:

[0016]

[0017]

[0018] In the formula, p0, p j ∈G respectively represent any spatial position p0 in the image and the coordinate offset of its initial perception region, Δp t is the set of all coordinate position offsets of the fixed region; By adjusting the visual perception region window at the last time, the transfer change of the human eye gaze spectrum can be adaptively simulated; A t (p0) represents the predicted human eye attention spectrum at time t under the excitation of the language description:

[0019]

[0020] In the formula, A t (p0) represents the importance score of the p0th image region at the current time t; The higher the score, the higher the attention of the human eye to the position; On the contrary, it indicates that the attention is low; H and W represent the height and width of the current attention spectrum, respectively; Finally, all the language-related human eye gaze spectra at the current time are predicted to explicitly simulate the visual attention perception region and transfer process of the human eye; Where τ represents the number of gaze spectra contained;

[0021] Finally, to optimize the prediction quality of gaze spectrum and the transfer process, the spectrum distribution metric loss function is adopted Linear correlation metric loss function And spectrum timing metric loss function The gaze spectrum prediction model is trained together, so that the network generates visual gaze spectrum that is more consistent with human perception and understanding process, that is:

[0022]

[0023] In the formula, L represents the loss function of the overall eye movement spectrum, N represents the number of recorded eye movement spectrum moments, KL divergence loss function is used to narrow the distribution of significant areas between the real gaze spectrum at the current moment t and the predicted spectrum, To measure the linear correlation between the predicted gaze spectrum and the real spectrum; For all moments of the predicted gaze spectrum, the transfer change process of the human eye gaze spectrum is accurately described;

[0024] Step 5: Based on the predicted language-guided human eye attention spectrum, the visual features are weighted and aggregated respectively to obtain cross-modal semantic features;

[0025] Based on the multi-head attention mechanism MATT(), a learnable feature vector is used as the target query, which is sequentially calculated with the cross-modal features at different moments to capture discriminative visual features, to simulate the human visual and reasoning process for the target; For the cross-modal features at each moment, layer normalization LN() and a forward propagation network FFN() composed of two fully connected layers are used for feature updating; At the same time, based on the updated target query feature, the boundary box coordinates of the language indicating target are predicted by using the multilayer perceptron MLP(); that is:

[0026] Q t+1 = LN(Q t + FFN(Q t ))

[0027] B t (x,y,w,h) = MLP(Q t )

[0028] In the formula, B t (x,y,w,h) represents the predicted target bounding box coordinates at the t moment, x,y,w,h represent the center point coordinates, width and height of the bounding box respectively; Q t represents the target query vector at the t moment;

[0029] Step 6: Indicating expression understanding network optimization

[0030] In the network training optimization phase, the positive and negative sample allocation process is removed, and the regression loss between the predicted target frame B t and the true label frame is directly calculated, so the loss function of the entire network is:

[0031]

[0032] In the formula, SmoothL1 loss function is used to constrain the four coordinates of the predicted bounding box to approach the true frame value. The purpose is to comprehensively compare the intersection over union between the predicted bounding box and the true bounding box; λ1 and λ2 are loss weight coefficients to control the balance between the two loss functions.

[0033] Further, the specific method of step 1 is:

[0034] First, according to the discrete fixation point graph calculation method, the collected fixation data of all subjects is statistically analyzed to obtain image fixation data corresponding to each input language; secondly, the Gaussian blur method is used, and the Gaussian filter G σ is used to perform convolution operation on all fixation points, so as to obtain the final human eye fixation heat spectrum f(x);

[0035]

[0036] S(x)=f(x)*G σ (x)

[0037] In the formula, x is a two-dimensional space coordinate vector of the image, x i (k) represents the coordinates of the i-th subject at the k-th fixation point, M a and N a respectively represent the number of fixation points of the i-th subject and the number of subjects in the current language-image pair; δ(·) represents the Kronecker function; when x-x i (k)=0, then δ(·)=1, otherwise δ(·)=0; σ is the standard deviation;

[0038] Since the collected fixation data is a continuous data, according to the time sequence of the fixation data, the fixation data with continuous fixation time less than the set threshold is removed, and finally a plurality of fixation spectra are obtained to indicate the attention shift process of human to language description.

[0039] Further, in step 4, a full connection layer is used to generate a position filter and a weight filter corresponding to the attention perception area at the current time.

[0040] ​Further, in step 4

[0041] Further, in step 4 By calculating A t and Y t Covariance Cov(A t ,Y t ) and standard deviation Var(A t )Var(Y t ), which measures the linear correlation between the predicted gaze spectrum and the true spectrum.

[0042] Further, in step 4

[0043] The proposed algorithm for understanding complex scene instructions based on cross-modal eye movement attention perception is based on a constructed database of human eye gaze spectra, capturing the changing characteristics of human eye attention perception and attention shifts under different language stimuli. Unlike traditional fixed feature fusion, this invention employs a dynamic, deformable attention mechanism for language perception, utilizing the eye gaze spectra as supervisory information. It adaptively captures the corresponding visual areas based on language features, and simultaneously designs an eye movement spectrum-driven Transformer decoder. By gradually fusing visual feature representations, it infers the target area location of language instructions, thereby explicitly simulating the human eye's visual attention perception area and shift process, effectively improving the accuracy of understanding complex scene instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the overall process framework diagram of the complex scene indication expression understanding algorithm based on cross-modal eye movement attention perception of the present invention.

[0045] Figure 2 It is the subjective effect diagram of the present invention. DETAILED DESCRIPTION

[0046] The present invention conducts experiments on a Pytorch-based deep learning platform, which specifically includes the following steps:

[0047] Step 1: Select a public database for sign language understanding, such as the RefCOCO, RefCOCO+, and RefCOCOg databases, consisting of images, sign language, and target region pairs. The RefCOCO and RefCOCO+ databases consist of a training set (train), a validation set (val), a test set (testA), and a test set (testB). TestA primarily covers the target class "person," while testB covers other categories. The sign language of the RefCOCOg database is relatively long and consists of a training set (train), a validation set (val), and a test set (test).

[0048] Step 2: Based on the published natural scene dataset, 10 experimenters with normal or corrected-to-normal vision and normal color vision are recruited to conduct eye movement experiments using a desktop eye tracker. At the same time, each pair of language-image contains eye movement fixation information of multiple subjects, and the influence of abnormal values is reduced and the reliability of the fixation information is ensured by further clustering and fusion of these data. Finally, the fixation information is analyzed and processed to obtain the human eye fixation spectrum and attention transfer process information.

[0049] During the annotation process, the computer will randomly display images and corresponding instruction category word descriptions. The subject needs to wear an eye tracker and find the target area corresponding to the word according to the instruction word description. During the observation process, the eye tracker can record the fixation position and dwell time changes of the subject on the image content by tracking the eye movement information. In order to ensure the quality of the fixation data, the eye movement fixation information of multiple subjects will be collected for each pair of language-image, and the influence of abnormal values is reduced and the reliability of the fixation information is ensured by further clustering and fusion of these data.

[0050] In order to facilitate the acquisition of the degree of attention perception of the human eye to different visual areas, the present application first calculates the discrete fixation point map according to the discrete fixation point map calculation method, and then obtains the image fixation data corresponding to each input language by statistical analysis of all the collected fixation data of the subjects. Secondly, by using the Gaussian blur method, the Gaussian filter G σ Convolution operation is performed on all fixation points to obtain the final human eye fixation heat spectrum, that is:

[0051]

[0052] S(x) = f(x) * G σ (x)

[0053] In the formula, x is a two-dimensional space coordinate vector of the image, x i (k) represents the coordinates of the i-th subject at the k-th fixation point, M a and N a respectively represent the number of fixation points of the i-th subject and the number of subjects in the current language-image pair. δ(·) represents the Kronecker function. When x-x i (k) = 0, then δ(·) = 1; otherwise δ(·) = 0. σ is the standard deviation, which is generally related to the size of the computer screen and the viewing distance, to reflect the area with higher attention around the subject.

[0054] Since the collected gaze data is a continuous data, according to the time sequence of the gaze data, the gaze data with a continuous gaze time less than a set threshold in a certain area is removed, so as to ensure the reliability of the finally obtained gaze points and avoid the interference of abnormal data. Finally, a plurality of gaze spectra are obtained to indicate the attention shift process of human beings to the language description.

[0055] Step 3: inputting the images in the database into an image feature encoder to extract coded visual features Wherein the image feature extractor adopts a ResNet-50 or ResNet-101 model, the input image size is 640*640, C=512 represents the visual feature channel dimension, and the initial learning rate is set to 10e-5.

[0056] Step 4: inputting the indicative language description in the database into a language feature encoder to extract corresponding indicative language features Wherein the language feature encoder adopts a Bert model, C=768 represents the language feature channel dimension, the maximum number of words is set to 40, and the initial learning rate is set to 10e-5.

[0057] Step 5: based on the extracted language and image features, in order to simulate the adaptive perception of human eyes to the language-related visual area and the shift change, the application designs a dynamic deformable attention mechanism. First, a linear mapping layer and a nonlinear activation function are used to generate a position filter and a weight filter corresponding to the attention perception area at the current moment by aggregating the current language features and the features after average pooling of the gaze spectrum at the last moment. Then, the position filter is used to convolve the visual features to generate the language-related visual perception area coordinate offset Δp t ; the weight filter is used to obtain the weight value w t of each coordinate position in the visual perception area. Thus, for each spatial position p0 in the image, the language-related visual attention perception area feature representation V t at the current moment can be obtained by dynamically weighting and aggregating the perception area features V related to it according to the coordinate offset and the weight value.

[0058]

[0059] In the formula, p0, p j ∈G respectively represent the coordinate offset of any spatial position p0 in the image and its initial perception area, represent the attention perception coordinate offset at the current moment t. By adjusting , the visual perception area window at the last moment can be changed, so that the shift change of the human eye gaze spectrum can be adaptively simulated. Here, A t represents the predicted human eye attention spectrum at time t under the excitation of the indicative language description:

[0060]

[0061] wherein, denotes the learnable weight matrix. A t (p0) denotes the importance score of the p0th image region at the current time t, the higher the score, the higher the attention of the human eye to the position. Conversely, it indicates that the attention is low. Finally, all the human eye gaze spectra related to the current language can be predicted explicitly simulate the visual attention perception area and transfer process of the human eye. Among them, denotes the number of gaze spectra contained.

[0062] Finally, in order to optimize the prediction quality of the gaze spectrum and the transfer process, the present application adopts a spectrum distribution measurement loss function linear correlation measurement loss function and spectrum timing measurement loss function train the gaze spectrum prediction model together, so that the network generates visual gaze spectra that are more consistent with human perception and understanding processes, that is:

[0063]

[0064] wherein, denotes the KL divergence loss function, which is used to narrow the significant area distribution between the real gaze spectrum at the current time t and the predicted spectrum. by calculating the ratio of the covariance of A t and Y t and the product of the standard deviation, the linear correlation between the predicted gaze spectrum and the real spectrum is measured. is used to sort the predicted gaze spectrum at all times to accurately depict the transfer change process of the human eye gaze spectrum.

[0065] The visual features and language features extracted in steps 3 and 4 are input into the cross-modal human eye attention spectrum prediction module to capture visual perception areas related to language at different times to obtain the human eye attention spectrum. Among them, G represents a 3x3 grid area, and the initial learning rate is set to 10e-4.

[0066] Step 6: The visual features are weighted and aggregated respectively based on the predicted language-guided eye movement fixation spectrum, resulting in cross-modal semantic features. Based on the multi-head attention mechanism, a learnable feature vector is used as the target query, which is sequentially associated with the cross-modal features at different times to capture discriminative visual features, simulating the human visual and reasoning process for the target. For the cross-modal features at each time, layer normalization LN and a forward propagation network FFN composed of two fully connected layers are used for feature updating. At the same time, based on the updated target query feature, a multilayer perceptron MLP is used to predict the bounding box coordinates of the language indicating target. That is:

[0067] Q t+1 = LN(Q t + FFN(Q t ))

[0068] B t (x, y, w, h) = MLP(Q t )

[0069] where B t (x, y, w, h) represents the predicted target bounding box coordinates at the t-th time, x, y, w, and h represent the center point coordinates, width, and height of the bounding box, respectively. Q t represents the target query vector at the t-th time.

[0070] The language-guided eye movement fixation spectrum predicted in step 5 and the visual features are input into the eye movement spectrum-driven target decoder to predict the bounding box coordinates of the language indicating target at different times. The number of decoding layers is set to 6, and the initial learning rate is set to 10e-4.

[0071] Step 7: The bounding box coordinates predicted in step 6 and the real bounding box coordinates are input into the indication expression understanding network optimization part to calculate the loss function of the entire network for network optimization. Therefore, the loss function of the entire network is:

[0072]

[0073] where is the SmoothL1 loss function, which respectively constrains the four coordinates of the predicted bounding box to approach the real box value. aims to comprehensively compare the intersection over union between the predicted bounding box and the real bounding box. λ1 and λ2 are loss weight coefficients to control the balance between the two loss functions. Where λ1 = 5, λ2 = 2, the AdamW optimizer is used for model training and optimization, and the training period is 90 rounds. After 60 rounds, the initial learning rate is reduced by 10 times.

[0074] The protection point of the application mainly lies in a complex scene indication expression understanding algorithm model structure of cross-modal eye movement attention perception, a cross-modal human eye gaze spectrum database is constructed, and the attention perception and transfer change characteristics of human eyes under different language excitations are collected. Meanwhile, a dynamic deformable attention mechanism of language perception is designed, which can adaptively capture the corresponding visual area according to the language characteristics and human eye gaze supervision information, and meanwhile, a gaze spectrum driven Transformer decoder is used to gradually fuse the visual feature representation, and then the target region position indicated by the language is inferred, so that the complex scene indication expression understanding precision is improved. The application shows advanced performance in RefCOCO, RefCOCO+ and RefCOCOg.

Claims

1. A complex scene instruction expression understanding method based on cross-modal eye movement attention perception, the method comprising: Step 1: constructing an eye gaze spectrum database; Collecting eye movement records of subjects using a desktop eye tracker; Recording the gaze position and dwell time variation information of the subjects on the image content; in order to ensure the quality of the gaze data, the eye gaze information of multiple subjects will be collected for each pair of language-image; Then, the collected data is clustered and fused to reduce the influence of outliers and ensure the reliability of the gaze information; Step 2: extracting language features of the instruction language; First, the input instruction language is segmented into a word sequence, then each word is encoded into a high-dimensional vector space using the Token Embedding layer, and finally the encoded language sequence is input into the Bert model to extract the hidden state of the input language, i.e. the language features L of the input language are obtained; Step 3: extracting visual feature spectrum of input image; First, the deep convolutional neural network ResNet is used to extract image features of different scales of the input image, then all scale feature spectra are aggregated, and a stacked Transformer encoding layer is used to interact and correlate between different visual region features to generate a visual feature spectrum V; Step 4: predicting cross-modal human eye attention spectrum; First, use a linear mapping layer and a nonlinear activation function to aggregate the current language features and the features after average pooling of the gaze spectrum at the previous time to generate a position filter and a weight filter corresponding to the attention perception region at the current time; Then, the visual features are convolved with the position filter to generate language-dependent visual perception region coordinate offsets Δp t ; and the weight values w of each coordinate position in the visual perception region are obtained with the weight filter t ; Thus, for each spatial position p0 in the image, the language-related current time visual attention perception region feature representation V is obtained by dynamically weighting and aggregating the perception region features V related thereto according to the coordinate offset and the weight value. t (p0): where p0, p j ∈G represent the coordinate offset of any spatial position p0 in the image and its initial perceptual region, respectively, represents the attention perceptual coordinate offset of the current time t at the jth position, Δp t is the set of coordinate offset of all fixed regions; by adjusting changing the visual perceptual region window at the last time, the transfer change of the human eye attention spectrum can be adaptively simulated; A t (p0) represents the predicted human eye attention spectrum at time t under the stimulation of the instruction language description: wherein denotes a learnable weight matrix, A t (p0) denotes the importance score of the p0th image region at the current time t, the higher the score, the higher the degree of attention of the human eye to the position; otherwise, it indicates a lower degree of attention. H and W represent the height and width of the current attention profile, respectively; all the current language-dependent human eye gaze profiles are predicted explicitly modeling the human eye's visual attention perception region and shifting process; wherein, represents the number of gaze profiles included Finally, to optimize the prediction quality of gaze spectrum and the transfer process, the spectrum distribution metric loss function is adopted Linear correlation metric loss function And spectrum timing metric loss function The gaze spectrum prediction model is trained together, so that the network generates visual gaze spectrum that is more consistent with human perception and understanding process, that is: wherein, a loss function representing the overall eye movement profile, the number of recorded eye movement profile time instants, a KL divergence loss function for pulling the significant region distribution between the real and predicted gaze profile at the current time instant t, is a measure of the linear correlation between the predicted and real gaze profile; for ranking the predicted gaze profiles for all time instants to accurately depict the transition change process of the human eye gaze profile; Step 5: based on the predicted language-guided human eye attention spectrum, the visual features are weighted and aggregated respectively to obtain cross-modal semantic features; Based on the multi-head attention mechanism MATT(), a learnable feature vector is used as the target query, which is sequentially calculated with cross-modal features at different times to capture discriminative visual features to simulate the human visual and reasoning process for the target; for the cross-modal features at each time, a layer normalization LN() and a forward propagation network FFN() composed of two fully connected layers are used for feature updating; at the same time, based on the updated target query feature, a multilayer perceptron MLP() is used to predict the bounding box coordinates of the language instruction target; that is: Q t+1 = LN(Q t + FFN(Q t )) B t (x,y,w,h) = MLP(Q t ) In the formula, B t (x, y, w, h) represents the target bounding box coordinates predicted at the t time, x, y, w, and h respectively represent the center point coordinates, width, and height of the bounding box; Q t represents the target query vector at the t time; Step 6: instruction expression understanding network optimization; In the network training optimization stage, the positive and negative sample assignment process is removed, and the predicted target box B is directly calculated t The regression loss between the real label box Therefore, the loss function of the whole network is: In the formula, SmoothL1 loss function, respectively, to constrain the four coordinates of the predicted bounding box to approach the true box value; Aim to compare the intersection over union between the predicted bounding box and the true bounding box; λ1 and λ2 are loss weight coefficients to control the balance between the two loss functions.

2. The method of claim 1, wherein the method is based on cross-modal eye movement attention perception for complex scene instruction representation understanding. The specific method of step 1 is: Firstly, according to the discrete fixation point figure calculation method, the collected fixation data of all subjects are statistically analyzed to obtain the image fixation data corresponding to each input language; secondly, the Gaussian blur method is adopted, and the Gaussian filter G σ Convolution operation is performed on all fixation points to obtain the final human eye fixation thermospectrum f(x); S(x) = f(x) * G σ (x) where x is the two-dimensional spatial coordinate vector of the image, x i (k) denotes the coordinate of the i-th subject at the k-th fixation point, M a and N a denote the number of fixation points and the number of subjects for the i-th subject in the current language-image pair, respectively; δ(·) denotes the Kronecker function; δ(·) = 1 when x - x i (k) = 0, otherwise δ(·) = 0; and σ is the standard deviation. Since the collected gaze data is a continuous data, according to the time sequence of the gaze data, the gaze data with continuous gaze time less than a set threshold is removed, and finally a plurality of gaze spectra are obtained to indicate the attention shift process of human on the language description.

3. The method of claim 1, wherein the method is based on cross-modal eye movement attention perception for complex scene instruction representation understanding. In step 4, a fully connected layer is used to generate a position filter and a weight filter corresponding to the attention perception region at the current time respectively.

4. The method of claim 1, wherein the method is based on cross-modal eye movement attention perception for complex scene instruction representation understanding. In step 4 5. The method of claim 1, wherein the method is based on cross-modal eye movement attention perception for complex scene instruction representation understanding. In step 4 By calculating A t and Y t ) with the standard deviation Var(A t , Y t ) of the product of the covariance Cov(A t , Y t ) between the predicted and true spectra.

6. The method of claim 1, wherein the method is based on cross-modal eye movement attention perception for complex scene instruction representation understanding. In step 4

Images