Bionic visual thinking chain cross-modal model based on eye movement tracking, visualization system and method

Abstract

The invention relates to a visual thinking chain, in particular to a bionic visual thinking chain cross-modal model based on eye movement tracking and a visualization system and method, and solves the technical problems that an existing visual thinking chain model lacks real visual thinking process simulation and is difficult to establish a visual thinking chain conforming to a real human visual thinking mode. According to the bionic visual thinking chain cross-modal model based on eye movement tracking, the to-be-detected image eye movement gazing area image blocks from eye movement tracking data and the gazing sequence and the gazing duration of the to-be-detected image eye movement gazing area image blocks are adopted, and the dynamic characteristics and the time sequence mode of human visual cognition can be truly reflected; the big language model is made to process visual information step by step according to the sequence of human eye movement tracks, a human real visual thinking mode is simulated, and therefore a real visual thinking chain is established; besides, an eye movement vector prediction matrix and an eye movement prediction text answer can be obtained, so that a bionic visual thinking chain and a text thinking chain are output at the same time, and the interpretability of the thinking process after the text and the image are fused is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to visual thinking chain, and in particular to a bionic visual thinking chain cross-modal model based on eye tracking, a visualization system and a method. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large language models have made remarkable achievements in visual understanding, question answering and reasoning tasks. Through the integration of text and image and other input modalities, it can perform complex cognitive tasks including logical reasoning, causal inference and analogy mapping.

[0003] Chain-of-Thought (CoT) is an important breakthrough in the field of large language models in recent years. It enhances the reasoning ability of the model by generating explicit intermediate reasoning steps. With the expansion of cross-modal technology into the image field, the concept of visual Chain-of-Thought (Visual CoT) emerged. It aims to demonstrate the reasoning path of large language models when processing visual information.

[0004] Multimodal-CoT (Multimodal Chain-of-Thought Reasoning in Language Models) proposed by Zhuosheng Zhang et al. in Transactions on Machine Learning Research, 2024, has become a foundational work in cross-modal thinking chain models. It incorporates visual information into the thinking chain model framework through a two-stage training strategy (reason generation → answer reasoning) and has made significant progress in benchmark tests such as ScienceQA (Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering, a dataset from NeurIPS 2022 (The 36th Conference on Neural Information Processing Systems)), proving the feasibility of cross-modal thinking chain models. Subsequently, a variety of methods have explored different technical paths, such as: The VoT model (Vision-of-Thought, from arXiv:2404.03622, 2024. Microsoft Research) simulates the cognitive process of human eyes and mind, generating mental images during reasoning, and tracking changes in visual state through a visual space board. However, the generated mental images are completely created by algorithms, lack objective human cognitive data support, and cannot verify whether they conform to real human visual thinking patterns.

[0005] The IPVR model (Interactive Prompting for Visual Reasoning, from See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning / arXiv:2301.05226, 2023) designs a three-stage structured framework: look-think-confirm. The look stage observes the input image as a whole and identifies key elements, the think stage conducts logical reasoning and relationship analysis based on the observation results, and the confirm stage verifies the reasoning results and makes the final decision. This progressive processing method enhances the structured degree of reasoning, but mainly relies on text prompts to guide the reasoning process, lacking a deep understanding mechanism for visual content.

[0006] The G-CoT model (Graph-guided Chain-of-Thought, from Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs / ACL 2024, arXiv:2404.07103) models the reasoning process as a graph structure, where nodes represent reasoning steps and edges represent logical relationships. This method integrates visual signals, historical signals, and language signals to generate decisions through graph information propagation. Although it has advantages in scenarios such as autonomous driving that require multi-time information fusion, the graph construction process is complex, the computational overhead is large, and it is difficult to handle large-scale dynamic change reasoning tasks.

[0007] Although the above models have made progress in cross-modal understanding and reasoning, there are still fundamental gaps compared to human visual cognitive processes: (1) Lack of real visual thinking process simulation: When humans understand complex visual information, they form a series of ordered visual thinking patterns from overall perception to local analysis, from main elements to secondary details, and so on. Such thinking process has clear time sequence and hierarchy. However, the above models cannot truly reflect the dynamic characteristics and time sequence of human visual cognition, and are essentially the product of computer algorithms rather than bionic simulation based on real human cognitive data.

[0008] (2) Lack of real visual reasoning ability: The above models all lack step-by-step reasoning analysis of image content and cannot perform deep reasoning through dynamic shift of visual attention like humans. More importantly, the above models do not actually establish a visual thinking chain that conforms to the real human visual thinking pattern. Their goal is mainly to improve the performance of models in cross-modal tasks by imitating a certain reasoning process, rather than truly constructing a reasoning mechanism based on visual cognition. Human visual thinking is not completely based on language text thinking, for example, human visual information acquisition process of face, dynamic motion, etc.

[0009] (3) Fundamental limitations of bounding box annotation: The widely used bounding box annotation method has multiple limitations. Traditional bounding boxes can only annotate entity objects with clear boundaries, and perform poorly on uncountable nouns and non-fixed shape objects, such as wind, flowing coffee, rising hot air, dispersing smoke, and flowing water. The above phenomena are all abstract information, but they are very important in daily visual understanding and cannot be accurately described by bounding boxes. At the same time, bounding boxes cannot accurately describe irregular shapes, gradual change areas, or scattered distribution of visual elements. In addition, bounding boxes can only identify positions and cannot express the degree of human cognitive investment in different regions, and the depth and importance of different thinking steps cannot be effectively expressed.

[0010] (4) Lack of visual thinking time sequence modeling: Human visual thinking process has a clear time sequence and logical hierarchy. When humans understand complex images, they will analyze the visual information according to specific cognitive strategies and form a thinking sequence with clear logic. However, the above models, although introducing a multi-step processing mechanism, fail to capture the real visual thinking time sequence. SUMMARY

[0011] The purpose of the present application is to solve the technical problems of the existing visual thinking chain model lacking real visual thinking process simulation and being difficult to establish a visual thinking chain that conforms to the real human visual thinking pattern, and to provide a bionic visual thinking chain cross-modal model based on eye tracking, a visualization system and a method.

[0012] To achieve the above purpose, the technical scheme adopted by the present application is: The application discloses an eye movement tracking-based bionic visual thinking chain cross-modal model, which is characterized by comprising an input module, a first image encoder, a second image encoder, a first multi-layer perceptron, a second multi-layer perceptron, a third multi-layer perceptron, a fourth multi-layer perceptron, a fifth multi-layer perceptron, a large language model, a text-image alignment module and an output module. The output ends of the input module are connected with the input ends of the first image encoder and the second image encoder and the first input end of the large language model; the input module is used for receiving a to-be-tested image, an image block of an eye movement fixation area of the to-be-tested image, a fixation sequence and a fixation time length of the image block, and a text question of the to-be-tested image, and transmitting the to-be-tested image, the image block of the eye movement fixation area of the to-be-tested image, the fixation sequence and the fixation time length of the image block and the text question of the to-be-tested image to the first image encoder, the second image encoder and the large language model respectively. The output end of the first image encoder is connected with the input ends of the first multi-layer perceptron and the second multi-layer perceptron, and is used for extracting features of the to-be-tested image; the output end of the first multi-layer perceptron is connected with the first input end of the text-image alignment module, and is used for mapping the features of the to-be-tested image to a feature space compatible with the text-image alignment module; the output end of the second multi-layer perceptron is connected with the second input end of the large language model, and is used for mapping the features of the to-be-tested image to a feature space compatible with the large language model. The output end of the second image encoder is connected with the input end of the third multi-layer perceptron, and is used for extracting features of the image block of the eye movement fixation area of the to-be-tested image, and fusing the features according to the fixation sequence and the fixation time length to obtain fused features of the image block of the eye movement fixation area of the to-be-tested image; the output end of the third multi-layer perceptron is connected with the third input end of the large language model, and is used for mapping the fused features of the image block of the eye movement fixation area of the to-be-tested image to a feature space compatible with the large language model. The first output end of the large language model is connected with the input end of the fourth multi-layer perceptron, the output end of the fourth multi-layer perceptron is connected with the second input end of the text-image alignment module, the output end of the text-image alignment module is connected with the input end of the fifth multi-layer perceptron, the output end of the fifth multi-layer perceptron is connected with the fourth input end of the large language model, and the second output end and the third output end of the large language model are connected with the first input end and the second input end of the output module respectively; the large language model is used for obtaining a predicted text answer of the to-be-tested image, an eye movement vector prediction matrix and an eye movement predicted text answer respectively, and the output module is used for outputting the eye movement vector prediction matrix and the eye movement predicted text answer of the to-be-tested image, so that a bionic visual thinking chain and a text thinking chain are obtained.

[0013] Further, the first image encoder and the second image encoder are both image feature extraction networks. The first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, the fourth multi-layer perceptron and the fifth multi-layer perceptron all adopt a residual connection and layer normalization structure.

[0014] Further, the first image encoder and the second image encoder employ CLIP or ViT. The large language model is GPT or LLaMA.

[0015] The application also provides a bionic visual thinking chain visualization system based on eye tracking, which is characterized by comprising an eye tracker for collecting eye tracking data of a to-be-tested image, a preprocessing module, the aforementioned bionic visual thinking chain cross-modal model based on eye tracking, and a visual thinking chain visualization module. The output end of the eye tracker is connected with the first input end of the preprocessing module, the second input end of the preprocessing module is used for receiving a to-be-tested image and a to-be-tested image text question, the output end is connected with the input end of an input module in the bionic visual thinking chain cross-modal model based on eye tracking, and the preprocessing module is used for acquiring a to-be-tested image eye fixation area image block and its fixation sequence and fixation duration as well as an eye vector matrix from the to-be-tested image according to the eye tracking data of the to-be-tested image, and transmitting the to-be-tested image eye fixation area image block, the fixation sequence and the fixation duration as well as the eye vector matrix to the input end of the input module in the bionic visual thinking chain cross-modal model based on eye tracking together with the to-be-tested image and the to-be-tested image text question. The output end of the output module in the bionic visual thinking chain cross-modal model based on eye tracking is connected with one input end of the visual thinking chain visualization module, the other input end of the visual thinking chain visualization module is used for receiving the to-be-tested image, and the visual thinking chain visualization module is used for visualizing and displaying the bionic visual thinking chain on the to-be-tested image.

[0016] Further, the sampling frequency of the eye tracker is greater than or equal to 60 Hz.

[0017] The application also provides a bionic visual thinking chain visualization method based on eye tracking, which is based on the aforementioned bionic visual thinking chain visualization system based on eye tracking and is characterized by comprising the following steps. Step 1, selecting a visual reasoning dataset comprising an original image, a text question and a text answer, then using an eye tracker to acquire eye tracking data of the original image in the visual reasoning dataset, and transmitting the eye tracking data and the original image to a preprocessing module for preprocessing to obtain eye fixation area image blocks corresponding to the original image and their fixation sequence and fixation duration as well as an eye vector matrix; labeling the eye vector matrix and the text answer on the corresponding original image; then inputting the original image and the corresponding text question, the eye fixation area image blocks and their fixation sequence and fixation duration as eye tracking data into a bionic visual thinking chain cross-modal model based on eye tracking. Step 2, the eye tracking based biomimetic visual thinking chain cross-modal model learns the features of an original image, corresponding eye fixation region image blocks and fixation order and fixation duration in the eye tracking data set, and reasons the text question according to the features to obtain an eye prediction text answer and an eye vector prediction matrix; Step 3, according to the eye prediction text answer, the eye vector prediction matrix, and the corresponding text answer and eye vector matrix, a loss function is calculated, and according to the loss function, the parameters of the first multilayer perceptron, the second multilayer perceptron, the third multilayer perceptron, the fourth multilayer perceptron, the fifth multilayer perceptron and the image-text alignment module in the eye tracking based biomimetic visual thinking chain cross-modal model are adjusted, and then the next original image in the eye tracking data set and the corresponding text question, eye fixation region image blocks and fixation order and fixation duration are used to train the eye tracking based biomimetic visual thinking chain cross-modal model, until the loss function converges, and the trained eye tracking based biomimetic visual thinking chain cross-modal model is obtained; Step 4, the eye tracking data of the to-be-tested image is collected by the eye tracker, and then it is transmitted to the pre-processing module together with the to-be-tested image; the pre-processing module acquires the eye fixation region image blocks of the to-be-tested image and their fixation order and fixation duration from the to-be-tested image according to the eye tracking data of the to-be-tested image, and transmits them to the input module of the trained eye tracking based biomimetic visual thinking chain cross-modal model together with the to-be-tested image and the text question of the to-be-tested image; Step 5, the trained eye tracking based biomimetic visual thinking chain cross-modal model learns the features of the to-be-tested image, the eye fixation region image blocks of the to-be-tested image and their fixation order and fixation duration, and reasons the text question of the to-be-tested image according to the features to obtain an eye prediction text answer and an eye vector prediction matrix of the to-be-tested image, and then transmits the eye vector prediction matrix of the to-be-tested image to the visual thinking chain visualization module; Step 6, the to-be-tested image is transmitted to the visual thinking chain visualization module, and the visual thinking chain visualization module draws fixation regions on the to-be-tested image according to the eye vector prediction matrix of the to-be-tested image, and labels the row number of the fixation region in the eye vector prediction matrix in the center of the fixation region to obtain a to-be-tested image containing a visual thinking chain, and complete the visualization of the biomimetic visual thinking chain.

[0018] Further, step 2 is specifically: Step 2.1, the input module transmits an original image in the eye tracking data set to the first image encoder, transmits the eye fixation region image blocks corresponding to the original image and their fixation order and fixation duration to the second image encoder, and transmits the text question corresponding to the original image to the large language model; Step 2.2, the first image encoder extracts the features of the original image, and transmits them to the first multi-layer perceptron and the second multi-layer perceptron respectively; the first multi-layer perceptron maps the features of the original image to a feature space compatible with the image-text alignment module, obtains original image alignment mapping features, and transmits them to the image-text alignment module; the second multi-layer perceptron maps the features of the original image to a feature space compatible with the large language model, obtains original image model mapping features, and transmits them to the large language model; The second image encoder extracts the features of the eye movement fixation region image block, and then performs feature fusion according to the fixation order and fixation duration of the eye movement fixation region image block to obtain eye movement fixation region image block fusion features and transmit them to the third multi-layer perceptron; the third multi-layer perceptron maps the eye movement fixation region image block fusion features to a feature space compatible with the large language model, obtains eye movement fixation region image block mapping features, and transmits them to the large language model; Step 2.3, the large language model reasons the text question according to the original image model mapping features, obtains a text answer, and transmits it to the fourth multi-layer perceptron; the fourth multi-layer perceptron maps the text answer to a feature space compatible with the image-text alignment module, obtains text answer mapping features, and transmits them to the image-text alignment module; Step 2.4, the image-text alignment module aligns and fuses the original image alignment mapping features and the text answer mapping features, obtains image-text alignment features, and transmits them to the fifth multi-layer perceptron; the fifth multi-layer perceptron maps the image-text alignment features to a feature space compatible with the large language model, obtains image-text alignment mapping features, and transmits them to the large language model; Step 2.5, the large language model reasons the text question according to the original image model mapping features and the eye movement fixation region image block mapping features obtained in step 2.2, and the image-text alignment mapping features obtained in step 2.4, obtains an eye movement prediction text answer and an eye movement vector prediction matrix, and transmits them to the output module for output.

[0019] Further, in step 3, the loss function is calculated by the following formula: wherein, is the loss function, is the visual question and answer loss, which is calculated according to the eye movement prediction text answer and the text answer; is the eye movement prediction loss, which is calculated according to the eye movement vector prediction matrix and the eye movement vector matrix; is the thought chain constraint loss, which is calculated according to the eye movement vector prediction matrix and the eye movement vector matrix; α, β, γ are the weights of the question and answer loss, the eye movement prediction loss, and the thought chain constraint loss, respectively.

[0020] Further, step 1 is specifically: Step 1.1, selecting a visual reasoning dataset including original images, text questions and text answers in a public dataset, then balancing selecting multiple original images from the visual reasoning dataset, and randomly and uniformly grouping; Step 1.2, setting an eye tracking calibration accuracy threshold and a gaze angular velocity threshold; Step 1.3, inviting test personnel, and respectively performing eye tracking calibration on the test personnel, if the eye tracking calibration accuracy is less than or equal to the eye tracking calibration accuracy threshold, then making the test personnel respectively read the text questions corresponding to one group of original images, and watch the group of original images, while recording eye tracking data using an eye tracker, then making the test personnel answer the text questions, obtaining eye tracking data and test personnel answers; otherwise, re-performing eye tracking calibration; Step 1.4, respectively checking whether the test personnel answers are correct according to the text answers, if yes, then retaining the corresponding eye tracking data and the corresponding original images, text questions and text answers, if not, then removing the corresponding eye tracking data and the corresponding original images, text questions and text answers, thereby obtaining an eye tracking initial dataset; Step 1.5, calculating the gaze angular velocity of all sampling points in each group of eye tracking data in the eye tracking initial dataset, if the gaze angular velocity of a sampling point is lower than the gaze angular velocity threshold, then determining the sampling point as a fixation point, otherwise determining the sampling point as a saccade, thereby obtaining fixation points of each group of eye tracking data in the eye tracking initial dataset respectively; Step 1.6, obtaining M fixation regions, fixation sequences and fixation durations according to the fixation points of each group of eye tracking data, and obtaining eye movement vector matrices of each group of eye tracking data according to the M fixation regions and fixation sequences, then calculating minimum circumscribed rectangular regions of the M fixation regions in each group of eye tracking data, wherein M is an integer and M≥1; Step 1.7, respectively cutting M eye fixation region image blocks from the corresponding original images according to the minimum circumscribed rectangular regions of the M fixation regions in each group of eye tracking data, and labeling the eye movement vector matrices and the text answers on the corresponding original images, then taking the original images, corresponding text questions, eye fixation region image blocks, fixation sequences and fixation durations as an eye tracking dataset, and inputting the eye tracking dataset into a biomimetic visual thinking chain cross-modal model based on eye tracking.

[0021] Further, step 1.6 is specifically: Step 1.6.1, judging whether adjacent fixation points in one group of eye tracking data in the eye tracking initial dataset satisfy: time interval < 200 ms, and angle of view < 5°, if yes, then merging the adjacent fixation points into one fixation point to obtain a merged fixation point; then obtaining M fixation regions and fixation sequences according to the merged fixation points; Step 1.6.2: Count the number of fixation points in each of the M fixation regions, and then calculate the fixation duration of each of the M fixation regions based on the sampling frequency of the eye tracker. Step 1.6.3: Assume that all M fixation regions are ellipses, and then solve for the elliptical equation parameters of the M fixation regions using the following formulas to obtain the elliptical equations of the M fixation regions: in, , Let x and y be the x and y coordinates of the i-th fixation point in the k-th fixation region, respectively, where k and i are integers, and 1 ≤ k ≤ M, 1 ≤ i ≤ N. k , Let A be the number of fixation points in the k-th fixation region. k B k C k D k E k F k All of these are the ellipse equation parameters for the k-th gaze region; Step 1.6.4: Calculate the geometric parameters of the center, semi-major axis, semi-minor axis, and rotation angle of the M gaze regions using the following formulas: in, , Let x and y be the geometric center coordinates of the k-th gaze region, respectively. , These are the major and minor semi-axis of the k-th fixation region, respectively. Let be the rotation angle of the k-th gaze region; Step 1.6.5: Based on the geometric parameters of the M fixation regions—center, semi-major axis, semi-minor axis, and rotation angle—obtain the eye movement vectors for each of the M fixation regions. Then, the eye movement vectors are arranged from top to bottom according to the fixation order of the fixation areas, resulting in an eye movement vector matrix of size M×5. The data in the k-th row corresponds to the k-th gaze region. Step 1.6.6: Calculate the minimum bounding rectangle region [x] for each of the M gaze regions using the following formula. mink ,y mink ,w k ,h k ]: wherein, , are respectively the horizontal coordinates of the left boundary and the right boundary of the minimum circumscribed rectangle region of the kth gaze region, , are respectively the vertical coordinates of the lower boundary and the upper boundary of the minimum circumscribed rectangle region of the kth gaze region, , are respectively the width and the height of the minimum circumscribed rectangle region of the kth gaze region; Step 1.6.7, repeat steps 1.6.1-1.6.6 until the minimum circumscribed rectangle region of the M gaze regions in each group of eye tracking data of the initial eye tracking data set is obtained.

[0022] Compared with the prior art, the application has the beneficial effects as follows: 1. The bionic visual thinking chain cross-modal model based on eye tracking provided by the application adopts image blocks of eye tracking regions of a to-be-tested image, and the fixation sequence and fixation duration thereof, which are derived from eye tracking data reflecting real human visual cognition, can truly reflect the dynamic characteristics and timing mode of human visual cognition, so that a large language model processes visual information in the order of human eye tracking trajectories, simulates the real visual thinking process of humans, and thus a visual thinking chain conforming to the real visual thinking mode of humans is established. 2. The bionic visual thinking chain cross-modal model based on eye tracking provided by the application can obtain an eye movement vector prediction matrix and an eye movement prediction text answer, so as to simultaneously output a bionic visual thinking chain and a text thinking chain, and improve the explainability of the thinking process after the fusion of text and image information. 3. The bionic visual thinking chain cross-modal model based on eye tracking provided by the application adopts residual connection and layer normalization structure for the multilayer perceptron, so as to improve the stability and expression ability of feature mapping. 4. The bionic visual thinking chain visualization system based on eye tracking provided by the application visualizes the visual thinking process on the original to-be-tested image through the visual thinking chain visualization module. 5. The bionic visual thinking chain visualization system based on eye tracking provided by the application has a sampling frequency of the eye tracker ≥60Hz, which can ensure that the detailed features of human visual cognition are captured. 6. The bionic visual thought chain visualization method based on eye tracking provided by this invention is based on real eye tracking data. It uses an eye tracking dataset containing image blocks of eye-tracking gaze regions and their gaze order and gaze duration to train a cross-modal model of the bionic visual thought chain. The eye tracking vector matrix is ​​integrated into the visual thought chain. Through real human eye tracking data, it guides the construction of the visual thinking process of the cross-modal model of the bionic visual thought chain based on eye tracking. 7. The bionic visual thinking chain visualization method based on eye tracking provided by this invention uses the eye tracking vector matrix to preserve the gaze order, which can reflect the human visual analysis strategy from the whole to the part and from the primary to the secondary. It also uses the gaze duration to reflect the degree of human cognitive investment in different gaze areas, enabling the cross-modal model of the bionic visual thinking chain based on eye tracking to learn the real path and thinking logic of human visual analysis, thereby improving the interpretability and accuracy of cross-modal reasoning. 8. The biomimetic visual thought chain visualization method based on eye tracking provided by this invention uses ellipses to describe the gaze area, which can handle uncountable nouns and non-fixed shape objects, accurately describe dynamic phenomena such as wind, flowing coffee, and rising hot air, solves the fundamental problem that traditional bounding boxes cannot accurately describe irregular shapes and dynamic phenomena, and significantly improves the ability to express visual information. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of an embodiment of the biomimetic visual thought chain cross-modal model based on eye tracking of the present invention; Figure 2 This is a schematic diagram of an original image from the GQA dataset selected in step 1.1 of the embodiment of the biomimetic visual thought chain visualization method based on eye tracking of the present invention; Figure 3 This is a flowchart of step 1.3 in an embodiment of the biomimetic visual thought chain visualization method based on eye tracking of the present invention; Figure 4 This is a schematic diagram of step 1.3, obtaining eye-tracking data, in an embodiment of the biomimetic visual thought chain visualization method based on eye tracking of the present invention. Figure 5 This is a schematic diagram of the gaze region obtained in step 1.6.1 of the embodiment of the bionic visual thought chain visualization method based on eye tracking of the present invention. In the figure, 1-6 are the serial numbers of the 6 gaze regions, indicating the gaze order. Figure 6 This is the eye-tracking gaze region image block obtained in step 1.7 of the embodiment of the biomimetic visual thought chain visualization method based on eye tracking of the present invention, wherein (a)-(f) respectively correspond to Figure 5 The fixation regions numbered 1-6 in the middle; Figure 7A schematic diagram of training the eye movement tracking based bionic visual thinking chain cross-modal model according to steps 2-3 in the embodiment of the eye movement tracking based bionic visual thinking chain visualization method of the present application. DETAILED DESCRIPTION

[0024] The eye movement tracking based bionic visual thinking chain cross-modal model, visualization system and method of the present application will be further described in detail below in combination with the drawings and specific embodiments. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application, and are not intended to limit the protection scope of the present application.

[0025] An eye movement tracking based bionic visual thinking chain cross-modal model, as shown in the figure, comprises an input module, a first image encoder, a second image encoder, a first multi-layer perceptron, a second multi-layer perceptron, a third multi-layer perceptron, a fourth multi-layer perceptron, a fifth multi-layer perceptron, a large language model, a text-image alignment module and an output module. Figure 1

[0026] The output end of the input module is connected to the input end of the first image encoder, the input end of the second image encoder and the first input end of the large language model. The input module is used to receive the to-be-tested image, the to-be-tested image eye movement fixation area image block and its fixation sequence and fixation duration, and the to-be-tested image text question, and transmit them to the first image encoder, the second image encoder and the large language model respectively.

[0027] The output end of the first image encoder is connected to the input end of the first multi-layer perceptron and the input end of the second multi-layer perceptron, for extracting the features of the to-be-tested image. The output end of the first multi-layer perceptron is connected to the first input end of the text-image alignment module, for mapping the features of the to-be-tested image to a feature space compatible with the text-image alignment module. The output end of the second multi-layer perceptron is connected to the second input end of the large language model, for mapping the features of the to-be-tested image to a feature space compatible with the large language model.

[0028] The output end of the second image encoder is connected to the input end of the third multi-layer perceptron, for extracting the features of the to-be-tested image eye movement fixation area image block and fusing the features according to the fixation sequence and fixation duration to obtain the to-be-tested image eye movement fixation area image block fused features. The output end of the third multi-layer perceptron is connected to the third input end of the large language model, for mapping the to-be-tested image eye movement fixation area image block fused features to a feature space compatible with the large language model.

[0029] ​The first output end of the large language model is connected with the input end of the fourth multilayer perceptron, the output end of the fourth multilayer perceptron is connected with the second input end of the image-text alignment module, the output end of the image-text alignment module is connected with the input end of the fifth multilayer perceptron, the output end of the fifth multilayer perceptron is connected with the fourth input end of the large language model, and the second output end and the third output end of the large language model are connected with the first input end and the second input end of the output module respectively. The large language model is used for obtaining the predicted text answer of the to-be-tested image, the eye movement vector prediction matrix and the eye movement prediction text answer respectively, and the output module is used for outputting the eye movement vector prediction matrix and the eye movement prediction text answer of the to-be-tested image, so that the biomimetic visual thinking chain and the text thinking chain are obtained.

[0030] The first image encoder and the second image encoder are both image feature extraction networks, which adopt CLIP (Contrastive Language-Image Pre-training, from Alec Radford, Jong Wook Kim, Chris Hallacy et al. Learning Transferable Visual Models From Natural Language Supervision. OpenAI, arXiv:2103.00020, 2021) or ViT (Vision Transformer, from Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. Vision Transformer (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale), Google Research, ICLR 2021, arXiv:2010.11929). The first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, the fourth multi-layer perceptron and the fifth multi-layer perceptron all adopt residual connection and layer normalization structure. The large language model is GPT (Generative Pre-trained Transformer, from Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever. Improving Language Understanding by Generative Pre-Training. OpenAI, 2018) or LLaMA (Large Language Model Meta AI, from Hugo Touvron, Thibaut Lavril, Gautier Izacard et al. LLaMA: Open and Efficient Foundation Language Models. Meta AI, arXiv:2302.13971, 2023).

[0031] Human cognition has the characteristics of visual multi-step reasoning. When processing image and text visual question answering tasks, large language models lack modeling of human visual reasoning processes. Eye tracking technology is based on high-speed optical imaging and can accurately capture eye movements and measure eye gaze points, i.e., the position and object of looking. Eye tracking technology can accurately record human visual behavior, providing an important window for understanding human visual cognition. The embodiment provides a bionic visual thinking chain cross-modal model based on eye tracking, which uses eye tracking data to construct a bionic visual thinking chain, can enhance the understanding and modeling capability of human visual cognitive processes, and can enable large language models to learn real human visual thinking patterns.

[0032] The embodiment also provides a bionic visual thinking chain visualization system based on eye tracking, which includes an eye tracker for collecting eye tracking data of a to-be-tested image, a preprocessing module, the above-mentioned bionic visual thinking chain cross-modal model based on eye tracking, and a visual thinking chain visualization module.

[0033] The output end of the eye tracker is connected to the first input end of the preprocessing module, the second input end of the preprocessing module is used to receive the to-be-tested image and a text question of the to-be-tested image, and the output end is connected to the input end of the input module in the bionic visual thinking chain cross-modal model based on eye tracking. The preprocessing module is used to acquire a to-be-tested image eye fixation region image block and its fixation sequence and fixation duration, as well as an eye vector matrix from the to-be-tested image according to the to-be-tested image eye tracking data, and transmit the to-be-tested image eye tracking data, the to-be-tested image, and the text question of the to-be-tested image to the input end of the input module in the bionic visual thinking chain cross-modal model based on eye tracking. The output end of the output module in the bionic visual thinking chain cross-modal model based on eye tracking is connected to one input end of the visual thinking chain visualization module, and the other input end of the visual thinking chain visualization module is used to receive the to-be-tested image. The visual thinking chain visualization module is used to visually display the bionic visual thinking chain on the to-be-tested image.

[0034] In the embodiment, the sampling frequency of the eye tracker is ≥60Hz, and Tobii (Tobii eye tracker, produced by Tobii AB Company in Sweden) or EyeLink (EyeLink eye tracker, produced by SR Research Ltd. Company in Canada) is used.

[0035] The embodiment also provides a bionic visual thinking chain visualization method based on eye tracking, which uses the above-mentioned bionic visual thinking chain visualization system based on eye tracking and includes the following steps: Step 1, select a visual reasoning dataset including original images, text questions and text answers, then use an eye tracker to obtain eye tracking data of the original images in the visual reasoning dataset respectively, and transmit them to the preprocessing module together with the original images for preprocessing, to obtain eye fixation area image blocks corresponding to the original images and their fixation sequences and fixation durations, and eye vector matrices respectively; label the eye vector matrices and text answers on the corresponding original images, then input the original images and corresponding text questions, eye fixation area image blocks and their fixation sequences and fixation durations as eye tracking datasets into the biomimetic visual thinking chain cross-modal model based on eye tracking. Specifically: Step 1.1, select a visual reasoning dataset including original images, text questions and text answers in the public dataset, then select multiple original images from the visual reasoning dataset in a balanced manner, and randomly and uniformly group them.

[0036] In this embodiment, the visual reasoning dataset is GQA dataset (GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering, GQA: New Dataset for Real-World Visual Reasoning and Compositional Question Answering, from Drew A. Hudson, Christopher D. Manning. 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Stanford University, arXiv:1902.09506). GQA dataset is a large-scale dataset for visual question answering task, which was proposed by Stanford University in 2019, aiming to promote the ability of artificial intelligence in visual reasoning and combined question answering. The GQA dataset contains 113,000 images and 22.66 million questions generated based on these images, covering diverse reasoning ability requirements, and the length of the questions and the required reasoning steps are different. The vocabulary of the dataset is 3097 words, containing 1878 possible answers, and each image in the dataset is associated with a scene graph that describes objects, attributes and relationships in detail; the question types cover object recognition, spatial reasoning, logical comparison and other complex tasks. As shown in Figure 2 The corresponding text question of the original image in the GQA dataset is how many types of chicks are there in the image.

[0037] Based on the GQA dataset, 10,000 original images are selected in a balanced manner in different image categories, which are randomly divided into 10 groups, and 10 test personnel are invited to watch the original images, each test personnel watches one group.

[0038] Step 1.2, set the eye tracking calibration accuracy threshold to 1° and the gaze angular velocity threshold to 20° / s.

[0039] Step 1.3, invite the test personnel, such as Figure 3 as shown, to perform eye tracking calibration respectively, if the eye tracking calibration accuracy is ≤1°, make the test personnel read the text questions corresponding to one set of original images, at the same time Figure 4 as shown, display the set of original images on the screen, make the test personnel watch, and record the eye tracking data using the eye tracking instrument, then make the test personnel answer the text questions, obtain the eye tracking data and the test personnel's answers; otherwise, re-perform eye tracking calibration.

[0040] In step 1.3, the text questions need to provide Chinese and English content at the same time; when watching the original images, the test personnel need to confirm that the watching is complete or the watching time reaches 30 seconds.

[0041] Step 1.4, check whether the test personnel's answers are correct according to the text answers respectively, if yes, keep the corresponding eye tracking data and the corresponding original images, text questions and text answers, if not, remove the corresponding eye tracking data and the corresponding original images, text questions and text answers, thereby obtaining the eye tracking initial data set.

[0042] Step 1.5, calculate the gaze angular velocity of all sampling points in each set of eye tracking data in the eye tracking initial data set, if the gaze angular velocity of the sampling point is lower than 20° / s, the sampling point is determined as a fixation point, otherwise it is determined as a saccade, thereby obtaining the fixation points of each set of eye tracking data in the eye tracking initial data set respectively. Wherein, the angular velocity of the sampling point is calculated by the following formula: Wherein, is the time interval between adjacent two sampling points, is the angle of line of sight between adjacent two sampling points.

[0043] Step 1.6, according to the fixation points of each set of eye tracking data in the eye tracking initial data set, obtain 6 fixation regions and their fixation order and fixation duration as shown in Figure 5 , and obtain the eye movement vector matrix of each set of eye tracking data according to the 6 fixation regions and their fixation order, then calculate the minimum circumscribed rectangular region of the 6 fixation regions in each set of eye tracking data. Specifically: Step 1.6.1, judge whether the adjacent fixation points in one set of eye tracking data in the initial data set of eye tracking satisfy: time interval < 200 ms, and the angle of view is < 5°, if yes, merge them into one fixation point, get the merged fixation point; then according to the merged fixation point, get the 6 fixation areas and their fixation order as shown in Figure 5

[0044] Step 1.6.2, respectively, the number of fixation points in the 6 fixation areas n, and then according to the sampling frequency f of the eye tracker, calculate the fixation time t of the 6 fixation areas, t = n x f.

[0045] Step 1.6.3, set the 6 fixation areas as ellipses, wherein the ellipse equation is: , and satisfy , A, B, C, D, E, F are parameters of the ellipse equation; then solve the ellipse equation parameters of the 6 fixation areas by the following formula respectively, get the ellipse equation of the 6 fixation areas: Wherein, , are the horizontal coordinate and vertical coordinate of the i-th fixation point in the k-th fixation area, k and i are integers, and 1 ≤ k ≤ 6, 1 ≤ i ≤ N k , is the number of fixation points in the k-th fixation area, A k , B k , C k , D k , E k , F k are the ellipse equation parameters of the k-th fixation area.

[0046] In step 1.6.3, the parameters of the fixation area ellipse equation can be solved by Lagrange multiplier method or directly solving the above formula. In other embodiments, the parameters of the fixation area ellipse equation can be optimized by regression method to improve the prediction accuracy.

[0047] Step 1.6.4, calculate the geometric parameters of the 6 fixation areas, center, major axis, minor axis and rotation angle respectively by the following formula: Wherein, , ​respectively the horizontal coordinate and the vertical coordinate of the geometric parameter center of the kth gaze region, 、 respectively the major semi-axis and the minor semi-axis of the kth gaze region, the rotation angle of the kth gaze region.

[0048] Step 1.6.5, obtain the eye movement vector of each of the 6 gaze regions according to the geometric parameter center, the major semi-axis, the minor semi-axis and the rotation angle of the 6 gaze regions, then arrange them from top to bottom according to the gaze region order to obtain an eye movement vector matrix of size 6x5 , wherein the kth row of data corresponds to the kth gaze region.

[0049] Step 1.6.6, calculate the minimum circumscribed rectangle region [x mink ,y mink ,w k ,h k ] of each of the 6 gaze regions by the following formula: wherein, 、 respectively the horizontal coordinate of the left boundary and the right boundary of the minimum circumscribed rectangle region of the kth gaze region, 、 respectively the vertical coordinate of the lower boundary and the upper boundary of the minimum circumscribed rectangle region of the kth gaze region, 、 respectively the width and the height of the minimum circumscribed rectangle region of the kth gaze region.

[0050] Step 1.6.7, repeat Step 1.6.1-Step 1.6.6 until the minimum circumscribed rectangle region of each of the 6 gaze regions in each group of eye movement tracking data of the initial eye movement tracking data set is obtained.

[0051] Step 1.7, according to the minimum circumscribed rectangle region of each of the 6 gaze regions in each group of eye movement tracking data, cut out 6 eye movement gaze region image blocks as shown in Figure 6 from the corresponding original image, label the eye movement vector matrix and the text answer on the corresponding original image, and then input the original image and the corresponding text question, the eye movement gaze region image block and the gaze order and gaze duration as the eye movement tracking data set into the biomimetic visual thinking chain cross-modal model based on eye movement tracking.

[0052] The training process of the biomimetic visual thinking chain cross-modal model based on eye tracking, i.e., steps 2-3, is as shown in Figure 7 .

[0053] Step 2, the biomimetic visual thinking chain cross-modal model based on eye tracking learns the features of an original image, the corresponding eye fixation region image block, and its fixation order and fixation duration in the eye tracking data set, and reasons the text question based on the features to obtain an eye prediction text answer and an eye vector prediction matrix. Specifically: Step 2.1, the input module transmits an original image in the eye tracking data set to the first image encoder, transmits the eye fixation region image block corresponding to the original image and its fixation order and fixation duration to the second image encoder, and transmits the text question corresponding to the original image to the large language model.

[0054] Step 2.2, the first image encoder extracts the features of the original image and transmits them to the first multilayer perceptron and the second multilayer perceptron respectively; the first multilayer perceptron maps the features of the original image to a feature space compatible with the image-text alignment module to obtain original image alignment mapping features and transmit them to the image-text alignment module, the dimension of the original image alignment mapping features is [1, m], where m is the feature dimension size of the image-text alignment module, which is a hyperparameter set according to the model architecture; the second multilayer perceptron maps the features of the original image to a feature space compatible with the large language model to obtain original image model mapping features and transmit them to the large language model; The second image encoder extracts the features of the eye fixation region image block, and then performs feature fusion according to the fixation order and fixation duration of the eye fixation region image block to obtain eye fixation region image block fusion features and transmit them to the third multilayer perceptron; the third multilayer perceptron maps the eye fixation region image block fusion features to a feature space compatible with the large language model to obtain eye fixation region image block mapping features and transmit them to the large language model.

[0055] Step 2.3, the large language model reasons the text question according to the original image model mapping features to obtain a text answer and transmits it to the fourth multilayer perceptron; the fourth multilayer perceptron maps the text answer to a feature space compatible with the image-text alignment module to obtain text answer mapping features and transmits them to the image-text alignment module, the dimension of the text answer mapping features is [1, m].

[0056] Step 2.4, the image-text alignment module aligns and fuses the original image alignment feature and the text answer alignment feature to obtain an image-text alignment feature and transmit it to the fifth multi-layer perceptron. The feature dimension of the image-text alignment feature is [2, m]; the fifth multi-layer perceptron maps the image-text alignment feature to a feature space compatible with the large language model to obtain an image-text alignment mapping feature and transmit it to the large language model.

[0057] Step 2.5, the large language model reasons the text question according to the original image model mapping feature obtained in step 2.2, the eye fixation region image block mapping feature, and the image-text alignment mapping feature obtained in step 2.4, to obtain an eye movement predicted text answer and an eye movement vector prediction matrix, and transmit them to the output module for output. The eye movement vector prediction matrix is a biomimetic visual thinking chain representing the reasoning process of the large language model.

[0058] Step 3, according to the eye movement predicted text answer, the eye movement vector prediction matrix, the corresponding text answer, and the eye movement vector matrix, a loss function is calculated, and according to the loss function, the parameters of the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, the fourth multi-layer perceptron, the fifth multi-layer perceptron, and the image-text alignment module in the biomimetic visual thinking chain cross-modal model based on eye tracking are adjusted, and then the process returns to step 2. The next original image and the corresponding text question, eye fixation region image block, and fixation sequence and fixation duration in the eye tracking data set are used to train the biomimetic visual thinking chain cross-modal model based on eye tracking until the loss function converges, obtaining the trained biomimetic visual thinking chain cross-modal model based on eye tracking. The loss function is calculated by the following formula: wherein, is the loss function, is the visual question and answer loss, calculated according to the predicted text answer and the text answer; is the eye movement prediction loss, calculated according to the eye movement vector prediction matrix and the eye movement vector matrix; is the thinking chain constraint loss, calculated according to the eye movement vector prediction matrix and the eye movement vector matrix; α, β, γ are the weights of the question and answer loss, the eye movement prediction loss, and the thinking chain constraint loss, respectively, and are all adjustable hyperparameters that can be optimized and adjusted according to specific task requirements. In this embodiment, α, β, γ are initially set to 0.3, 0.3, and 0.4, and are dynamically adjusted during the training process.

[0059] The loss function in the training process includes eye movement prediction loss, visual question and answer loss, and thought chain constraint loss. Among them, the eye movement prediction loss uses KL divergence to calculate the difference between the predicted eye movement distribution and the real eye movement distribution, and obtains the loss of the eye movement tracking data and the predicted eye movement data; the visual question and answer loss takes the text answer as the reference benchmark; the thought chain constraint loss adopts a group relative strategy to optimize the dynamic adjustment strategy in the multi-step text thinking process, and combines the strategy gradient loss and the KL divergence constraint to calculate the loss function in the multi-step reasoning process.

[0060] In this embodiment, end-to-end training provides a unified optimization target, adjusts parameters according to the loss function, and reversely propagates, which can balance the learning effect of multi-class loss.

[0061] Step 4, the eye movement tracking data of the to-be-tested image is collected by using the eye movement tracker, and then the eye movement tracking data and the to-be-tested image are transmitted to the pretreatment module; the pretreatment module acquires the eye movement fixation area image block of the to-be-tested image and the fixation sequence and fixation duration thereof from the to-be-tested image according to the eye movement tracking data of the to-be-tested image, and transmits the eye movement fixation area image block of the to-be-tested image and the fixation sequence and fixation duration thereof, the to-be-tested image, and the to-be-tested image text question to the input module of the trained eye movement tracking-based bionic visual thought chain cross-modal model.

[0062] In other embodiments, the eye movement tracking-based bionic visual thought chain cross-modal model supports online learning and incremental training, and can continuously learn and improve the generation quality of the bionic visual thought chain from new eye movement data.

[0063] Step 5, the trained eye movement tracking-based bionic visual thought chain cross-modal model learns the features of the to-be-tested image, the eye movement fixation area image block of the to-be-tested image, and the fixation sequence and fixation duration thereof, and performs reasoning on the to-be-tested image text question according to the features, to obtain the eye movement prediction text answer and the eye movement vector prediction matrix of the to-be-tested image, and then transmits the eye movement vector prediction matrix of the to-be-tested image to the visual thought chain visualization module.

[0064] Step 6, the to-be-tested image is transmitted to the visual thought chain visualization module, and the visual thought chain visualization module draws the fixation area on the to-be-tested image according to the eye movement vector prediction matrix of the to-be-tested image, and labels the row number of the fixation area in the eye movement vector prediction matrix at the center of the fixation area, to obtain the to-be-tested image containing the visual thought chain, and complete the visualization of the bionic visual thought chain.

[0065] In other embodiments, the visual thought chain visualization can adopt a semi-transparent ellipse superposition mode to sequentially display the fixation areas according to the fixation sequence, and intuitively present the visual thinking process.

[0066] When processing image-text information, human beings have a reasoning process for text information, and when watching images, the brain has a reasoning process for image analysis, which is more difficult to explain than text, but can be characterized by eye movement trajectories. In this embodiment, eye movement fixation region image blocks and their fixation order and fixation duration are used to enable large language models to understand human reasoning processes for images. Specifically, the observation process of human beings for images is characterized by eye movement vector matrices, which include fixation regions and fixation order. During the training process, eye movement fixation region image blocks and their fixation order and fixation duration obtained from real eye movement tracking data are input into the eye movement tracking-based biomimetic visual thinking chain cross-modal model. During the reasoning process, the eye movement tracking-based biomimetic visual thinking chain cross-modal model outputs the calculated biomimetic visual thinking chain, and the visual reasoning process can be visualized on the image.

[0067] The data flow of the reasoning process is divided into two stages, as shown in Figure 7 The first stage performs image-text alignment (blue line in Figure 7 ), including two parts: text question→large language model→fourth multilayer perceptron→image-text alignment module; original image→first image encoder→first multilayer perceptron→image-text alignment module. The second stage performs deep reasoning based on aligned features and eye movement information (red line in Figure 7 ), including five parts: first image encoder→second multilayer perceptron→large language model; eye movement fixation region image block and its fixation order and fixation duration→second image encoder→third multilayer perceptron→large language model; image-text alignment module→fifth multilayer perceptron→large language model; large language model→eye movement predicted text answer; large language model→eye movement vector prediction matrix→biomimetic visual thinking chain.

Claims

1. A biomimetic visual thought chain cross-modal model based on eye tracking, characterized in that: It includes an input module, a first image encoder, a second image encoder, a first multilayer perceptron, a second multilayer perceptron, a third multilayer perceptron, a fourth multilayer perceptron, a fifth multilayer perceptron, a large language model, an image-text alignment module, and an output module; The output of the input module is connected to the input of the first image encoder, the second image encoder, and the first input of the large language model, respectively. The input module is used to receive the image to be tested, the image block of the eye-tracking fixation region of the image to be tested and its fixation sequence and fixation duration, and the text question of the image to be tested, and transmit them to the first image encoder, the second image encoder, and the large language model respectively; The output of the first image encoder is connected to the input of the first multilayer perceptron and the second multilayer perceptron, respectively, for extracting features of the image to be tested; the output of the first multilayer perceptron is connected to the first input of the image-text alignment module, for mapping the features of the image to be tested to a feature space compatible with the image-text alignment module; the output of the second multilayer perceptron is connected to the second input of the large language model, for mapping the features of the image to be tested to a feature space compatible with the large language model. The output of the second image encoder is connected to the input of the third multilayer perceptron, which is used to extract the features of the eye-tracking fixation region image block of the image under test, and fuse the features according to the fixation order and fixation duration to obtain the fused features of the eye-tracking fixation region image block of the image under test. The output of the third multilayer perceptron is connected to the third input of the large language model, and is used to map the fusion features of the eye-tracking fixation region of the image to be tested to a feature space compatible with the large language model. The first output of the large language model is connected to the input of the fourth multilayer perceptron. The output of the fourth multilayer perceptron is connected to the second input of the image-text alignment module. The output of the image-text alignment module is connected to the input of the fifth multilayer perceptron. The output of the fifth multilayer perceptron is connected to the fourth input of the large language model. The second and third outputs of the large language model are connected to the first and second inputs of the output module, respectively. The large language model is used to obtain the predicted text response of the image under test, as well as the eye-tracking vector prediction matrix and the eye-tracking predicted text response. The output module is used to output the eye-tracking vector prediction matrix and the eye-tracking predicted text response of the image under test, thereby obtaining the biomimetic visual thought chain and the text thought chain.

2. The biomimetic visual thought chain cross-modal model based on eye tracking according to claim 1, characterized in that: Both the first image encoder and the second image encoder are image feature extraction networks; The first, second, third, fourth, and fifth multilayer perceptrons all employ residual connections and layer normalization structures.

3. The biomimetic visual thought chain cross-modal model based on eye tracking according to claim 2, characterized in that: The first image encoder and the second image encoder use CLIP or ViT; The large language model is GPT or LLaMA.

4. A biomimetic visual thought chain visualization system based on eye tracking, characterized in that: The invention includes an eye tracker for acquiring eye-tracking data of an image to be tested, a preprocessing module, a bionic visual thought chain cross-modal model based on eye tracking as described in any one of claims 1-3, and a visual thought chain visualization module. The output of the eye tracker is connected to the first input of the preprocessing module. The second input of the preprocessing module is used to receive the image to be tested and the text question of the image to be tested. The output is connected to the input of the input module in the bionic visual thinking chain cross-modal model based on eye tracking. The preprocessing module is used to obtain the eye-tracking gaze region image block, its gaze order and gaze duration, and the eye-tracking vector matrix from the image to be tested based on the eye-tracking data of the image to be tested. It then transmits these, along with the image to be tested and the text question of the image to be tested, to the input of the input module in the bionic visual thinking chain cross-modal model based on eye tracking. In the eye-tracking-based bionic visual thinking chain cross-modal model, the output end of the output module is connected to one input end of the visual thinking chain visualization module. The other input end of the visual thinking chain visualization module is used to receive the image to be tested, and the visual thinking chain visualization module is used to visualize the bionic visual thinking chain on the image to be tested.

5. The biomimetic visual thought chain visualization system based on eye tracking according to claim 2, characterized in that: The sampling frequency of the eye tracker is ≥60Hz.

6. A biomimetic visual thought chain visualization method based on eye tracking, based on the biomimetic visual thought chain visualization system based on eye tracking as described in claim 4 or 5, characterized in that, Includes the following steps: Step 1: Select a visual reasoning dataset including original images, text questions, and text answers. Then, use an eye tracker to acquire eye-tracking data of the original images in the visual reasoning dataset. Transfer the data along with the original images to the preprocessing module for preprocessing to obtain the eye-tracking gaze region image patches, their gaze order, gaze duration, and eye-tracking vector matrix corresponding to the original images. Annotate the eye-tracking vector matrix and the text answer on the corresponding original images. Then, use the original images, the corresponding text questions, the eye-tracking gaze region image patches, their gaze order, and gaze duration as the eye-tracking dataset and input it into the bionic visual thinking chain cross-modal model based on eye-tracking. Step 2: Based on eye tracking, the bionic visual thinking chain cross-modal model learns the features of an original image, the corresponding eye-tracking gaze region image patch, its gaze order, and gaze duration from the eye-tracking dataset, and uses these features to reason about text questions to obtain eye-tracking predicted text answers and eye-tracking vector prediction matrices. Step 3: Calculate the loss function based on the eye-tracking predicted text answer, the eye-tracking vector prediction matrix and the corresponding text answer, and the eye-tracking vector matrix. Then, adjust the parameters of the first multilayer perceptron, the second multilayer perceptron, the third multilayer perceptron, the fourth multilayer perceptron, the fifth multilayer perceptron and the image-text alignment module in the eye-tracking-based bionic visual thought chain cross-modal model according to the loss function. Then, return to Step 2 and train the eye-tracking-based bionic visual thought chain cross-modal model using the next original image and the corresponding text question, the eye-tracking gaze region image patch and its gaze order and gaze duration in the eye-tracking dataset until the loss function converges. Step 4: Use an eye tracker to collect eye-tracking data of the image to be tested, and then transmit it along with the image to be tested to the preprocessing module; The preprocessing module extracts the eye-tracking region image blocks, their gaze order, and gaze duration from the eye-tracking data of the image to be tested, and transmits them, along with the image to be tested and the text question in the image to be tested, to the input module of the trained bionic visual thought chain cross-modal model based on eye tracking. Step 5: The trained eye-tracking-based bionic visual thinking chain cross-modal model learns the features of the test image, the eye-tracking gaze region image block of the test image, and its gaze sequence and gaze duration. Based on these features, it infers the text question of the test image to obtain the eye-tracking predicted text answer and eye-tracking vector prediction matrix of the test image. Then, the eye-tracking vector prediction matrix of the test image is transmitted to the visual thinking chain visualization module. Step 6: Transmit the image to be tested to the visual thinking chain visualization module. The visual thinking chain visualization module draws the gaze region on the image to be tested according to the eye movement vector prediction matrix of the image to be tested, and marks the row number of the gaze region in the eye movement vector prediction matrix at the center of the gaze region, thus obtaining the image to be tested containing the visual thinking chain and completing the visualization of the bionic visual thinking chain.

7. The biomimetic visual thought chain visualization method based on eye tracking according to claim 6, characterized in that, Step 2 is as follows: Step 2.1: The input module transmits an original image from the eye-tracking dataset to the first image encoder, transmits the eye-tracking gaze region image block corresponding to the original image, along with its gaze order and gaze duration, to the second image encoder, and transmits the text question corresponding to the original image to the large language model; Step 2.2: The first image encoder extracts the features of the original image and transmits them to the first multilayer perceptron and the second multilayer perceptron respectively; the first multilayer perceptron maps the features of the original image to a feature space compatible with the image-text alignment module to obtain the original image alignment mapping features and transmits them to the image-text alignment module; the second multilayer perceptron maps the features of the original image to a feature space compatible with the large language model to obtain the original image model mapping features and transmits them to the large language model. The second image encoder extracts features from the eye-tracking fixation region image patch, and then performs feature fusion based on the fixation order and fixation duration of the eye-tracking fixation region image patch to obtain the eye-tracking fixation region image patch fused features, which are then transmitted to the third multilayer perceptron; the third multilayer perceptron maps the eye-tracking fixation region image patch fused features to a feature space compatible with the large language model to obtain the eye-tracking fixation region image patch mapped features, which are then transmitted to the large language model. Step 2.3: The large language model infers the text question based on the mapping features of the original image model, obtains the text answer, and transmits it to the fourth multilayer perceptron; the fourth multilayer perceptron maps the text answer to a feature space compatible with the image-text alignment module, obtains the text answer mapping features, and transmits them to the image-text alignment module. Step 2.4: The image-text alignment module aligns and fuses the original image alignment mapping features and text response mapping features to obtain image-text alignment features, which are then transmitted to the fifth multilayer perceptron. The fifth multilayer perceptron maps the image-text alignment features to a feature space compatible with the large language model, obtains the image-text alignment mapping features, and transmits them to the large language model. Step 2.5: Based on the original image model mapping features and eye-tracking fixation region image patch mapping features obtained in Step 2.2, and the image-text alignment mapping features obtained in Step 2.4, the large language model performs reasoning on the text question to obtain the eye-tracking predicted text answer and eye-tracking vector prediction matrix, and transmits them to the output module for output.

8. The biomimetic visual thought chain visualization method based on eye tracking according to claim 6 or 7, characterized in that, In step 3, the loss function is calculated using the following formula: ； in, For loss function, The visual question-answering loss is calculated based on eye-tracking predicted text responses and text answers. The eye-tracking prediction loss is calculated based on the eye-tracking vector prediction matrix and the eye-tracking vector matrix. The thought chain constraint loss is calculated based on the eye-tracking vector prediction matrix and the eye-tracking vector matrix; α, β, and γ are the weights of the question-answering loss, eye-tracking prediction loss, and thought chain constraint loss, respectively.

9. The biomimetic visual thought chain visualization method based on eye tracking according to claim 8, characterized in that, Step 1 is as follows: Step 1.1: Select a visual reasoning dataset from the public dataset that includes the original images, text questions, and text answers. Then, select multiple original images from the visual reasoning dataset in an equalized manner and randomly and uniformly group them. Step 1.2: Set the eye-tracking calibration accuracy threshold and the gaze angle speed threshold; Step 1.3: Invite testers and perform eye-tracking calibration on them individually. If the eye-tracking calibration accuracy is less than or equal to the eye-tracking calibration accuracy threshold, have the testers read the text questions corresponding to one set of original images and watch the set of original images. At the same time, use an eye tracker to record eye-tracking data. Then have the testers answer the text questions to obtain the eye-tracking data and the testers' answers. Otherwise, recalibrate the eye-tracking. Step 1.4: Check whether the tester's answer is correct based on the text answer. If it is correct, retain the corresponding eye-tracking data and the corresponding original image, text question and text answer. If not, remove the corresponding eye-tracking data and the corresponding original image, text question and text answer to obtain the initial eye-tracking dataset. Step 1.5: Calculate the gaze velocity of all sampling points in each group of eye tracking data in the initial eye tracking dataset. If the gaze velocity of a sampling point is lower than the gaze velocity threshold, the sampling point is determined as a fixation point; otherwise, it is determined as a saccade. This way, the fixation points of each group of eye tracking data in the initial eye tracking dataset are obtained. Step 1.6: Based on the fixation points of each group of eye-tracking data in the initial eye-tracking dataset, obtain M fixation regions, their fixation order, and fixation duration. Based on the M fixation regions and their fixation order, obtain the eye movement vector matrix of each group of eye-tracking data. Then, calculate the minimum bounding rectangle region of the M fixation regions in each group of eye-tracking data, where M is an integer and M≥1. Step 1.7: Based on the minimum bounding rectangle of the M gaze regions in each set of eye-tracking data, extract M eye-tracking gaze region image patches from the corresponding original images, and label the eye-tracking vector matrix and text answer on the corresponding original images. Then, use the original images, the corresponding text questions, the eye-tracking gaze region image patches and their gaze order and gaze duration as the eye-tracking dataset, and input them into the bionic visual thought chain cross-modal model based on eye-tracking.

10. The biomimetic visual thought chain visualization method based on eye tracking according to claim 9, characterized in that, Step 1.6 specifically involves: Step 1.6.1: Determine whether adjacent fixation points in one set of eye-tracking data in the initial eye-tracking dataset meet the following conditions: time interval < 200ms and gaze angle < 5°. If so, merge them into one fixation point to obtain the merged fixation point; then obtain M fixation regions and their fixation order based on the merged fixation point. Step 1.6.2: Count the number of fixation points in each of the M fixation regions, and then calculate the fixation duration of each of the M fixation regions based on the sampling frequency of the eye tracker. Step 1.6.3: Assume that all M fixation regions are ellipses, and then solve for the elliptical equation parameters of the M fixation regions using the following formulas to obtain the elliptical equations of the M fixation regions: ； in, , Let x and y be the x and y coordinates of the i-th fixation point in the k-th fixation region, respectively, where k and i are integers, and 1 ≤ k ≤ M, 1 ≤ i ≤ N. k , Let A be the number of fixation points in the k-th fixation region. k B k C k D k E k F k All of these are the ellipse equation parameters for the k-th gaze region; Step 1.6.4: Calculate the geometric parameters of the center, semi-major axis, semi-minor axis, and rotation angle of the M gaze regions using the following formulas: ； ； ； ； ； in, , Let x and y be the geometric center coordinates of the k-th gaze region, respectively. , These are the major and minor semi-axis of the k-th fixation region, respectively. Let be the rotation angle of the k-th gaze region; Step 1.6.5: Based on the geometric parameters of the M fixation regions—center, semi-major axis, semi-minor axis, and rotation angle—obtain the eye movement vectors for each of the M fixation regions. Then, the eye movement vectors are arranged from top to bottom according to the fixation order of the fixation areas, resulting in an eye movement vector matrix of size M×5. The data in the k-th row corresponds to the k-th gaze region. Step 1.6.6: Calculate the minimum bounding rectangle region [x] for each of the M gaze regions using the following formula. mink ,y mink ,w k ,h k ]: ； ； ； ； ； ； in, , These are the x-coordinates of the left and right boundaries of the minimum bounding rectangle of the k-th gaze region, respectively. , These are the ordinates of the lower and upper boundaries of the minimum bounding rectangle of the k-th gaze region, respectively. , These are the width and height of the minimum bounding rectangle of the k-th gaze region, respectively; Step 1.6.7: Repeat steps 1.6.1-1.6.6 until the minimum bounding rectangle of the M gaze regions in each group of eye-tracking data in the initial eye-tracking dataset is obtained.

Images