Method for generating virtual human eye movement trajectory based on multi-modal information input

Abstract

The application discloses an algorithm for generating virtual human eye movement track based on multi-modal information input, when the input is text, voice, image or a combination of two or three of them, the virtual human eye can produce natural action according to the semantic and visual, voice stimulation, so as to more truly and effectively simulate the eye movement. The application is automatically generated without manual operation; through the application of computer vision, hearing, multi-modal and other technologies, the manpower and time required for generating virtual human eye movement sequence in different scenes are greatly reduced. The generation of human eye movement is the result of multiple modal information stimulation. Through the deep summary of the eye movement law in psychology, the multi-modal input and stimulation are combined, the algorithm is used to generate the virtual human eye movement track, so that the eye movement is more real and natural.

Description

Technical Field

[0001] This invention relates to the technical fields of "image recognition", "semantic matching" and "multimodal" in the field of artificial intelligence, and in particular to an algorithm for generating virtual human eye movement trajectories based on multimodal information input. Background Technology

[0002] Currently, the industry primarily uses two methods to depict the eye movements of virtual characters: The first is motion capture, where workers wear devices that capture motion and facial expressions, reacting to external stimuli in specific tasks and scenarios. Hardware captures and records real-time motion data (including eye movement data), which is then reflected in the virtual character. The second method is animation, where special effects artists and animators use professional software (such as UE5) to design scripts for all eye movements, directly depicting the eye actions.

[0003] Meanwhile, in a paper presented at this year's top conference ISPN (2022), the Eyesyn research team at Duke University developed a virtual eye. By summarizing psychological literature, they customized eye-tracking rules and algorithms to simulate the changes in eye fixation points in four scenarios: text reading, face-to-face communication, viewing images, and watching videos. When using Eyesyn, the algorithm can generate eye-tracking sequences of arbitrary length simply by inputting an image or video, thus solving the problem of the difficulty in obtaining eye-tracking datasets to some extent.

[0004] Existing solutions, such as motion capture-based virtual human eye movement simulation, require significant manpower and time. This process involves repeated trials and sampling, increasing workload and costs. Similarly, animation-based methods require developers to design a separate eye movement script for each task. However, algorithmic eye movement prediction technology can significantly reduce this repetitive work.

[0005] Human eye movements are the result of stimulation from multiple modalities. However, current eye-tracking algorithms often focus only on visual modal stimuli. In the algorithm developed by researchers at Duke University, the prediction of the human eye's gaze point is based solely on images or videos, ignoring stimuli from other external modalities such as real-time audio. This limits the applicability of the developed eye-tracking model and makes it less realistic. The model in this application, however, considers both visual and auditory modal stimuli, synthesizing eye-tracking trajectories by comprehensively taking into account both visual and auditory information.

[0006] Virtual human movements and expressions often exhibit unnaturalness, which is particularly evident in the eyes. The eyes are the windows to the soul, but current technology is not yet perfect in simulating virtual human eye movements. This patent combines psychological knowledge and research to summarize and refine several patterns in human eye movements, and uses artificial intelligence technology to predict eye movements, enabling more realistic and effective simulation of eye movements, making virtual humans feel more authentic and natural.

[0007] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0008] The purpose of this invention is to provide an algorithm for generating virtual human eye movement trajectories based on multimodal information input, so as to solve the problems existing in the prior art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] This invention provides an algorithm for generating virtual human eye movement trajectories based on multimodal information input. When the input is text, speech, image, or a combination of two or three of them, the virtual human's eyes can produce natural movements based on semantic and visual and speech stimuli, so as to simulate eye movements more realistically and effectively.

[0011] As a further technical solution, the algorithm for generating virtual human eye movement trajectories based on multimodal information input specifically includes the following steps:

[0012] S1: Image / Video Subject Recognition: Using image recognition technology to identify the subject in an image / video frame and define the subject within the frame;

[0013] S2: Speech-to-Text Segmentation: First, TTS technology is used to convert speech into text; assuming that the smallest unit of auditory stimulus is a word, word segmentation is performed on the text obtained by TTS to divide a sentence into a series of words, resulting in a sequence of auditory stimuli; the order in which the words appear in the sentence is the arrangement of auditory stimuli in the time dimension.

[0014] S3: Image and text feature extraction: Encode the visual and auditory stimuli from steps S1 and S2; Use the visual and text encoder of the multimodal pre-trained dual-tower model to encode all the subjects segmented in the first step and the series of words obtained in the second step to obtain vectorized representations of images and words.

[0015] S4: Perform subject matching for each word segment: For each word segment heard, the corresponding gaze point is the subject with the highest relevance to this word among all the subjects identified in the image;

[0016] S5: Generate a virtual human eye movement coordinate sequence and add perturbations that conform to the laws of human eye movement: Input a series of auditory stimuli and use steps S1-S4 to calculate and generate a series of fixation point coordinates; the order of the coordinates is the order of changes in the eye fixation point; at the same time, a series of noise points are constructed using a Gaussian normal distribution to control the changes in the fixation point for each obtained fixation point coordinate, in order to more realistically simulate the subtle eye movements of humans.

[0017] S6: Apply eye-tracking sequences to virtual human models. Eye-tracking simulation: Connect the output eye-tracking coordinate sequence with the virtual human's relevant interface to transform the data sequence into the virtual human's actual pupil movement, thus demonstrating the virtual human's eye-tracking effect.

[0018] S7: Test the model's effectiveness using real eye-tracking data and subjective assessment scores.

[0019] As a further technical solution, the subject in step S1 includes the objects appearing in the image / video and their background environment. Each identified subject is a sub-image of the original image. Subject recognition can provide potential focal points for the line of sight, and the potential focal point of the line of sight is the center position of all identified subjects.

[0020] As a further technical solution, the pre-trained dual-tower model in step S3 refers to the Wenlan multimodal pre-trained model. This model incorporates the MoCo method into an improved contrastive learning algorithm, proposing a dual-tower structure pre-trained model based on the weak association assumption between image-text pairs. The Wenlan cross-modal pre-trained model is defined based on the image-text retrieval task; therefore, the main goal of Wenlan training is to learn two encoders that can embed image and text samples into the same space, thereby achieving effective image-text retrieval. To enhance this cross-modal embedding learning, the Wenlan model introduces contrastive learning with InfoNCE loss. Specifically, for a given text embedding, Wenlan's learning objective is to find the best image embedding from a batch of image embeddings. Similarly, for a given image embedding, Wenlan's learning objective is to find the best text embedding from a batch of text embeddings. In summary, the Wenlan pre-trained model learns the cross-modal vector space by jointly training image and text encoders to maximize the cosine similarity of the embedding vectors of correct image and text pairs in each sample, while minimizing the cosine similarity of the embedding vectors of other incorrect image and text pairs (e.g., ...). Figure 2 (As shown).

[0021] As a further technical solution, step S4 specifically involves obtaining the vector encoding of word segmentation and image subject through the image-text pre-training model in step S3. In S4, a similarity formula can be selected to calculate the similarity between image-text vectors, which is used to represent the relevance between images and text.

[0022] For nouns in a sentence, the corresponding fixation point is the center position of one or more subjects in the image that are most relevant to this word segment. For pronouns, verbs, and adjectives in a sentence, denoted as word 'a', the first step is to calculate the center position of the subject in the image that is most relevant to word segment 'a'. The second step is to use a self-attention model to calculate the other words in the sentence that are most relevant to word segment 'a', denoted as word segment 'b'. The image-text relevance calculated in the first step and the word relevance calculated in the second step are compared. The coordinates of the fixation point corresponding to word segment 'a' are determined by the center position of the subject with the highest relevance or the coordinates of the fixation point corresponding to word segment 'b'.

[0023] As a further technical solution, the specific algorithm for step S5 is as follows: A series of auditory stimuli are input, and a series of fixation point coordinates are calculated using steps S1-S4. The order of the coordinates corresponds to the order of changes in the eye's fixation point. Simultaneously, considering that the human eye does not remain stationary at a fixation point but continuously undergoes "micro-saccades"—the eye's focal point always trembles irregularly near the object being fixed—a model for micro-saccades is considered. Referring to academic research on micro-saccades, this model sets hyperparameters such as the mean and variance of a Gaussian distribution and the sampling frequency, first obtaining a set of parameters derived from 1 / f... α The filtered Gaussian white noise is used, where f is the frequency of the signal. The angle θ between the gaze point and saccade point is then calculated based on this noise. The formula can be expressed as F(α,β), which is 1 / f. α The filter has a power α at the inverse frequency and β is the input following the frequency response. Gaussian distributed white noise. Converting its units to radians yields the angle value. Given the distance between the eye and the object being gazed upon, multiplying this distance by the tangent of the angle gives the fixation point for the microsaccades, calculated as (x, y) = (x0 + distance × tanθ, y0 + distance × tanθ). Finally, a series of new fixation points are added to the fixation point coordinate system to more realistically simulate the minute eye movements of the human eye.

[0024] As a further technical solution, step S7 tests the model's effectiveness in two ways: Method 1 uses real eye-tracking data as a baseline, defines a loss function, and compares the loss magnitudes of the eye-tracking sequences generated by the algorithm and the randomly generated eye-tracking sequences; Method 2 designs a subjective evaluation, asks subjects to subjectively evaluate the eye-tracking sequences generated by the algorithm and the randomly generated eye-tracking sequences, and compares the scores of the two sequences.

[0025] By adopting the above technical solution, the present invention has the following beneficial effects:

[0026] 1. Fully automated generation, no manual intervention required. By applying technologies such as computer vision, hearing, and multimodal processing, the manpower and time required to generate virtual human eye movement sequences in different scenarios are greatly reduced.

[0027] 2. Multimodal stimulation for greater realism. Human eye movements are the result of stimulation from multiple modal information. Through a profound summary of the psychological principles of eye movement, combined with multimodal input and stimulation, algorithms are used to generate the eye movement trajectory of a virtual human, making eye movements more realistic and natural.

[0028] 3. Handling weakly correlated cases. Most pre-trained models assume a strong semantic correlation between the input text and the image, but this assumption often doesn't hold true in real-world scenarios. In many cases, the information we hear and the information we see are not strongly correlated. (For example, if an image shows a beach, sea, and sunset, we might not directly name these objects, but are more likely to use words like "comfortable," "vast," or "beautiful" to describe them.) Therefore, we utilize superior image-text pre-trained models that encode deep semantic information to handle cases where the image and text are weakly correlated. Attached Figure Description

[0029] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 The flowchart shows the algorithm for generating virtual human eye movement trajectories based on multimodal information input, as provided in an embodiment of the present invention.

[0031] Figure 2 This is a structural diagram of the pre-trained dual-tower model provided in an embodiment of the present invention.

[0032] Figure 3 A specific example flowchart is provided for embodiments of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0035] Human eye movements are the result of stimulation from multiple modalities. It is hypothesized that eye movements differ depending on whether there is only visual stimulation, only auditory stimulation, or both. Furthermore, it is hypothesized that when there is both visual and auditory stimulation, upon hearing the information, the eye's fixation point tends to fall on the part of the visual field more relevant to the information heard at that moment, and subtle eye movements occur in that vicinity. Based on this hypothesis, this application uses an image / video as input to simulate visual stimulation and audio input to simulate auditory stimulation, designing the following technical solution for simulating eye movements (e.g., ...). Figure 1 As shown):

[0036] Step 1: Image / Video Subject Recognition.

[0037] For images / videos, the first step is to use image recognition technology to identify the main subject within the image. The subject here includes objects appearing in the image / video and their background environment; each identified subject is a sub-image of the original image. Subject identification provides potential focal points for the line of sight. We define the potential focal point as the center position of all identified subjects.

[0038] Step 2: Segment the speech into text.

[0039] First, we use TTS (Text-to-Speech) technology to convert speech into text. We assume that the smallest unit of auditory stimulus is a word. We then perform word segmentation on the text obtained from TTS, dividing a sentence into a series of segments, thus obtaining a sequence of auditory stimuli. The order in which the segments appear in the sentence represents the temporal arrangement of the auditory stimuli.

[0040] Step 3: Image and text feature extraction.

[0041] To facilitate image-text matching, the visual and auditory stimuli from the first two steps need to be encoded. Using a pre-trained image-text encoding model, and employing a multimodal pre-trained dual-tower model for both visual and text encoders, all subjects segmented in the first step and the series of word segments obtained in the second step are encoded to obtain vectorized representations of the images and word segments. This model incorporates the MoCo method into an improved contrastive learning algorithm, proposing a dual-tower structure pre-trained model based on the assumption of weak association between image-text pairs. The Wenlan cross-modal pre-trained model is defined based on the image-text retrieval task; therefore, the main goal of Wenlan's training is to learn two encoders that can embed image and text samples into the same space, thereby achieving effective image-text retrieval. To enhance this cross-modal embedding learning, the Wenlan model introduces contrastive learning with InfoNCE loss. Specifically, for a given text embedding, Wenlan's learning objective is to find the best image embedding from a set of image embeddings. Similarly, for a given image embedding, Wenlan's learning objective is to find the best text embedding from a set of text embeddings. In summary, the Wenlan pre-trained model learns a cross-modal vector space by jointly training image and text encoders to maximize the cosine similarity of the embedding vectors of correct image and text pairs in each sample, while minimizing the cosine similarity of the embedding vectors of other incorrect image and text pairs (e.g., ...). Figure 2 (As shown).

[0042] The dual-tower structure trained in this way has a series of high-dimensional vectors obtained by the image encoder from all the subjects segmented in the first step of the eye-tracking model. The series of words obtained in the second step of the eye-tracking model can also be mapped into high-dimensional vectors in the same space by the text encoder, which facilitates matching.

[0043] Step 4: Perform subject matching for each word segment.

[0044] For each segmentation heard, the corresponding fixation point should be the subject with the highest relevance to that segmentation among all the subjects identified in the image.

[0045] In the third step, we used the image-text pre-trained model to obtain the vector encoding of the word segmentation and the main body of the image. In this step, we can select a similarity formula (such as the cosine similarity formula) to calculate the similarity between the image and text vectors, which is used to represent the relevance between the image and text.

[0046] For nouns in a sentence, the corresponding fixation point is the center position of one or more entities in the image that are most relevant to this word segment. For other words in the sentence (pronouns, verbs, adjectives, etc., denoted as word 'a'), the first step is to calculate the center position of the entity in the image that is most relevant to word 'a'. The second step is to use a self-attention model to calculate the other words in the sentence that are most relevant to word 'a', denoted as word 'b'. By comparing the image-text relevance calculated in the first step with the word-segment relevance calculated in the second step (self-attention), the coordinates of the fixation point corresponding to word 'a' are determined by either the center position of the entity with the highest relevance or the coordinates of the fixation point corresponding to word 'b'.

[0047] Because we utilize a pre-trained image-text encoding model to encode deeper semantic information and employ self-attention to analyze the correlation between word segments, our image-text relevance matching is not limited to matching concrete words (such as nouns) with the main subject of the image. For weakly related words or phrases such as "beautiful" or "towering," this method can also find reasonable fixations. Furthermore, a superior image-text encoder enables cross-modal image-text matching, resulting in better matching performance.

[0048] Step 5: Generate a virtual human eye movement coordinate sequence and add perturbations that conform to the laws of human eye movement.

[0049] The key to simulating eye movements under visual and auditory stimuli is determining the order in which gaze points change on a given image / video, based on the guided sequence of auditory stimuli. A series of auditory stimuli (i.e., a series of segmented codes) are input, and the above method generates a series of gaze point coordinates. The order of these coordinates corresponds to the order in which the eye gazes change. Simultaneously, modeling and sampling are performed around each obtained gaze point coordinate to obtain a series of new gaze points. This is used to more realistically simulate subtle human eye movements (research shows that when a person perceives themselves as gazing at a point, their eyes actually make subtle movements around that point), making the eye movement sequence more realistic.

[0050] A series of auditory stimuli are input, and steps S1-S4 are used to calculate a series of fixation point coordinates. The order of the coordinates corresponds to the order in which the eye's fixation point changes. Simultaneously, considering that the human eye does not focus statically on a fixation point but constantly undergoes "micro-saccades"—the eye's focal point always jitters irregularly near the object being fixed—a model of micro-saccades is considered. Referring to academic research on micro-saccades, this model sets hyperparameters such as the mean and variance of a Gaussian distribution and the sampling frequency, first obtaining a set of hyperparameters derived from 1 / f... αThe filtered Gaussian white noise is used, where f is the frequency of the signal. The angle θ between the gaze point and saccade point is then calculated based on this noise. The formula can be expressed as F(α,β), which is 1 / f. α The filter has a power α at the inverse frequency; β is the input following N(0,σ) 2 Gaussian distributed white noise is used. Converting this noise to radians yields the angle value. Given the distance between the eye and the object being gazed upon, multiplying this distance by the tangent of the angle gives the fixation point for the microsaccades, calculated as (x, y) = (x0 + distance × tanθ, y0 + distance × tanθ). Finally, a series of new fixation points are added to the fixation point coordinate system to more realistically simulate the minute eye movements of the human eye.

[0051] Step 6: Apply the eye-tracking sequence to the virtual human model to simulate eye-tracking gaze.

[0052] The output eye-tracking coordinate sequence is connected to the virtual human interface to transform the data sequence into the actual pupil movement of the virtual human, thus demonstrating the eye-tracking effect of the virtual human.

[0053] Step 7: Test the model's effectiveness using real eye-tracking data and subjective assessment scores.

[0054] Step 7 tests the model's performance in two ways: Method 1 uses real eye-tracking data as a baseline, defines a loss function, and compares the loss of the eye-tracking sequence generated by the algorithm with that of the randomly generated eye-tracking sequence; Method 2 designs a subjective evaluation, asks subjects to give subjective evaluations of the eye-tracking sequence generated by the algorithm and the randomly generated eye-tracking sequence, and compares the scores of the two sequences.

[0055] Method 1 is detailed below: Using an EyeLink eye tracker, information about the subject's actual fixation sequence is obtained (including the coordinates (x_i, y_i) of fixation point i, the initial time t_i when the gaze falls on that fixation point, and the total fixation time T for the image. An image and text are input, and an eye-tracking simulation model generates an eye-tracking sequence of total duration T, including the initial time t_j when reading to a keyword j, and the positions (x_j1, y_j1), (x_j2, y_j2), ..., (x_jn, y_jn) of all subjects in the image corresponding to that keyword (assuming keyword j corresponds to n subjects in the image). These positions are the potential fixation points in the simulated eye-tracking sequence.

[0056] Assuming the initial time t_i when the gaze falls on the true fixation point satisfies t_j <= t_i <= t_{j+1}, then the keyword corresponding to fixation point i is j. Find the point (x_j', y_j') with the smallest distance to (x_i, y_i) among all potential fixations (x_j1, y_j1), (x_j2, y_j2), ..., (x_jn, y_jn). Let d be the error of fixation point i. For each true fixation point, calculate the error value using the above method, and finally calculate the average error.

[0057] Meanwhile, due to the human eye's field of vision and the existence of micro-saccades, we stipulate that if the distance d between the real fixation point and the simulated fixation point is less than 100 pixels, then the error at that fixation point is 0.

[0058] The simulated eye-tracking sequence and the randomly generated eye-tracking sequence were tested on 21 images of size 1024×768. The MAE and RMSE of the two sequences are shown in the table below:

[0059] 326.7 254.3 512.8 399.5

[0060] It can be seen that the MAE and RMSE losses of the eye-tracking sequences simulated by the model are both smaller than those of the randomly generated eye-tracking sequences, indicating that the eye-tracking sequences simulated by the model perform better.

[0061] Method 2 is detailed below: To better verify the model's effectiveness, we designed a subjective evaluation method, using a scoring system to evaluate the model. The model-generated gaze-point sequence and a randomly generated sequence were presented to real users for scoring in two aspects: "consistency between gaze-points and multimodal information" and "realism of gaze-point movement." The scoring criteria are as follows:

[0062]

[0063] The scores for the model-generated and randomly generated implementation landing sequences are shown below. The model also outperforms the randomly generated results in the overall subjective evaluation.

[0064]

[0065] Currently, there are very few eye-tracking datasets for multimodal stimuli. Most eye-tracking datasets only involve single-modal visual stimuli, neglecting the influence of auditory stimuli on eye movements. Therefore, we designed our own experimental data to simulate multiple modal stimuli and invited subjects to observe and record their eye movements using an eye tracker, thus constructing a multimodal eye-tracking dataset. We also designed a subjective evaluation criterion for eye-tracking sequences to obtain a subjective score for the model's effectiveness.

[0066] The experimental design is as follows: Download copyright-free images from websites, write a short statement related to the image content, and record it as audio. During the experiment, the audio content is played while the images are shown to the subjects, and their eye movements are recorded.

[0067] The implementation process of this invention will be further explained below with specific examples:

[0068] like Figure 3 As shown, the input image and text are: "The palm trees on the beach became silhouettes against the sunset and the sea". Then, topic recognition and word segmentation are performed ("The palm trees on the beach / became silhouettes against the sunset and the sea"). Then, image and word segmentation encoding and image-text relevance matching are performed, and then an eye-tracking sequence is generated.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating virtual human eye movement trajectories based on multimodal information input, characterized in that, When the input is text, voice, image, or a combination of two or three of them, the virtual human's eyes can produce natural movements based on semantic and visual and voice stimuli, so as to simulate eye movements more realistically and effectively. The method specifically includes the following steps: S1: When the input is an image: Image recognition technology is used to identify the subject in the image / video and frame the subject in the image; When the input is speech: First, TTS technology is used to convert speech into text; Assuming that the smallest unit of auditory stimulus is a word, word segmentation is performed on the text obtained from TTS. After segmenting a sentence into a series of words, an auditory stimulus sequence is obtained; The order in which the words appear in the sentence is the arrangement of auditory stimuli in the time dimension; S2: Image and text feature extraction: Encode the visual and auditory stimuli from step S1; use the visual and text encoders of the multimodal pre-trained dual-tower model to encode all the subjects segmented in the first step and the series of word segments obtained in the second step, and obtain the vectorized representations of the image and word segments. S3: Perform subject matching for each word segment: For each word segment heard, the corresponding gaze point is the subject with the highest relevance to this word among all the subjects identified in the image; S4: Generate a virtual human eye movement coordinate sequence and add perturbations that conform to the laws of human eye movement: Input a series of auditory stimuli and use steps S1-S3 to calculate and generate a series of fixation point coordinates; the order of the coordinates is the order of changes in the eye fixation points; at the same time, use a Gaussian normal distribution to construct a series of noise points for each obtained fixation point coordinate to control the changes in the fixation points, in order to more realistically simulate the subtle eye movements of humans. S5: Apply eye-tracking sequences to virtual human models. Eye-tracking simulation: Connect the output eye-tracking coordinate sequence with the virtual human's relevant interface to transform the data sequence into the virtual human's actual pupil movement, thus demonstrating the virtual human's eye-tracking effect. S6: Test the model's effectiveness using real eye-tracking data and subjective assessment scores; Step S3 specifically involves obtaining the word segmentation and image subject vector encodings using the image-text pre-training model in step S2, and selecting a similarity formula in S3 to calculate the similarity between the image and text vectors, which is used to represent the relevance between the image and text. For nouns in a sentence, the corresponding fixation point is the center position of one or more subjects in the image that are most relevant to this word segment. For pronouns, verbs, and adjectives in a sentence, denoted as word 'a', the first step is to calculate the center position of the subject in the image that is most relevant to word segment 'a'. The second step is to use a self-attention model to calculate the other words in the sentence that are most relevant to word segment 'a', denoted as word segment 'b'. The image-text relevance calculated in the first step and the word relevance calculated in the second step are compared. The coordinates of the fixation point corresponding to word segment 'a' are determined by the center position of the subject with the highest relevance or the coordinates of the fixation point corresponding to word segment 'b'.

2. The method for generating virtual human eye movement trajectories based on multimodal information input according to claim 1, characterized in that, The subject in step S1 includes the objects appearing in the image and the background environment. Each identified subject is a sub-image of the original image. Subject recognition can provide potential focal points for the line of sight. The potential focal point of the line of sight is the center position of all identified subjects.

3. The method for generating virtual human eye movement trajectories based on multimodal information input according to claim 1, characterized in that, The pre-trained dual-tower model in step S2 refers to the Wenlan multimodal pre-trained model. This model incorporates the MoCo method into an improved contrastive learning algorithm, proposing a dual-tower structure pre-trained model based on the weak association assumption between image-text pairs. The Wenlan multimodal pre-trained model is defined based on the image-text retrieval task; therefore, its training objective is to learn two encoders that can embed image and text samples into the same space, thus achieving effective image-text retrieval. To enhance this cross-modal embedding learning, the Wenlan multimodal pre-trained model introduces contrastive learning with InfoNCE loss. Specifically, for a given text embedding, the learning objective of the Wenlan multimodal pre-trained model is to find the best image embedding from a batch of image embeddings; similarly, for a given image embedding, the learning objective of the Wenlan multimodal pre-trained model is to find the best text embedding from a batch of text embeddings. In summary, the Wenlan multimodal pre-trained model learns a cross-modal vector space by jointly training image and text encoders to maximize the cosine similarity of the embedding vectors of the correct image and text pairs in each sample, while minimizing the cosine similarity of the embedding vectors of other incorrect image and text pairs.

4. The method for generating virtual human eye movement trajectories based on multimodal information input according to claim 1, characterized in that, Step S4 specifically involves constructing a series of noise points using a Gaussian normal distribution. The method involves setting the mean, variance, and sampling frequency hyperparameters of the Gaussian normal distribution, first obtaining a set of... Gaussian white noise after filtering, where The frequency of the signal is given; then, the angle between the line of sight at the "fixation point" and the "saccade point" is calculated based on this noise. Calculation formula This is The filter has a power at the reverse frequency. ; For the input to obey Gaussian distributed white noise; converting its units to radians yields the angle value; given the distance between the eye and the object being gazed upon, multiplying by the tangent of this angle gives the fixation point of the microsaccade, as shown in the formula: Finally, a series of new fixation points were obtained and added to the fixation point coordinates to more realistically simulate the subtle eye movements of the human eye.

5. The method for generating virtual human eye movement trajectories based on multimodal information input according to claim 1, characterized in that, Step S6 tests the model's performance in two ways: Method 1 uses real eye-tracking data as a baseline, defines a loss function, and compares the loss of the eye-tracking sequence generated by the algorithm with that of the randomly generated eye-tracking sequence; Method 2 designs a subjective evaluation, asks subjects to give subjective evaluations of the eye-tracking sequence generated by the algorithm and the randomly generated eye-tracking sequence, and compares the scores of the two sequences.

Images