A method and system for understanding long image sequences

Abstract

The present invention proposes a long image sequence understanding method and system, which belongs to the field of artificial intelligence technology, including: obtaining a long image, obtaining high-level semantic information of the image, and vectorizing it to obtain an image vector; obtaining a text description, obtaining a basic feature matrix of the text description, and vectorizing it to obtain a text vector; inputting the image vector and the text vector into a common attention module, and outputting a sequence structure of image blocks; performing image-text alternating sequence processing on the sequence structure to obtain a text feature Htext and a visual feature Himg; inputting a graphic-text interactive attention module to obtain a fusion feature; and enhancing the model's ability to understand the image sequence by introducing adjacent visual attention masks and image-text interlaced rotation position embedding, so that the model can not only effectively process long image sequences, but also show excellent performance in a variety of application scenarios such as image description generation, multi-image reasoning, video understanding, and multimodal question answering.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a long image sequence understanding method and system. Background Art

[0002] Multimodal large language models have demonstrated powerful capabilities at the intersection of natural language processing and computer vision. In particular, in tasks that process a single image, these models are able to perform complex operations based on given instructions, such as image description generation, object recognition, scene understanding, etc. This capability is due to the pre-training process of the model on a large-scale image-text dataset, which enables the model to learn rich cross-modal representations.

[0003] Nevertheless, the performance of multimodal large language models still needs to be improved when processing long image sequences, such as video clips, image storyboards, comic series, etc. Existing methods mainly rely on large-scale datasets of image-text pairs to train models with basic multimodal understanding capabilities. However, these methods have limitations in the following aspects: Current methods are generally unable to effectively handle complex patterns in long sequences, especially those events or action sequences that span multiple images. Maintaining contextual consistency and coherence is a challenge when processing long image sequences. Existing models often have difficulty capturing long-term dependencies between images. As the length of the sequence grows, the computational resources required to process each image also increase, which is an obstacle for real-time or resource-constrained applications. Therefore, there is an urgent need to develop a new method to address these issues to achieve more efficient and in-depth understanding of long image sequences and support advanced features such as multi-image reasoning, video understanding, and multimodal question-answering systems. Summary of the invention

[0004] The present invention aims at solving the problems in the prior art, and the technical solution adopted in this application is as follows:

[0005] In a first aspect, the present invention provides a long image sequence understanding method, comprising:

[0006] Obtain a long image, extract the high-level semantic information of the long image, and vectorize it to obtain an image vector;

[0007] Obtain the text description, extract the text description to obtain the basic feature matrix of the text description, and vectorize it to obtain the text vector;

[0008] The image vector and text vector are fed into the joint attention module, and the sequence construction of the output image patches is constructed;

[0009] The sequence structure is processed by alternating image and text sequences to obtain text features Htext and visual features Himg with position encoding vectors; image and text alternating sequence processing introduces image and text interleaved rotation position embedding to enhance understanding;

[0010] The text feature Htext and the visual feature Himg are input into the graphic-text interactive attention module to obtain the fusion feature; the graphic-text interactive attention module introduces the neighboring visual attention mask to ensure the directionality of the information flow;

[0011] Use the fused features as input data for multi-image reasoning, video understanding, or multimodal question answering systems.

[0012] Furthermore, the specific steps of extracting high-level semantic information of a long image include: performing image feature extraction on the long image to obtain image blocks, performing basic visual feature extraction on the image blocks to obtain basic visual features, performing high-level semantic feature refinement on the basic visual features to obtain high-level semantic information of the image, and vectorizing the high-level semantic information of the image to obtain an image vector.

[0013] Furthermore, the specific steps of extracting the basic feature matrix of the text description include: obtaining the text description, performing text preprocessing on the text description to obtain the basic feature matrix of the text description by extracting text features, and vectorizing the basic feature matrix to obtain a text vector.

[0014] Furthermore, the specific steps of inputting the image vector and the text vector into the common attention module include: in the common attention module, the image features and the text features interact with each other through the cross-attention mechanism. For the image features, Q and K come from the text features, and V comes from the image features. Finally, the high-level semantic features of the output image block are represented as a sequence structure [I1, I2, ..., In].

[0015] Furthermore, the specific steps of alternating the sequence structure with the image and text include: inputting the sequence structure, the sequence structure is a sequence consisting of alternating text blocks and image blocks, in the form of S=[T1,I1,T2,I2,T3,I3], where Ti represents the i-th text segment and Ii represents the i-th image block;

[0016] The visual features extracted from each image block Ii are adjusted to the same dimension as the language model through linear projection; the projected visual features are expressed as ;

[0017] Create a plain text representation for each image block as a placeholder so that the text sequence and the image sequence can be interleaved to construct the text sequence Stext;

[0018] Input the text sequence Stext into the word embedding layer to obtain the corresponding text feature Htext; use the image-text interleaved rotation position embedding to pre-record the position index i of the placeholder Timg_i of each image block Ii in the interleaved sequence Stext. For each image block Ii, use the formula to calculate the position encoding vector.

[0019] Furthermore, the specific steps of obtaining the fusion feature by the graphic-text interaction attention module include:

[0020] The query Q of the image-text interaction attention module comes from the text feature Htext; the key K and value V come from the visual feature Himg; a specific image weight matrix is ​​used To generate the keys and values ​​of visual features, the image weight matrix Initialize using the weights of K and V from the language model;

[0021] A neighboring visual attention mask is used in the cross attention. For the text sequence Stext=[T1,Timg_1,T2,Timg_2,T3,,Timg_3,…], each text token can only access the previous context information and the adjacent image information;

[0022] By designing an adaptive mechanism, the image-text interactive attention module dynamically determines the number of interactive attention layers according to the characteristics of the input data, and obtains the output result after processing the text feature Htext and the visual feature Himg through the interactive attention layers with the set number of layers;

[0023] The output of the interactive attention module is passed through a fully connected layer to further fuse the information of text and image to obtain fusion features.

[0024] In a second aspect, the present invention provides a long image sequence understanding system, comprising:

[0025] The long image processing module is configured to: obtain a long image, extract the long image to obtain high-level semantic information of the image, and vectorize the long image to obtain an image vector;

[0026] The text description processing module is configured to: obtain the text description, extract the text description to obtain a basic feature matrix of the text description, and vectorize it to obtain a text vector;

[0027] The joint attention module is configured to: input the image vector and the text vector into the joint attention module, and output a sequence construction of image patches;

[0028] The image-text alternation module is configured to: process the sequence structure into an image-text alternation sequence to obtain a text feature Htext and a visual feature Himg with a position encoding vector; the image-text alternation sequence processing introduces image-text interleaved rotation position embedding to enhance understanding;

[0029] The graphic-text interaction attention module is configured as follows: the text feature Htext and the visual feature Himg are input into the graphic-text interaction attention module to obtain the fusion feature; the graphic-text interaction attention module introduces the neighboring visual attention mask to ensure the directionality of the information flow;

[0030] The output module is configured to use the fused features as input data for multi-image reasoning, video understanding, or multimodal question answering systems.

[0031] In a third aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a long image sequence understanding method described in the first aspect.

[0032] In a fourth aspect, the present invention provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute a long image sequence understanding method described in the first aspect.

[0033] Compared with the prior art, the advantages and positive effects of the present invention are:

[0034] The present invention proposes a method and system for understanding long image sequences, which enhances the model's ability to understand image sequences by introducing neighboring visual attention masks and image-text interlaced rotation position embedding. Specifically, the neighboring visual attention mask limits each text tag to only pay attention to the visual features adjacent to it, which helps the model better capture the local contextual relationships in the image sequence. The image-text interlaced rotation position embedding further enhances the model's sensitivity to the sequential information of the image sequence. This design enables the model to not only effectively process long image sequences, but also to perform well in a variety of application scenarios such as image description generation, multi-image reasoning, video understanding, and multimodal question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 The figure is a flow chart of a long image sequence understanding method of the present invention. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0039] Embodiment 1, as Figure 1 As shown, this embodiment provides a long image sequence understanding method, which maintains the consistency and coherence of the context by capturing and understanding the long-term dependencies in the image sequence. The steps include: S1, image text feature extraction; S2, image text interleaved sequence processing; S3, image text interactive understanding; S4, long image sequence understanding application;

[0040] Specifically, in step S1, S1.1, image feature extraction:

[0041] Image block processing: First, define a suitable block size (32×32 pixels), use a sliding window to traverse the entire image, move one block size at a time until the entire image is covered, and use overlapping blocks (set the step size to P / 2) to capture more contextual information.

[0042] Basic visual feature extraction: Each image block is processed by a pre-trained deep convolutional neural network to extract basic visual features, including low-level visual elements such as edges, textures, and color patterns. The pre-trained deep convolutional neural network uses DenseNet (Densely Connected Convolutional Networks); DenseNet uses dense connections to allow each layer to be directly connected to all subsequent layers, thereby encouraging feature reuse and reducing redundant learning. This structure can effectively alleviate the gradient vanishing problem, enhance feature propagation, and strengthen the flow of supervisory signals throughout the network.

[0043] High-level semantic feature extraction: Based on the basic visual features obtained, the self-attention mechanism is further used to extract high-level semantic information of the image, such as identifying the relationship between objects, actions, and scene background. For example, in images of sports events, the self-attention mechanism can help identify the actions of athletes and their interactions with other athletes.

[0044] Steps to extract high-level semantic information of images using the self-attention mechanism:

[0045] Step S1.11, feature extraction: First, obtain the basic visual features extracted in the above steps;

[0046] Step S1.12, feature reorganization: The extracted feature map is reorganized into a series of vectors that can represent different parts of the image. For example, the feature map can be divided into multiple non-overlapping blocks and each block is flattened into a vector.

[0047] Step S1.13, calculate the self-attention matrix: For each feature vector, calculate a query vector (Query) and a key vector (Key). The query vector is used to ask questions, and the key vector is used to answer these queries. The dot product between the query vector and the key vector gives the similarity score between the query and the key.

[0048] In step S1.13, after calculating the attention score, it is converted into a probability distribution through the softmax function, indicating the importance of each position relative to the query.

[0049] Step S1.14, weighted summation: Use the calculated attention distribution to perform weighted summation on the value vector (Value) to generate a new feature representation. The value vector contains the information to be aggregated by the attention mechanism.

[0050] Step S1.15, Multi-head Attention, uses a multi-head attention mechanism, that is, repeating the above process multiple times under different linear transformations to capture information in different subspaces and concatenating the results to form the final output.

[0051] Step S1.16, feature fusion: Finally, the features processed by the self-attention mechanism can be fused with the original features or other information sources to further enhance the model's understanding ability.

[0052] Step S1.17, high-level semantic information extraction: Through the above steps, the model can extract more advanced semantic information from basic visual features, such as the relationship between objects, actions, and scene background.

[0053] Generate structured vector representation: Ultimately, these high-level semantic features are aggregated into a structured vector representation to facilitate the next step of fusion processing. Aggregation into a structured vector representation can be achieved by adding one or more fully connected layers, which can map high-level semantic features to a new space, thereby creating a more compact vector representation. Fully connected layers can learn complex relationships between features and may capture more information than simple pooling.

[0054] S1.2, Text feature extraction: Text preprocessing: Use the SentencePiece word segmentation tool to preprocess the text data and split it into meaningful fragments or vocabulary units.

[0055] Extraction of basic text features: The open source large language model Qwen2 is used to perform semantic analysis on the text to capture the deep meaning of the text. At the same time, the text data is also processed through the self-attention layer to capture the key semantic information in the text and its association with the image. The basic feature matrix of the text description is output.

[0056] Text vectorization: The output text features are represented in vector form, which is convenient for fusion with image features.

[0057] S1.3, image-text joint attention, inputs the image high-level semantic features generated in step S1.1 and the text high-level semantic features generated in step S1.2 into the image-text joint attention module. In the joint attention module, image features and text features interact with each other through a cross-attention mechanism. For image features, Q and K come from text features, and V comes from image features. The high-level semantic features of the final output image block are expressed as [I1, I2, ..., In]. This module can dynamically highlight key semantic features. When understanding the image content, it not only pays attention to the objects themselves in the image, but also pays attention to high-level semantic information such as text and events in the picture.

[0058] In step S2, the network structure of this method is optimized based on Transformer, and a cross attention module is introduced in the Transformer block in parallel with the multi-head self-attention module. The cross attention module is a cross between text and image; the adaptive sparse interactive attention adds an adaptive sparse mechanism to the cross attention, and the adaptive sparse interactive attention can dynamically decide when to add an interactive attention layer according to the characteristics of the input data.

[0059] S2.1, input sequence construction: The input sequence S is a sequence consisting of alternating text blocks and image blocks, in the form of S=[T1,I1,T2,I2,T3,I3], where Ti represents the i-th text segment and Ii represents the i-th image block.

[0060] First, the visual features extracted from each image block Ii are adjusted to the same dimension as the language model through linear projection. The projected features are represented as .

[0061] Then, a plain text representation is created for each image block as a placeholder so that the text sequence and the image sequence can be interleaved. For each image block Ii, a plain text representation Timg_i is created as a placeholder for the image Ii. The text sequence Stext=[T1,Timg_1,T2,Timg_2,T3,,Timg_3,…] is constructed, where Timg_i represents the position of the image block Ii.

[0062] S2.2, word embedding and position encoding: Input the text sequence Stext into the word embedding layer to obtain the corresponding text features Htext. Each text segment and image placeholder has a corresponding vector representation.

[0063] In order to accurately represent the original position of the image in the interleaved sequence, an image-text interleaved rotation position embedding is used. The position index i of the placeholder Timg_i of each image block Ii in the interleaved sequence Stext is pre-recorded.

[0064] For each image block Ii, the position encoding vector is calculated using the following formula ,

[0065]

[0066] Where N is the length of the interleaved sequence, i is the position index of the image block in the interleaved sequence, d is the dimension index of the position encoding vector, and is a frequency factor related to the dimension and is defined as:

[0067]

[0068] Where D is the total dimension of the position encoding vector.

[0069] For each image block in the image, a position encoding vector is generated, which reflects the two-dimensional coordinates of the image block in the image and the position in the interleaved sequence. All image blocks belonging to the same image share the same position encoding vector.

[0070] In step S3, S3.1 Image-text interaction attention: In the interactive understanding stage, the image-text interaction attention module is used to promote the mutual understanding and fusion between image features and text features. Specifically: the query (Q) comes from the text feature Htext; the key (K) and value (V) come from the visual feature Himg. A specific image weight matrix is ​​used To generate the keys and values ​​of the visual features, the matrix is ​​initialized using the weights of K and V from the language model. In this process, the text features look for relevant information by querying the keys of the visual features, and then update their representations based on the values ​​of this information.

[0071] S3.2, Proximal Visual Attention Mask: A Proximal Visual Attention Mask is used in the cross attention. For the text sequence Stext=[T1,Timg_1,T2,Timg_2,T3,,Timg_3,…], each text token can only pay attention to the visual features of the previous and next one. This mask ensures the directionality of the information flow, that is, the text token can only access the contextual information before it and the image information adjacent to it.

[0072] Specifically, create a zero matrix M whose size is the sequence length times the sequence length. For each text token Ti (i=1,2,3...), find its position index pos(i). If there is a previous image feature , then set If there is a following image feature , then set . A mask matrix M is applied in the attention mechanism to ensure that only allowed positions can be used for attention calculation. For the attention score matrix A, a mask matrix M is applied, that is, the positions in the matrix A corresponding to 0 in M ​​are set to a very small number to ensure that the attention scores of these positions are 0.

[0073] For example, for the interleaved sequence Seq=[T1,I1,T2,I2,T3,I3], the mask matrix M can be constructed as follows:

[0074]

[0075] T1 can only see I1;

[0076] T2 can see I1 and I2;

[0077] T3 can see I2 and I3.

[0078] S3.3, Adaptive Sparse Interactive Attention, designs an adaptive mechanism so that the model can dynamically decide when to add interactive attention layers based on the characteristics of the input data. For example, when a complex scene is detected, more interactive attention layers are automatically added.

[0079] The adaptive mechanism includes: A1, initial evaluation, input evaluation: Before the model starts processing the input data, it is first evaluated to determine its complexity. This is done by calculating the statistical characteristics of the input data (such as the degree of pixel variation, color distribution, edge density, etc.).

[0080] A2, Complexity threshold setting, Setting thresholds: Based on experience or observations on the training data set, set one or more complexity thresholds. For example, a threshold T can be set, and when the complexity of the input data exceeds this threshold, the condition of adding an interactive attention layer is triggered. Example threshold: Assume that it is found through experiments that when the edge density in the image exceeds a certain value (such as the number of edge pixels per square centimeter of the image area exceeds 10), it indicates that the scene is more complex. Threshold setting: T=10 edge pixels / square centimeter.

[0081] A3, Adaptive layer insertion, conditional check: Check whether the set complexity threshold is exceeded based on the result of the initial evaluation. Layer insertion: If the complexity of the input data exceeds the set threshold, an additional interactive attention layer is inserted into the model. This can be achieved through dynamic graph construction, that is, adjusting its structure as needed during the execution of the model.

[0082] A4, dynamic graph construction, dynamic adjustment: Use a dynamic graph construction framework (such as PyTorch's dynamic graph) to dynamically adjust the model structure based on the results of conditional checks. Modular design: Design the interactive attention layer as a modular component, which can be easily inserted or removed as needed.

[0083] After completing the interactive attention processing, a richer representation is generated by fusing text features and image features. The output of the interactive attention module is passed through a fully connected layer to further fuse the information of text and image.

[0084] S3.4, output representation, finally, after multi-layer interactive attention processing, a new feature representation that combines text and image information is obtained, namely the fused feature. It can be used for subsequent tasks, such as multi-image reasoning, video understanding, and multimodal question-answering system data input, and processed to obtain text describing the entire image sequence, the answer to a specific question, or a summary of the image sequence.

[0085] Embodiment 2: This embodiment provides a long image sequence understanding system, including:

[0086] The long image processing module is configured to: obtain a long image, extract the long image to obtain high-level semantic information of the image, and vectorize the long image to obtain an image vector;

[0087] The text description processing module is configured to: obtain the text description, extract the text description to obtain a basic feature matrix of the text description, and vectorize it to obtain a text vector;

[0088] The joint attention module is configured to: input the image vector and the text vector into the joint attention module, and output a sequence construction of image patches;

[0089] The image-text alternation module is configured to: process the sequence structure into an image-text alternation sequence to obtain a text feature Htext and a visual feature Himg with a position encoding vector; the image-text alternation sequence processing introduces image-text interleaved rotation position embedding to enhance understanding;

[0090] The graphic-text interaction attention module is configured as follows: the text feature Htext and the visual feature Himg are input into the graphic-text interaction attention module to obtain the fusion feature; the graphic-text interaction attention module introduces the neighboring visual attention mask to ensure the directionality of the information flow;

[0091] The output module is configured to use the fused features as input data for multi-image reasoning, video understanding, or multimodal question answering systems.

[0092] Embodiment 3, this embodiment provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a long image sequence understanding method described in Embodiment 1.

[0093] Embodiment 4. This embodiment provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute a long image sequence understanding method described in Embodiment 1.

[0094] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for understanding a long image sequence, characterized in that: include: Obtain a long image, extract the high-level semantic information of the long image, and vectorize it to obtain an image vector; Obtain the text description, extract the text description to obtain the basic feature matrix of the text description, and vectorize it to obtain the text vector; The image vector and text vector are fed into the joint attention module, and the sequence construction of the output image patches is constructed; The sequence structure is processed by alternating image and text sequences to obtain text features Htext and visual features Himg with position encoding vectors; image and text alternating sequence processing introduces image and text interleaved rotation position embedding to enhance understanding; Input the text feature Htext and the visual feature Himg into the graph-text interactive attention module to obtain the fusion feature; The graph-text interaction attention module introduces neighboring visual attention masks to ensure the directionality of information flow; Use the fused features as input data for multi-image reasoning, video understanding, or multimodal question answering systems; The specific steps of processing the sequence structure into an image-text alternating sequence include: Input sequence construction, the sequence construction is a sequence consisting of alternating text blocks and image blocks, in the form of S=[T1,I1,T2,I2,T3,I3], where Ti represents the i-th text segment and Ii represents the i-th image block; The visual features extracted from each image block Ii are adjusted to the same dimension as the language model through linear projection; the projected visual features are expressed as ; Create a plain text representation for each image block as a placeholder so that the text sequence and the image sequence can be interleaved to construct the text sequence Stext; Input the text sequence Stext into the word embedding layer to obtain the corresponding text feature Htext; Use the image-text interleaved rotation position embedding to pre-record the position index i of the placeholder Timg_i of each image block Ii in the interleaved sequence Stext. For each image block Ii, use the formula to calculate the position encoding vector; For each image block Ii, the position encoding vector is calculated using the following formula , Where N is the length of the interleaved sequence, i is the position index of the image block in the interleaved sequence, d is the dimension index of the position encoding vector, and is a frequency factor related to the dimension and is defined as: Where D is the total dimension of the position encoding vector; The specific steps of obtaining the fusion feature by the graphic-text interactive attention module include: The query Q of the image-text interaction attention module comes from the text feature Htext; the key K and value V come from the visual feature Himg; a specific image weight matrix is ​​used To generate the keys and values ​​of visual features, the image weight matrix Initialize using the weights of K and V from the language model; A neighboring visual attention mask is used in the cross attention. For the text sequence Stext=[T1,Timg_1,T2,Timg_2,T3,Timg_3,…], each text token can only access the previous context information and the adjacent image information; By designing an adaptive mechanism, the image-text interactive attention module dynamically determines the number of interactive attention layers according to the characteristics of the input data, and obtains the output result after processing the text feature Htext and the visual feature Himg through the interactive attention layers with the set number of layers; the adaptive mechanism includes: A1, initial evaluation, input evaluation: before the model starts to process the input data, the input data is first preliminarily evaluated to determine the complexity of the input data; A2, complexity threshold setting, setting threshold: based on experience or observations on the training data set, one or more complexity thresholds are set; A3, adaptive layer insertion, conditional check: check whether the set complexity threshold is exceeded according to the result of the initial evaluation; layer insertion: if the complexity of the input data exceeds the set threshold, insert an additional interactive attention layer into the model; A4, dynamic graph construction, dynamic adjustment: use the dynamic graph construction framework to dynamically adjust the model structure according to the result of the conditional check; The output of the interactive attention module is passed through a fully connected layer to further fuse the information of text and image to obtain fusion features.

2. A long image sequence understanding method as claimed in claim 1, characterized in that: The specific steps of extracting high-level semantic information of a long image include: extracting image features from the long image to obtain image blocks, extracting basic visual features from the image blocks to obtain basic visual features, refining high-level semantic features from the basic visual features to obtain high-level semantic information of the image, and vectorizing the high-level semantic information of the image to obtain an image vector.

3. A long image sequence understanding method as claimed in claim 1, characterized in that: The specific steps of extracting the basic feature matrix of the text description include: obtaining the text description, performing text preprocessing on the text description and extracting text features to obtain the basic feature matrix of the text description, and vectorizing the basic feature matrix to obtain a text vector.

4. A long image sequence understanding method as claimed in claim 1, characterized in that: The specific steps of inputting the image vector and the text vector into the common attention module include: in the common attention module, the image features and the text features interact with each other through the cross-attention mechanism. For the image features, Q and K come from the text features, and V comes from the image features. Finally, the high-level semantic features of the output image block are represented as a sequence structure [I1, I2, ..., In].

5. A long image sequence understanding system, characterized in that: include: The long image processing module is configured to: obtain a long image, extract the long image to obtain high-level semantic information of the image, and vectorize the long image to obtain an image vector; The text description processing module is configured to: obtain the text description, extract the text description to obtain a basic feature matrix of the text description, and vectorize it to obtain a text vector; The joint attention module is configured to: input the image vector and the text vector into the joint attention module, and output a sequence construction of image patches; The image-text alternation module is configured to: process the sequence structure into an image-text alternation sequence to obtain a text feature Htext and a visual feature Himg with a position encoding vector; the image-text alternation sequence processing introduces image-text interleaved rotation position embedding to enhance understanding; The graphic-text interaction attention module is configured to: input the text feature Htext and the visual feature Himg into the graphic-text interaction attention module to obtain a fusion feature; The graph-text interaction attention module introduces neighboring visual attention masks to ensure the directionality of information flow; The output module is configured to: use the fused features as input data for multi-image reasoning, video understanding, or a multimodal question answering system; The specific steps of processing the sequence structure into an image-text alternating sequence include: Input sequence construction, the sequence construction is a sequence consisting of alternating text blocks and image blocks, in the form of S=[T1,I1,T2,I2,T3,I3], where Ti represents the i-th text segment and Ii represents the i-th image block; The visual features extracted from each image block Ii are adjusted to the same dimension as the language model through linear projection; the projected visual features are expressed as ; Create a plain text representation for each image block as a placeholder so that the text sequence and the image sequence can be interleaved to construct the text sequence Stext; Input the text sequence Stext into the word embedding layer to obtain the corresponding text feature Htext; Use the image-text interleaved rotation position embedding to pre-record the position index i of the placeholder Timg_i of each image block Ii in the interleaved sequence Stext. For each image block Ii, use the formula to calculate the position encoding vector; For each image block Ii, the position encoding vector is calculated using the following formula , Where N is the length of the interleaved sequence, i is the position index of the image block in the interleaved sequence, d is the dimension index of the position encoding vector, and is a frequency factor related to the dimension and is defined as: Where D is the total dimension of the position encoding vector; The specific steps of obtaining the fusion feature by the graphic-text interactive attention module include: The query Q of the image-text interaction attention module comes from the text feature Htext; the key K and value V come from the visual feature Himg; a specific image weight matrix is ​​used To generate the keys and values ​​of visual features, the image weight matrix Initialize using the weights of K and V from the language model; A neighboring visual attention mask is used in the cross attention. For the text sequence Stext=[T1,Timg_1,T2,Timg_2,T3,Timg_3,…], each text token can only access the previous context information and the adjacent image information; By designing an adaptive mechanism, the image-text interactive attention module dynamically determines the number of interactive attention layers according to the characteristics of the input data, and obtains the output result after processing the text feature Htext and the visual feature Himg through the interactive attention layers with the set number of layers; the adaptive mechanism includes: A1, initial evaluation, input evaluation: before the model starts to process the input data, the input data is first preliminarily evaluated to determine the complexity of the input data; A2, complexity threshold setting, setting threshold: based on experience or observations on the training data set, one or more complexity thresholds are set; A3, adaptive layer insertion, conditional check: check whether the set complexity threshold is exceeded according to the result of the initial evaluation; layer insertion: if the complexity of the input data exceeds the set threshold, insert an additional interactive attention layer into the model; A4, dynamic graph construction, dynamic adjustment: use the dynamic graph construction framework to dynamically adjust the model structure according to the result of the conditional check; The output of the interactive attention module is passed through a fully connected layer to further fuse the information of text and image to obtain fusion features.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute a long image sequence understanding method as described in any one of claims 1-4.

7. An electronic device, characterized in that: It comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute a long image sequence understanding method as described in any one of claims 1-4.

Images