Image and text processing method, image and text processing model training method and electronic device

Through the method of aggregating and deleting image tokens, the problem of high cost of multimodal large model calculation is solved, the efficiency and accuracy of graphics and text processing are improved, and the application of models in different industries is promoted.

CN118537683BActive Publication Date: 2024-11-15ALIBABA CLOUD FEITIAN (HANGZHOU) CLOUD COMPUTING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410997542.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-15
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

Multimodal large models are costly to process cross-modal data, resulting in low training and inference efficiency and difficult to achieve a balance of performance and efficiency.

Method used

By acquiring the pending images and text, aggregate the multi-layer attention weight of the image token, deletion of unnecessary image tokens, and combining large language models for graphics and text processing, reducing the number of image tokens, improving the robustness of the model and Processing speed.

Benefits of technology

The training and inference acceleration of multimodal large models are realized, the speed and accuracy of graphics and text processing are improved, the calculation costs are reduced, and the application of models in different industries is promoted.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118537683B_ABST
    Figure CN118537683B_ABST
Patent Text Reader

Abstract

The present application provides a method for image and text processing, a training method for an image and text processing model, and an electronic device, which relate to the field of artificial intelligence technology, including: obtaining an image to be processed and a text to be processed corresponding to the image to be processed; aggregating the attention weights of multiple layers corresponding to multiple image tokens of the image to be processed to obtain the weights after aggregation corresponding to the multiple image tokens; deleting multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens; and obtaining the image and text processing results based on the remaining image tokens, multiple first text tokens of the text to be processed, and a large language model. In this embodiment, multiple layers of attention weights are aggregated, and image tokens are deleted according to the weights after aggregation, thereby reducing the number of image tokens. By aggregating multiple layers of attention weights, the selective attention of image tokens can be made more robust, which can improve the speed of image and text processing while ensuring the accuracy of the results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for processing images and texts, a method for training an image and text processing model, and an electronic device. Background Art

[0002] Multimodal large models (e.g., Vision Language Models, VLMs) are a type of artificial intelligence model that can process and integrate information from multiple different modalities (such as text, images, audio, etc.). These models can perform complex tasks in multiple fields, such as image description, video understanding, cross-modal retrieval, etc., by learning the intrinsic relationships and representations of different types of data. Due to their powerful ability to process cross-modal data, multimodal large models have attracted widespread attention from academia and industry.

[0003] However, the computational cost brought by the increasing model size and the huge number of image tokens has seriously hindered the downstream application of multimodal large models. In related technologies, the amount of computation in the training process of multimodal large models is reduced by training only a small number of introduced parameters. However, the introduction of additional parameters increases the inference cost. Therefore, how to achieve a balance between the performance and efficiency of multimodal large models has become a problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method for image and text processing, a method for training an image and text processing model, and an electronic device to accelerate the training and reasoning of multimodal large models and promote the application of multimodal large models in different industries.

[0005] In a first aspect, an embodiment of the present application provides a method for image and text processing, the method comprising: obtaining an image to be processed and a text to be processed corresponding to the image to be processed; aggregating the attention weights of multiple layers corresponding to multiple image tokens of the image to be processed to obtain the weights after aggregation corresponding to the multiple image tokens; deleting the multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens to obtain the remaining image tokens; obtaining the image and text processing result based on the remaining image tokens, multiple first text tokens of the text to be processed and a large language model.

[0006] In a second aspect, an embodiment of the present application provides a training method for an image-text processing model, wherein the image-text processing model includes a visual encoder, a text encoder, and a large language model, wherein the method includes: the visual encoder obtains a sample image, and aggregates the attention weights of multiple layers corresponding to multiple image tokens of the sample image to obtain the weights after aggregation corresponding to the multiple image tokens, and deletes the multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens to obtain the remaining image tokens; the text encoder obtains multiple first text tokens of the sample text corresponding to the sample image; and the image-text processing model is trained based on the remaining image tokens, the multiple first text tokens, and the large language model.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, any of the above methods is implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements any of the above methods.

[0010] Compared with the prior art, this application has the following advantages:

[0011] The present application provides a method for image and text processing, which obtains an image to be processed and a text to be processed corresponding to the image to be processed; aggregates the attention weights of multiple layers corresponding to multiple image tokens of the image to be processed to obtain the weights after aggregation corresponding to the multiple image tokens; deletes the multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens to obtain the remaining image tokens; obtains the image and text processing results based on the remaining image tokens, multiple first text tokens of the text to be processed and a large language model. In this embodiment, the attention weights of multiple layers corresponding to multiple image tokens of the image to be processed are aggregated, and the multiple image tokens are deleted according to the weights after aggregation, thereby reducing the number of image tokens. By aggregating the attention weights of multiple layers, the selective attention of the image tokens can be made more robust, which can improve the speed of image and text processing while ensuring the accuracy of the results.

[0012] The present application provides a training method for an image-text processing model, which includes a visual encoder, a text encoder, and a large language model. The visual encoder obtains a sample image, and aggregates the attention weights of multiple layers corresponding to multiple image tokens of the sample image to obtain the weights after aggregation corresponding to the multiple image tokens, and deletes the multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens to obtain the remaining image tokens; the text encoder obtains multiple first text tokens of the sample text corresponding to the sample image; and the image-text processing model is trained based on the remaining image tokens, the multiple first text tokens, and the large language model. In this embodiment, the attention weights of multiple layers corresponding to multiple image tokens of the sample image are aggregated, and the multiple image tokens are deleted according to the weights after aggregation, thereby reducing the number of image tokens. By aggregating the attention weights of multiple layers, the selective attention of the image tokens can be made more robust, while improving the training speed and ensuring the accuracy of the results.

[0013] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0015] Figure 1 A schematic diagram of an application scenario of the image and text processing method provided in this application.

[0016] Figure 2 FIG. 1 is a schematic diagram of a GTP module according to an embodiment of the present application.

[0017] Figure 3 This is a flowchart of a method for processing images and texts according to an embodiment of the present application.

[0018] Figure 4 This is a flowchart of a method for processing images and texts according to an embodiment of the present application.

[0019] Figure 5 This is a flowchart of a method for training a graphic and text processing model according to an embodiment of the present application.

[0020] Figure 6 This is a structural block diagram of a graphic processing device according to an embodiment of the present application.

[0021] Figure 7 This is a structural block diagram of a training device for a graphic and text processing model according to an embodiment of the present application.

[0022] Figure 8 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0024] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0025] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] Figure 1 A schematic diagram of an application scenario of the image and text processing method provided in this application. Figure 1 As shown, get the image and text to be processed, such as Figure 1 As shown in the figure, the image to be processed is a photo of a beer bottle, and the text to be processed can be a question, for example, “What kind of beer is this?” The text to be processed can also include text obtained by performing optical character recognition (OCR) on the image to be processed: “NINK, NK, BOWING, CC, STON, SUE, SELF, KGy, ALE”. The image to be processed is processed by a visual encoder to obtain multiple image tokens (such as Figure 1The image token includes a patch token and a classification token (CLS token). The visual encoder includes a Group-wise Token Pruning (GTP) module. The GTP module is used to prune the patch tokens in the image token to obtain the pruned image token, including the classification token and the remaining patch tokens.

[0027] The GTP module that performs image token deletion is as follows Figure 2 The GTP module is inserted between the attention module and the feed forward network of any Transformer layer, where the Transformer layer is implemented based on the Transformer model, as shown in Figure 2 The "converter model" shown in . Multiple image tokens are processed by the attention module, the sequence output by the attention module is fused with the input image token, and the intermediate result value matrix Value status (such as Figure 2 ”A1” shown in ) and the attention map Attention map (as shown in Figure 2 The “A5” shown in , that is, the aggregated attention weight) is multiplied to obtain the importance score, and then sorted to obtain the sorted importance score CLS Score (such as Figure 2 According to the importance score, the image token (here refers to the patch token, such as Figure 2 , select the image tokens with higher scores, delete the image tokens with lower scores, and then Figure 2 The output of the feedforward network ("A4" shown in the figure) is passed through a feedforward network, the output of the feedforward network is added to the selected image token, and then the classification token is added as the final output result.

[0028] Among them, the aggregated attention weight is obtained by aggregating the attention weight of the current Transformer layer and the attention weight of the Transformer layer before the current layer.

[0029] The calculation formula for the attention weight of each Transformer layer is as follows:

[0030] (1)

[0031] in, Q Represents the query matrix Query , K Represents the key matrix Key , Trepresents the transpose of a matrix, d represents the embedding feature dimension, A mask represents the attention mask, N Indicates the number of patch tokens.

[0032] The attention weights are aggregated using the attention rollout mechanism. In one example, the aggregation of attention weights is calculated by recursive multiplication, and the calculation formula is as follows:

[0033] (2)

[0034] in, represents the attention weight after aggregation, l 1 represents the first Transformer layer, l 2 represents the last Transformer layer, l Represents the current Transformer layer, I Represents the identity matrix. represents the attention weight of the first Transformer layer, represents the attention weight of the current Transformer layer, Represents the aggregated attention weights of the Transformer layer before the current Transformer layer.

[0035] For example, if the current Transformer layer is the first layer, the attention weight of the current layer is used as the aggregation result; if the current Transformer layer is the second layer, the attention weights of the first and second layers are aggregated. If the current Transformer layer is the third layer, the attention weights of the first, second, and third layers are aggregated.

[0036] The importance score is obtained by multiplying the aggregated attention weight by the value matrix (value status), and the calculation formula is as follows:

[0037] (3)

[0038] in, S i Indicates i The importance score of each patch token, Indicates i The aggregated attention weights corresponding to the patch tokens, i.e., the classification tokens for the i The level of attention of patch tokens, j Represents a value from 0 to N, N represents the number of patch tokens, Vi Indicates i The value matrix corresponding to the patch tokens, V j Indicates j The value matrix corresponding to the patch tokens, N Indicates the number of patch tokens.

[0039] The text to be processed is input into the text segmenter (i.e., CLIP Text Tokenizer) of the Contrastive Language-Image Pre-Training (CLIP) model and the text segmenter (i.e., LLM Text Tokenizer) of the Large Language Model (LLM) respectively. A mapping relationship is established between the multiple second text tokens output by the CLIP Text Tokenizer and the multiple third text tokens output by the LLM Text Tokenizer, and the text features output by the text encoder (i.e., CLIP Text Encoder) of the image-text correlation matching model are matched with the deleted image tokens (for example, similarity is calculated), and the image tokens that do not match the text features are deleted to obtain the remaining image tokens. The text features output by the CLIP Text Encoder correspond one-to-one to the values ​​of each dimension of the second text token output by the CLIP Text Tokenizer. According to the mapping relationship between the multiple second text tokens and the multiple third text tokens, the multiple text features (such as Figure 1 ” T2 ”) is mapped into a plurality of third text tokens (such as Figure 1 ), obtain multiple third text tokens matching the pruned image token, aggregate the multiple third text tokens matching the pruned image token, and obtain a text classification token (ie, PseudoCLS).

[0040] In order to ensure the semantic alignment of image-text information, CLIP Text Tokenizer and CLIP TextEncoder are introduced into the multimodal large model. According to the mapping relationship between the text tokens output by CLIP Text Tokenizer and LLM Text Tokenizer respectively, the third text token that will match the deleted image token is obtained. Then the third text tokens matching the image token are aggregated to obtain pseudo-classification tokens, i.e., text classification tokens. The text classification tokens are used to characterize the semantic information of the text to be processed. In this way, although the text tokens output by LLM Text Tokenizer do not contain semantic information, semantic information is obtained by aggregating the third text tokens matching the image tokens, so that the GTP module can be extended to LLM.

[0041] Encode the remaining image tokens to obtain image coding features; use a mapping layer (which can be implemented by a multi-layer perceptron) to map the image coding features to the text space to obtain mapped features; encode multiple first text tokens (third text tokens and text classification tokens) to obtain text coding features; concatenate the mapped features and the text coding features, and input the concatenated features into the large language model (first part) and the large language model (second part) to obtain the image and text processing results.

[0042] The large language model (part 1) includes multiple Transformer layers, in which the GTP module is inserted for deleting image tokens. The large language model (part 2) includes multiple Transformer layers, and the large language model (part 2) no longer prunes image tokens. The specific number of layers of the first and second parts can be determined according to specific needs. For example, the first part includes 12 Transformer layers, and image tokens are deleted every 3 layers, for a total of 4 times. The second part includes 20 Transformer layers, which are not deleted.

[0043] LLM also includes a fully connected layer (not shown in the figure), and finally outputs the image and text processing result. The image and text processing result is the probability corresponding to each character in the character library. According to the image and text processing result, the answer corresponding to the question information "What kind of beer is this?" in the text to be processed is obtained.

[0044] In this embodiment, the attention weights of multiple levels are aggregated, and the importance scores are determined by using the aggregated attention weights. The image tokens are deleted according to the importance scores. By aggregating the attention weights of multiple levels, the selective attention of the image tokens can be made more robust. According to the matching results with the text information, the image tokens are deleted again to further reduce the number of image tokens, and the image tokens related to the text information are retained, so as to improve the image and text processing speed while ensuring the accuracy of the processing results.

[0045] The present application embodiment provides a method for image and text processing. The method in this embodiment can be applied to a computing device, which may include: a server, a user terminal, etc. The image and text processing method in this embodiment uses an image and text processing model to process images and text. The image and text processing model is a multimodal large model, specifically including a visual encoder, a text encoder, and a large language model. Figure 3 The flowchart of the image and text processing method of one embodiment of the present application is shown, including:

[0046] Step S301, obtaining an image to be processed and a text to be processed corresponding to the image to be processed.

[0047] The image to be processed may be any image input by a user, or may be any image acquired from a current terminal device or a network.

[0048] The text to be processed corresponding to the image to be processed may be question information raised for the image to be processed. Optionally, in addition to the question information, the text to be processed may also include text recognized for the image to be processed, or other text related to the image to be processed.

[0049] Step S302, aggregate the attention weights of the multiple layers corresponding to the multiple image tokens of the image to be processed to obtain the aggregated weights corresponding to the multiple image tokens.

[0050] The image to be processed is divided into multiple non-overlapping blocks using a visual encoder, and each block is embedded to obtain multiple image tokens, which include patch tokens and classification tokens.

[0051] The attention weight is the token attention between the patch token and the classification token, which is used to determine their importance in the visual encoder. The attention module in the Transformer model is used to obtain the attention maps of multiple levels corresponding to each image token as the attention weight, and the attention weights of multiple levels are aggregated to obtain the aggregated weight corresponding to each image token.

[0052] Each level may be a Transformer layer obtained based on the Transformer model.

[0053] Step S303 , deleting the plurality of image tokens according to the weights after aggregation corresponding to the plurality of image tokens, to obtain remaining image tokens.

[0054] The visual encoder prunes the patch tokens in the image tokens according to the aggregated weights corresponding to each image token, and obtains the classification tokens and the pruned patch tokens as the remaining image tokens.

[0055] Step S304, obtaining the image and text processing result based on the remaining image tokens, the plurality of first text tokens of the text to be processed and the large language model.

[0056] The text to be processed is subjected to word segmentation and embedding processing by using a text encoder to obtain a plurality of first text tokens.

[0057] The large language model is used to process the remaining image tokens and the multiple first text tokens of the text to be processed, and the image-text processing results are output. According to the image-text processing results, answer information corresponding to the question information can be obtained.

[0058] The image and text processing method provided in the embodiment of the present application obtains the image to be processed and the text to be processed corresponding to the image to be processed; aggregates the attention weights of the multiple layers corresponding to the multiple image tokens of the image to be processed to obtain the weights after aggregation corresponding to the multiple image tokens; deletes the multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens to obtain the remaining image tokens; obtains the image and text processing results based on the remaining image tokens, the multiple first text tokens of the text to be processed and the large language model. In this embodiment, the attention weights of the multiple layers corresponding to the multiple image tokens of the image to be processed are aggregated, and the multiple image tokens are deleted according to the weights after aggregation, thereby reducing the number of image tokens. By aggregating the attention weights of the multiple layers, the selective attention of the image tokens can be made more robust, which can improve the speed of image and text processing while ensuring the accuracy of the results.

[0059] The following introduces the specific process of each of the above steps through multiple implementation methods:

[0060] In one implementation, step S302 aggregates the attention weights of multiple layers corresponding to multiple image tokens of the image to be processed to obtain aggregated weights corresponding to the multiple image tokens, including: aggregating the attention weights of the current layer corresponding to the multiple image tokens and the attention weights of a preset number of layers before the current layer to obtain aggregated weights corresponding to the multiple image tokens.

[0061] The attention module in the Transformer model is used to obtain multiple levels of attention maps corresponding to each image token as attention weights, and the attention weights of multiple levels are aggregated to obtain the aggregated weights corresponding to each image token. Each level can be each Transformer layer obtained based on the Transformer model. The preset number of layers before the current layer can be at least one continuous Transformer layer before the current layer, and the preset number can be configured according to specific needs.

[0062] For example, if the current Transformer layer is the first layer, the attention weight of the current layer is used as the aggregation result; if the current Transformer layer is the second layer, the attention weights of the first and second layers are aggregated. If the current Transformer layer is the third layer, the attention weights of the first, second, and third layers are aggregated. If the current Transformer layer is the fourth layer, the attention weights of the second, third, and fourth layers are aggregated.

[0063] In this embodiment, by aggregating the attention weights of multiple layers, the selective attention of the image token can be made more robust.

[0064] In one implementation, step S303, based on the weights after aggregation respectively corresponding to the multiple image tokens, multiple image tokens are deleted to obtain the remaining image tokens, including: based on the weights after aggregation respectively corresponding to the multiple image tokens, importance scores respectively corresponding to the multiple image tokens are determined; based on the importance scores respectively corresponding to the multiple image tokens, multiple image tokens are deleted to obtain the remaining image tokens.

[0065] In practical applications, the aggregated weights can be multiplied by the value matrix to obtain the importance score of each image token. According to the importance score, image tokens with lower importance are deleted to reduce the number of image tokens.

[0066] In this embodiment, image tokens are deleted based on multi-level attention weight aggregation. Compared with single-layer attention weight, the inter-layer transmission of attention information is taken into account and it is more stable.

[0067] In one implementation, multiple image tokens are pruned according to importance scores corresponding to the multiple image tokens to obtain remaining image tokens, including: pruned according to importance scores corresponding to the multiple image tokens to obtain pruned image tokens; semantically aligning the pruned image tokens with multiple text features of the text to be processed to obtain semantic alignment results, wherein the text features are obtained by processing the text to be processed using a text encoder of a graph-text correlation matching model; and pruned again according to the semantic alignment results to obtain remaining image tokens.

[0068] Optionally, a text encoder of a graph-text correlation matching model is used to encode the text to be processed to obtain multiple text features.

[0069] First, the image tokens are pruned according to their corresponding importance scores, and then the pruned image tokens are semantically aligned with multiple text features to obtain image tokens related to the text to be processed and pruned irrelevant image tokens, thereby further reducing the number of image tokens.

[0070] The large language model in the embodiment of the present application is a large language model of only a decoder, which does not contain classification tags representing text prompts, and lacks semantic alignment between images and text. CLIP's text encoder is used to more specifically select image tokens related to the text to be processed, ensuring that the fine image token filtering is consistent with the context of the text to be processed.

[0071] In one implementation, semantic alignment is performed on the deleted image tokens and multiple text features of the text to be processed to obtain a semantic alignment result, including: matching calculation is performed on the deleted image tokens and multiple text features to obtain a semantic alignment result.

[0072] In practical applications, matching calculation can be performed by calculating similarity, and image tokens whose similarity exceeds a preset similarity threshold are used as image tokens that match text features.

[0073] In one implementation, based on the semantic alignment results, the pruned image tokens are pruned again to obtain remaining image tokens, including: based on the semantic alignment results, image tokens that do not match multiple text features are determined in the pruned image tokens and pruned to obtain remaining image tokens.

[0074] The semantic alignment result includes the similarity between the deleted image tokens and multiple text features. The image tokens whose similarity with the text features does not exceed a preset similarity threshold are deleted, and the image tokens with higher similarity with the text features are retained as the remaining image tokens.

[0075] In one implementation, the method further includes: establishing a mapping relationship between multiple second text tokens and multiple third text tokens, the second text tokens are obtained by processing the text to be processed using a text segmenter of a picture-text correlation matching model, and the third text tokens are obtained by processing the text to be processed using a text segmenter of a large language model; according to the mapping relationship, mapping multiple text features that match the deleted image tokens to multiple third text tokens to obtain multiple third text tokens that match the deleted image tokens; performing aggregation calculation on the matched multiple third text tokens to obtain text classification tokens; the multiple first text tokens include the matched multiple third text tokens and text classification tokens, and the text classification tokens are used to characterize the semantic information of the text to be processed.

[0076] The text features of the text to be processed correspond one-to-one to the numerical values ​​of each dimension of the second text token. According to the mapping relationship between multiple second text tokens and multiple third text tokens, multiple text features matching the deleted image tokens are mapped to multiple third text tokens to obtain multiple third text tokens matching the deleted image tokens.

[0077] The aggregation calculation may include any aggregation method, for example, calculating an average value of a plurality of third text tokens that match the deleted image token as a text classification token.

[0078] In this embodiment, through the mapping relationship between the second text token and the third text token, the third text token can be associated with the image token, thereby improving the association strength between the image and the text and improving the accuracy of the image and text processing results.

[0079] In one implementation, based on the remaining image tokens, multiple first text tokens of the text to be processed and a large language model, an image-text processing result is obtained, including: encoding the remaining image tokens to obtain image encoding features; mapping the image encoding features to the text space to obtain mapped features; encoding multiple first text tokens to obtain text encoding features; splicing the mapped features and the text encoding features, and inputting the spliced ​​features into the large language model to obtain the image-text processing result.

[0080] The image features are mapped to the text space using a mapping layer (for example, a multi-layer perceptron), and then the mapped features and text features are concatenated and input into the large language model for processing. The image-text processing result can be the probability corresponding to each character in the character library, and the answer corresponding to the question information in the text to be processed is obtained according to the image-text processing result.

[0081] The present application embodiment provides a method for processing graphics and text. The method in the present embodiment can be applied to a computing device, which may include a server, a user terminal, etc. Figure 4 The flowchart of the image and text processing method of one embodiment of the present application is shown, including:

[0082] Step S401, obtaining an image to be processed and a text to be processed corresponding to the image to be processed.

[0083] The image to be processed may be any image input by a user, or may be any image acquired from a current terminal device or a network.

[0084] The text to be processed corresponding to the image to be processed may be question information raised for the image to be processed. Optionally, in addition to the question information, the text to be processed may also include text recognized for the image to be processed, or other text related to the image to be processed.

[0085] Step S402, aggregate the attention weights of the current layer corresponding to the multiple image tokens and the attention weights of a preset number of layers before the current layer to obtain the aggregated weights corresponding to the multiple image tokens.

[0086] The image to be processed is divided into multiple non-overlapping blocks using a visual encoder, and each block is embedded to obtain multiple image tokens, which include patch tokens and classification tokens.

[0087] The attention weight is the token attention between the patch token and the classification token, which is used to determine their importance in the visual encoder. The attention module in the Transformer model is used to obtain the attention maps of multiple levels corresponding to each image token as the attention weight, and the attention weights of multiple levels are aggregated to obtain the aggregated weight corresponding to each image token.

[0088] Among them, each level can be a Transformer layer obtained based on the Transformer model.

[0089] Step S403, determining importance scores corresponding to the multiple image tokens according to the aggregated weights corresponding to the multiple image tokens; and deleting the multiple image tokens according to the importance scores corresponding to the multiple image tokens.

[0090] In practical applications, the weight after aggregation can be multiplied by the value matrix to obtain the importance score of each image token. According to the importance score, the image tokens with lower importance are deleted to reduce the number of image tokens. Based on the multi-level attention weight aggregation, the image tokens are deleted. Compared with the single-layer attention weight, the inter-layer transmission of attention information is considered and it is more stable.

[0091] Step S404, semantically aligning the deleted image tokens with multiple text features of the text to be processed to obtain a semantic alignment result, where the text features are obtained by using a text encoder of the image-text correlation matching model; based on the semantic alignment result, the deleted image tokens are deleted again to obtain the remaining image tokens.

[0092] The text to be processed is encoded using a text encoder of the image-text correlation matching model to obtain multiple text features. The deleted image tokens are semantically aligned with multiple text features to obtain image tokens related to the text to be processed, and irrelevant image tokens are deleted, thereby further reducing the number of image tokens.

[0093] Step S405, establish a mapping relationship between multiple second text tokens and multiple third text tokens, the second text tokens are obtained by using the text segmenter of the image-text correlation matching model, and the third text tokens are obtained by using the text segmenter of the large language model; according to the mapping relationship, map multiple text features that match the deleted image tokens to multiple third text tokens to obtain multiple third text tokens that match the deleted image tokens; aggregate the matched multiple third text tokens to obtain text classification tokens; the multiple first text tokens include the matched multiple third text tokens and the text classification tokens.

[0094] The text classification token is used to represent the semantic information of the text to be processed. The aggregation calculation may include any aggregation method, for example, calculating the average value of multiple third text tokens that match the deleted image token as the text classification token.

[0095] Step S406, encode the remaining image tokens to obtain image coding features; map the image coding features to the text space to obtain mapped features; encode multiple first text tokens to obtain text coding features; concatenate the mapped features and the text coding features, input the concatenated features into the large language model, and obtain the image and text processing results.

[0096] The image encoding features are mapped to the text space using a mapping layer (for example, a multi-layer perceptron), and then the mapped features and the text encoding features are concatenated and input into the large language model for processing. The image-text processing result can be the probability corresponding to each character in the character library, and the answer corresponding to the question information in the text to be processed is obtained according to the image-text processing result.

[0097] In this embodiment, the attention weights of multiple levels are aggregated, and the importance scores are determined by using the aggregated attention weights. The image tokens are deleted according to the importance scores. By aggregating the attention weights of multiple levels, the selective attention of the image tokens can be made more robust. According to the matching results with the text information, the image tokens are deleted again to further reduce the number of image tokens, and the image tokens related to the text information are retained, so as to improve the image and text processing speed while ensuring the accuracy of the processing results.

[0098] The present application embodiment provides a training method for a graph-text processing model. The method in this embodiment can be applied to a computing device, which may include: a server, a user terminal, etc. The graph-text processing model includes a visual encoder, a text encoder, and a large language model, such as Figure 5 The flowchart of the training method of the image and text processing model according to one embodiment of the present application is shown, including:

[0099] In step S501, the visual encoder obtains a sample image and aggregates the multi-layer attention weights corresponding to multiple image tokens of the sample image to obtain the aggregated weights corresponding to the multiple image tokens. According to the aggregated weights corresponding to the multiple image tokens, the multiple image tokens are deleted to obtain the remaining image tokens.

[0100] Step S502: the text encoder obtains a plurality of first text tokens of the sample text corresponding to the sample image.

[0101] Step S503: training a graphic-text processing model based on the remaining image tokens, the plurality of first text tokens and the large language model.

[0102] In this embodiment, the training samples of the image-text processing model include sample images and sample texts corresponding to the sample images. The sample texts may include question information about the sample images, sample labels, or other texts related to the sample images.

[0103] A visual encoder is used to divide the sample image into multiple non-overlapping blocks, and each block is embedded to obtain multiple image tokens, which include patch tokens and classification tokens.

[0104] The attention weight is the token attention between the patch token and the classification token, which is used to determine their importance in the visual encoder. The attention module in the Transformer model is used to obtain the attention maps of multiple levels corresponding to each image token as the attention weight, and the attention weights of multiple levels are aggregated to obtain the aggregated weight corresponding to each image token. Each level can be each Transformer layer obtained based on the Transformer model.

[0105] According to the weight after aggregation corresponding to each image token, the patch tokens in the image tokens are deleted to obtain classification tokens and deleted patch tokens. The sample text is segmented and embedded using a text encoder to obtain multiple first text tokens. The remaining image tokens and multiple first text tokens are used as training data to train the image-text processing model until the preset training termination condition is met, and the training is terminated to obtain a trained image-text processing model.

[0106] The training method of the image and text processing model provided in the embodiment of the present application includes a visual encoder, a text encoder and a large language model. The visual encoder obtains a sample image, and aggregates the attention weights of the multiple layers corresponding to the multiple image tokens of the sample image to obtain the weights after aggregation corresponding to the multiple image tokens. According to the weights after aggregation corresponding to the multiple image tokens, the multiple image tokens are deleted to obtain the remaining image tokens; the text encoder obtains the multiple first text tokens of the sample text corresponding to the sample image; based on the remaining image tokens, the multiple first text tokens and the large language model, the image and text processing model is trained. In this embodiment, the attention weights of the multiple layers corresponding to the multiple image tokens of the sample image are aggregated, and according to the weights after aggregation, the multiple image tokens are deleted, thereby reducing the number of image tokens. By aggregating the attention weights of the multiple layers, the selective attention of the image tokens can be made more robust, while improving the training speed and ensuring the accuracy of the results.

[0107] The following introduces the specific process of each of the above steps through multiple implementation methods:

[0108] In one implementation, the attention weights of multiple layers corresponding to multiple image tokens of a sample image are aggregated to obtain the aggregated weights corresponding to the multiple image tokens, including: the visual encoder aggregates the attention weights of the current layer corresponding to the multiple image tokens and the attention weights of a preset number of layers before the current layer to obtain the aggregated weights corresponding to the multiple image tokens.

[0109] The attention module in the Transformer model is used to obtain multiple levels of attention maps corresponding to each image token as attention weights, and the attention weights of multiple levels are aggregated to obtain the aggregated weights corresponding to each image token. Each level can be each Transformer layer obtained based on the Transformer model. The preset number of layers before the current layer can be at least one continuous Transformer layer before the current layer, and the preset number can be configured according to specific needs.

[0110] For example, if the current Transformer layer is the first layer, the attention weight of the current layer is used as the aggregation result; if the current Transformer layer is the second layer, the attention weights of the first and second layers are aggregated. If the current Transformer layer is the third layer, the attention weights of the first, second, and third layers are aggregated. If the current Transformer layer is the fourth layer, the attention weights of the second, third, and fourth layers are aggregated.

[0111] In this embodiment, by aggregating the attention weights of multiple layers, the selective attention of the image token can be made more robust.

[0112] In one implementation, multiple image tokens are pruned based on their respective weights after aggregation to obtain remaining image tokens, including: determining importance scores corresponding to the multiple image tokens based on their respective weights after aggregation; and pruned based on their respective importance scores to obtain remaining image tokens.

[0113] In practical applications, the aggregated weights can be multiplied by the value matrix to obtain the importance score of each image token. According to the importance score, image tokens with lower importance are deleted to reduce the number of image tokens.

[0114] In this embodiment, image tokens are deleted based on multi-level attention weight aggregation. Compared with single-layer attention weight, the inter-layer transmission of attention information is taken into account and it is more stable.

[0115] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a graphic processing device. Figure 6 The figure is a structural block diagram of a graphic processing device according to an embodiment of the present application, the device comprising:

[0116] The acquisition module 601 is used to acquire the image to be processed and the text to be processed corresponding to the image to be processed.

[0117] Aggregation module 602 is used to aggregate the attention weights of multiple layers corresponding to multiple image tokens of the image to be processed, so as to obtain the aggregated weights corresponding to the multiple image tokens.

[0118] The deletion module 603 is used to delete the multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens to obtain the remaining image tokens.

[0119] The processing module 604 is used to obtain the image-text processing result based on the remaining image tokens, the multiple first text tokens of the text to be processed and the large language model.

[0120] The image and text processing device provided in the embodiment of the present application obtains the image to be processed and the text to be processed corresponding to the image to be processed; aggregates the attention weights of the multiple layers corresponding to the multiple image tokens of the image to be processed to obtain the weights after aggregation corresponding to the multiple image tokens; deletes the multiple image tokens according to the weights after aggregation corresponding to the multiple image tokens to obtain the remaining image tokens; obtains the image and text processing results based on the remaining image tokens, the multiple first text tokens of the text to be processed and the large language model. In this embodiment, the attention weights of the multiple layers corresponding to the multiple image tokens of the image to be processed are aggregated, and the multiple image tokens are deleted according to the weights after aggregation, thereby reducing the number of image tokens. By aggregating the attention weights of the multiple layers, the selective attention of the image tokens can be made more robust, which can improve the speed of image and text processing while ensuring the accuracy of the results.

[0121] In one implementation, the aggregation module 602 is used to aggregate the attention weights of the current layer corresponding to the multiple image tokens and the attention weights of a preset number of layers before the current layer to obtain the aggregated weights corresponding to the multiple image tokens.

[0122] In one implementation, the deletion module 603 is used to: determine the importance scores corresponding to the multiple image tokens according to the aggregated weights corresponding to the multiple image tokens; and delete the multiple image tokens according to the importance scores corresponding to the multiple image tokens to obtain the remaining image tokens.

[0123] In one implementation, when the deletion module 603 deletes multiple image tokens according to the importance scores respectively corresponding to the multiple image tokens to obtain the remaining image tokens, it is used to: delete the multiple image tokens according to the importance scores respectively corresponding to the multiple image tokens to obtain deleted image tokens; semantically align the deleted image tokens with multiple text features of the text to be processed to obtain a semantic alignment result, where the text features are obtained by processing the text to be processed using a text encoder of a graphic-text correlation matching model; and delete the deleted image tokens again according to the semantic alignment result to obtain the remaining image tokens.

[0124] In one implementation, the semantic alignment module, when semantically aligning the deleted image tokens with multiple text features of the text to be processed to obtain the semantic alignment result, is used to: match and calculate the deleted image tokens with the multiple text features to obtain the semantic alignment result.

[0125] In one implementation, the semantic alignment module, when further deleting the deleted image tokens based on the semantic alignment results to obtain the remaining image tokens, is used to: determine the image tokens that do not match the multiple text features in the deleted image tokens based on the semantic alignment results and delete them to obtain the remaining image tokens.

[0126] In one implementation, the device also includes a matching module, which is used to: establish a mapping relationship between multiple second text tokens and multiple third text tokens, the second text tokens are obtained by processing the text to be processed using a text segmenter of a picture-text correlation matching model, and the third text tokens are obtained by processing the text to be processed using a text segmenter of a large language model; according to the mapping relationship, multiple text features that match the deleted image tokens are mapped to multiple third text tokens to obtain multiple third text tokens that match the deleted image tokens; aggregate the matched multiple third text tokens to obtain text classification tokens; the multiple first text tokens include the matched multiple third text tokens and text classification tokens, and the text classification tokens are used to represent the semantic information of the text to be processed.

[0127] In one implementation, the processing module 604 is used to: encode the remaining image tokens to obtain image encoding features; map the image encoding features to the text space to obtain mapped features; encode multiple first text tokens to obtain text encoding features; concatenate the mapped features and the text encoding features, and input the concatenated features into the large language model to obtain image and text processing results.

[0128] The functions of each module in the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0129] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a training device for a graphic processing model. The graphic processing model includes a visual encoder, a text encoder and a large language model, such as Figure 7 The figure is a structural block diagram of a training device for a graphic processing model according to an embodiment of the present application, the device comprising:

[0130] The deletion module 701 is used to obtain a sample image using a visual encoder, and aggregate the multi-layer attention weights corresponding to multiple image tokens of the sample image to obtain the aggregated weights corresponding to the multiple image tokens, and delete the multiple image tokens according to the aggregated weights corresponding to the multiple image tokens to obtain the remaining image tokens.

[0131] The acquisition module 702 is used to acquire a plurality of first text tokens of the sample text corresponding to the sample image by using a text encoder.

[0132] The training module 703 is used to train the image-text processing model based on the remaining image tokens, multiple first text tokens and the large language model.

[0133] The training method of the image and text processing model provided in the embodiment of the present application includes a visual encoder, a text encoder and a large language model. The visual encoder obtains a sample image, and aggregates the attention weights of the multiple layers corresponding to the multiple image tokens of the sample image to obtain the weights after aggregation corresponding to the multiple image tokens. According to the weights after aggregation corresponding to the multiple image tokens, the multiple image tokens are deleted to obtain the remaining image tokens; the text encoder obtains the multiple first text tokens of the sample text corresponding to the sample image; based on the remaining image tokens, the multiple first text tokens and the large language model, the image and text processing model is trained. In this embodiment, the attention weights of the multiple layers corresponding to the multiple image tokens of the sample image are aggregated, and according to the weights after aggregation, the multiple image tokens are deleted, thereby reducing the number of image tokens. By aggregating the attention weights of the multiple layers, the selective attention of the image tokens can be made more robust, while improving the training speed and ensuring the accuracy of the results.

[0134] The following introduces the specific process of each of the above steps through multiple implementation methods:

[0135] In one implementation, when the deletion module 701 aggregates the attention weights of multiple layers corresponding to multiple image tokens of a sample image to obtain the aggregated weights corresponding to the multiple image tokens, it is used to: use a visual encoder to aggregate the attention weights of the current layer corresponding to the multiple image tokens and the attention weights of a preset number of layers before the current layer to obtain the aggregated weights corresponding to the multiple image tokens.

[0136] In one implementation, when the deletion module 701 deletes multiple image tokens according to the aggregated weights respectively corresponding to the multiple image tokens to obtain the remaining image tokens, it is used to: determine the importance scores respectively corresponding to the multiple image tokens according to the aggregated weights respectively corresponding to the multiple image tokens; delete the multiple image tokens according to the importance scores respectively corresponding to the multiple image tokens to obtain the remaining image tokens.

[0137] The functions of each module in the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0138] Figure 8 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 8 As shown, the electronic device includes: a memory 810 and a processor 820. The memory 810 stores a computer program that can be run on the processor 820. When the processor 820 executes the computer program, the method in the above embodiment is implemented. The number of the memory 810 and the processor 820 can be one or more.

[0139] The electronic device also includes:

[0140] The communication interface 830 is used to communicate with external devices and perform data exchange transmission.

[0141] If the memory 810, the processor 820 and the communication interface 830 are implemented independently, the memory 810, the processor 820 and the communication interface 830 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0142] Optionally, in a specific implementation, if the memory 810, the processor 820 and the communication interface 830 are integrated on a chip, the memory 810, the processor 820 and the communication interface 830 can communicate with each other through an internal interface.

[0143] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0144] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0145] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0146] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0147] It should be understood that the above processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0148] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0149] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0151] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0152] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0153] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0154] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0155] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0156] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for processing images and texts, It is characterized in that The method comprises: Acquire an image to be processed and text to be processed corresponding to the image to be processed; Aggregate the attention weights of multiple layers corresponding to the multiple image tokens of the image to be processed to obtain the aggregated weights corresponding to the multiple image tokens; determine the importance scores corresponding to the multiple image tokens according to the aggregated weights corresponding to the multiple image tokens; delete the multiple image tokens according to the importance scores corresponding to the multiple image tokens to obtain deleted image tokens; semantically align the deleted image tokens with multiple text features of the text to be processed to obtain semantic alignment results; delete the deleted image tokens again according to the semantic alignment results to obtain the remaining image tokens; the text features are obtained by processing the text to be processed using a text encoder of a graph-text correlation matching model; wherein, semantically aligning the deleted image tokens with multiple text features of the text to be processed includes: matching and calculating the deleted image tokens with the multiple text features to obtain the semantic alignment results; Based on the remaining image tokens, the plurality of first text tokens of the text to be processed and the large language model, an image-text processing result is obtained.

2. The method according to claim 1, It is characterized in that The step of aggregating the attention weights of the multiple layers corresponding to the multiple image tokens of the image to be processed to obtain the aggregated weights corresponding to the multiple image tokens includes: Aggregate the attention weights of the current layer corresponding to the multiple image tokens and the attention weights of a preset number of layers before the current layer to obtain the aggregated weights corresponding to the multiple image tokens.

3. The method according to claim 1, It is characterized in that The method of further deleting the deleted image tokens according to the semantic alignment result to obtain the remaining image tokens includes: According to the semantic alignment result, the image tokens that do not match the multiple text features are determined in the deleted image tokens and deleted to obtain the remaining image tokens.

4. The method according to claim 1, It is characterized in that The method further comprises: Establishing a mapping relationship between a plurality of second text tokens and a plurality of third text tokens, wherein the second text tokens are obtained by processing the text to be processed using the text segmenter of the image-text correlation matching model, and the third text tokens are obtained by processing the text to be processed using the text segmenter of the large language model; According to the mapping relationship, a plurality of text features matching the deleted image tokens are mapped to the plurality of third text tokens to obtain a plurality of third text tokens matching the deleted image tokens; The matched multiple third text tokens are aggregated and calculated to obtain text classification tokens; the multiple first text tokens include the matched multiple third text tokens and the text classification tokens; the text classification tokens are used to represent the semantic information of the text to be processed.

5. The method according to claim 1 or 4, It is characterized in that The step of obtaining the image-text processing result based on the remaining image tokens, the plurality of first text tokens of the text to be processed and the large language model comprises: Encoding the remaining image tokens to obtain image encoding features; Mapping the image coding features to a text space to obtain mapped features; Encoding the plurality of first text tokens to obtain text encoding features; The mapped features and the text encoding features are concatenated, and the concatenated features are input into the large language model to obtain the image-text processing result.

6. A training method for an image and text processing model, It is characterized in that The image and text processing model includes a visual encoder, a text encoder and a large language model, and the method includes: The visual encoder obtains a sample image, and aggregates the attention weights of multiple layers corresponding to multiple image tokens of the sample image to obtain the aggregated weights corresponding to the multiple image tokens, and determines the importance scores corresponding to the multiple image tokens according to the aggregated weights corresponding to the multiple image tokens; and deletes the multiple image tokens according to the importance scores corresponding to the multiple image tokens to obtain the remaining image tokens; The text encoder obtains a plurality of first text tokens of the sample text corresponding to the sample image; The image-text processing model is trained based on the remaining image tokens, the plurality of first text tokens and the large language model.

7. The method according to claim 6, It is characterized in that The step of aggregating the attention weights of the multiple layers corresponding to the multiple image tokens of the sample image to obtain the aggregated weights corresponding to the multiple image tokens includes: The visual encoder aggregates the attention weights of the current layer corresponding to the multiple image tokens and the attention weights of a preset number of layers before the current layer to obtain the aggregated weights corresponding to the multiple image tokens.

8. An electronic device, It is characterized in that The electronic device comprises a memory, a processor and a computer program stored in the memory, and the processor implements the method of any one of claims 1-5 or 6-7 when executing the computer program.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 or 6 to 7 is implemented.

10. A computer program product, It is characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 or 6 to 7 is implemented.