Text generation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a text generation method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring image features and text features corresponding to an input image and an input text; splicing the image-text features to obtain the image-text features; inputting the image features, the text features and the image-text features into a multi-modal large language model, and performing feature extraction on the image features, the image-text features and the text features to obtain image key features, image-text key features and text key features; respectively interacting the image key feature and the text key feature with the image-text key feature to obtain a first interaction feature and a second interaction feature; and obtaining a target fusion feature according to the first interaction feature, the second interaction feature and the image-text feature, and obtaining a target generated text according to the target fusion feature. According to the method, deep fusion of image features and text features can be realized, and the language description capability of the model in a complex visual scene is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text generation method, device, electronic device and readable storage medium. Background Art

[0002] The multimodal large language model is an artificial intelligence model that can process multiple types of data input, such as text, images, etc., and can fuse data of different modalities to provide richer and more accurate language output.

[0003] The shallow registration method commonly used by existing large multimodal models connects the pre-trained visual encoder and language model through a trainable conversion layer (such as a linear layer or a multi-layer perceptron). However, since the weights of the language model are frozen and optimized for text processing, this leads to serious mismatch problems when visual features are mapped to the text input space. As a result, existing methods only regard image features as auxiliary to text, without achieving deep fusion of visual and language features, which limits the model's ability to understand complex visual scenes and generate language descriptions.

[0004] It can be seen that the target text generated by the existing methods in complex visual scenes has the problem of inaccurate description. Summary of the invention

[0005] In view of this, embodiments of the present application provide a text generation method, device, electronic device and readable storage medium to solve the problem of inaccurate text description generated in complex visual scenes in the prior art.

[0006] In a first aspect of an embodiment of the present application, a text generation method is provided, the method comprising:

[0007] An input image and an input text are obtained, and feature processing is performed on the input image and the input text respectively to obtain image features and text features; the image features and the text features are spliced ​​to obtain image-text features; the image features, the text features and the image-text features are input into a multimodal large language model, and the multimodal large language model is used to extract the image features, the image-text features and the text features respectively to obtain image key features, image-text key features and text key features; the image key features are interacted with the image-text key features to obtain a first interactive feature, and the text key features are interacted with the image-text key features to obtain a second interactive feature; a target fusion feature is obtained based on the first interactive feature, the second interactive feature and the image-text features, and a target generated text is obtained based on the target fusion feature.

[0008] According to a second aspect of the embodiments of the present application, a text generation device is provided, the device comprising:

[0009] The acquisition module is configured to acquire an input image and an input text, perform feature processing on the input image and the input text respectively, and obtain image features and text features; the splicing module is configured to splice the image features and the text features to obtain image-text features; the extraction module is configured to input the image features, the text features and the image-text features into a multimodal large language model, and use the multimodal large language model to perform feature extraction on the image features, the image-text features and the text features respectively to obtain image key features, image-text key features and text key features; the interaction module is configured to interact the image key features with the image-text key features to obtain a first interaction feature, and interact the text key features with the image-text key features to obtain a second interaction feature; the generation module is configured to obtain a target fusion feature based on the first interaction feature, the second interaction feature and the image-text features, and obtain a target generated text based on the target fusion feature.

[0010] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0011] According to a fourth aspect of an embodiment of the present application, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0012] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0013] An input image and an input text are obtained, and feature processing is performed on the input image and the input text respectively to obtain image features and text features; the image features and the text features are spliced ​​to obtain image-text features; the image features, the text features and the image-text features are input into a multimodal large language model, and the multimodal large language model is used to extract the image features, the image-text features and the text features respectively to obtain image key features, image-text key features and text key features; the image key features are interacted with the image-text key features to obtain a first interactive feature, and the text key features are interacted with the image-text key features to obtain a second interactive feature; a target fusion feature is obtained based on the first interactive feature, the second interactive feature and the image-text features, and a target generated text is obtained based on the target fusion feature. By extracting features from image features, graphic features, and text features respectively, we can obtain image key features, graphic key features, and text key features. This allows the model to focus on features that need to be paid attention to during the processing and ignore other unimportant features to prevent unimportant features from affecting the output of the model. At the same time, the image key features interact with the graphic key features, and the text key features interact with the graphic key features, which helps the model to better understand the relationship between images and texts and improve the performance of the task. Finally, based on the interacting features and graphic features, a target fusion feature that integrates all the original information is generated to further deepen the degree of fusion between features, thereby improving the relevance and accuracy of the generated text, avoiding the problem that the existing technology only regards image features as an aid to text, but does not achieve deep fusion of visual and language features, which limits the model's ability to understand complex visual scenes and generate language descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 It is a flowchart of a text generation method provided in an embodiment of the present application;

[0016] Figure 2 It is a structural diagram of a multimodal large language model provided in an embodiment of the present application;

[0017] Figure 3 It is a schematic diagram of the structure of another multimodal large language model provided in an embodiment of the present application;

[0018] Figure 4 It is a structural schematic diagram of a text generation device provided in an embodiment of the present application;

[0019] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0021] A text generation method, device, electronic device and readable storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0022] Figure 1 is a flow chart of a text generation method provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0023] S101, obtaining an input image and an input text, and performing feature processing on the input image and the input text respectively to obtain image features and text features;

[0024] S102, combining the image features and the text features to obtain image and text features;

[0025] S103, inputting the image features, the text features, and the graphic-text features into the multimodal large language model, and using the multimodal large language model to extract the image features, the graphic-text features, and the text features, respectively, to obtain the image key features, the graphic-text key features, and the text key features;

[0026] S104, interacting the image key feature with the image-text key feature to obtain a first interactive feature, and interacting the text key feature with the image-text key feature to obtain a second interactive feature;

[0027] S105, obtaining a target fusion feature according to the first interaction feature, the second interaction feature and the image-text feature, and obtaining a target generated text according to the target fusion feature.

[0028] Specifically, the input image and input text are obtained, and feature processing is performed on the input image and input text respectively to obtain image features and text features; feature processing is a key step in data preprocessing, which aims to extract useful information for subsequent tasks (such as classification, regression, clustering, etc.) from the original data. This process is particularly important for the processing of images and texts, because the original image data is usually a collection of pixel values, and the original text data is a sequence of characters or words. These forms of data are often difficult for machine learning models to directly understand and use. Therefore, feature processing is needed to convert the input image and input text into image features and text features that can be understood and used by the machine learning model. Then, the image features and text features are spliced ​​to obtain image and text features. The spliced ​​image and text feature vector can contain information of both the image and the text, which is used for subsequent image and text recognition or fusion tasks. Splicing methods include but are not limited to direct splicing, weighted splicing, and feature cascading.

[0029] It can be understood that after obtaining the image features, text features and graphic-text features, since the image features, text features and graphic-text features only convert the original input image and input text into information that the model can understand, the image features, text features and graphic-text features are input into the multimodal large language model. The multimodal large language model can extract features of the image features, graphic-text features and text features respectively to obtain image key features, graphic-text key features and text key features, so that the model can focus on the features that need to be paid attention to during the processing process and ignore other unimportant features to prevent unimportant features from affecting the output of the model. How to extract the key features from the image features, graphic-text features and text features to obtain the image key features, graphic-text key features and text key features will be explained in detail later and will not be repeated here.

[0030] Furthermore, the image key features are interacted with the image-text key features to obtain the first interactive features, and the text key features are interacted with the image-text key features to obtain the second interactive features; the purpose of this process is to fuse the image and text information and extract richer feature representations through interactive learning. The image key features and text key features represent the core key information of the input image and input text respectively, while the image-text key features contain the new information generated when the image and text are combined. By interacting the image key features with the image-text key features, new features reflecting the combination of image and text (first interactive features) can be obtained. Similarly, by interacting the text key features with the image-text key features, new features reflecting the combination of text and image (second interactive features) can be obtained. Such an interactive process helps the model better understand the relationship between image and text and improve the performance of the task.

[0031] Furthermore, according to the first interactive feature, the second interactive feature and the image-text feature, a target fusion feature is obtained, and according to the target fusion feature, a target generated text is obtained. By combining the first interactive feature, the second interactive feature and the image-text feature, a target fusion feature that integrates all of this information can be generated. The target fusion feature can more comprehensively capture the complexity and diversity of the input image and the input text, and at the same time, further fuse the feature information of the input image and the input text, thereby improving the relevance and accuracy of the generated text. Ultimately, the text generated based on this target fusion feature will be more in line with the actual needs and context of the user. This avoids the problem that the prior art only regards image features as an aid to text, but does not achieve deep fusion of visual and language features, limiting the model's ability to understand complex visual scenes and generate language descriptions.

[0032] In some examples, for example, input image: a landscape photo taken by a user. Input text: a description information entered by the user, for example: "This photo was taken at the Great Wall on a sunny day." Perform image feature processing on the input image to extract image features of the input image, such as the architectural style, color, shape, etc. of the Great Wall. Perform text feature processing on the input text to extract keywords and semantic information in the text, such as "Great Wall", "sunny", "taken". Splice the image features and text features to obtain image-text features, combining visual and text information. Input the image features, text features, and image-text features into a multimodal large language model. The model extracts features from the image features to obtain key image features, such as typical architectural elements of the Great Wall. The model extracts features from the image-text features to obtain key image-text features, combining image and text information. The model extracts features from the text features to obtain key text features, such as emotional color and description details in the text. The image key features interact with the image-text key features to obtain the first interactive features, further strengthening the combination of image and text. The key features of the text are interacted with the key features of the image and text to obtain the second interactive feature, which further strengthens the combination of text and image. According to the first interactive feature, the second interactive feature and the image and text features, the target fusion feature is obtained to integrate all information. According to the target fusion feature, the target generated text is generated, for example: "This photo vividly shows the magnificent scenery of the Great Wall. The sunny weather makes the whole picture brighter and more vivid. The Great Wall is winding and undulating, like a giant dragon lying between the mountains, magnificent. Against the backdrop of blue sky and white clouds, the bricks and stones of the Great Wall appear clearer, as if every brick and tile are telling the vicissitudes of history. The mountains in the distance are layered and green, which contrasts sharply with the ancient vicissitudes of the Great Wall, making the whole picture more vivid and layered." Finally, a descriptive text is generated based on the image and text information provided by the user to help users better understand and share their travel experiences.

[0033] According to the technical solution provided in the embodiment of the present application, an input image and an input text are obtained, and feature processing is performed on the input image and the input text respectively to obtain image features and text features; the image features and the text features are spliced ​​to obtain image-text features; the image features, the text features and the image-text features are input into a multimodal large language model, and the multimodal large language model is used to extract the image features, the image-text features and the text features respectively to obtain image key features, the image-text key features and the text key features; the image key features are interacted with the image-text key features to obtain a first interactive feature, and the text key features are interacted with the image-text key features to obtain a second interactive feature; according to the first interactive feature, the second interactive feature and the image-text features, a target fusion feature is obtained, and according to the target fusion feature, a target generated text is obtained. By extracting features from image features, graphic features, and text features respectively, we can obtain image key features, graphic key features, and text key features. This allows the model to focus on features that need to be paid attention to during the processing and ignore other unimportant features, thereby preventing unimportant features from affecting the output of the model. At the same time, the image key features interact with the graphic key features, and the text key features interact with the graphic key features, which helps the model to better understand the relationship between images and texts and improve the performance of the task. Finally, based on the interacting features and graphic features, a target fusion feature that integrates all the original information is generated, which further deepens the degree of fusion between features, thereby improving the relevance and accuracy of the generated text, and avoiding the problem that the existing technology only regards image features as an aid to text, but does not achieve deep fusion of image features and text features, which limits the model's ability to understand complex visual scenes and generate language descriptions.

[0034] In some embodiments, feature processing is performed on the input image and input text respectively to obtain image features and text features, including: encoding the input image using an image encoder, and optimizing the features of the encoded input image using an initial multi-layer perceptron to obtain image features; vectorizing the input text to obtain text features.

[0035] Specifically, the image encoder can convert the input image into a form that can be processed by the model, that is, it can extract the features of the input image and generate a corresponding encoding representation. Exemplarily, in the process of processing the input image and the features to obtain the image features, a VIT (Vision Transformer) image encoder, Swin Transformer (SIT), Data-efficient Image Transformers (DeiT), etc. can be used. In order to effectively capture the global dependencies in the input image, this embodiment preferentially uses the VIT image encoder to perform encoding operations on the input image.

[0036] Furthermore, the encoded features are input into the initial multi-layer perceptron (MLP) for feature optimization. The initial MLP contains multiple fully connected layers, and the features are further abstracted and combined through nonlinear activation functions. In addition, during the training process of the model, the weights of the MLP can be adjusted through the back-propagation algorithm to minimize the loss function in the feature optimization process, thereby obtaining more accurate and discriminative image features.

[0037] It can be understood that the vectorized representation of the input text refers to the use of numerical vectors to represent the semantics of the text. For the input text, the text can be segmented into words or phrases, and then a word vector model, a sentence vector model, etc. can be included to convert each word or phrase into a vector representation of a fixed dimension, which can specifically include a bag of words model (Bag of Words Model), term frequency-inverse document frequency (TF-IDF), word-vector model (Word2vec), document-vector model (Doc2vec), etc., and the vectorized representation method is not limited here. Exemplarily, in the process of vectorizing the input text and obtaining text features, the input text can be converted into text features by using word embedding in the word vector model.

[0038] According to the technical solution provided by the embodiment of the present application, an image encoder is used to encode the input image, and an initial multi-layer perceptron is used to optimize the features of the encoded input image to obtain image features; the input text is vectorized to obtain text features. The input image and input text can be converted into image features and text features that can be understood by the model, thereby improving the processing efficiency of the model and the accuracy of the input data recognized by the model.

[0039] In some embodiments, the multimodal large language model includes an image processing branch, a graphic and text processing branch, and a text processing branch. The multimodal large language model is used to extract image features, graphic and text features, and text features, respectively, to obtain image key features, graphic and text key features, and text key features, including:

[0040] The image features are processed using the first self-attention layer and the first multi-layer perceptron layer that are successively set in the image processing branch to obtain the key features of the image; the text features are processed using the second self-attention layer and the second multi-layer perceptron layer that are successively set in the text processing branch to obtain the key features of the text; the image and text features are processed using the third self-attention layer, the cross-attention layer, and the third multi-layer perceptron layer that are successively set in the image and text processing branch, and in the processing process of the cross-attention layer, a learnable target query vector is added to obtain the key features of the image and text.

[0041] In order to better understand this embodiment, Figure 2 The embodiments are described as follows. Figure 2 As shown: Specifically, the multimodal large language model includes an image processing branch, an image and text processing branch, and a text processing branch, wherein the image processing branch has a first self-attention layer and a first multi-layer perceptron layer that are continuously set. After the image features are input into the image processing branch, they are first processed by the first self-attention layer, which allows the model to capture long-distance dependencies between different regions of the image. Then, the processing results of the first self-attention layer are input into the first multi-layer perceptron layer for further feature extraction and nonlinear transformation, and finally the key features of the image are obtained. Similarly, in the text processing branch, the text features are first processed by the second self-attention layer, which can capture long-distance dependencies in the text sequence, thereby retaining the intrinsic structure and semantic information of the text. Subsequently, the output of the second self-attention layer is sent to the second multi-layer perceptron layer to further extract key information from the text features and form the key features of the text.

[0042] It can be understood that in the image and text processing branch, the image and text features first pass through the third self-attention layer. The third self-attention layer uses a self-attention mechanism to capture the relationship between the image and the text, that is, dynamically pays attention to the relevant parts between the image and the text, and enhances the model's ability to understand the image and text content. Then, the output of the third self-attention layer is sent to the cross-attention layer. It should be noted that in the processing of the cross-attention layer, a learnable target query vector can be introduced, wherein the learnable target query vector refers to a specific vector learned through a training process in a machine learning or deep learning model. The learnable target query vector can be used to represent the intent or target of the query so that the model can more accurately find relevant data or information. By performing cross-attention calculations on the output of the third self-attention layer and the learnable target query vector, the model can process the image and text features more accurately, and at the same time, the model focuses on the image area most relevant to the text during the processing process. Finally, the features output by the third self-attention layer are sent to the third multi-layer perceptron layer for further feature changes and information extraction, thereby extracting the key features of the image and text.

[0043] As an example, for example, taking the image description task as an example, there is an image containing multiple objects, such as a picture containing "dog" and "cat", and there is a description text "a brown dog chasing a white cat on the grass". In the processing of the image-text processing branch, the image-text features first pass through the third self-attention layer, which dynamically pays attention to the cat and dog in the picture and the keywords such as "cat" and "dog" in the text description. Through the self-attention mechanism, the model can understand the correspondence between the cat and dog in the picture and the text description, thereby enhancing the understanding of the image-text content. The output of the third self-attention layer is sent to the cross-attention layer. In this layer, a learnable target query vector is introduced, for example, this vector represents the attention to "cat". By performing cross-attention calculations on the output of the third self-attention layer and this target query vector, the model can more accurately identify the area in the picture related to "cat", that is, the cat in the picture, while ignoring the part that is not related to "cat", such as the dog in the picture. In this way, when processing image-text features, the model can focus on the image area that is most relevant to the text description, thereby improving the accuracy of image-text matching.

[0044] According to the technical solution extracted by the embodiment of the present application, the first self-attention layer and the first multi-layer perceptron layer which are continuously arranged in the image processing branch are used to process the image features to obtain the key features of the image; the second self-attention layer and the second multi-layer perceptron layer which are continuously arranged in the text processing branch are used to process the text features to obtain the key features of the text; the third self-attention layer, the cross-attention layer and the third multi-layer perceptron layer which are continuously arranged in the image-text processing branch are used to process the image-text features, and in the processing process of the cross-attention layer, a learnable target query vector is added to obtain the key features of the image-text. Through the processing of each branch, an in-depth understanding of the image features, text features and image-text features can be achieved, and the key features which need more attention in the task processing can be obtained, while the unimportant features can be ignored. Thereby, better performance can be achieved in the multimodal text generation task, and the accuracy of the text generation task can be improved.

[0045] In some embodiments, the image key feature interacts with the image-text key feature to obtain a first interactive feature, and the text key feature interacts with the image-text key feature to obtain a second interactive feature, including:

[0046] The first gated network layer in the image processing branch is used to interact the image key features with the image-text key features to obtain the first interactive features; the second gated network layer in the text processing branch is used to interact the text key features with the image-text key features to obtain the second interactive features.

[0047] Specifically, Figure 2As shown, the image processing branch also includes a first gated network layer, which can control the flow of information, and enhance the fusion and expression of key features by interacting with feature information from different sources, thereby improving the processing performance of the model. In this embodiment, after obtaining the image key features output by the first multi-layer perceptron and the image-text key features output by the third multi-layer perceptron, the image key features and the image-text key features are input into the first gated network layer for interaction, so that the first gated network layer can effectively integrate the image key features and the image-text key features, obtain richer first interactive features, and enhance the recognition and understanding capabilities of the model.

[0048] It can be understood that the text processing branch also includes a second gated network layer, wherein the function of the second gated network layer is the same as that of the first gated network layer. After obtaining the text key features output by the second multi-layer perceptron and the image-text key features output by the third multi-layer perceptron, the text key features and the image-text key features are input into the second gated network layer for interaction, so that the second gated network layer can effectively integrate the text key features and the image-text key features to obtain richer second interactive features, thereby improving the recognition and understanding capabilities of the model.

[0049] According to the technical solution provided in the embodiment of the present application, the first gated network layer in the image processing branch is used to interact the image key features with the image-text key features to obtain the first interactive features; the second gated network layer in the text processing branch is used to interact the text key features with the image-text key features to obtain the second interactive features. By using the gated network layer to integrate the image key features with the image-text key features, and integrating the text key features with the image-text key features, richer first interactive features and second interactive features are obtained, achieving a deep fusion of image, text and image-text features. By introducing a learnable target query vector and optimized feature representation, the model can process the image-text content more accurately, thereby achieving better performance in multimodal text generation tasks and improving the recognition and comprehension capabilities of the model.

[0050] In some embodiments, the target fusion feature is obtained according to the first interaction feature, the second interaction feature and the image-text feature, including:

[0051] The first interaction feature, the second interaction feature and the image-text feature are fused to obtain an initial fusion feature; and the target fusion feature is optimized using the fourth multi-layer perceptron layer to obtain a target fusion feature.

[0052] Specifically, after obtaining the first interactive feature and the second interactive feature, further fusion is required to obtain the final target fusion feature. Specifically, the first interactive feature, the second interactive feature and the image-text feature are first fused. This fusion process can be completed by a simple splicing operation, or by a more complex fusion strategy, such as using an attention mechanism to weight the importance of different features. The fused initial fusion feature contains comprehensive information of images, texts and image-text interactions, but in order to further improve the performance of the model, these features need to be optimized. To this end, the initial fusion feature is sent to the fourth multi-layer perceptron layer for feature optimization. The fourth multi-layer perceptron layer can learn more complex feature representations through a multi-layer neural network structure, so that the final target fusion feature has a stronger expressive ability.

[0053] It should be noted that, during the optimization process, the fourth multi-layer perceptron layer can use various activation functions, such as ReLU, tanh or sigmoid, to introduce nonlinear factors and enhance the generalization ability of the model. In addition, regularization techniques, such as L1 or L2 regularization, can be used to prevent the model from overfitting and ensure that the model has good generalization performance when facing new data.

[0054] It can be understood that through the above fusion and optimization process, the model can obtain a target fusion feature containing image, text and image-text interaction information. This feature not only contains rich information, but also has high expressive power, which can provide strong support for subsequent multimodal text generation tasks.

[0055] According to the technical solution provided in the embodiment of the present application, the first interactive feature, the second interactive feature and the image-text feature are fused to obtain the initial fused feature; the fourth multi-layer perceptron layer is used to optimize the target fused feature to obtain the target fused feature. The target fused feature can contain comprehensive information of the image, text and image-text interactive information, and at the same time, the target fused feature has a higher expressive ability, thereby improving the accuracy of the target generated text.

[0056] In some embodiments, the multimodal large language model includes at least two language processing modules, each of which includes an image processing branch, a graphic and text processing branch, and a text processing branch. After obtaining the target fusion feature according to the first interaction feature, the second interaction feature, and the graphic and text feature, the following further includes:

[0057] The first interactive feature is residually connected with the image feature to obtain the image residual feature, and the image residual is input as a new image feature to the image processing branch of the next language processing module connected to the current language processing module for iteration until the first interactive feature output by the last language processing module is obtained; the second interactive feature is residually connected with the text feature to obtain the text residual feature, and the text residual feature is input as a new text feature to the text processing branch of the next language processing module connected to the current language processing module for iteration until the second interactive feature output by the last language processing module is obtained; the target fusion feature is input as a new image-text feature to the image-text processing branch of the next language processing module connected to the current language processing module for iteration until the target fusion feature output by the last language processing module is obtained.

[0058] Specifically, in order to better understand this embodiment, Figure 3 This embodiment is described as follows. Figure 3 As shown: The multimodal large language model includes at least two language processing modules, each of which includes an image processing branch, a graphic and text processing branch, and a text processing branch. Each language processing module can use the output of the previous language processing module as input for iterative processing. Using at least two language processing modules for iteration can enhance the model's ability to learn and integrate features and improve the effect of processing complex tasks. Each module deepens its understanding of image features, text features, and graphic and text features through repeated iterations, thereby improving the performance and accuracy of the model.

[0059] It can be understood that the first interactive feature is residually connected with the image feature to obtain the image residual feature, wherein the residual connection is a structure commonly used in deep neural networks, which can add the output of a certain layer to the output of the directly previous layer as the input of the next layer. It is used to alleviate the gradient vanishing problem in the deep network training process so that the network can learn richer feature representations. In addition, the first interactive feature is residually connected with the original image feature to retain important information in the image feature, while referencing the new features in the first interactive feature to enhance the image feature representation, so that the model can better capture the complex features in the image and improve the robustness and expression ability of the model. Furthermore, the image residual is input as a new image feature to the image processing branch of the next language processing module connected to the current language processing module for iteration. The structure of the image processing branch is the same. The specific processing process refers to the processing flow of the above embodiment, which will not be repeated here. It should be noted that, in the processing process of the last language processing module, it is sufficient to obtain the first interactive feature input by the last language processing module. Since it is the last language processing module, the first interactive feature is not residually connected with the image feature to obtain the image residual feature. The image residual is input as a new image feature to the image processing branch of the next language processing module connected to the current language processing module to perform this iterative step.

[0060] It should be noted that the structure of the text processing branch is the same as that of the image processing branch. The processing flow of at least two text processing branches can refer to the processing flow of the image processing branch in this embodiment, which will not be described in detail here.

[0061] It can be understood that the target fusion feature is input as a new graphic feature into the graphic processing branch of the next language processing module connected to the current language processing module for iteration until the target fusion feature output by the last language processing module is obtained. This process helps the model to gradually optimize and refine the fusion of graphic and text information at different levels.

[0062] In each iteration, the output of the current language processing module will be input into the next language processing module for further processing and refinement, making the final output features more accurate and expressive. The iterative mechanism can not only enhance the model's ability to understand graphic information, but also improve its adaptability and generalization ability in complex scenarios.

[0063] It should be noted that in practical applications, the iterative processing method of the multimodal large language model can be applied to a variety of tasks, such as image description generation, visual question answering, image-text matching, etc. For example, in the image description generation task, the model needs to understand the image content and generate descriptive text. Through iterative processing, the model can gradually integrate image and text information to generate richer and more accurate descriptions. In the visual question answering task, the model needs to answer relevant questions based on the image content. Through iterative processing, the model can better understand the relationship between the image and the question, thereby providing more accurate answers.

[0064] In addition, the iterative processing method of the multimodal large language model has good scalability. With the addition of more language processing modules, the model can handle more complex tasks and adapt to data from different fields. This makes the model more adaptable when facing diverse application scenarios.

[0065] According to the technical solution provided in the embodiment of the present application, the first interactive feature is residually connected with the image feature to obtain the image residual feature, and the image residual is input as a new image feature to the image processing branch of the next language processing module connected to the current language processing module for iteration until the first interactive feature output by the last language processing module is obtained; the second interactive feature is residually connected with the text feature to obtain the text residual feature, and the text residual feature is input as a new text feature to the text processing branch of the next language processing module connected to the current language processing module for iteration until the second interactive feature output by the last language processing module is obtained; the target fusion feature is input as a new graphic feature to the graphic processing branch of the next language processing module connected to the current language processing module for iteration until the target fusion feature output by the last language processing module is obtained. The multimodal large language model can effectively fuse image, text and graphic information through iterative processing, thereby improving the performance and accuracy of the model in multimodal tasks.

[0066] In some embodiments, the image residual features are input into the image processing branch of the next language processing module connected to the current language processing module for iteration, including: using the fifth multi-layer perceptron layer to perform feature optimization on the image residual features, and inputting the optimized features into the image processing branch of the next language processing module connected to the current language processing module for iteration; the text residual features are input into the text processing branch of the next language processing module connected to the current language processing module for iteration, including: using the sixth multi-layer perceptron layer to perform feature optimization on the text residual features, and inputting the optimized features into the text processing branch of the next language processing module connected to the current language processing module for iteration.

[0067] Specifically, the fifth multi-layer perceptron layer and the sixth multi-layer perceptron layer can be neural network structures with the same or different numbers of hidden layers. The purpose of these multi-layer perceptron layers is to further refine and optimize the residual features so as to provide more accurate image and text feature representations in the iterative process. In this way, the model can learn more complex and abstract features, thereby achieving better performance in multimodal tasks.

[0068] In addition, during the iteration process, each language processing module may include one or more multi-layer perceptron layers for processing and fusing features from the image processing branch, the text processing branch, and the image-text processing branch. These multi-layer perceptron layers may use different activation functions, such as ReLU, tanh, or sigmoid, to meet the needs of different feature fusion.

[0069] According to the technical solution provided by the embodiment of the present application, the fifth multi-layer perceptron layer is used to optimize the image residual features, and the optimized features are input to the image processing branch of the next language processing module connected to the current language processing module for iteration, and the sixth multi-layer perceptron layer is used to optimize the text residual features, and the optimized features are input to the text processing branch of the next language processing module connected to the current language processing module for iteration. Thus, before each iteration, the model can learn more complex and abstract features, so as to achieve better performance in multimodal tasks.

[0070] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of the present application.

[0071] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0072] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0073] Figure 4 Schematic diagram of the structure of a text generation device provided in an embodiment of the present application. Figure 4 As shown, the device comprises:

[0074] The acquisition module 401 is configured to acquire an input image and an input text, and perform feature processing on the input image and the input text respectively to obtain image features and text features;

[0075] A splicing module 402 is configured to splice the image features and the text features to obtain image and text features;

[0076] The extraction module 403 is configured to input the image features, the text features and the graphic-text features into the multimodal large language model, and use the multimodal large language model to extract the image features, the graphic-text features and the text features, respectively, to obtain the image key features, the graphic-text key features and the text key features;

[0077] The interaction module 404 is configured to interact the image key feature with the image-text key feature to obtain a first interaction feature, and interact the text key feature with the image-text key feature to obtain a second interaction feature;

[0078] The generation module 405 is configured to obtain a target fusion feature according to the first interaction feature, the second interaction feature and the image-text feature, and obtain a target generated text according to the target fusion feature.

[0079] In some embodiments, the acquisition module 401 is also configured to use an image encoder to encode the input image, and use an initial multi-layer perceptron to optimize the features of the encoded input image to obtain image features; and vectorize the input text to obtain text features.

[0080] In some embodiments, the multimodal large language model includes an image processing branch, a graphic and text processing branch, and a text processing branch. The extraction module 403 is also configured to use the first self-attention layer and the first multi-layer perceptron layer that are continuously set in the image processing branch to process image features to obtain image key features; use the second self-attention layer and the second multi-layer perceptron layer that are continuously set in the text processing branch to process text features to obtain text key features; use the third self-attention layer, the cross-attention layer, and the third multi-layer perceptron layer that are continuously set in the graphic and text processing branch to process graphic and text features, and in the processing process of the cross-attention layer, add a learnable target query vector to obtain graphic and text key features.

[0081] In some embodiments, the interaction module 404 is also configured to utilize the first gated network layer in the image processing branch to interact the image key features with the graphic key features to obtain the first interaction features; and utilize the second gated network layer in the text processing branch to interact the text key features with the graphic key features to obtain the second interaction features.

[0082] In some embodiments, the generation module 405 is further configured to fuse the first interaction feature, the second interaction feature and the image and text feature to obtain an initial fusion feature; and optimize the target fusion feature using the fourth multi-layer perceptron layer to obtain a target fusion feature.

[0083] In some embodiments, the text generation device also includes an iteration module, and the iteration module 406 is configured to perform a residual connection on the first interaction feature and the image feature to obtain an image residual feature, and input the image residual as a new image feature to the image processing branch of the next language processing module connected to the current language processing module for iteration until the first interaction feature output by the last language processing module is obtained; perform a residual connection on the second interaction feature and the text feature to obtain a text residual feature, and input the text residual feature as a new text feature to the text processing branch of the next language processing module connected to the current language processing module for iteration until the second interaction feature output by the last language processing module is obtained; and input the target fusion feature as a new image-text feature to the image-text processing branch of the next language processing module connected to the current language processing module for iteration until the target fusion feature output by the last language processing module is obtained.

[0084] In some embodiments, the iteration module 406 is also configured to use the fifth multi-layer perceptron layer to perform feature optimization on the image residual features, and input the optimized features to the image processing branch of the next language processing module connected to the current language processing module for iteration; input the text residual features to the text processing branch of the next language processing module connected to the current language processing module for iteration, including: using the sixth multi-layer perceptron layer to perform feature optimization on the text residual features, and input the optimized features to the text processing branch of the next language processing module connected to the current language processing module for iteration.

[0085] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0086] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0087] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0088] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0089] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0091] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A text generation method, characterized in that: include: Acquire an input image and an input text, and perform feature processing on the input image and the input text respectively to obtain image features and text features; Concatenating the image feature and the text feature to obtain a graphic feature; Inputting the image features, the text features, and the graphic-text features into a multimodal large language model, and using the multimodal large language model to perform feature extraction on the image features, the graphic-text features, and the text features, respectively, to obtain image key features, graphic-text key features, and text key features; The image key feature interacts with the image-text key feature to obtain a first interactive feature, and the text key feature interacts with the image-text key feature to obtain a second interactive feature; A target fusion feature is obtained according to the first interaction feature, the second interaction feature and the image-text feature, and a target generated text is obtained according to the target fusion feature.

2. The method according to claim 1, characterized in that The performing feature processing on the input image and the input text respectively to obtain image features and text features includes: Encoding the input image using an image encoder, and optimizing the features of the encoded input image using an initial multi-layer perceptron to obtain the image features; The input text is vectorized to obtain the text features.

3. The method according to claim 1, characterized in that The multimodal large language model includes an image processing branch, a graphic and text processing branch, and a text processing branch. The multimodal large language model is used to extract features of the image, the graphic and text features, and the text features to obtain key features of the image, key features of the graphic and text, and key features of the text, including: Processing the image features using a first self-attention layer and a first multi-layer perceptron layer that are continuously arranged in the image processing branch to obtain the key features of the image; Processing the text features using a second self-attention layer and a second multi-layer perceptron layer that are continuously arranged in the text processing branch to obtain the text key features; The image and text features are processed using the third self-attention layer, cross-attention layer, and third multi-layer perceptron layer that are continuously set in the image and text processing branch, and in the processing process of the cross-attention layer, a learnable target query vector is added to obtain the key features of the image and text.

4. The method according to claim 3, characterized in that The step of interacting the image key feature with the image-text key feature to obtain a first interactive feature, and interacting the text key feature with the image-text key feature to obtain a second interactive feature includes: Using the first gated network layer in the image processing branch to interact the image key feature with the graphic key feature to obtain the first interactive feature; The second gated network layer in the text processing branch is used to interact the text key feature with the image-text key feature to obtain the second interactive feature.

5. The method according to claim 4, characterized in that The obtaining a target fusion feature according to the first interaction feature, the second interaction feature and the image-text feature includes: Fusing the first interaction feature, the second interaction feature, and the image-text feature to obtain an initial fused feature; The target fusion feature is optimized by using a fourth multi-layer perceptron layer to obtain the target fusion feature.

6. The method according to claim 5, characterized in that The multimodal large language model includes at least two language processing modules, each of which includes the image processing branch, the image and text processing branch, and the text processing branch. After obtaining the target fusion feature according to the first interaction feature, the second interaction feature, and the image and text feature, the method further includes: Performing a residual connection between the first interactive feature and the image feature to obtain an image residual feature, and inputting the image residual as a new image feature into the image processing branch of the next language processing module connected to the current language processing module for iteration until the first interactive feature output by the last language processing module is obtained; Performing a residual connection between the second interactive feature and the text feature to obtain a text residual feature, and inputting the text residual feature as a new text feature into the text processing branch of the next language processing module connected to the current language processing module for iteration until the second interactive feature output by the last language processing module is obtained; The target fusion feature is input as the new image-text feature to the image-text processing branch of the next language processing module connected to the current language processing module for iteration until the target fusion feature output by the last language processing module is obtained.

7. The method according to claim 6, characterized in that The step of inputting the image residual feature to the image processing branch of the next language processing module connected to the current language processing module for iteration comprises: Utilizing a fifth multi-layer perceptron layer to optimize the image residual features, and inputting the optimized features into the image processing branch of the next language processing module connected to the current language processing module for iteration; The step of inputting the text residual feature into the text processing branch of the next language processing module connected to the current language processing module for iteration comprises: The text residual features are optimized using a sixth multi-layer perceptron layer, and the optimized features are input into the text processing branch of the next language processing module connected to the current language processing module for iteration.

8. A text generation device, characterized in that: include: An acquisition module is configured to acquire an input image and an input text, and perform feature processing on the input image and the input text respectively to obtain image features and text features; A splicing module, configured to splice the image feature and the text feature to obtain an image-text feature; An extraction module is configured to input the image features, the text features, and the graphic-text features into a multimodal large language model, and use the multimodal large language model to perform feature extraction on the image features, the graphic-text features, and the text features, respectively, to obtain image key features, graphic-text key features, and text key features; An interaction module is configured to interact the image key feature with the image-text key feature to obtain a first interaction feature, and interact the text key feature with the image-text key feature to obtain a second interaction feature; The generation module is configured to obtain a target fusion feature according to the first interaction feature, the second interaction feature and the image-text feature, and obtain a target generated text according to the target fusion feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images