A text generation method, device, electronic equipment, program product and storage medium

Abstract

The application provides a text generation method and device, electronic equipment, program product and storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: receiving an input image and an input text, and encoding the input image and the input text respectively to obtain an image feature vector and a text feature vector; performing cross-attention processing on the text feature vector by using the image feature vector, and performing self-attention processing on the cross-attention processing result to obtain a feature extraction vector; and performing logical calculation on the feature extraction vector to obtain an output text generated according to the semantics of the input image and the input text. The feature interaction mode between the image and the text can be adjusted, the calculation increase of the square of the length of the input sequence can be avoided, the computing power requirement of the model can be reduced, and the use experience of the user can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a text generation method, apparatus, electronic device, program product, and storage medium. Background Technology

[0002] A multimodal model is a machine learning model that involves multiple data sources and data types. It can extract and integrate information from different perceptual modalities (such as text, images, speech, and video), perform comprehensive analysis and prediction, and generate results in the form required by the user (such as text or images). For example, a multimodal visual-language question-answering model can receive a question input containing both image and text information, and perform feature extraction and analysis on both the image and text information to generate the corresponding question answer.

[0003] In related technologies, multimodal models typically concatenate image representation vectors with text representation vectors when extracting features from images and text, and then perform self-attention processing based on the concatenated vector to extract features from the images and text. However, since image representation vectors are usually large in dimensionality, and this dimensionality increases with higher image resolution, the concatenated vector becomes excessively large. Furthermore, the time complexity of the self-attention mechanism is O(N^2). 2 (N can represent the dimension of the concatenated vector). Therefore, when the dimension of the concatenated vector is too large, it will generate a large amount of processing, which will not only easily occupy computing resources, but also easily prolong the processing time, thus bringing a bad user experience. Summary of the Invention

[0004] The purpose of this invention is to provide a text generation method, apparatus, electronic device, program product, and storage medium that can reduce the computational power requirements of the model by adjusting the feature interaction method between images and text, thereby effectively improving the user experience.

[0005] To address the aforementioned technical problems, this invention provides a text generation method, comprising:

[0006] Receive an input image and input text, and encode the input image and input text respectively to obtain an image representation vector and a text representation vector;

[0007] The image representation vector is used to perform cross-attention processing on the text representation vector, and the cross-attention processing result is then subjected to self-attention processing to obtain the feature extraction vector.

[0008] Logical calculations are performed on the feature extraction vectors to obtain the output text generated based on the semantics of both the input image and the input text.

[0009] Optionally, the step of performing cross-attention processing on the text representation vector using the image representation vector includes:

[0010] Both the text representation vector and the image representation vector are subjected to root mean square normalization.

[0011] Set the processed text representation vector as the query data, and set the processed image representation vector as the key data and value data;

[0012] Cross-attention processing is performed on the query data, the key data, and the value data, and the processing result is processed using a linear layer.

[0013] Optionally, the input image includes a high-resolution image and a low-resolution image, wherein the image size of the high-resolution image is larger than a preset size, and the image size of the low-resolution image is not larger than the preset size, and the high-resolution image representation vector of the high-resolution image and the low-resolution image representation vector of the low-resolution image are sequentially subjected to cross-attention processing with the text representation vector.

[0014] Optionally, the step of performing cross-attention processing on the text representation vector using the image representation vector, and performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector, includes:

[0015] Both the high-resolution image representation vector and the text representation vector are input into the first cross-attention unit, so that the first cross-attention unit uses the high-resolution image representation vector to perform the cross-attention processing on the text representation vector to obtain the first feature vector;

[0016] The first feature vector is input into the first self-attention unit, so that the first self-attention unit performs the self-attention processing on the first feature vector to obtain the second feature vector;

[0017] Both the low-resolution image representation vector and the second feature vector are input into the second cross-attention unit, so that the second cross-attention unit uses the low-resolution image representation vector to perform the cross-attention processing on the second feature vector to obtain the third feature vector;

[0018] The third feature vector is input into the second self-attention unit, so that the second self-attention unit performs the self-attention processing on the third feature vector to obtain the feature extraction vector.

[0019] Optionally, an input image is received, including:

[0020] Receive the raw input image;

[0021] The resolution of the original input image is adjusted to obtain the high-resolution image and the low-resolution image.

[0022] Optionally, both the first cross-attention unit and the second cross-attention unit include a cross-attention layer and a first multilayer perceptron layer, wherein the first multilayer perceptron performs multilayer perceptron processing on the output of the cross-attention layer; both the first self-attention unit and the second self-attention unit include a self-attention layer and a second multilayer perceptron layer, wherein the second multilayer perceptron performs multilayer perceptron processing on the output of the sub-attention layer.

[0023] Optionally, the first cross-attention unit, the first self-attention unit, the second cross-attention unit, and the second self-attention unit constitute a feature extraction unit, and multiple feature extraction units are connected in series.

[0024] Optionally, encoding the input image includes:

[0025] Convert the color mode of the input image to a preset color mode, and determine the length and width of the input image;

[0026] The background color is set according to the average color value of the input image. The maximum value of the length and width of the input image is used as the length and width value of the background image. A background image is generated based on the background color and the length and width value of the background image. The input image is then set in the background image to obtain a new input image.

[0027] The new input image is converted into a first tensor, and the first tensor is regularized to obtain a second tensor;

[0028] The second tensor is encoded using an image encoder to obtain the third tensor;

[0029] The third tensor is processed by a linear layer to obtain the image representation vector.

[0030] Optionally, encoding the input text includes:

[0031] The input text is subjected to vector embedding processing to obtain the text representation vector.

[0032] Optionally, the step of performing cross-attention processing on the text representation vector using the image representation vector, and performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector, includes:

[0033] The image feature vector and the text feature vector are sent to a heterogeneous computing device so that the heterogeneous computing device can perform the steps of using the image representation vector to perform cross-attention processing on the text representation vector, and performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector.

[0034] Optionally, the heterogeneous computing device is a graphics card device.

[0035] The present invention also provides a text generation apparatus, comprising:

[0036] An encoding module is used to receive an input image and input text, and encode the input image and the input text respectively to obtain an image representation vector and a text representation vector;

[0037] The feature extraction module is used to perform cross-attention processing on the text representation vector using the image representation vector, and to perform self-attention processing on the cross-attention processing result to obtain the feature extraction vector.

[0038] The logic calculation module is used to perform logical calculations on the feature extraction vector to obtain the output text generated based on the semantics of both the input image and the input text.

[0039] The present invention also provides an electronic device, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor for implementing the text generation method described above when executing the computer program.

[0042] Optionally, the electronic device further includes:

[0043] A heterogeneous computing device is used to receive image representation vectors and text representation vectors sent by the processor, perform cross-attention processing on the text representation vectors using the image representation vectors, and perform self-attention processing on the cross-attention processing results to obtain feature extraction vectors.

[0044] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the text generation method described above.

[0045] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the text generation method described above.

[0046] This invention provides a text generation method, comprising: receiving an input image and input text, and encoding the input image and the input text respectively to obtain an image representation vector and a text representation vector; performing cross-attention processing on the text representation vector using the image representation vector, and performing self-attention processing on the cross-attention processing result to obtain a feature extraction vector; and performing logical calculations on the feature extraction vector to obtain output text generated based on the semantics of the input image and the input text.

[0047] As can be seen, after receiving the input image and input text, this invention first encodes the input image and input text separately to obtain image representation vectors and text representation vectors. Then, unlike related technologies, this invention does not concatenate the image representation vector and text representation vector and perform self-attention processing based on the concatenated vector. Instead, it uses the image representation vector to perform cross-attention processing on the text representation vector, and then performs self-attention processing on the cross-attention result to obtain a feature extraction vector. This allows the image representation vector to be used to perform feature interaction on the text representation vector independently, and then self-attention interaction is performed using the interaction result corresponding to the text representation vector. Thus, because the image processing in the model uses a cross-attention mechanism, there is no increase in the calculation of the square of the input sequence length; and because the dimension of the text representation vector is usually small, the computational power required for self-attention interaction is reduced, thereby reducing the computational power requirement of the model and effectively improving the user experience. This invention also provides a text generation device, electronic device, computer program product, and computer-readable storage medium, which have the above-mentioned beneficial effects. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 A flowchart of a text generation method provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of a cross-attention processing method provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of a multimodal model provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of another multimodal model provided in an embodiment of the present invention;

[0053] Figure 5 This is a structural block diagram of a text generation device provided in an embodiment of the present invention;

[0054] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] A multimodal model is a machine learning model that involves multiple data sources and data types. It can extract and integrate information from different perceptual modalities (such as text, images, speech, and video), perform comprehensive analysis and prediction, and generate results in the form required by the user (such as text or images). For example, a multimodal visual-language question-answering model can receive a question input containing both image and text information, and perform feature extraction and analysis on both the image and text information to generate the corresponding question answer.

[0057] In related technologies, multimodal models typically concatenate image representation vectors with text representation vectors when extracting features from images and text, and then perform self-attention processing based on the concatenated representation vector to extract features from the image and text. However, since the dimensionality of the image representation vector is related to the image resolution—the higher the image resolution, the larger the dimensionality of the image representation vector—high-resolution images will directly lead to excessively large dimensionality in the concatenated representation vector. Furthermore, the time complexity of the self-attention mechanism is O(N). 2 ), where represents the size of the input sequence, which is the dimension of the concatenated representation vector. Therefore, images with excessively high resolution will increase the computational cost by the square of the dimension for self-attention processing, which will not only easily consume a lot of computing resources, but also prolong the processing time, resulting in a poor user experience.

[0058] In related technologies, to avoid excessive computational burden on the model due to excessively high-resolution images, image resolution is typically limited, such as to 336*336 or 448*448 pixels. However, this directly affects the amount of information the model can extract from the image. Related technologies also crop high-resolution images into several small blocks and extract features from each block individually. However, image cropping leads to discontinuous image content, affecting the model's extraction of image information and potentially causing incorrect responses. This is particularly evident in image document processing or charts involving OCR (Optical Character Recognition). To increase the resolution of the input image without significantly increasing computational burden, while avoiding image cropping, this invention provides a text generation method. By adjusting the feature interaction between the image and text, the computational requirements of the model can be reduced, thereby effectively improving the user experience.

[0059] It should be noted that the embodiments of the present invention do not limit the type of hardware device executing this method, and can be set according to actual application needs. For example, it can be a personal computer, server, or other dedicated device, and can be set according to actual application needs. In addition, all steps in this method can be executed by a central processing unit (CPU); to improve processing efficiency, some steps can also be dispatched by the central processing unit to other heterogeneous computing devices for execution. Heterogeneous computing devices refer to computing devices other than the central processing unit, such as graphics cards (GPUs), FPGAs (Field Programmable Gate Arrays), etc.

[0060] For easier understanding, please refer to Figure 1 , Figure 1 A flowchart of a text generation method provided in an embodiment of the present invention, the method may include:

[0061] S101. Receive an input image and input text, and encode the input image and input text respectively to obtain an image representation vector and a text representation vector.

[0062] One possible form of image and text input is:

[0063] {

[0064] "index":"0",

[0065] "input":[

[0066] {

[0067] "image":" / image / path / namg.jpeg"

[0068] "text":"Please describe this image."

[0069] } ]

[0071] }

[0072] Where index is the input sequence number, input represents the actual input content, image is the path to the input image, and text is the input text.

[0073] It should be noted that the embodiments of the present invention do not limit the resolution of the input image or the length of the input text, and can be set according to actual application needs. It is worth pointing out that, because the embodiments of the present invention improve the way the model extracts image and text features, the embodiments of the present invention can achieve a better reduction in computational load when the input image has a large resolution (i.e., image size). Furthermore, to facilitate model processing, the resolution of the input image can be multiple preset resolutions of different sizes. The embodiments of the present invention do not limit the specific value of the preset resolution, and can be set according to actual application needs.

[0074] Furthermore, to perform feature extraction and feature interaction on the input image and input text, this embodiment of the invention first needs to encode the input image and input text to convert them into image representation vectors and text representation vectors that the model can process. This embodiment of the invention does not limit how the input image is encoded; it can be set according to actual application requirements. For example, to facilitate model processing, the input image can first be converted into a square. Subsequently, the input image with the shape transformed can be converted into a tensor, and the tensor can be regularized. Then, an image encoder (such as the Clip-Vit-L14 model) can be used to encode the tensor to obtain the encoded tensor. Subsequently, linear layer processing can be applied to the encoded tensor to further adjust the hidden layer dimensions in the tensor, obtaining the image representation vector. For the process of converting the input image into a square, the color mode of the input image is first converted to a preset color mode, which can be RGB, CMYK, HSB, Lab color mode, etc., so that the model can perform feature extraction processing on the image based on the preset color mode. Then, the length and width of the input image can be determined. Subsequently, the background color can be set according to the average color value of the input image, and the maximum value of the length and width of the input image can be used as the length and width value of the background image. Then, a background image is generated based on the background color and the length and width value of the background image, and the input image is set in the background image (e.g., set in the center of the background image) to obtain the input image after shape transformation.

[0075] Based on this, encoding the input image may include:

[0076] Step 11: Convert the color mode of the input image to a preset color mode, and determine the length and width of the input image.

[0077] Step 12: Set the background color according to the average color value of the input image, take the maximum value of the length and width of the input image as the length and width value of the background image, generate a background image based on the background color and the length and width value of the background image, and set the input image in the background image to obtain a new input image;

[0078] Step 13: Convert the new input image into a first tensor, and regularize the first tensor to obtain a second tensor;

[0079] Step 14: Encode the second tensor using an image encoder to obtain the third tensor;

[0080] Step 15: Perform linear layer processing on the third tensor to obtain the image representation vector.

[0081] It should be noted that, since the resolution of the input images may be different, the dimensions of the first, second, and third tensors corresponding to different images may be different.

[0082] Furthermore, the embodiments of the present invention do not limit how the input text is encoded, and can be set according to actual application requirements, such as performing vector embedding processing on the input text to obtain the text representation vector.

[0083] Based on this, encoding the input text may include:

[0084] Step 21: Perform vector embedding processing on the input text to obtain the text representation vector.

[0085] S102. The image representation vector is used to perform cross-attention processing on the text representation vector, and the cross-attention processing result is subjected to self-attention processing to obtain the feature extraction vector.

[0086] This step aims to extract features from image representation vectors and text representation vectors, enabling the model to acquire semantic information from the input image and text, and generate corresponding response text based on this semantic information. Unlike related technologies, this embodiment does not concatenate image representation vectors and text representation vectors and then perform self-attention processing on the resulting concatenated representation vector. Instead, it first performs cross-attention processing on the text representation vector using the image representation vector, and then performs self-attention processing on the cross-attention result to obtain the feature extraction vector. This reduces computational cost because the computational cost of self-attention processing on the concatenated representation vector is the square of the concatenated representation vector's dimension, while the computational cost of cross-attention processing on the text representation vector using the image representation vector is the product of the dimensions of the image and text representation vectors. Since both the dimensions of the image and text representation vectors are smaller than the dimension of the concatenated representation vector, the product of their dimensions is clearly smaller than the square of the concatenated representation vector's dimension. Furthermore, the dimension of the cross-attention result obtained by performing cross-attention processing on the text representation vector using the image representation vector is actually the dimension of the text representation vector. Typically, because the input text is short, the dimension of the text representation vector is also small. Therefore, performing self-attention processing on the cross-attention result with a dimension equal to the dimension of the text representation vector results in less computation. Thus, this invention significantly reduces the computational cost and improves processing efficiency by improving the way multimodal models extract features from images and text.

[0087] Specifically, this invention sets the text representation vector as query data (Query, Q) and the image representation vector as key data (Key, K) and value data (Value, V), and performs cross-attention processing based on this query data, key data, and value data. Of course, to improve model processing efficiency, the text representation vector and image representation vector can first undergo root mean square layer normalization (RMSNorm) before cross-attention processing; and after completing the cross-attention processing, a linear layer can be applied.

[0088] Based on this, the cross-attention processing of the text representation vector using the image representation vector may include:

[0089] Step 31: Perform root mean square normalization on both the text representation vector and the image representation vector.

[0090] Step 32: Set the processed text representation vector as the query data, and set the processed image representation vector as the key data and value data.

[0091] Step 33: Perform cross-attention processing on the query data, the key data, and the value data, and perform linear layer processing on the processing results.

[0092] Specifically, the query data, the key data, and the value data undergo cross-attention processing according to the following formula:

[0093]

[0094] Wherein, softmax() represents the normalization exponential function, Q represents the query data, K represents the key data, V represents the value data, T represents the transpose operation, and scale represents the dimension corresponding to each attention head among multiple attention heads. The attention heads are used to split the query data, the key data, and the value data.

[0095] To better understand the processing flow of cross-attention, please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of a cross-attention processing method provided in an embodiment of the present invention.

[0096] Furthermore, step S102 can be executed multiple times. It should be noted that, starting from the second round of execution, step S102 changes to: using the image representation vector to perform cross-attention processing on the feature extraction vector of the previous round, and performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector of the current round.

[0097] S103. Perform logical calculations on the feature extraction vector to obtain the output text generated based on the semantics of both the input image and the input text.

[0098] In this step, after feature extraction, only logits need to be performed on the extracted feature vectors to obtain the output text generated based on the semantics of both the input image and the input text. For example, if the input image contains the content "a little girl flying a kite" and the input text is "please describe this image," then the output text could be: "The image depicts a child flying a kite." It should be noted that this embodiment of the invention does not limit the specific process of the logits; relevant machine learning techniques can be referenced.

[0099] Based on the above embodiments, after receiving the input image and input text, the present invention first encodes the input image and input text separately to obtain image representation vectors and text representation vectors. Then, unlike related technologies, the present invention does not concatenate the image representation vector and text representation vector and perform self-attention processing based on the concatenated vector. Instead, it uses the image representation vector to perform cross-attention processing on the text representation vector, and then performs self-attention processing on the cross-attention result to obtain a feature extraction vector. This allows the image representation vector to be used alone to interact with the text representation vector, and then self-attention interaction is performed using the interaction result corresponding to the text representation vector. Thus, because the image processing in the model uses a cross-attention mechanism, there is no increase in the calculation of the square of the input sequence length; and because the dimension of the text representation vector is usually small, the computational power required for self-attention interaction is reduced, thereby reducing the model's computational power requirements and effectively improving the user experience.

[0100] Based on the above embodiments, to further reduce the computational load generated by the interaction between image features and text features, this embodiment of the invention can input a high-resolution image and a low-resolution image, and alternately use the high-resolution image features and low-resolution image features to perform cross-attention processing interaction with the text features, so as to further reduce the computational load by using the low-resolution image. In other words, the input image includes a high-resolution image and a low-resolution image, the image size of the high-resolution image is larger than a preset size, the image size of the low-resolution image is not larger than the preset size, and the high-resolution image representation vector of the high-resolution image and the low-resolution image representation vector of the low-resolution image are sequentially cross-attention processed with the text representation vector. In addition, the high-resolution image and the low-resolution image differ only in image size, but contain the same image content.

[0101] To facilitate model processing, embodiments of the present invention can also set corresponding preset resolutions for high-resolution and low-resolution images, meaning that the resolution of the input high-resolution and low-resolution images must be the same as the preset resolution. Adjustments to image resolution can be performed automatically by the electronic device for user convenience. For example, after receiving the original input image, the electronic device can adjust the resolution of the original input image (e.g., zoom in or out) to obtain the desired high-resolution and low-resolution images.

[0102] Based on this, receiving an input image may include:

[0103] Step 41: Receive the original input image;

[0104] Step 42: Adjust the resolution of the original input image to obtain the high-resolution image and the low-resolution image.

[0105] The following describes the process of alternately applying cross-attention processing to the text representation vector using high-resolution and low-resolution image representation vectors. In one possible scenario, the process of applying cross-attention processing to the text representation vector using the image representation vectors, and then applying self-attention processing to the cross-attention result to obtain the feature extraction vector, may include:

[0106] S201. Input both the high-resolution image representation vector and the text representation vector into the first cross-attention unit, so that the first cross-attention unit can use the high-resolution image representation vector to perform the cross-attention processing on the text representation vector to obtain the first feature vector.

[0107] The high-resolution image representation vector is the image representation vector obtained by encoding a high-resolution image. In this embodiment of the invention, the high-resolution image representation vector can first be used to perform cross-attention processing on the text representation vector. Specifically, this embodiment of the invention can set up a first cross-attention unit dedicated to performing cross-attention processing on the text representation vector using the high-resolution image representation vector. It should be noted that the relevant descriptions of the cross-attention processing are the same as in the above embodiments, and therefore will not be repeated here.

[0108] S202. Input the first feature vector into the first self-attention unit, so that the first self-attention unit performs the self-attention processing on the first feature vector to obtain the second feature vector.

[0109] After completing the first round of cross-attention processing, this embodiment of the invention will perform self-attention processing on the obtained first feature vector. This embodiment of the invention may set up a first self-attention unit, which is dedicated to performing self-attention processing on the first feature vector.

[0110] S203. Input both the low-resolution image representation vector and the second feature vector into the second cross-attention unit, so that the second cross-attention unit can use the low-resolution image representation vector to perform the cross-attention processing on the second feature vector to obtain the third feature vector.

[0111] The low-resolution image representation vector is an image representation vector obtained by encoding a low-resolution image. In this embodiment of the invention, the low-resolution image representation vector can be used to further perform cross-attention processing on the second feature vector. Specifically, this embodiment of the invention can set up a second cross-attention unit dedicated to performing cross-attention processing on the second feature vector using the low-resolution image representation vector.

[0112] S204. The third feature vector is input into the second self-attention unit, so that the second self-attention unit performs the self-attention processing on the third feature vector to obtain the feature extraction vector.

[0113] After completing the second round of cross-attention processing, this embodiment of the invention will perform self-attention processing on the obtained third feature vector. This embodiment of the invention may include a second self-attention unit specifically designed for performing self-attention processing on the third feature vector.

[0114] As can be seen, the embodiments of the present invention can first use high-resolution image representation vectors to perform feature interaction with text representation vectors, and then continue feature interaction using low-resolution image representation vectors. In this way, in multiple rounds of feature interaction, the embodiments of the present invention can avoid continuously using high-resolution image representation vectors for interaction, but can instead use high-resolution image representation vectors and low-resolution image representation vectors in turn for interaction, thereby effectively reducing the amount of computation.

[0115] Furthermore, to improve the processing performance of the above units, each unit may also include a multilayer perceptron (MLP). Specifically, the first cross-attention unit and the second cross-attention unit each include a cross-attention layer and a first multilayer perceptron layer, whereby the first multilayer perceptron performs multilayer perceptron processing on the output of the cross-attention layer; the first self-attention unit and the second self-attention unit each include a self-attention layer and a second multilayer perceptron layer, whereby the second multilayer perceptron performs multilayer perceptron processing on the output of the sub-attention layer. For easier understanding, please refer to [link to documentation]. Figure 3 , Figure 3 This is a schematic diagram of a multimodal model provided in an embodiment of the present invention. CrossAttention represents a cross-attention layer, MLP represents a multilayer perceptron, and SelfAttention represents a self-attention layer. CrossAttention and its subsequent MLP form a cross-attention unit, and SelfAttention and its subsequent MLP form a self-attention unit. Furthermore, in this embodiment of the invention, the first cross-attention unit, the first self-attention unit, the second cross-attention unit, and the second self-attention unit can be used to form a feature extraction unit. To improve feature extraction capabilities, multiple feature extraction units can be cascaded to perform multi-round feature interactions on high-resolution image representation vectors, low-resolution image representation vectors, and text representation vectors.

[0116] Furthermore, for a better understanding of the overall structure of the multimodal model, please refer to [reference needed]. Figure 4 , Figure 4This is a schematic diagram of another multimodal model provided in an embodiment of the present invention. The model may include a high-resolution image preprocessing and encoding module, a low-resolution image preprocessing and encoding module, an embedding module (text embedding module), an image and text feature extraction module, and a Logits calculation module (logic calculation module). The high-resolution image preprocessing and encoding module, the low-resolution image preprocessing and encoding module, and the image and text feature extraction module will be described in detail below.

[0117] 1. High-resolution image preprocessing and encoding module

[0118] like Figure 4 The dashed box in the upper left corner shows the high-resolution image preprocessing and encoding module in the multimodal model. In this module, image encoding can use the Clip-ViT-L14 model, and image preprocessing includes the following steps:

[0119] Step 1: Open the image, convert it to RGB mode, and calculate the length and width of the image.

[0120] Step 2: Based on the larger of the length and width of the original image as a fixed length, and the mean of the normalized value in the image coding model preprocessing, calculate the background image of a fixed length with the mean of the normalized value as the background color.

[0121] Step 3: Copy the original image to the center of the background image to obtain a new input image.

[0122] Step 4: Resize the new input image obtained in Step 3 to a resolution of 1120*1120.

[0123] Step 5: Convert the 1120*1120 pixel image obtained in Step 4 into a Tensor that represents image information.

[0124] Step 6: Regularize the Tensor representing image information obtained in Step 5.

[0125] Step 7: Input the regularized Tensor from Step 6 into the Clip-ViT-L14 model to encode image information, resulting in a Tensor of length [6400, 896].

[0126] Step 8: Take the Tensor of length [6400, 896] from Step 7 and pass it through a Linear layer to change the hidden layer dimension from [6400, 896] to [6400, 2048].

[0127] 2. Low-resolution image preprocessing and encoding module

[0128] like Figure 4The dashed box in the upper right corner shows the low-resolution image preprocessing and encoding module in the multimodal model. In this module, image encoding can use the Clip-ViT-G14 model, and image preprocessing includes the following steps:

[0129] Step 1: Open the image, convert it to RGB mode, and calculate the length and width of the image.

[0130] Step 2: Based on the larger of the length and width of the original image as a fixed length, and the mean of the normalized value in the image coding model preprocessing, calculate the background image of a fixed length with the mean of the normalized value as the background color.

[0131] Step 3: Copy the original image to the center of the background image to obtain a new input image.

[0132] Step 4: Resize the new input image obtained in Step 3 to a resolution of 224*224.

[0133] Step 5: Convert the 224*224 pixel image obtained in Step 4 into a Tensor that represents image information.

[0134] Step 6: Regularize the Tensor representing image information obtained in Step 5.

[0135] Step 7: Input the regularized Tensor from Step 6 into the Clip-ViT-G14 model to encode image information, resulting in a Tensor of length [256, 1664].

[0136] Step 8: Take the [256, 1664] length Tensor from Step 7 and pass it through a Linear layer to change the hidden layer dimension from [256, 1664] to [256, 2048].

[0137] 3. Image and text feature extraction module

[0138] like Figure 4 The main structure consists of an N-layer image and text feature extraction module, which has three inputs: input text information, high-resolution image features, and low-resolution image features.

[0139] For text information, it first goes through the Embedding layer, which converts the text into a vector representation. A simple example is that if you input a sentence: "Please describe this image", this sentence will become a [9,2048] Tensor after passing through the Embedding layer.

[0140] Figure 4The core of the algorithm is an N-layer feature extraction module consisting of cross-attention layers, self-attention layers, and a multilayer perceptron (MLP). In this module, features from the high-resolution image ([6400, 2048]) and the output from the text embedding module ([9, 2048]) undergo cross-attention processing. Specifically, the input text features are first normalized by root mean square (RMSNorm) and then linearly processed to obtain the query data Query(Q). The high-resolution image features are first normalized by RMS and then linearly processed to obtain the key data Key(K) and value data Value(V). The three values ​​Q, K, and V are then used to perform a Softmax (QKT)V / scale calculation, where scale represents the dimension of each attention head in the calculation. When the number of attention heads is 16, the scale is (2048 / 16). 0.5 After the above calculations, the data passes through a linear layer, resulting in text feature output after image and text interaction. Following cross-attention processing, the model goes through a multilayer perceptron layer, then a self-attention layer, followed by another multilayer perceptron layer. After self-attention and multilayer perceptron processing, the text features interact with low-resolution image features. In the low-resolution cross-attention processing, the interaction between image and text features is similar to the high-resolution calculation process described above, only the number of image features represented differs, which will not be elaborated further here.

[0141] After the final model performs Logits calculations, it maps the text output features to the output text, transforming the text feature values ​​into the output text. An example output is: "The picture depicts a child flying a kite."

[0142] As can be seen from the above introduction, when introducing image features into the multimodal model, this invention first introduces two image encoders of different sizes. The interaction method adopts cross-attention processing. While introducing high-resolution features, the computational load of the model is further reduced by interacting with text features through high-resolution and low-resolution images respectively. This is different from the interaction method of concatenating image features and text features and then performing self-attention processing. This method limits the computational load of the model while improving image resolution.

[0143] Based on the above embodiments, to fully leverage the advantages of various computing devices, for feature extraction operations, the central processing unit of the electronic device can distribute image representation vectors and text representation vectors to heterogeneous computing devices, so that the heterogeneous computing devices can perform feature extraction on the image representation vectors and text representation vectors. Based on this, the step of performing cross-attention processing on the text representation vector using the image representation vector, and then performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector, may include:

[0144] S301. The image representation vector and the text representation vector are sent to a heterogeneous computing device so that the heterogeneous computing device can perform the steps of using the image representation vector to perform cross-attention processing on the text representation vector and performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector.

[0145] It should be noted that the embodiments of the present invention are not limited to the type of heterogeneous computing device; for example, it can be a graphics card device.

[0146] The text generation device provided in the embodiments of the present invention will be described below. The smart gateway described below and the message information processing method of the home smart gateway described above can be referred to each other.

[0147] Please refer to Figure 5 , Figure 5 This is a structural block diagram of a text generation device provided in an embodiment of the present invention. The device may include:

[0148] The encoding module 501 is used to receive an input image and input text, and encode the input image and the input text respectively to obtain an image representation vector and a text representation vector;

[0149] The feature extraction module 502 is used to perform cross-attention processing on the text representation vector using the image representation vector, and to perform self-attention processing on the cross-attention processing result to obtain the feature extraction vector.

[0150] The logic calculation module 503 is used to perform logical calculations on the feature extraction vector to obtain the output text generated based on the semantics of the input image and the input text.

[0151] Optionally, the feature extraction module 502 includes:

[0152] The normalization processing submodule is used to perform root mean square normalization processing on both the text representation vector and the image representation vector.

[0153] The settings submodule is used to set the processed text representation vector as query data and the processed image representation vector as key data and value data;

[0154] The cross-attention processing submodule is used to perform cross-attention processing on the query data, the key data, and the value data, and to perform linear layer processing on the processing results.

[0155] Optionally, the cross-attention processing submodule can be used to:

[0156] The query data, the key data, and the value data are subjected to cross-attention processing according to the following formula:

[0157] softmsacxa(lQeK) T V;

[0158] Wherein, softmax() represents the normalization exponential function, Q represents the query data, K represents the key data, V represents the value data, T represents the transpose operation, and scale represents the dimension corresponding to each attention head among multiple attention heads. The attention heads are used to split the query data, the key data, and the value data.

[0159] Optionally, the input image includes a high-resolution image and a low-resolution image, wherein the image size of the high-resolution image is larger than a preset size, and the image size of the low-resolution image is not larger than the preset size;

[0160] The feature extraction module 502 may include:

[0161] The first input submodule is used to input both the high-resolution image representation vector and the text representation vector into the first cross-attention unit, so that the first cross-attention unit uses the high-resolution image representation vector to perform the cross-attention processing on the text representation vector to obtain the first feature vector.

[0162] The second input submodule is used to input the first feature vector into the first self-attention unit, so that the first self-attention unit performs the self-attention processing on the first feature vector to obtain the second feature vector.

[0163] The third input submodule is used to input both the low-resolution image representation vector and the second feature vector into the second cross-attention unit, so that the second cross-attention unit can use the low-resolution image representation vector to perform the cross-attention processing on the second feature vector to obtain the third feature vector.

[0164] The fourth input submodule is used to input the third feature vector into the second self-attention unit, so that the second self-attention unit performs the self-attention processing on the third feature vector to obtain the feature extraction vector.

[0165] Optionally, the encoding module 501 may include:

[0166] The image receiving submodule is used to receive the raw input image;

[0167] The image resolution adjustment submodule is used to adjust the resolution of the original input image to obtain the high-resolution image and the low-resolution image.

[0168] Optionally, both the first cross-attention unit and the second cross-attention unit include a cross-attention layer and a first multilayer perceptron layer, wherein the first multilayer perceptron performs multilayer perceptron processing on the output of the cross-attention layer; both the first self-attention unit and the second self-attention unit include a self-attention layer and a second multilayer perceptron layer, wherein the second multilayer perceptron performs multilayer perceptron processing on the output of the sub-attention layer.

[0169] Optionally, the first cross-attention unit, the first self-attention unit, the second cross-attention unit, and the second self-attention unit constitute a feature extraction unit, and multiple feature extraction units are connected in series.

[0170] Optionally, the encoding module 501 may include:

[0171] The first preprocessing submodule is used to convert the color mode of the input image to a preset color mode and determine the length and width of the input image;

[0172] The second preprocessing submodule is used to set the background color according to the color mean of the input image, take the maximum value of the length and width of the input image as the length and width value of the background image, generate a background image based on the background color and the length and width value of the background image, and set the input image in the background image to obtain a new input image;

[0173] The tensor transformation submodule is used to convert the new input image into a first tensor and regularize the first tensor to obtain a second tensor.

[0174] The image encoding submodule is used to encode the second tensor using an image encoder to obtain the third tensor;

[0175] The linear processing submodule is used to perform linear layer processing on the third tensor to obtain the image representation vector.

[0176] Optionally, the encoding module 501 may include:

[0177] The text encoding submodule is used to perform vector embedding processing on the input text to obtain the text representation vector.

[0178] Optionally, the feature extraction module 502 includes:

[0179] The task distribution submodule is used to distribute the image representation vector and the text representation vector to a heterogeneous computing device, so that the heterogeneous computing device can perform the steps of using the image representation vector to perform cross-attention processing on the text representation vector, and performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector.

[0180] Please refer to Figure 6 , Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The present invention provides an electronic device 60, including a processor 61 and a memory 62; wherein, the memory 62 is used to store a computer program; the processor 61 is used to execute the text generation method provided in the foregoing embodiment when executing the computer program.

[0181] For details regarding the specific process of the above text generation method, please refer to the relevant content provided in the foregoing embodiments, which will not be repeated here.

[0182] Furthermore, the memory 62, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0183] In addition, the electronic device 60 also includes a power supply 63, a communication interface 64, an input / output interface 65, and a communication bus 66; wherein, the power supply 63 is used to provide operating voltage for the various hardware devices on the electronic device 60; the communication interface 64 can create a data transmission channel between the electronic device 60 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface 65 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0184] Optionally, the electronic device may further include: a heterogeneous computing device, configured to receive image representation vectors and text representation vectors sent by the processor, perform cross-attention processing on the text representation vectors using the image representation vectors, and perform self-attention processing on the cross-attention processing results to obtain feature extraction vectors.

[0185] Optionally, the heterogeneous computing device is a graphics card device.

[0186] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the text generation method described in the above embodiments.

[0187] Since the embodiments of the computer program product part correspond to the embodiments of the text generation method part, please refer to the description of the embodiments of the text generation method part for the embodiments of the computer program product part, and will not be repeated here.

[0188] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the text generation method described in the above embodiments.

[0189] Since the embodiments of the computer-readable storage medium portion correspond to the embodiments of the text generation method portion, the embodiments of the storage medium portion are described in the text generation method portion description and will not be repeated here.

[0190] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0191] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0192] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0193] The foregoing has provided a detailed description of the text generation method, apparatus, electronic device, program product, and storage medium provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A text generation method characterized by, include: Receive an input image and input text, and encode the input image and input text respectively to obtain an image representation vector and a text representation vector; The image representation vector is used to perform cross-attention processing on the text representation vector, and the cross-attention processing result is then subjected to self-attention processing to obtain the feature extraction vector. Logical calculations are performed on the feature extraction vectors to obtain the output text generated based on the semantics of both the input image and the input text; The receiving of the input image includes: Receive the raw input image; The original input image is resolution adjusted to obtain a high-resolution image and a low-resolution image; the image size of the high-resolution image is larger than a preset size, and the image size of the low-resolution image is not larger than the preset size. The process of applying cross-attention to the text representation vector using the image representation vector includes: Both the text representation vector and the image representation vector are subjected to root mean square normalization. Set the processed text representation vector as the query data, and set the processed image representation vector as the key data and value data; Cross-attention processing is performed on the query data, the key data, and the value data, and the processing result is processed using a linear layer. The image representation vector includes a high-resolution image representation vector of the high-resolution image and a low-resolution image representation vector of the low-resolution image. The step of performing cross-attention processing on the text representation vector using the image representation vector, and then performing self-attention processing on the cross-attention result to obtain the feature extraction vector, includes: Both the high-resolution image representation vector and the text representation vector are input into the first cross-attention unit, so that the first cross-attention unit uses the high-resolution image representation vector to perform the cross-attention processing on the text representation vector to obtain the first feature vector; The first feature vector is input into the first self-attention unit, so that the first self-attention unit performs the self-attention processing on the first feature vector to obtain the second feature vector; Both the low-resolution image representation vector and the second feature vector are input into the second cross-attention unit, so that the second cross-attention unit uses the low-resolution image representation vector to perform the cross-attention processing on the second feature vector to obtain the third feature vector; The third feature vector is input into the second self-attention unit, so that the second self-attention unit performs the self-attention processing on the third feature vector to obtain the feature extraction vector.

2. The text generation method of claim 1, wherein, Both the first cross-attention unit and the second cross-attention unit include a cross-attention layer and a first multilayer perceptron layer. The first multilayer perceptron performs multilayer perceptron processing on the output of the cross-attention layer. Both the first self-attention unit and the second self-attention unit include a self-attention layer and a second multilayer perceptron layer. The second multilayer perceptron performs multilayer perceptron processing on the output of the self-attention layer.

3. The text generation method of claim 1, wherein, The first cross-attention unit, the first self-attention unit, the second cross-attention unit, and the second self-attention unit constitute a feature extraction unit, and multiple feature extraction units are connected in series.

4. The text generation method of claim 1, wherein, Encoding the input image includes: Convert the color mode of the input image to a preset color mode, and determine the length and width of the input image; The background color is set according to the average color value of the input image. The maximum value of the length and width of the input image is used as the length and width value of the background image. A background image is generated based on the background color and the length and width value of the background image. The input image is then set in the background image to obtain a new input image. The new input image is converted into a first tensor, and the first tensor is regularized to obtain a second tensor; The second tensor is encoded using an image encoder to obtain the third tensor; The third tensor is processed by a linear layer to obtain the image representation vector.

5. The text generation method of claim 1, wherein, Encoding the input text includes: The input text is subjected to vector embedding processing to obtain the text representation vector.

6. The text generation method of claim 1, wherein, The step of performing cross-attention processing on the text representation vector using the image representation vector, and then performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector, includes: The image representation vector and the text representation vector are sent to a heterogeneous computing device so that the heterogeneous computing device can perform the steps of using the image representation vector to perform cross-attention processing on the text representation vector and performing self-attention processing on the cross-attention processing result to obtain the feature extraction vector.

7. The text generation method according to claim 6, characterized in that, The heterogeneous computing device is a graphics card device.

8. A text generation device, characterized in that, include: An encoding module is used to receive an input image and input text, and encode the input image and the input text respectively to obtain an image representation vector and a text representation vector; The feature extraction module is used to perform cross-attention processing on the text representation vector using the image representation vector, and to perform self-attention processing on the cross-attention processing result to obtain the feature extraction vector. The logic calculation module is used to perform logical calculations on the feature extraction vector to obtain the output text generated based on the semantics of both the input image and the input text; The encoding module includes: The image receiving submodule is used to receive the raw input image; The image resolution adjustment submodule is used to adjust the resolution of the original input image to obtain a high-resolution image and a low-resolution image; the image size of the high-resolution image is larger than a preset size, and the image size of the low-resolution image is not larger than the preset size. The feature extraction module includes: The normalization processing submodule is used to perform root mean square normalization processing on both the text representation vector and the image representation vector. The settings submodule is used to set the processed text representation vector as query data and the processed image representation vector as key data and value data; The cross-attention processing submodule is used to perform cross-attention processing on the query data, the key data, and the value data, and to perform linear layer processing on the processing results; The image representation vector includes a high-resolution image representation vector of the high-resolution image and a low-resolution image representation vector of the low-resolution image. The feature extraction module includes: The first input submodule is used to input both the high-resolution image representation vector and the text representation vector into the first cross-attention unit, so that the first cross-attention unit can use the high-resolution image representation vector to perform the cross-attention processing on the text representation vector to obtain the first feature vector. The second input submodule is used to input the first feature vector into the first self-attention unit, so that the first self-attention unit performs the self-attention processing on the first feature vector to obtain the second feature vector. The third input submodule is used to input both the low-resolution image representation vector and the second feature vector into the second cross-attention unit, so that the second cross-attention unit can use the low-resolution image representation vector to perform the cross-attention processing on the second feature vector to obtain the third feature vector; The fourth input submodule is used to input the third feature vector into the second self-attention unit, so that the second self-attention unit can perform the self-attention processing on the third feature vector to obtain the feature extraction vector.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the text generation method as described in any one of claims 1 to 7.

10. The electronic device according to claim 9, characterized in that, Also includes: A heterogeneous computing device is used to receive image representation vectors and text representation vectors sent by the processor, perform cross-attention processing on the text representation vectors using the image representation vectors, and perform self-attention processing on the cross-attention processing results to obtain feature extraction vectors.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the text generation method as described in any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the text generation method as described in any one of claims 1 to 7.

Images