Text generation method, device and computing device

Abstract

Provided are a text generation method, apparatus, and computing device. The text generation method may include: obtaining input data of multiple modalities; generating fused feature representations from the input data using multiple encoders corresponding to the input data of the multiple modalities; and generating output text associated with the input data of the multiple modalities based on the fused feature representations using a decoder, wherein the decoder includes multiple decoding layer blocks and a hybrid cross-attention mechanism module included in each decoding layer block, wherein the hybrid cross-attention mechanism module in each decoding layer block includes a cross-attention module, multiple conversion parameter group modules for the cross-attention module, and a selection module, wherein the selection module is used to select one or more conversion parameter group modules from the multiple conversion parameter group modules based on the fused feature representations to generate a conversion parameter group used by the cross-attention module.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, and computing device for generating text based on input data of multiple modalities. Background Art

[0002] A large language model (LLM) is a large natural language processing model trained with huge amounts of text data and obtained based on deep learning. It can understand the meaning of language text or generate natural language text. As ChatGPT and other algorithms have demonstrated extraordinary capabilities in various fields, multimodal large language models (MLLMs) have recently become a hot topic of research. A multimodal large language model is a model that combines multiple different types (different modalities) of input data. Different types of input can be, for example, text, images, audio, etc. The multimodal large language model uses a powerful large language model (LLM) as a "brain" to perform various multimodal tasks. The goal of a multimodal large language model is to learn to extract useful information from multiple input data of different types and fuse them into a unified representation. For example, a multimodal large language model can use image and text input at the same time and output a text description of the image.

[0003] Therefore, how to more efficiently and accurately use large language models for text generation based on input data of multiple modalities is currently a research focus in this field. Summary of the invention

[0004] In one aspect of the present application, a method for generating text is provided. The method includes: obtaining input data of multiple modalities; using multiple encoders corresponding to the input data of the multiple modalities to generate fused feature representations from the input data; using a decoder to generate output text associated with the input data of the multiple modalities based on the fused feature representations, wherein the decoder includes multiple decoding layer blocks and a hybrid cross-attention mechanism module included in each decoding layer block, wherein the hybrid cross-attention mechanism module in each decoding layer block includes a cross-attention module, multiple conversion parameter group modules for the cross-attention module, and a selection module, wherein the selection module is used to select one or more conversion parameter group modules from the multiple conversion parameter group modules based on the fused feature representations to generate a conversion parameter group used by the cross-attention module.

[0005] According to an embodiment of the present application, the output text associated with the input data of the multiple modalities is generated based on the fused feature representation using a decoder, including: for each decoding layer block, based on the fused feature representation, using the conversion parameter group in each conversion parameter group module included in the hybrid cross-attention mechanism module in the decoding layer block to determine a plurality of key-value combinations; based on the fused feature representation, using the selection module to determine the key-value combination for the cross-attention module in the decoding layer block from the plurality of key-value combinations; and based on the determined key-value combination, using the cross-attention module to generate the output representation of the decoding layer block for generating input information for the next decoding layer block or for generating the output text.

[0006] According to an embodiment of the present application, the fused feature representation includes feature representations corresponding to multiple word-grams obtained after the input data is tokenized, wherein each of the multiple key-value combinations includes information associated with the multiple word-grams, wherein determining the key-value combination for the cross-attention module in the decoding layer block includes: using the selection module and based on the fused feature representation, for each word-gram obtained after the input data is tokenized, determining the conversion parameter group module corresponding to the word-gram; using the selection module and based on the conversion parameter group module corresponding to each word-gram, determining the key-value combination corresponding to each word-gram from the multiple key-value combinations; and determining the information associated with the word-gram from the key-value combination corresponding to each word-gram, wherein the determined information associated with each word-gram constitutes the key-value combination of the cross-attention module for the fused feature representation.

[0007] According to an embodiment of the present application, each key-value combination includes a key matrix K and a value matrix V, wherein determining multiple key-value combinations includes: using the conversion parameter group in each conversion parameter group module in the decoding layer block to perform a linear transformation on the fused feature representation to determine the key matrix K and value matrix Q corresponding to each conversion parameter group module.

[0008] According to an embodiment of the present application, the method also includes: performing a linear transformation on the input information of the decoding layer block to determine a query matrix Q for the cross-attention module in the decoding layer block, wherein the input information of the decoding layer block includes the output information of the previous decoding layer block or the encoded embedded information of the user instruction.

[0009] According to an embodiment of the present application, the method also includes: training the multiple encoders and the decoders, wherein the model parameters in the hybrid cross-attention mechanism module in the multiple encoders and each decoding layer block are adjusted during the training process, wherein the model parameters in each decoding layer block except the model parameters in the hybrid cross-attention mechanism module are pre-trained and do not need to be adjusted during the training process.

[0010] According to an embodiment of the present application, the input information of the decoder includes a user instruction as a start instruction for the text generation operation.

[0011] According to an embodiment of the present application, the method also includes: tokenizing the input data of each modality in the input data, and converting it into a latent space matrix of the same dimension as the corresponding encoder as the input of the corresponding encoder, wherein the multiple encoders respectively encode the corresponding latent space matrices of the input to obtain multiple encoding output matrices, and the multiple encoding output matrices are merged to obtain the fused feature representation.

[0012] According to an embodiment of the present application, the architecture of each of the multiple encoders adopts the encoder architecture in the Transformer-based large language model, and the architecture of the decoder adds the hybrid cross-attention mechanism module to the decoder architecture in the Transformer-based large language model.

[0013] According to an embodiment of the present application, the latent space dimension of each encoder among the multiple encoders is smaller than the latent space dimension of the decoder.

[0014] According to another aspect of the present application, a device for generating text is provided, comprising: a data acquisition unit for acquiring input data of multiple modalities; an encoding unit for generating a fused feature representation from the input data using multiple encoders corresponding to the input data of the multiple modalities; and a decoding unit for generating an output text associated with the input data of the multiple modalities based on the fused feature representation using a decoder, wherein the decoder comprises multiple decoding layer blocks and a hybrid cross-attention mechanism module included in each decoding layer block, wherein the hybrid cross-attention mechanism module in each decoding layer block comprises a cross-attention module, multiple conversion parameter group modules for the cross-attention module, and a selection module, wherein the selection module is used to select one or more conversion parameter group modules from the multiple conversion parameter group modules based on the fused feature representation for generating a conversion parameter group used by the cross-attention module.

[0015] According to another aspect of the present application, a computing device is provided, comprising: one or more processors; and one or more memories on which a computer program or computer instructions are stored, wherein the computer program or computer instructions implement the method as described above when executed by the one or more processors.

[0016] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program or computer instructions are stored. When the computer program or computer instructions are executed by the one or more processors, the method as described above is implemented.

[0017] According to another aspect of the present application, a computer program product is provided, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by the one or more processors, the method as described above is implemented.

[0018] According to the embodiments of the present application, by using multiple different encoders to process input data of different modalities, each modality has a dedicated encoder, so it can flexibly adapt to different multi-modal application scenarios; in addition, a hybrid cross-attention mechanism is introduced into the existing decoder structure, the encoder's latent space information is directly mixed into the attention module of each layer of the large language model, and different conversion parameter group modules (for example, KV weight matrix) can be selected for different word units, which has a stronger ability to understand input data of multiple modalities. In addition, due to the latent space dimension (D E ) is smaller than the latent space dimension of the decoder (D D ), using an encoder + decoder structure is better than a pure decoder structure (such as Figure 2 ) can compress long sequence inputs more efficiently. In addition, in terms of training complexity, since only the hybrid cross-attention mechanism module of the encoder dedicated to each modality and the decoder in the multimodal large language model needs to be trained for different multi-modal input data, without fine-tuning the multimodal large language model as a whole, the training cost is lower than training the traditional encoder + decoder model from scratch. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0020] Figure 1 An exemplary system architecture that can be used to implement the text generation process is shown.

[0021] Figure 2 The example architecture of a large language model based on a Transformer model, i.e., an encoder and a decoder, is shown.

[0022] Figure 3 An example architecture of a large language model based on a decoder of a Transformer model is shown.

[0023] Figure 4 A flowchart of a text generation method according to an embodiment of the present application is shown.

[0024] Figure 5 and Figure 6 A schematic diagram of the architecture of a multimodal large language model according to an embodiment of the present application is shown.

[0025] Figure 7 Shows Figure 4 More details of step S430 in the method are shown.

[0026] Figure 8 A schematic diagram of a K-hot matrix for multiple word units is shown.

[0027] Fig. 9 A schematic structural diagram of a text generation device according to an embodiment of the present application is shown.

[0028] Fig.10 A schematic block diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] Application scenarios of multimodal large language models may include financial analysis (for example, being able to integrate text, numerical, table, and image data, so that after training, it can perform well in processing financial documents and data, such as realizing financial report analysis, market trend forecasting, etc.); or, it can also be used in the medical field. For example, users can input input data in multiple modalities including patient voice descriptions, CT images, MRI images, and electronic medical record texts, as well as user instructions (for example, inquiry instructions regarding the patient's condition). Based on the architecture and method provided in the embodiments of the present application, output text can be generated, and the output text may include diagnosis and treatment recommendations for the patient's condition, thereby realizing intelligent consultation services.

[0031] In order to facilitate the understanding of the embodiments of the present application, Figure 1 An exemplary system architecture for implementing the text generation process is shown.

[0032] like Figure 1 As shown, the system architecture 100 includes: a server 110 , a network 120 and a user device 130 .

[0033] The server 110 can provide text generation services to users. In the process of using the text generation service, the user can input input data of multiple modes through the user device 130 and send it to the server through the network 120. The server 110 can also download the input data of multiple modes from the network in response to the user instruction sent from the user device 130 via the network. Considering the large storage requirements and processing and operation requirements of the multimodal large language model (as will be described in detail later), the multimodal large language model is generally arranged at the server instead of at the user device (this application takes this as an example). Of course, if the computing power and storage capacity of the user device are sufficient, it can also be set at the user device.

[0034] The network 120 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

[0035] The server 110 can use the multimodal large language model of the embodiment of the present application (as will be described in detail later) according to the proposed text generation method to process the acquired input data of multiple modalities and generate output text, which can be returned to the user device 130 through the network. The multimodal large language model can be trained at the server 310, and the trained multimodal large language model can be used to generate output text according to actual input data of multiple modalities.

[0036] The server 110 may be one or more servers and may be set up in an independent or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in a cloud computing service system to solve the defects of difficult management and weak service scalability in traditional physical hosts and virtual private servers (VPS) services.

[0037] The user equipment 130 may include, but is not limited to, a smart mobile terminal, a smart home device, a wearable device, a PC (Personal Computer), etc.

[0038] As previously mentioned, there is a need to more efficiently and accurately use large language models for text generation based on input data from multiple modalities.

[0039] According to the embodiments of the present application, some multimodal large language models are based on the architecture of the Transformer model, that is, a neural network model including an encoder and a decoder, which uses an attention mechanism to model sequence data, and is widely used in language models, such as Figure 2 In addition, other multimodal large language models (for example, State-of-the-art large language models such as GPT and LLaMA) only use the decoder part of the Transformer model architecture, and uniformly convert the input data of multiple modalities into the latent space of the decoder part as the input of the large language model, as shown in Figure 3 shown.

[0040] However, for the above two model architectures, the architecture based on the traditional Transformer model (including encoder and decoder) requires training and parameter adjustment of both the encoder and decoder. Experimental verification shows that the text output effect is not as good as the model architecture including a pure decoder.

[0041] In addition, for Figure 2 The model architecture shown, including a pure decoder, also has the following problems:

[0042] The decoder is essentially a single model that can only recognize one modality, i.e., its ability to understand multiple modalities is weak.

[0043] Due to the limitation of the context window of the decoder and the large dimension of the latent space of the decoder, the decoder cannot handle long sequence inputs that exceed the training scale of its base model (for example, if the input text data is long, the image is large, etc.).

[0044] The large language model (multiple decoders) needs to be trained and adjusted as a whole for input data of different modalities, so the training cost is high.

[0045] It can be seen that the above-mentioned multimodal large language model has the problems of high training complexity, limitation based on the sequence length of input data and insufficient recognition ability.

[0046] Therefore, the present application proposes a text generation method, apparatus and computing device based on an improved model architecture, wherein the improved model can process input data of different types (modalities) so as to be flexibly adapted to different multimodal application scenarios, and can have strong comprehension capabilities. In addition, it can compress long sequence inputs and reduce training complexity.

[0047] The following will be combined with the attached Figure 4-7 The embodiments of the present application are described in detail.

[0048] Figure 4 A flowchart of a text generation method according to an embodiment of the present application is shown. The method can be Figure 3 Server implementation in the system architecture shown.

[0049] like Figure 4 As shown, in step S410, input data of multiple modalities are obtained.

[0050] For example, multimodality refers to expressing or perceiving things from multiple modalities, including text, images, audio, video (which can be regarded as a combination of images and audio), etc., that is, multimodality can be divided into text modality and non-text modality, among which non-text modality can include images, audio, video, etc.

[0051] In step S420, a plurality of encoders corresponding to the input data of the plurality of modalities are used to generate a fused feature representation for the input data.

[0052] Optionally, the structure of each encoder is the same. For example, the architecture of each encoder adopts the encoder architecture in the large language model based on Transformer, for example, it can include a plurality of cascaded coding layer blocks, each coding layer block includes a cascaded self-attention mechanism layer (Self-Attention), a first residual connection & layer normalization layer (Add&layerNorm or also written as Add&Norm), a feed forward layer (Feed Forward) and a second residual connection & layer normalization layer (Add&layerNorm), their functions and operation processing procedures are well known in the art, for example, residual connection is used to prevent network damage, and layer normalization is used to normalize the startup value of each layer, in order to avoid blurring this application, so a detailed description thereof is omitted here. The various model parameters inside the encoder (for example, parameters of the attention mechanism layer and multilayer perceptron, etc.) can be learned during the training process.

[0053] The input of each coding layer block starting from the second coding layer block is the output of the previous coding layer block, and the input of the first coding layer block is the feature representation of the input data of the corresponding modality after preprocessing, that is, before the input data of each modality is input into the corresponding encoder, the input data of each modality needs to be preprocessed to align them to the latent space of the encoder.

[0054] Optionally, different preprocessing methods can be used for text modalities and non-text modalities. The input data of each modality can be tokenized (e.g., text tokens, image tokens, audio tokens, and video tokens) and then converted to the latent space of the encoder using a feature extractor (e.g., an audio encoder, a video encoder, or an image encoder) and an optional linear projection layer to obtain a latent space matrix with the same dimension as the encoder. The dimension of the latent space matrix obtained by converting the input data of each modality is [L, D E ], where L is the length of the word sequence of the input data of the modality (the number of tokens), D E is the latent space dimension of the encoder.

[0055] For example, for audio, video or image input data, after tokenization (for example, pixel segmentation of an image, spatial and temporal segmentation of input data of other modalities), the input data can be converted to the latent space of the encoder using a corresponding feature extractor (for example, an audio encoder, a video encoder or an image encoder) and a linear projection layer. In this way, the audio, video or image input data is also converted into a latent space matrix with the same dimension as the encoder. Optionally, the image encoder can be implemented based on a deep CNN (Convolutional Network) or other such as an autoencoder, and the audio encoder can be implemented based on a deep RNN (Recurrent Neural Network) or Transformer structure to capture the temporal features in the audio; the video encoder can be implemented based on a visual transformation network ViT or a three-dimensional convolutional neural network.

[0056] For text input data, the text input data can be tokenized and then embedded using the embedding layer corresponding to the corresponding encoder to obtain a latent space matrix with the same dimension as the encoder. The embedding layer usually embeds the text sequence corresponding to the input data of the text modality based on Token (element) to obtain the feature representation of the input data of the text modality (also called embedding representation). The above-mentioned token-based embedding processing can include word embedding, and can further include position embedding, etc. Word embedding is to encode each token into a word vector to obtain a word vector representation. Position embedding is to encode the position of each token in the text sequence to obtain a position representation. Similarly, non-text modalities such as image tokens and audio tokens can also be combined with position representations during encoding and projection (not shown in the figure).

[0057] That is, different feature extractors are used for input data of different modalities (the embedding layer for text input data is also a type of feature extractor), and then the feature representations of each modality are aligned and uniformly converted to the latent space (i.e., feature space) used by the encoder, that is, the feature representations of each modality are aligned in the latent space of the encoder. Optionally, the latent space dimension can be comprehensively selected based on the size of the input data of each modality.

[0058] In this way, the input of each encoder is the latent space matrix of its corresponding mode. After encoding the latent space matrix, the encoder obtains the encoding output matrix. All encoders encode the corresponding latent space matrix of the input respectively to obtain multiple encoding output matrices, and then merge the multiple encoding output matrices to obtain the fusion feature representation [∑L i ,D E ], where ∑L i is the sum of the lengths of word sequences of all modalities.

[0059] In step S430, a decoder is used to generate output text associated with the input data of the multiple modalities based on the fused feature representation, wherein the hybrid cross-attention mechanism module in each decoding layer block includes a cross-attention module, multiple transformation parameter group modules for the cross-attention module, and a selection module, wherein the selection module is used to select one or more transformation parameter group modules from the multiple transformation parameter group modules based on the fused feature representation to generate a transformation parameter group used by the cross-attention module.

[0060] For ease of understanding, Figure 5 and Figure 6 A schematic diagram of the architecture of a multimodal large language model according to an embodiment of the present application is shown.

[0061] exist Figure 5 In , the number of encoders is the same as the number of modalities, and each encoder can adopt the encoder architecture used in the large Transformer-based language model. As mentioned earlier, and in Figure 5 and Figure 6 As shown in FIG, the input end of the encoder for the non-text modality also receives the latent space matrix after the feature extractor and the linear projection layer, and the input end of the encoder for the text modality also receives the latent space matrix after the embedding process.

[0062] in addition, Figure 5 and Figure 6Taking the decoder architecture in the Transformer-based large language model (which is also the LLM architecture currently used by ChatGPT, etc.) as an example, improvements based on it are made. Those skilled in the art should understand that there may be other encoder and decoder architectures at present or in the future, as long as the hybrid cross attention mechanism module can be combined with its processing process.

[0063] Figure 5 and Figure 6 Each module inside the decoder shown in the figure may be implemented by a computer program or an instruction set running on a processor.

[0064] like Figure 5 and Figure 6 As shown, the decoder includes multiple (Nd) decoding layer blocks, each decoding layer block is additionally added with a mixed cross attention mechanism module (Mixture-of-attention cross-attention, MoA attention) and a corresponding residual connection & layer normalization layer (Add&layerNorm) compared to each decoding layer block in the current Transformer model architecture. Each of the multiple conversion parameter group modules of the mixed cross attention mechanism module can receive the fusion feature representation [∑L i ,D E ], and use the K weight matrix and V weight matrix to calculate the K matrix and V matrix required in the attention mechanism. After training, the selection module can select the most appropriate set of conversion parameters for each word in the fused feature representation based on the current fused feature representation, thereby providing more accurate K and V matrices to the corresponding cross-attention module, making it more capable of understanding input data of multiple modalities.

[0065] In the Transformer model, the attention mechanism may involve self-attention and cross-attention, and optionally, multi-head attention (similar to self-attention). Self-attention is a method of assigning weights to each word in the input sequence by calculating the correlation between the query (Q), key (K), and value (V). It builds a global context representation by comparing elements at different positions, while cross-attention introduces additional input sequences based on self-attention to fuse information from multiple sources, which interact with each other through the cross-attention mechanism, thereby better capturing the dependencies between different input sequences. For example, in Figure 6In the hybrid cross attention mechanism module shown, the decoder input includes user instructions and / or a generated partial sequence (after embedding processing), and a query vector is generated at each position, which is used to calculate the attention weight at all positions of the encoder output (for example, the fused feature representation in the previous text). A set of key vectors and value vectors are generated at all positions of the encoder output, and the query vector and the key vector are used for dot product operation, and the attention weight is obtained through the softmax function. The attention weight is then multiplied by the value vector and the result is summed to obtain the encoder-adjusted output, which is then passed through the residual connection & layer normalization layer and the feedforward layer and another residual connection & layer normalization layer as the input information of the next decoder layer or further linearly transformed and softmax function to generate a new word as part of the output text.

[0066] The working principle and operation process of the attention mechanism are well known in the art, so a detailed description thereof is omitted here. In this way, the embodiment of the present application can better understand and integrate information of different modalities through the model of the cross attention mechanism, thereby improving the ability to process complex scenes.

[0067] Furthermore, the conversion parameter group suitable for each word may not come from the same conversion parameter group module. Therefore, in some embodiments of the present application, the most suitable conversion parameter group can be selected from multiple conversion parameter groups for each word, so that a new conversion parameter group for fusion feature representation can be recombined.

[0068] like Figure 7 As shown, the step S430 of generating the output text may further include the following sub-steps: The following sub-steps are performed in each decoding layer block.

[0069] In sub-step S430-1, multiple key-value combinations can be determined based on the fused feature representation using the transformation parameter groups in each transformation parameter group module included in the hybrid cross-attention mechanism module in the decoding layer block.

[0070] Optionally, each conversion parameter group module outputs a corresponding key-value combination based on the input fused feature representation and its conversion parameter group, and the key-value combination may include a key matrix K and a value matrix V. When determining the multiple key-value combinations, the conversion parameter group in each conversion parameter group module in the decoding layer block can be used to perform a linear transformation on the fused feature representation from multiple encoders to determine the corresponding key matrix K and value matrix V of each expert attention mechanism.

[0071] For example, the transformation parameter group of each transformation parameter group module is implemented as a key weight matrix W k and a value weight matrix Wv The key weight matrix W k And the dimension of the value weight matrix Wv is [D E ,D D ], where D D is the size of the latent space of the decoder. The fusion feature is represented by (dimension [∑L i ,D E ]) and the weight matrices corresponding to the multiple conversion parameter group modules are multiplied to obtain the K and V matrices of each conversion parameter group module, and their dimensions are [∑L i ,D D ] Key weight matrix W k And the value weight matrix W v And the query weight matrix W mentioned later q The values ​​of are learned during the model training process. Each weight matrix in each conversion parameter group module will be applied to each word in the fusion feature representation. For example, the first word and the second word need to pass through the key weight matrix W k And the value weight matrix W v To achieve the same linear transformation, the key matrix K and value matrix V output by the conversion parameter group module are obtained.

[0072] In addition, since a query matrix Q is also required in the cross-attention mechanism, the method may further include determining a query matrix of the cross-attention mechanism in each decoding layer block. For example, the query matrix Q for the cross-attention module in the decoding layer block may be determined based on the input information of the decoding layer block and the query weight parameter, wherein the input information of the decoding layer block includes the output information of the previous decoding layer block or the user instruction and / or the encoding embedding information of the generated partial sequence. For example, the input information or the encoding embedding information is combined with the corresponding query weight parameter W q Multiply them to obtain the query matrix Q.

[0073] In sub-step S430-2, the key-value combination for the cross-attention module in the decoding layer block can be determined from the multiple key-value combinations using the selection module based on the fused feature representation.

[0074] The fused feature representation output by the encoder includes the feature representations corresponding to multiple word units obtained after the input data is tokenized, for example, as described above, the total ∑L i As described above, each weight matrix in each conversion parameter group module will be applied to each word in the fused feature representation, so the key-value combination output by each corresponding conversion parameter group module should include information associated with the multiple word-grams (i.e., all word-grams in the fused feature representation).

[0075] Optionally, a selection module can be used to determine, for each word-gram obtained after tokenization of the input data based on the fused feature representation, a conversion parameter group module corresponding to the word-gram; a key-value combination corresponding to each word-gram can be determined from the multiple key-value combinations based on the selection module and the conversion parameter group module corresponding to each word-gram; and information associated with the word-gram can be determined from the key-value combination corresponding to each word-gram, wherein the determined information associated with each word-gram constitutes the key-value combination of the cross-attention module for the fused feature representation.

[0076] Optionally, when the key-value combination corresponding to the current word is multiple key-value combinations, the multiple key-value combinations are added together to serve as the key-value combination corresponding to the current word.

[0077] Optionally, the selection module may include a hybrid routing layer-MoA routing layer (MoA Router) and a masked selector (Masked Selector).

[0078] The hybrid routing layer can be composed of a linear projection layer and a TopK selection layer, both of which can learn better model parameter values ​​through training. The dimension of the weight of the linear projection layer is [D E ,N Expert ], where N Expert is the total number of transformation parameter group modules in the hybrid attention mechanism. The fused feature representation from the encoder passes through the linear projection layer, and the output size is [∑L i ,N Expert ], which means the probability of selecting a certain conversion parameter group module for each word. Finally, the TopK selection layer selects K conversion parameter group modules based on these probabilities and obtains a [∑Li,N Expert ] size is used to represent the transformation parameter group module (for example, referred to by ID) for each word unit selection, such as Figure 8 shown.

[0079] For example, in Figure 8 In the figure, the ID corresponding to each word in the fusion feature representation is represented by the K-hot method. It can be seen that word 0 selects conversion parameter group modules 1 and 2, and word 1 selects conversion parameter group modules 3 and 5.

[0080] Thus, for word 0 (dimensions are [1,D E ]), the conversion parameter group W in the conversion parameter group module 1 and 2 is required k1 and W v1 (Dimensions are [D E ,D D ]) and conversion parameter set W k2 and Wv2 (Dimensions are [D E ,D D ]). The conversion parameter group modules 1 and 2 are used to represent the fusion features to obtain the K matrix and V matrix (both dimensions are [∑L i ,D D ]), and can be considered as for ∑L i word units (dimensions are [1,D E ]) to obtain the K matrix and V matrix. Therefore, the submatrix (vector) corresponding to the word element vector of word element 0 can be extracted from each K matrix and each V matrix (two in this case), with dimensions (dimensions are [1,D D ]). Add the two K sub-matrices together, and add the two V sub-matrices together to obtain the K sub-matrix and V sub-matrix corresponding to the word 0.

[0081] Similarly, for word unit 1, the K submatrix and V submatrix corresponding to word unit 1 can also be determined.

[0082] In this way, ∑L i The K sub-matrices corresponding to each word unit can be combined to obtain the K matrix (dimension [∑L i ,D D ]), and ∑L i The V sub-matrices corresponding to each word unit can be combined to obtain the V matrix (dimension [∑L i ,D D ]).

[0083] In sub-step S430-3, the cross-attention module can be used to generate the output representation of the decoding layer block based on the determined key-value combination, which is used to generate input information of the next decoding layer block or to generate the output text.

[0084] For example, the output representation obtained by the cross-attention module of the current decoding layer block based on the determined K, V, Q matrices can be input into a residual connection & layer normalization layer, and then after passing through another feed-forward layer and a residual connection & layer normalization layer, the output representation of the decoding layer block is obtained.

[0085] When the current decoding layer model is the last layer of the decoder, its output can pass through the linear layer and softmax to generate a new word as part of the output text.

[0086] When the current decoding layer block is not the last layer of the decoder, its output can be used as the input information of the next decoding layer block. This process is similar to the output method in the existing LLM model, so its detailed description is omitted.

[0087] Thus, through the above Figure 5 and Figure 6 The benefits of the hybrid structure (including multiple conversion parameter group modules) shown in the text include allowing the conversion parameter group modules of different parts of the multimodal large language model to have enhanced understanding capabilities for different modalities after training, without compressing all learned knowledge into the same conversion parameter group module. This allows the model to more efficiently distinguish and understand information of different modalities, improving the ability to understand input data of each modality in practical applications to generate text with higher quality.

[0088] In addition, as mentioned above, user instructions can be used to start decoding operations or text generation operations. User instructions can be input data in text form or text data obtained through audio-text conversion, which is input into the embedding layer on the decoder side for encoding and processing to obtain the feature representation of the user instruction. User instructions are usually texts entered by users when using text generation services to describe their needs or service intentions. For example, when a user wants to get a reply, he can enter an instruction containing a question. For another example, when a user wants to get a summary, he can enter an instruction to instruct a multimodal large language model to generate a summary based on the current multi-modal input data, and so on.

[0089] As above reference Figure 3-Figure 8 A process for text generation based on input data of multiple modalities based on a multimodal large language model with an improved architecture is described, wherein the multimodal large language model has been trained and can be directly used for text generation.

[0090] The following is a detailed introduction to the training process of the large language model.

[0091] As mentioned above and combined Figure 5-Figure 6 The multimodal large language model shown is an improvement based on the existing LLM model (such as the large model of ChatGPT), and the model parameters of the LLM model can be directly used for the same parts as the LLM model, and only the model parameters of the hybrid attention mechanism modules introduced in multiple encoders (corresponding to multiple modalities) and decoders are adjusted during the training process. That is, the model parameters in the hybrid cross attention mechanism module in each encoder and each decoding layer block of the decoder are adjusted during the training process, and the other model parameters in each decoding layer block except the model parameters in the hybrid cross attention mechanism module are pre-trained and do not need to be adjusted in the current training process (that is, these other pre-trained model parameters are frozen).

[0092] For example, each piece of training data in the training data set for training model parameters in the hybrid cross-attention mechanism module in multiple encoders and decoders may include sample input data of multiple modalities and corresponding reference output text data, as well as optional user instruction text data (determined according to the purpose that the model is ultimately to achieve). The sample input data of the multiple modalities include at least two of picture sample data, sound sample data, video sample data or text sample data. For example, the training data may be a pairing form of multimodal data and text data (and optional user instruction text data), for example, data in the form of pictures, sounds, videos, etc. and their descriptive text data (and the user instructions based on which the text data is output) are combined and paired to train the model to understand the features of multimodal data and its textual expression.

[0093] For example, the goal of training multiple encoders and decoders is to minimize the difference between the output text of the multimodal large language model and the corresponding reference output text data. A loss function can be constructed based on the above training objectives, such as a cross entropy loss function. In each round of iteration, the loss function is obtained using the output text of the multimodal large language model and the corresponding reference output text data, and each model parameter is updated using methods such as gradient descent and error back propagation until the preset training end condition is met. For example, the model parameters to be learned may include the model parameters in each coding layer block within each encoder and the weight parameters of each linear projection layer (for example, the model parameters in the feature extractor before the linear projection layer can be fixed), and the model parameters in each hybrid cross attention mechanism module in the decoder (for example, K / V weight matrix parameters, Q weight matrix parameters, linear projection layer weight parameters, etc.).

[0094] The termination conditions of the above-mentioned training process may include, for example, that the value of the loss function is less than or equal to a preset loss function threshold, or the number of iterations reaches a preset number threshold.

[0095] Through the above combination Figure 3-Figure 8 The text generation method according to the embodiment of the present application is described. By using multiple different encoders to process input data of different modalities, each modality has a dedicated encoder, so it can flexibly adapt to different multi-modal application scenarios; in addition, a hybrid cross-attention mechanism is introduced into the existing decoder structure, and the latent space information of the encoder is directly mixed into the attention module of each layer of the large language model, and different conversion parameter group modules (for example, KV weight matrix) can be selected for different word units, which has a stronger ability to understand input data of multiple modalities. In addition, due to the latent space dimension (D E ) is smaller than the latent space dimension of the decoder (D D), using an encoder + decoder structure is better than a pure decoder structure (such as Figure 2 ) can compress long sequence inputs more efficiently. For example, assuming D E =512,D D =4096, and the sequence length is L, then the encoder's latent space matrix is ​​[L,512], which is 8 times smaller than the decoder's latent space matrix [L,4096]. In addition, in terms of training complexity, since only the hybrid cross-attention mechanism module of the encoder dedicated to each modality and the decoder in the multimodal large language model need to be trained for different multi-modal input data, without fine-tuning the multimodal large language model as a whole, the training cost is lower than training the traditional encoder + decoder model from scratch.

[0096] According to another aspect of the present application, a text generation device is also provided, which can generate text based on input data of multiple modalities.

[0097] Fig. 9 The schematic diagram of the structure of the text generation device according to the embodiment of the present application is shown. The text generation device can be as follows Figure 1 The server shown.

[0098] like Fig. 9 As shown, the text generating device 900 may include a data acquiring unit 910 , an encoding unit 920 and a decoding unit 930 .

[0099] Among them, the data acquisition unit 910 can acquire input data of multiple modes. The encoding unit 920 can use multiple encoders corresponding to the multiple modes to generate a fused feature representation from the input data. The fused feature representation includes feature representations corresponding to multiple lemmas obtained after the input data is lemmatized. The decoding unit 930 can use a decoder to generate output text associated with the input data of the multiple modes based on the fused feature representation.

[0100] For example, the encoding unit 920 may include Figure 5-Figure 6 The multiple encoders shown correspond to the multiple modalities, as well as the feature extractors and linear projection layers (or embedding layers for text data) corresponding to the respective encoders.

[0101] For another example, the decoding unit 930 may include: Figure 5-Figure 6The decoder shown includes multiple decoding layer blocks and a hybrid cross-attention mechanism module included in each decoding layer block, wherein the hybrid cross-attention mechanism module in each decoding layer block includes a cross-attention module, multiple transformation parameter group modules for the cross-attention module, and a selection module, wherein the selection module is used to select one or more transformation parameter group modules from the multiple transformation parameter group modules based on the fused feature representation to generate a transformation parameter group used by the cross-attention module.

[0102] For example, when the decoding unit 930 generates output text using the decoder, it can be configured to: determine multiple key (K)-value (V) combinations based on the fused feature representation using the conversion parameter groups in each conversion parameter group module included in the hybrid cross-attention mechanism module in the decoding layer block; determine the key-value combination for the cross-attention module in the decoding layer block from the multiple key-value combinations based on the fused feature representation using the selection module; and generate the output representation of the decoding layer block using the cross-attention module based on the determined key-value combination, which is used to generate input information for the next decoding layer block or to generate the output text.

[0103] Specifically, the fused feature representation includes feature representations corresponding to multiple word-grams obtained after the input data is tokenized, and each of the multiple key-value combinations includes information associated with the multiple word-grams. When determining the key-value combination for the cross-attention module in the decoding layer block, the decoding unit 930 can be configured as follows: using the selection module and based on the fused feature representation, determine a conversion parameter group module for each word-gram obtained after the input data is tokenized; using the selection module and based on the conversion parameter group module determined for each word-gram, determine a key-value combination for each word-gram from the multiple key-value combinations; and determine information associated with the word-gram from the key-value combination for each word-gram, wherein the determined information associated with each word-gram constitutes the key-value combination of the cross-attention module for the fused feature representation.

[0104] Optionally, the text generation device 900 may further include a training unit or may be coupled to a training device so that corresponding model parameters in the encoding unit and the decoding unit can be trained.

[0105] For example, the training unit or training device can be configured to train the multiple encoders in the encoding unit 920 and the decoder in the decoding unit 930, wherein the model parameters in the hybrid cross-attention mechanism module in each decoding layer block in the multiple encoders and the decoder are adjusted during the training process.

[0106] Optionally, the text generation device 900 may also include an instruction acquisition unit, which may be coupled or integrated with the decoding unit 930 to obtain user instructions. As described above, user instructions may be used to start a decoding operation or a text generation operation. The user instruction may be input data in text form or text data obtained by audio-text conversion, and the instruction acquisition unit may acquire the user instruction, which is encoded in an embedded layer to obtain its feature representation (the embedded layer may be in the instruction acquisition unit or in the decoding unit). The user instruction is usually a text that the user inputs when using the text generation service, describing his or her needs or service intentions. In this way, the text generation device can respond to the user instruction by outputting text based on the current multiple modal input data.

[0107] For more details of the various operations of the text generation device 900, please refer to the above combined Figure 3-Figure 8 , so it will not be repeated here.

[0108] In addition, although Fig. 9 The above modules are shown in an exemplary manner, but it should be understood that the device 900 can be divided into more or fewer modules according to different functions, or each module can be divided into further sub-modules. In some example embodiments, each module or further divided sub-module can be implemented by electronic hardware (e.g., a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.), computer software, program or instruction set (e.g., can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), etc.) or a combination of the two.

[0109] Therefore, by referring to Fig. 9 The described text generation device processes input data of different modalities by using multiple different encoders. Each modality has a dedicated encoder, so it can flexibly adapt to different multi-modal application scenarios. In addition, a hybrid cross-attention mechanism is introduced into the existing decoder structure to directly mix the encoder's latent space information into the attention module of each layer of the large language model, and different conversion parameter group modules (for example, KV weight matrix) can be selected for different word units, so that the understanding ability of input data of multiple modalities is stronger. In addition, due to the latent space dimension (D E ) is smaller than the latent space dimension of the decoder (D D ), using an encoder + decoder structure is better than a pure decoder structure (such as Figure 2) can compress long sequence inputs more efficiently. In addition, in terms of training complexity, since only the hybrid cross-attention mechanism module of the encoder dedicated to each modality and the decoder in the multimodal large language model needs to be trained for different multi-modal input data, without fine-tuning the multimodal large language model as a whole, the training cost is lower than training the traditional encoder + decoder model from scratch.

[0110] Fig.10 A schematic block diagram of a computing device according to an embodiment of the present application is shown. The computing device may be or include: Figure 3 The server shown.

[0111] like Fig.10 As shown, the computing device 1000 may include one or more processors, one or more memories, a network interface, an input device, and a display screen connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computing device stores an operating system and may also store a computer executable program. When the computer executable program is executed by the processor, the processor may implement various operations of the text generation method as described above. The internal memory may also store a computer executable program. When the computer executable program is executed by the processor, the processor may perform various operations described in each step of the text generation method.

[0112] As an example, the processor may include an integrated circuit chip having a signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, for implementing or executing the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The processor may also include a memory or be combined with a memory, and the memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and may also store a computer executable program, which, when executed, enables the computing device to implement various operations as described above. The internal memory may also store a computer executable program, which, when executed, enables the computing device to implement various operations as described above.

[0113] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable categories of memory.

[0114] The display screen of the computing device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computing device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the terminal shell, or an external keyboard, touchpad or mouse.

[0115] The computing device may be a terminal or a server. The terminal may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc.; a variety of clients (applications, APPs) may run in the terminal, such as a multimedia player client, a social client, a browser client, an information stream client, an education client, etc. The server may be a reference Figure 3 The server described can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0116] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the methods and devices according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, or the various modules mentioned, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The embodiments of the present application described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present application, and such modifications should fall within the scope of the present application.

Claims

1. A method for generating text, comprising: Obtain input data in multiple modalities; Using a plurality of encoders corresponding to the input data of the plurality of modalities, respectively, to generate a fused feature representation from the input data; as well as generating, using a decoder, output text associated with the input data of the multiple modalities based on the fused feature representation, The decoder includes a plurality of decoding layer blocks and a hybrid cross attention mechanism module included in each decoding layer block, The hybrid cross-attention mechanism module in each decoding layer block includes a cross-attention module, a plurality of conversion parameter group modules for the cross-attention module, and a selection module, Among them, the fused feature representation includes feature representations corresponding to multiple word-grams obtained after the input data is tokenized, and the selection module is used to select one or more conversion parameter group modules from the multiple conversion parameter group modules for each of the multiple word-grams to generate a conversion parameter group used by the cross-attention module.

2. The method according to claim 1, wherein: Generating output text associated with the input data of the multiple modalities based on the fused feature representation using a decoder, comprising: for each decoding layer block, Determine a plurality of key-value combinations based on the fused feature representation using the transformation parameter groups in each transformation parameter group module included in the hybrid cross attention mechanism module in the decoding layer block; Determining, using the selection module based on the fused feature representation, a key-value combination for the criss-cross attention module in the decoding layer block from the plurality of key-value combinations; and Based on the determined key-value combination, the cross-attention module is used to generate the output representation of the decoding layer block, which is used to generate input information of the next decoding layer block or to generate the output text.

3. The method according to claim 2, wherein: Each key-value combination of the plurality of key-value combinations includes information associated with the plurality of word-grams, Wherein, determining a key-value combination for a criss-cross attention module in the decoding layer block comprises: Determine, by using the selection module and based on the fused feature representation, for each word-gram in the plurality of word-grams, a conversion parameter group module corresponding to the word-gram; Determine the key-value combination corresponding to each word-unit from the plurality of key-value combinations by using the selection module and based on the conversion parameter group module corresponding to each word-unit; and Determine information associated with each word-gram from the key-value combination corresponding to the word-gram, The information determined to be associated with each word-unit constitutes a key-value combination for the cross-attention module in the decoding layer block.

4. The method according to claim 3, wherein: When the key-value combination corresponding to the current word is a plurality of key-value combinations, the plurality of key-value combinations are added together to form the key-value combination corresponding to the current word.

5. The method according to claim 2, wherein: Each key-value combination consists of a key matrix K and a value matrix V. Among them, multiple key-value combinations are determined, including: The fused feature representation is linearly transformed using the conversion parameter group in each conversion parameter group module in the decoding layer block to determine a key matrix K and a value matrix V corresponding to each conversion parameter group module.

6. The method according to claim 5, further comprising: The input information of the decoding layer block is linearly transformed to determine the query matrix Q used for the cross-attention module in the decoding layer block, wherein the input information of the decoding layer block includes the output information of the previous decoding layer block or the encoded embedded information of the user instruction.

7. The method according to claim 1, further comprising: The plurality of encoders and the decoder are trained, wherein model parameters in the hybrid cross-attention mechanism modules in the plurality of encoders and each decoding layer block are adjusted during the training process, Among them, the model parameters in each decoding layer block except the model parameters in the hybrid cross-attention mechanism module are pre-trained and do not need to be adjusted during the training process.

8. The method according to claim 7, wherein: The multiple encoders and the decoders are trained using a training data set, and each training data in the training data set includes at least sample input data of multiple modalities and corresponding reference output text data, wherein the sample input data of multiple modalities includes at least two of picture sample data, sound sample data, video sample data or text sample data.

9. The method according to claim 1, wherein: The input information of the decoder includes user instructions as start instructions for text generation operation.

10. The method according to claim 1, further comprising: The input data of each modality in the input data is tokenized and converted into a latent space matrix of the same dimension as the corresponding encoder as the input of the corresponding encoder, The multiple encoders respectively encode the corresponding latent space matrices of the input to obtain multiple encoding output matrices, and the multiple encoding output matrices are merged to obtain the fused feature representation.

11. The method according to claim 1, wherein: The architecture of each encoder among the multiple encoders adopts the encoder architecture in the Transformer-based large language model, and the architecture of the decoder adds the hybrid cross-attention mechanism module in each decoding layer block of the decoder architecture in the Transformer-based large language model.

12. The method according to claim 1, wherein: The latent space dimension of each encoder of the plurality of encoders is smaller than the latent space dimension of the decoder.

13. A device for generating text, comprising: A data acquisition unit, used to acquire input data of multiple modes; An encoding unit, configured to generate a fused feature representation from the input data using a plurality of encoders corresponding to the input data of the plurality of modalities; as well as A decoding unit, configured to generate output text associated with the input data of the multiple modalities based on the fused feature representation using a decoder, The decoder includes a plurality of decoding layer blocks and a hybrid cross attention mechanism module included in each decoding layer block, The hybrid cross-attention mechanism module in each decoding layer block includes a cross-attention module, a plurality of conversion parameter group modules for the cross-attention module, and a selection module, Among them, the fused feature representation includes feature representations corresponding to multiple word-grams obtained after the input data is tokenized, and the selection module is used to select one or more conversion parameter group modules from the multiple conversion parameter group modules for each of the multiple word-grams to generate a conversion parameter group used by the cross-attention module.

14. The device according to claim 13, wherein: When the decoding unit generates the output text using the decoder, the decoding unit is configured to: for each decoding layer block, Determine a plurality of key-value combinations based on the fused feature representation using the transformation parameter groups in each transformation parameter group module included in the hybrid cross attention mechanism module in the decoding layer block; Determining, using the selection module based on the fused feature representation, a key-value combination for the criss-cross attention module in the decoding layer block from the plurality of key-value combinations; as well as Based on the determined key-value combination, the cross-attention module is used to generate the output representation of the decoding layer block, which is used to generate input information of the next decoding layer block or to generate the output text.

15. The device according to claim 14, wherein: Each key-value combination of the plurality of key-value combinations includes information associated with the plurality of word-grams, Wherein, the decoding unit, when determining the key-value combination for the cross-attention module in the decoding layer block, is configured to: Determine a conversion parameter group module for each word-gram of the plurality of word-grams using the selection module and based on the fused feature representation; Determining a key-value combination for each word-gram from the plurality of key-value combinations using the selection module and based on the conversion parameter set module determined for each word-gram; and determining information associated with each word-gram from the key-value combination for the word-gram, The information determined to be associated with each word-unit constitutes a key-value combination for the cross-attention module in the decoding layer block.

16. The apparatus according to claim 13, further comprising: A training unit is used to train the multiple encoders and the decoder, wherein the model parameters in the hybrid cross attention mechanism module in the multiple encoders and each decoding layer block are adjusted during the training process, Among them, the model parameters in each decoding layer block except the model parameters in the hybrid cross-attention mechanism module are pre-trained and do not need to be adjusted during the training process.

17. The device according to claim 13, wherein: The latent space dimension of each encoder of the plurality of encoders is smaller than the latent space dimension of the decoder.

18. A computing device comprising: one or more processors; as well as One or more memories storing computer programs or computer instructions, which, when executed by the one or more processors, implement the method according to any one of claims 1 to 12.

19. A computer-readable storage medium having a computer program or computer instructions stored thereon, wherein the computer program or computer instructions, when executed by the one or more processors, implement the method according to any one of claims 1 to 12.

Images