Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of image processing, in particular to the technical fields of artificial intelligence, large models, multi-modal fusion and the like. According to the specific implementation scheme, multiple pieces of multi-modal data input by a user are obtained; encoding the multi-modal data, and obtaining an embedded vector corresponding to the multi-modal data; arranging the embedding vectors corresponding to the multiple pieces of multi-modal data according to a user input sequence to obtain a multi-modal embedding vector; and performing prediction according to the multi-modal embedded vector, and generating answer data corresponding to the multi-modal data.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, in particular to the fields of artificial intelligence, large models, multimodal fusion, etc. Specifically, the present disclosure relates to a data processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In recent years, LLM (Large Language Model) represented by ChatGPT and LlaMa has made significant progress.

[0003] The emergent capability generated by the increase in training data enables the large model to have better generalization capabilities for unknowable tasks, greatly enhancing the versatility of the large model in different downstream tasks, making the large model more and more used in different fields. Summary of the invention

[0004] The present disclosure provides a data processing method and device, an electronic device, and a computer-readable storage medium.

[0005] According to a first aspect of the present disclosure, a data processing method is provided, the method comprising:

[0006] Acquire multiple multimodal data input by a user; at least one multimodal data has a different modality from the other multimodal data;

[0007] Encode the multimodal data and obtain the embedding vector corresponding to the multimodal data;

[0008] Arrange the embedding vectors corresponding to the multiple multimodal data according to the order input by the user to obtain the multimodal embedding vector;

[0009] Predictions are made based on the multimodal embedding vectors to generate answer data corresponding to the multimodal data.

[0010] According to a second aspect of the present disclosure, there is provided a data processing device, the device comprising:

[0011] A data module, used for acquiring a plurality of multimodal data input by a user; at least one multimodal data has a different modality from the other multimodal data;

[0012] An encoding module, used to encode multimodal data and obtain an embedding vector corresponding to the multimodal data;

[0013] A fully connected module is used to arrange the embedding vectors corresponding to multiple multimodal data according to the order input by the user to obtain a multimodal embedding vector;

[0014] The prediction module is used to make predictions based on the multimodal embedding vectors and generate answer data corresponding to the multimodal data.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] A memory in communication with the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned data processing method.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above data processing method when executed by a processor.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present disclosure;

[0024] Figure 2 It is a flowchart of some steps of a data processing method provided by an embodiment of the present disclosure;

[0025] Figure 3 It is a flowchart of some steps of a data processing method provided by an embodiment of the present disclosure;

[0026] Figure 4 It is a flowchart of some steps of a data processing method provided by an embodiment of the present disclosure;

[0027] Figure 5 It is a flowchart of some steps of a data processing method provided by an embodiment of the present disclosure;

[0028] Figure 6 It is a flowchart of some steps of a data processing method provided by an embodiment of the present disclosure;

[0029] Figure 7 It is a structural schematic diagram of a multimodal model that can implement a data processing method provided by an embodiment of the present disclosure;

[0030] Figure 8 is a structural schematic diagram of a data processing device provided by an embodiment of the present disclosure;

[0031] Fig. 9 It is a block diagram of an electronic device used to implement the data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] Due to the limitations of the large model’s structure, the large model can only understand discrete texts, but is “blind” to vision, that is, the large model cannot understand images. …

[0034] Among some related technologies, MLLM (Multimodal Large Language Model) not only contains the backbone of a large language model, but also has a visual encoder and a multimodal interface, which enables it to have image, video, and audio understanding capabilities that LLM does not have.

[0035] Among them, the visual encoding can be CLIP, Ospery, Fuyu and other networks, and the multimodal interface is realized through symbol-level and feature-level means.

[0036] However, MLLM is inferior to LLM in ICL (InContext Learning) capability, and its capability is weaker and its robustness is poor.

[0037] The data processing method and device, electronic device, and computer-readable storage medium provided by the embodiments of the present disclosure are intended to solve at least one of the above technical problems in the prior art.

[0038] The data processing method provided in the embodiments of the present disclosure may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a vehicle-mounted device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method may be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method may be executed by a server.

[0039] Figure 1 FIG. 1 is a flow chart showing a data processing method provided by an embodiment of the present disclosure. Figure 1 As shown in , the data processing method provided by the embodiment of the present disclosure may include step S110, step S120, step S130, and step S140.

[0040] In step S110, a plurality of multimodal data input by a user is obtained; at least one multimodal data has a different modality from the other multimodal data;

[0041] In step S120, the multimodal data is encoded to obtain an embedding vector corresponding to the multimodal data;

[0042] In step S130, the embedding vectors corresponding to the plurality of multimodal data are arranged in the order input by the user to obtain a multimodal embedding vector;

[0043] In step S140, prediction is performed based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

[0044] For example, the data processing method provided in the embodiments of the present disclosure can be used to process data input by users in human-machine dialogue scenarios such as chat robots.

[0045] In some possible implementations, in step S110, obtaining multiple multimodal data input by the user may be obtaining multiple multimodal data input by the user during multiple rounds of human-computer dialogue. The data input by the user during one round of dialogue may be used as one multimodal data, or may be divided into multiple multimodal data.

[0046] Among them, modality refers to the form of expression or sensory channel of information. For example, image data is data in the visual modality, voice data is data in the auditory modality, and text data is data in the text modality.

[0047] Each multimodal data is data of one mode, and the mode of at least one multimodal data among the multiple multimodal data is different from the modes of the other multimodal data. That is to say, there are at least two modes of data among the multiple multimodal data.

[0048] In some possible implementations, in step S120, multiple multimodal data are encoded respectively to obtain an Embedding (embedding vector) corresponding to each multimodal data.

[0049] Among them, multimodal data of different modes can be encoded using different methods, and multimodal data of the same mode can be encoded using the same method.

[0050] The embedding vector corresponding to each multimodal data may be an embedding vector understandable by an LLM (Large Language Model).

[0051] In some possible implementations, in step S130, the embedding vectors corresponding to the plurality of multimodal data are arranged in the order input by the user and fused to generate a multimodal embedding vector.

[0052] Among them, arranging according to the order of user input can be that the embedding vectors corresponding to the multimodal data input first by the user are arranged in the front, and the embedding vectors corresponding to the multimodal data input later are arranged in the back.

[0053] Fusing the embedded vectors corresponding to the arranged multiple multimodal data may be mapping the embedded vectors corresponding to the arranged multiple multimodal data to the same space.

[0054] Specifically, a dense layer may be used to fuse the embedded vectors corresponding to the arranged multiple multimodal data.

[0055] Dense layer is a commonly used neural network layer in deep learning, also known as fully connected layer. It plays the role of fully connecting input data with output results in the neural network. Therefore, using dense layer to fuse the embedding vectors corresponding to multiple multimodal data can generate a multimodal embedding vector for prediction.

[0056] In the case where the embedding vector corresponding to the multimodal data is an embedding vector that can be understood by the large language model, the dense layer can be a dense layer that can be used by the large language model.

[0057] In some possible implementations, in step S140, reasoning and understanding are performed based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

[0058] In step S130, the embedding vectors corresponding to the multiple multimodal data are arranged in the order of user input, that is, the embedding vectors corresponding to the multimodal data in the context can be arranged according to the logic of the user conversation and combined into an interleaved representation similar to the user input. Therefore, the context information can be saved in the fused multimodal embedding vector. Therefore, in step S140, the multimodal embedding vector can be understood and reasoned using the context information in the multimodal embedding vector to generate answer data that is more in line with the context information.

[0059] In the data method provided in the embodiment of the present disclosure, not only can single-modal information be processed, but also the context of modal information can be effectively learned and inferred. It can adapt to a variety of tasks and application scenarios, especially long-term conversations and complex task scenarios, and has strong versatility.

[0060] The data processing method provided by the embodiment of the present disclosure is introduced in detail below.

[0061] As described above, each multimodal data is data of one modality, and the modality of at least one multimodal data among the multiple multimodal data is different from the modalities of the other multimodal data.

[0062] Visual modality data (such as image data) and text data are the most commonly used data in human-computer interaction, and the demand for cross-modal understanding of image data and text data is the greatest and most urgent. The data processing method provided in the embodiment of the present disclosure can be applied to cross-modal understanding of image data and text data.

[0063] In some possible implementations, at least one of the multiple multimodal data obtained from the user input is image data; and at least one of the multimodal data is text data.

[0064] Since image data and text data are data of different modalities, the encoding methods for different modalities are different. Figure 2 A flowchart of an implementation method of encoding multimodal data and obtaining an embedding vector corresponding to the multimodal data is shown in the case where at least one of the multiple multimodal data is image data and at least one is text data, such as Figure 2 As shown, it may include step S210 and step S220.

[0065] In step S210, when the multimodal data is image data, the image data is encoded, image data features of the image data are extracted, and the image data features are converted into image embedding vectors;

[0066] In step S220, when the multimodal data is text data, the text data is encoded and converted into a text embedding vector.

[0067] In some possible implementations, in step S210, a pre-trained image encoder can be used to encode the image data, and high-level features of the image data can be extracted as image data features of the image data. A Q-former (a neural network architecture that performs visual-language alignment between the image encoder and the LLM) can be used as a VPG (Visual Prompt Generator) to convert the acquired image data features into an embedding vector that can be understood by the LLM, i.e., an image embedding vector.

[0068] In some possible implementations, in step S220, the text data may be encoded by the tokenizer of the LLM itself, and the natural language in the text data may be converted into an embedding vector that the LLM can understand, namely, a text embedding vector.

[0069] By obtaining the image embedding vector corresponding to the image data and the text embedding vector corresponding to the text data, we can solve the problem that the image data and text data are in different embedding spaces and it is difficult to achieve effective alignment of the image data and text data. This can better capture the deep correlation between the quality inspection of image data and text data and improve the quality of the generated answer data.

[0070] After obtaining the image embedding vector corresponding to the image data and the text embedding vector corresponding to the text data, in the process of arranging and fusing the image embedding vector and the text embedding vector, since there is a contextual relationship not only between the image data and the text data, but also between image data and image data, and between text data and text data, it is necessary to distinguish between the image embedding vector and the text embedding vector in the process of arranging and fusing the image embedding vector and the text embedding vector.

[0071] Figure 3 A flowchart of an implementation method for distinguishing image embedding vectors from text embedding vectors is shown, such as Figure 3 As shown, distinguishing the image embedding vector from the text embedding vector may include step S310, step S320, and step S330.

[0072] In step S310, an image label is assigned to the image embedding vector to form an image representation corresponding to the image data;

[0073] In step S320, a text label is assigned to the text embedding vector to form a text representation corresponding to the text data;

[0074] In step S330, the image representation and the text representation are arranged according to the input order of the image data and the text data input by the user to obtain a multimodal embedding vector.

[0075] In some possible implementations, in step S310, an image tag is inserted at the beginning of each image embedding vector and an image tag is inserted at the end of each image embedding vector to form an image representation corresponding to the image data.

[0076] Among them, the image tag inserted at the beginning of the image embedding vector, that is, the image start tag, and the image tag inserted at the end of the image embedding vector, that is, the image end tag, are different image tags. The final image representation can be as follows <boi>[IMG0 Embed] <eoi>in the form of <boi>is the image start tag, <eoi>is the end tag of the image, and [IMG0Embed] is the image embedding vector.

[0077] In some possible implementations, in step S320, a text tag is inserted at the beginning of each text embedding vector and a text tag is inserted at the end of each text embedding vector to form a text representation corresponding to the text data.

[0078] Among them, the text tag inserted at the beginning of the text embedding vector, that is, the text start tag, and the text tag inserted at the end of the text embedding vector, that is, the text end tag, are different text tags. The final text representation can be as follows <bot>[Text Embed] <eot>in the form of <bot>is the start text tag, <eot>is the text end tag, and [TextEmbed] is the text embedding vector.

[0079] In some possible implementations, in step S330, the corresponding image representation and text representation are arranged according to the input order of the user input image data and text, and the image representation and text representation are fused using a dense layer to generate a multimodal embedding vector.

[0080] By inserting image labels and text labels, the image embedding vectors corresponding to the image data and the text embedding vectors corresponding to the text data can be distinguished through the labels. Compared with the method of pre-setting the positions and quantities of image embedding vectors and text embedding vectors, using labels to distinguish image embedding vectors and text embedding vectors allows image data and text data to appear in any order and quantity in the context. Users can also enter text data and image data arbitrarily in the context, effectively improving the user interaction experience.

[0081] In some possible implementations, the multiple multimodal data obtained are multiple multimodal data input by the user during multiple rounds of human-computer dialogue. In order to further enhance the understanding of the context, tags are also inserted at the beginning and end of the dialogue to distinguish different dialogues.

[0082] Figure 4 A flowchart showing an implementation method of inserting tags at the beginning and end of a conversation to distinguish different conversations is shown, such as Figure 4 As shown, it may include step S410, step S420, and step S430.

[0083] In step S410, the embedding vectors corresponding to at least one of the multimodal data input by the user during a round of human-computer dialogue are arranged according to the user input order to obtain the round embedding vector corresponding to the round of human-computer dialogue;

[0084] In step S420, a turn label is assigned to the turn embedding vector to form a vector representation corresponding to the round of human-computer dialogue;

[0085] In step S430, the vector representations corresponding to each round of human-computer dialogue are sorted and fused according to the order of user input, and dialogue labels are assigned to the sorting and fusion results to obtain a multimodal embedding vector.

[0086] In some possible implementations, in step S410, the corresponding embedding vectors, specifically image embedding vectors and text embedding vectors, are sorted and fused according to the input order of multimodal data, specifically image data and text data, input by the user during a round of dialogue to obtain the round embedding vector corresponding to the round of human-computer dialogue.

[0087] Alternatively, the image embedding vector and the text embedding vector may be sorted and fused after the corresponding labels are inserted into them. That is, the corresponding image representation and text representation may be sorted and fused according to the input order of the image data and text data input by the user during a round of dialogue to obtain the round embedding vector corresponding to the round of human-computer dialogue.

[0088] In some possible implementations, in step S420, a round label is inserted at the beginning of each round embedding vector obtained, and a round label is inserted at the end of each round embedding vector to form a vector representation corresponding to the round of human-computer dialogue.

[0089] Among them, the round label inserted at the beginning of the round embedding vector, that is, the round start label, and the round label inserted at the end of the round embedding vector, that is, the round end label, are different round labels. The corresponding vector representation finally formed can be as follows <bor> <bot>[Text Embed] <eot> <boi>[IMG0 Embed] <eoi> <eor>of the form, in which <bor>is the round start label, <eor>is the round end tag, <bot>[Text Embed] <eot> <boi>[IMG0Embed] <eoi>The corresponding meanings are as described above and will not be repeated here.

[0090] In some possible implementations, in step S430, the vector representations corresponding to each round of human-computer dialogue are sorted and fused according to the order of the rounds of each round of human-computer dialogue, and a dialogue tag is inserted at the beginning of the sorted fusion result and at the end of the sorted fusion result to form a multimodal embedding vector.

[0091] Among them, the conversation tag inserted at the beginning of the sorting fusion result, that is, the conversation start tag, and the conversation tag inserted at the end of the sorting fusion result, that is, the conversation end tag, are different conversation tags. The final multimodal embedding vector representation can be as follows <bos> <bor> <bot>[Text Embed] <eot> <boi>[IMG0 Embed] <eoi> <eor> <bor> <bot>[Text Embed] <eot> <boi>[IMG1 Embed] <eoi> <eor> <eos>of the form, in which <bos>is the conversation starter tag. <eos>is the conversation end tag, <bor> <bot>[Text Embed] <eot> <boi>[IMG0][Embed] <eoi> <eor>and <bor> <bot>[Text Embed] <eot> <boi>[IMG1 Embed] <eoi> <eor>It is the vector representation corresponding to different rounds of human-computer dialogue.

[0092] The multimodal data input by the user in different rounds of human-computer dialogue can be distinguished through labels. The complete contextual logical relationship can be grasped through the multimodal embedding vector, and the image embedding vector and the text embedding vector are also distinguished. Therefore, the NTP (Next Token Prediction) method can be used to make predictions based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

[0093] Figure 5 A flowchart of an implementation method of using the NTP method to predict based on the multimodal embedding vector to generate answer data corresponding to the multimodal data is shown, as shown in Figure 5 As shown, it may include step S510 and step S520.

[0094] In step S510, a prediction token sequence is generated using a next token prediction method according to the multimodal embedding vector, the generated prediction token sequence is used as an input token sequence, a target token sequence is predicted according to the input token sequence, and the target token sequence is used as a new input token sequence to generate a new target token sequence, until a target token sequence that meets the preset requirements is generated;

[0095] In step S520, the target token sequence is decoded to generate answer data corresponding to the multimodal data.

[0096] Among them, NTP is mainly used to predict the content of the next token in a given text sequence. Its core idea is to predict the next token when the first n tokens of a sentence are known. This prediction method is similar to cloze test, where the model needs to infer the next most likely token from the given context.

[0097] NTP is not only applicable to text generation, but can also be used to generate images. The pixel value range [0,255] can be regarded as a table containing 256 different "tokens". Image generation can be regarded as a prediction process for the next pixel point (token).

[0098] In some possible implementations, in step S510, since the image embedding vector and the text embedding vector in the multimodal embedding vector are distinguished by labels and are arranged according to the logic of the user conversation, the image embedding vector and the text embedding vector can be processed separately.

[0099] A predicted token sequence is generated according to the contextual relationship corresponding to the text embedding vector in the multimodal embedding vector, and the generated predicted token sequence is used as the input token sequence, i.e., the first few tokens, to predict the next token, and the input token sequence and the predicted next token are combined into a target token sequence. If the target token sequence does not meet the preset requirements (such as being unable to form a complete sentence), the target token sequence is used as a new input token sequence, i.e., the first few tokens, and the next token is continued to be predicted, and the execution is repeated until a target token sequence that meets the preset requirements is generated.

[0100] Similarly, for the image embedding vector, a prediction token sequence is generated according to the contextual relationship corresponding to the image embedding vector in the multimodal embedding vector, and the generated prediction token sequence is used as the input token sequence, i.e., the first few tokens, to predict the next token, and the input token sequence and the predicted next token are combined into a target token sequence. If the target token sequence does not meet the preset requirements (such as it cannot form a complete sentence), the target token sequence is used as the new input token sequence, i.e., the first few tokens, and the next token is predicted. The execution is repeated until a target token sequence that meets the preset requirements is generated.

[0101] In some possible implementations, in step S520, since the image embedding vector and the text embedding vector are processed separately in step S510, the modality corresponding to the tokens in the generated target token sequence can be known.

[0102] Different decoding methods are used to decode the tokens corresponding to different modes in the target token sequence. For example, the tokens corresponding to the image mode are decoded using an image decoder to generate image data, and the tokens corresponding to the text mode are decoded using an LLM decoder to generate text data.

[0103] The corresponding image data and text data are combined into answer data according to the order of the tokens in the target token sequence.

[0104] Compared with the method in MLLM that requires the use of a complex extended model to convert images into text, and then convert the text into images after understanding the text, the prediction using NTP can be achieved using the prediction module of LLM itself, without the need for an additional model. This simplifies the traditional visual language perception process, making it possible to generate accurate and contextual outputs with lower computational complexity. NTP can also better align the context and generate more on-topic and logical answers for users.

[0105] At the same time, since the embedding vectors corresponding to the multimodal data are sorted according to the order of user input in the process of generating multimodal embedding vectors, the complete contextual logical relationship can be grasped through the multimodal embedding vectors. Based on the complete contextual logical relationship, potential Context Hijacking, that is, attention diversion, can be identified and located, and adjustments can be made through the feedback mechanism to improve the accuracy and robustness of the predicted generated answer data.

[0106] Figure 6 is a flow chart showing an implementation method of identifying and locating attention shift, such as Figure 6 As shown, it may include step S610 and step S620.

[0107] In step S610, an attention weight matrix is ​​obtained, wherein the number of rows of the attention weight matrix is ​​the length of the target token sequence, the number of columns of the attention weight matrix is ​​the length of the input token sequence, each row of the attention weight matrix is ​​the attention distribution of a token in the target token sequence to all tokens in the input token sequence; each element of the attention weight matrix is ​​the attention weight of a token in the target token sequence to a token in the input token sequence;

[0108] In step S620, attention shift is identified according to the attention weight matrix, and the attention shift is adjusted through a feedback mechanism.

[0109] In some possible implementations, in step S610, since NTP is used for prediction, the attention weight matrix generated in the NTP prediction process can be used. Specifically, the attention weight matrix is ​​generated based on the attention mechanism, which describes the degree of attention to each token in the input token sequence when generating each token.

[0110] The specific form of the attention weight matrix can be: A = Δ T×S , where T is the length of the target token sequence (i.e., the number of tokens it contains), S is the length of the input token sequence, each line is the attention distribution of a token in the target token sequence to all tokens in the input token sequence, and each element is the attention weight of a token in the target token sequence to a token in the input token sequence.

[0111] In some possible implementations, in step S620, based on the acquired attention weight matrix, attention can be diverted in a variety of ways. After the attention diversion is identified, the attention is diverted back through a feedback mechanism to improve the accuracy of the generated answer data.

[0112] The following introduces several different specific implementation methods of identifying attention shift based on the attention weight matrix.

[0113] In some possible implementations, an input token is determined from all tokens in an input token sequence according to an attention weight matrix; the input token is the token of the input token sequence corresponding to the largest attention weight among the attention weights corresponding to the target tokens of the target token sequence; and when the input token is a noise token, it is determined that attention transfer has occurred to the target token.

[0114] It is expected that each generated token should focus mainly on the input tokens related to the current context (such as the previous text sequence or image-related information). If the attention weight is too focused on some irrelevant tokens (such as noise tokens), it may indicate that the token diverts the generation process of the model.

[0115] Among them, the token in the target token attention weight concentration is the input token, which is determined according to all the attention weights corresponding to the target token, and the token in the input token sequence corresponding to the attention weight with the largest value is the input token. The token in the input token sequence corresponding to the attention weight is determined according to the position of the attention weight in the attention weight matrix, specifically, the column corresponding to the attention weight in the attention weight matrix.

[0116] At the same time, if the attention weight corresponding to the target token is stable within the normal context range, but a mutation occurs at a certain layer or at a certain moment, and the attention is suddenly focused on an irrelevant token or image part, this is also an attention shift.

[0117] In some possible implementations, the distribution of input tokens corresponding to target tokens of a target token sequence in multiple time steps is calculated based on an attention weight matrix; when the distribution of input tokens corresponding to the target tokens changes suddenly, it is determined that the target token has undergone attention transfer.

[0118] That is to say, the distribution of the input tokens corresponding to the target token in multiple time steps is statistically analyzed to see whether they have been stably concentrated in the relevant context. If the distribution suddenly changes, that is to say, the input tokens corresponding to the target token are not stably concentrated in the relevant context, but are transferred to other unattainable tokens, it can be determined that the target token has undergone attention transfer.

[0119] In some possible implementations, the difference between the attention weight of the target command of the target command sequence to each input command in the input command sequence and the pre-acquired attention weight average is calculated; when the sum of all the differences is greater than a preset value, it is determined that the target command has undergone attention transfer.

[0120] In order to quantify the attention deviation behavior, an "Attention Deviation Score" can be designed, which is defined as the difference between the attention weight distribution and the normal weight distribution at a certain moment, that is, the difference between the attention weight of the target token of the target token sequence to each input token in the input token sequence and the pre-acquired attention weight mean.

[0121] Specifically, the formula to calculate the difference.

[0122] Among them, A t,i is the attention weight of the target token generated at time t to the i-th token in the input token sequence, is the mean of the normal attention distribution of the ith token in the input token sequence (which can be obtained by statistics on the training data).

[0123] When D t When it increases significantly, it may mean that the current attention distribution has deviated from the normal pattern, indicating that attention shift has occurred.

[0124] After identifying that attention has shifted, the specific method of redirecting attention back through a feedback mechanism is not limited in the embodiments of the present disclosure, and any method that can achieve the corresponding function is within the protection scope of the embodiments of the present disclosure.

[0125] In some possible implementations, the data processing method provided by the embodiments of the present disclosure may be implemented based on a trained multimodal model. Figure 7 A structural diagram of a multimodal model that can implement the data processing method provided by an embodiment of the present disclosure is shown.

[0126] like Figure 7 As shown, the multimodal data input by the user is obtained in the order in which the user inputs, such as Figure 7 Instruction (introduction), Image1 (image data 1), Text (text data), Image2 (image data 2), Query (query).

[0127] The encoder of the pre-trained multimodal model is used to encode the multimodal data to obtain the corresponding embedding vector of the multimodal data. The image data uses the image encoder ViT (Vision Transformer, an architecture that uses sequence models based on attention mechanisms in computer vision tasks) to extract image data features, and uses Q-former19 as VPG to convert the image into an image embedding vector that LLM can understand; the text data is encoded using the LLM's own Tokenizer to generate the corresponding text embedding vector.

[0128] The fully connected module (i.e., Tokenizer & Embedding) of the pre-trained multimodal model will arrange and fuse the embedding vectors corresponding to multiple multimodal data in the order of user input to obtain the multimodal embedding vector. Specifically, the above method is used to generate a similar <bos> <bor> <bot>[Text Embed] <eot> <boi>[IMG0Embed] <eoi> <eor> <bor> <bot>[Text Embed] <eot> <boi>[IMG1Embed] <eoi> <eor> <eos>Multimodal embedding vectors in the form of .

[0129] The prediction module using the pre-trained multimodal model is Figure 7 Next Token Prediction uses the NTP method, Auto Regression predicts based on the multimodal embedding vector, generates a token sequence corresponding to the multimodal data, and performs Hijack Detection during the prediction process.

[0130] Different decoding methods are used to decode the token sequences corresponding to different modalities. For example, the tokens corresponding to the image modality are decoded using the image decoder (Vision_Decoder) to generate image data, and the tokens corresponding to the text modality are decoded using the LLM decoder (Decoder) to generate text data.

[0131] Of course, the multimodal data input by the user can also be data of other modalities. In this case, you only need to use the encoder corresponding to the modal data to encode it and generate the corresponding embedding vector. The subsequent processing steps are consistent with the processing methods of image data and text data, so they will not be repeated here.

[0132] like Figure 7 As shown, the multimodal model can process cross-modal information in a unified manner, so the multimodal model is more efficient and easy to deploy, reducing reasoning and computing costs.

[0133] In some possible implementations, the encoder of the multimodal model is trained separately, that is, the encoder is trained using multimodal training data to obtain a trained encoder.

[0134] The fully connected module and the prediction module are trained based on the trained encoder, that is, the parameters of the encoder are locked, and the fully connected module and the prediction module of the multimodal model are trained using the training data to obtain a pre-trained multimodal model.

[0135] By training different modules separately, the training speed of the multimodal model is accelerated, avoiding the situation where the model fails to converge.

[0136] The data processing method provided in the embodiments of the present disclosure can be applied to AI emotional companionship products to improve the reply quality of the chatbot and provide more accurate conversations that are in line with the user context, thereby enhancing user immersion and experience, and helping to enhance the market competitiveness of the product.

[0137] Based on Figure 1 The same principle as shown in the method, Figure 8 A schematic diagram of the structure of a data processing device provided by an embodiment of the present disclosure is shown. Figure 8 As shown, the data processing device 80 may include:

[0138] The data module 810 is used to obtain a plurality of multimodal data input by a user; at least one multimodal data has a different modality from the other multimodal data;

[0139] An encoding module 820, configured to encode the multimodal data and obtain an embedding vector corresponding to the multimodal data;

[0140] A fully connected module 830, configured to arrange the embedding vectors corresponding to the plurality of multimodal data in the order input by the user to obtain a multimodal embedding vector;

[0141] The prediction module 840 is used to make predictions based on the multimodal embedding vectors to generate answer data corresponding to the multimodal data.

[0142] In the data processing device provided in the embodiment of the present disclosure, it is not only capable of processing single-modal information, but also capable of effectively learning and reasoning about the context of modal information. It can adapt to a variety of tasks and application scenarios, especially long-term conversations and complex task scenarios, and has strong versatility.

[0143] In some possible implementations, at least one multimodal data is image data; at least one multimodal data is text data; the encoding module includes: an image submodule, which is used to encode the image data, extract image data features of the image data, and convert the image data features into image embedding vectors when the multimodal data is image data; and a text submodule, which is used to encode the text data and convert the text data into text embedding vectors when the multimodal data is text data.

[0144] In some possible implementations, the fully connected module includes: a first label unit, used to assign image labels to image embedding vectors to form image representations corresponding to image data; assign text labels to text embedding vectors to form text representations corresponding to text data; and a sorting unit, used to arrange image representations and text representations according to the input order of image data and text data input by a user to obtain a multimodal embedding vector.

[0145] In some possible implementations, the data module is used to: obtain multiple multimodal data input by the user during multiple rounds of human-computer dialogue; the fully connected module includes: a second label unit, which is used to arrange the embedding vectors corresponding to at least one of the multimodal data input by the user during a round of human-computer dialogue in the order of user input, and obtain the round embedding vector corresponding to the round of human-computer dialogue; assign round labels to the round embedding vectors to form a vector representation corresponding to the round of human-computer dialogue; a third label unit, which is used to sort and fuse the vector representations corresponding to each round of human-computer dialogue in the order of user input, and assign dialogue labels to the sorting and fusion results to obtain a multimodal embedding vector.

[0146] In some possible implementations, the prediction module is used to: generate a predicted command sequence using a next command prediction method based on a multimodal embedding vector, use the generated predicted command sequence as an input command sequence, predict a target command sequence based on the input command sequence, and use the target command sequence as a new input command sequence to generate a new target command sequence until a target command sequence that meets preset requirements is generated; and decode the target command sequence to generate answer data corresponding to the multimodal data.

[0147] In some possible implementations, the prediction module includes: an attention unit, used to obtain an attention weight matrix, the number of rows of the attention weight matrix is ​​the length of the target token sequence, the number of columns of the attention weight matrix is ​​the length of the input token sequence, each row of the attention weight matrix is ​​the attention weight distribution of a token in the target token sequence to all tokens in the input token sequence; each element of the attention weight matrix is ​​the attention weight of a token in the target token sequence to a token in the input token sequence; an offset recognition unit, used to identify attention shifts according to the attention weight matrix, and adjust the attention shifts through a feedback mechanism.

[0148] In some possible implementations, the offset identification unit is used to: determine the input command from all the commands in the input command sequence according to the attention weight matrix; the input command is the command of the input command sequence corresponding to the attention weight with the largest weight value among the attention weights corresponding to the target commands of the target command sequence; and when the input command is a noise command, determine that attention shift has occurred to the target command.

[0149] In some possible implementations, the offset identification unit is used to: calculate the distribution of input commands corresponding to the target commands of the target command sequence in multiple time steps based on the attention weight matrix; and determine that the target command has an attention shift when the distribution of the input commands corresponding to the target command changes suddenly.

[0150] In some possible implementations, the offset identification unit is used to: calculate the difference between the attention weight of the target command of the target command sequence for each input command in the input command sequence and the pre-acquired attention weight average; and determine that the target command has an attention shift when the sum of all differences is greater than a preset value.

[0151] In some possible implementations, the encoding module is used to: encode the multimodal data using an encoder of a pre-trained multimodal model to obtain an embedding vector corresponding to the multimodal data; the fully connected module is used to: arrange the embedding vectors corresponding to multiple multimodal data in the order of user input using the fully connected module of the pre-trained multimodal model to obtain the multimodal embedding vector; the prediction module is used to use the prediction module of the pre-trained multimodal model to make predictions based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

[0152] In some possible implementations, the data processing device also includes a training module, which is used to: train the encoder using multimodal training data to obtain a trained encoder; lock the parameters of the encoder, and use the training data to train the fully connected module and the prediction module of the multimodal model to obtain a pre-trained multimodal model.

[0153] It can be understood that the above modules of the data processing device in the embodiment of the present disclosure have the function of implementing Figure 1 The functions of the corresponding steps of the data processing method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above data processing device, please refer to Figure 1 The corresponding description of the data processing method in the embodiment shown in will not be repeated here.

[0154] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0155] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0156] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method provided in the embodiment of the present disclosure.

[0157] Compared with the existing technology, this electronic device can not only process single-modal information, but also effectively learn and reason about the context of modal information. It can adapt to a variety of tasks and application scenarios, especially long-term conversations and complex task scenarios, and has strong versatility.

[0158] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a data processing method as provided in an embodiment of the present disclosure.

[0159] Compared with the existing technology, this readable storage medium can not only process single-modal information, but also effectively learn and reason about the context of modal information. It can adapt to a variety of tasks and application scenarios, especially long-term conversations and complex task scenarios, and has strong versatility.

[0160] The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the data processing method provided in the embodiment of the present disclosure.

[0161] Compared with the existing technology, this computer program product can not only process single-modal information, but also effectively learn and reason about the context of modal information. It can adapt to a variety of tasks and application scenarios, especially long-term conversations and complex task scenarios, and has strong versatility.

[0162] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0163] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0164] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0165] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0166] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0170] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0171] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0172] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0173] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.< / eos> < / eor> < / eoi> < / boi> < / eot> < / bot> < / bor> < / eor> < / eoi> < / boi> < / eot> < / bot> < / bor> < / bos> < / eor> < / eoi> < / boi> < / eot> < / bot> < / bor> < / eor> < / eoi> < / boi> < / eot> < / bot> < / bor> < / eos> < / bos> < / eos> < / eor> < / eoi> < / boi> < / eot> < / bot> < / bor> < / eor> < / eoi> < / boi> < / eot> < / bot> < / bor> < / bos> < / eoi> < / boi> < / eot> < / bot> < / eor> < / bor> < / eor> < / eoi> < / boi> < / eot> < / bot> < / bor> < / eot> < / bot> < / eot> < / bot> < / eoi> < / boi> < / eoi> < / boi>

Claims

1. A data processing method, comprising: Obtain multiple multimodal data input by the user; At least one of the multimodal data has a different modality from the other multimodal data; Encode the multimodal data to obtain an embedding vector corresponding to the multimodal data; Arranging the embedding vectors corresponding to the plurality of multimodal data according to the order input by the user to obtain a multimodal embedding vector; Prediction is performed based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

2. The method according to claim 1, wherein: At least one of the multimodal data is image data; at least one of the multimodal data is text data; The encoding of the multimodal data to obtain an embedding vector corresponding to the multimodal data includes: In the case where the multimodal data is image data, encoding the image data, extracting image data features of the image data, and converting the image data features into image embedding vectors; In the case where the multimodal data is text data, the text data is encoded and converted into a text embedding vector.

3. The method according to claim 2, wherein: Arranging the embedding vectors corresponding to the plurality of multimodal data in the order input by the user to obtain the multimodal embedding vector includes: Assigning an image label to the image embedding vector to form an image representation corresponding to the image data; Assigning a text label to the text embedding vector to form a text representation corresponding to the text data; The image representation and the text representation are arranged according to the input order of the image data and the text data by a user to obtain the multimodal embedding vector.

4. The method according to claim 1, wherein: The obtaining of a plurality of multimodal data input by a user comprises: Acquire multiple multimodal data input by the user during multiple rounds of human-computer dialogue; Arranging the embedding vectors corresponding to the plurality of multimodal data in the order input by the user to obtain the multimodal embedding vector includes: Arrange the embedding vectors corresponding to at least one of the multimodal data input by the user during a round of human-computer dialogue according to the user input order, and obtain the round embedding vector corresponding to the round of human-computer dialogue; Assigning a round label to the round embedding vector to form a vector representation corresponding to the round of human-computer dialogue; The vector representations corresponding to each round of human-computer dialogue are sorted and fused according to the order of user input, and dialogue labels are assigned to the sorting and fusion results to obtain a multimodal embedding vector.

5. The method according to claim 1, wherein: The predicting according to the multimodal embedding vector to generate answer data corresponding to the multimodal data includes: Generate a predicted command sequence using a next command prediction method according to the multimodal embedding vector, use the generated predicted command sequence as an input command sequence, predict a target command sequence according to the input command sequence, and use the target command sequence as a new input command sequence to generate a new target command sequence, until a target command sequence that meets preset requirements is generated; The target token sequence is decoded to generate answer data corresponding to the multimodal data.

6. The method according to claim 5, wherein: The predicting of the target token sequence according to the input token sequence comprises: Obtain an attention weight matrix, wherein the number of rows of the attention weight matrix is ​​the length of the target token sequence, the number of columns of the attention weight matrix is ​​the length of the input token sequence, each row of the attention weight matrix is ​​the attention weight distribution of a token in the target token sequence to all tokens in the input token sequence; each element of the attention weight matrix is ​​the attention weight of a token in the target token sequence to a token in the input token sequence; Attention shift is identified based on the attention weight matrix, and the attention shift is adjusted through a feedback mechanism.

7. The method according to claim 6, wherein: The step of identifying attention shift according to the attention weight matrix comprises: According to the attention weight matrix, an input token is determined from all tokens in the input token sequence; the input token is a token in the input token sequence corresponding to the largest attention weight among the attention weights corresponding to the target tokens in the target token sequence; In the case where the input command is a noise command, it is determined that attention diversion occurs to the target command.

8. The method according to claim 7, wherein: The step of identifying attention shift according to the attention weight matrix comprises: Calculating, according to the attention weight matrix, the distribution of input tokens corresponding to the target tokens of the target token sequence in a plurality of time steps; When the distribution of the input command corresponding to the target command changes suddenly, it is determined that the attention shift occurs to the target command.

9. The method according to claim 6, wherein: The step of identifying attention shift according to the attention weight matrix comprises: Calculating a difference between an attention weight of a target token of the target token sequence to each input token in the input token sequence and a pre-acquired mean of the attention weights; When the sum of all the differences is greater than a preset value, it is determined that the target symbol has shifted attention.

10. The method according to claim 1, wherein: The encoding of the multimodal data to obtain an embedding vector corresponding to the multimodal data includes: Encode the multimodal data using an encoder of a pre-trained multimodal model to obtain an embedding vector corresponding to the multimodal data; Arranging the embedding vectors corresponding to the plurality of multimodal data in the order input by the user to obtain the multimodal embedding vector includes: Arrange the embedding vectors corresponding to the plurality of multimodal data in the order input by the user using a fully connected module of a pre-trained multimodal model to obtain a multimodal embedding vector; The predicting according to the multimodal embedding vector to generate answer data corresponding to the multimodal data includes: Using a prediction module of a pre-trained multimodal model, prediction is performed based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

11. The method according to claim 10, further comprising: Using multimodal training data to train the encoder to obtain a trained encoder; The parameters of the encoder are locked, and the fully connected module and the prediction module of the multimodal model are trained using the training data to obtain a pre-trained multimodal model.

12. A data processing device, comprising: A data module is used to obtain multiple multimodal data input by the user; At least one of the multimodal data has a different modality from the other multimodal data; An encoding module, used to encode the multimodal data to obtain an embedding vector corresponding to the multimodal data; A fully connected module, used to arrange the embedding vectors corresponding to the plurality of multimodal data according to the order input by the user to obtain a multimodal embedding vector; A prediction module is used to make predictions based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

13. The device according to claim 12, wherein: At least one of the multimodal data is image data; at least one of the multimodal data is text data; The encoding module comprises: An image submodule, for encoding the image data, extracting image data features of the image data, and converting the image data features into an image embedding vector when the multimodal data is image data; The text submodule is used to encode the text data when the multimodal data is text data, and convert the text data into a text embedding vector.

14. The device according to claim 13, wherein: The fully connected module includes: A first labeling unit is used to assign an image label to the image embedding vector to form an image representation corresponding to the image data; and to assign a text label to the text embedding vector to form a text representation corresponding to the text data; A sorting unit is used to arrange the image representation and the text representation according to the input order of the image data and the text data by a user to obtain the multimodal embedding vector.

15. The device according to claim 12, wherein: The data module is used to: obtain a plurality of multimodal data input by the user during multiple rounds of human-computer dialogue; The fully connected module includes: The second labeling unit is used to arrange the embedding vectors corresponding to at least one of the multimodal data input by the user during a round of human-computer dialogue according to the user input order, and obtain the round embedding vector corresponding to the round of human-computer dialogue; assign a round label to the round embedding vector to form a vector representation corresponding to the round of human-computer dialogue; The third labeling unit is used to sort and fuse the vector representations corresponding to each round of human-computer dialogue according to the order of user input, and assign dialogue labels to the sorting and fusion results to obtain a multimodal embedding vector.

16. The device according to claim 12, wherein: The prediction module is used to: Generate a predicted command sequence using a next command prediction method according to the multimodal embedding vector, use the generated predicted command sequence as an input command sequence, predict a target command sequence according to the input command sequence, and use the target command sequence as a new input command sequence to generate a new target command sequence, until a target command sequence that meets preset requirements is generated; The target token sequence is decoded to generate answer data corresponding to the multimodal data.

17. The device according to claim 16, wherein: The prediction module comprises: An attention unit, used to obtain an attention weight matrix, wherein the number of rows of the attention weight matrix is ​​the length of the target token sequence, the number of columns of the attention weight matrix is ​​the length of the input token sequence, each row of the attention weight matrix is ​​the attention weight distribution of a token in the target token sequence to all tokens in the input token sequence; each element of the attention weight matrix is ​​the attention weight of a token in the target token sequence to a token in the input token sequence; The offset identification unit is used to identify the attention shift according to the attention weight matrix and adjust the attention shift through a feedback mechanism.

18. The device according to claim 17, wherein: The offset identification unit is used for: According to the attention weight matrix, an input token is determined from all tokens in the input token sequence; the input token is a token in the input token sequence corresponding to the largest attention weight among the attention weights corresponding to the target tokens in the target token sequence; In the case where the input command is a noise command, it is determined that attention diversion occurs to the target command.

19. The device according to claim 18, wherein: The offset identification unit is used for: Calculating, according to the attention weight matrix, the distribution of input tokens corresponding to the target tokens of the target token sequence in a plurality of time steps; When the distribution of the input command corresponding to the target command changes suddenly, it is determined that the attention shift occurs to the target command.

20. The device according to claim 17, wherein: The offset identification unit is used for: Calculating a difference between an attention weight of a target token of the target token sequence to each input token in the input token sequence and a pre-acquired mean of the attention weights; When the sum of all the differences is greater than a preset value, it is determined that the target symbol has shifted attention.

21. The device according to claim 12, wherein: The encoding module is used to: encode the multimodal data using an encoder of a pre-trained multimodal model to obtain an embedding vector corresponding to the multimodal data; The fully connected module is used to: use the fully connected module of the pre-trained multimodal model to arrange the embedding vectors corresponding to the plurality of multimodal data in the order input by the user to obtain a multimodal embedding vector; The prediction module is used to use a prediction module of a pre-trained multimodal model to make predictions based on the multimodal embedding vector to generate answer data corresponding to the multimodal data.

22. The apparatus according to claim 21, further comprising a training module for: Using multimodal training data to train the encoder to obtain a trained encoder; The parameters of the encoder are locked, and the fully connected module and the prediction module of the multimodal model are trained using the training data to obtain a pre-trained multimodal model.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images