Multi-modal data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a multi-modal data processing method and device, a storage medium and electronic equipment. The method comprises the steps that multi-modal data to be recognized are acquired, and the multi-modal data comprise image data and text data; obtaining an image embedding vector corresponding to the image data, and converting the image embedding vector into a pre-fusion coding vector based on a pre-training fusion word list; the pre-training fusion word list is a coding book which is obtained according to an image training sample and is used for reducing the coding quantity of image features; and combining the pre-fused coding vector and a text embedding vector corresponding to the text data to obtain a target multi-modal vector. According to the method and the device, the technical problems that a multi-modal large language model occupies more computing resources, multi-modal information is difficult to effectively understand and fuse and the overall performance is poor in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal data processing method, device, storage medium and electronic device. Background Art

[0002] When processing multimodal information such as images and text, multimodal large language models in related technologies usually process images into a large number of independent codes. This method not only takes up a lot of computing resources, but also makes it difficult to capture the structural information and semantic associations in the image, making it difficult for the multimodal large language model to effectively understand and fuse multimodal information. In addition, large language models usually have a fixed input length limit, which will lead to information loss when processing high-dimensional information such as images, and will also limit the amount of multimodal information processed simultaneously, thus resulting in poor overall performance of the multimodal large language model. Summary of the invention

[0003] The embodiments of the present application provide a multimodal data processing method, device, storage medium and electronic device to at least solve the technical problems in the related art that large multimodal language models occupy a lot of computing resources, are difficult to effectively understand and integrate multimodal information, and have poor overall performance.

[0004] According to one aspect of an embodiment of the present application, a multimodal data processing method is provided, including: obtaining multimodal data to be identified, the multimodal data including image data and text data; obtaining an image embedding vector corresponding to the image data, and converting the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples for reducing the coding amount of image features; and combining the pre-fused coding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

[0005] According to another aspect of an embodiment of the present application, a multimodal data processing device is also provided, including: a first acquisition unit, used to acquire multimodal data to be identified, the multimodal data including image data and text data; a second acquisition unit, used to acquire an image embedding vector corresponding to the image data, and convert the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples for reducing the coding amount of image features; a combination unit, used to combine the pre-fused coding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

[0006] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the multimodal data processing method through the computer program.

[0007] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned multimodal data processing method when running.

[0008] In the embodiment of the present application, a method is adopted to obtain multimodal data to be identified, wherein the multimodal data includes image data and text data; obtain an image embedding vector corresponding to the image data, and convert the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained according to image training samples for reducing the amount of coding of image features; the pre-fused coding vector and the text embedding vector corresponding to the text data are combined to obtain a target multimodal vector; in the above method, based on the preset pre-trained fusion vocabulary, the model can learn the connection between visual features and training corpus more easily, and can also greatly reduce the number of codes required to represent each image, so that the model can process more image information or larger images within a limited context length. It can not only greatly save the computing resources occupied by the large language model, but also improve the performance of the multimodal large language model in understanding and fusing multimodal information; it solves the technical problem that the multimodal large language model in the related art occupies more computing resources, is difficult to effectively understand and fuse multimodal information, and has poor overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0010] Figure 1 is a schematic diagram of an application environment of an optional multimodal data processing method according to an embodiment of the present application;

[0011] Figure 2 is a schematic diagram of an application environment of another optional multimodal data processing method according to an embodiment of the present application;

[0012] Figure 3 is a flowchart of an optional multimodal data processing method according to an embodiment of the present application;

[0013] Figure 4 is a schematic diagram of a multimodal large language model framework of an optional pre-fused vocabulary according to an embodiment of the present application;

[0014] Figure 5 is a schematic diagram of an optional VQGAN model architecture according to an embodiment of the present application;

[0015] Figure 6 is an optional adjacency matrix vocabulary expansion training flow chart according to an embodiment of the present application;

[0016] Figure 7 is a schematic diagram of an optional Transformer network structure based on an attention mechanism according to an embodiment of the present application;

[0017] Figure 8 is a schematic structural diagram of an optional multimodal data processing device according to an embodiment of the present application;

[0018] Fig. 9 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] According to one aspect of an embodiment of the present application, a multimodal data processing method is provided. Optionally, as an optional implementation, the multimodal data processing method can be but is not limited to being applied to Figure 1In the application environment shown. The application environment includes: a terminal device 102 for human-computer interaction with a user, a network 104, and a server 106. A user 108 can perform human-computer interaction with the terminal device 102, and a multimodal data processing application client runs in the terminal device 102. The above-mentioned terminal device 102 includes a human-computer interaction screen 1022, a processor 1024 and a memory 1026. The human-computer interaction screen 1022 is used to display image data and text data; the processor 1024 is used to obtain multimodal data to be identified. The memory 1026 is used to store the multimodal data to be identified and the target multimodal vector corresponding to the multimodal data.

[0022] In addition, the server 106 includes a database 1062 and a processing engine 1064, wherein the database 1062 is used to store the multimodal data to be identified and the target multimodal vector corresponding to the multimodal data. The processing engine 1064 is used to obtain the multimodal data to be identified, wherein the multimodal data includes image data and text data; obtain the image embedding vector corresponding to the image data, and convert the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples for reducing the coding amount of image features; and combine the pre-fused coding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

[0023] In one or more embodiments, the multimodal data processing method of the present application can be applied to Figure 2 In the application environment shown. Figure 2 As shown, human-computer interaction can be performed between user 202 and user device 204. User device 204 includes memory 206 and processor 208. In this embodiment, user device 204 can refer to but not be limited to executing the operations performed by the above terminal device 102 to obtain the target multimodal vector corresponding to the multimodal data.

[0024] Optionally, the terminal device 102 and the user device 204 include but are not limited to mobile phones, tablet computers, laptop computers, PCs, vehicle-mounted electronic devices, wearable devices and other terminals, and the network 104 may include but is not limited to wireless networks or wired networks. Among them, the wireless network includes: WIFI and other networks that realize wireless communication. The wired network may include but is not limited to: wide area network, metropolitan area network, local area network. The server 106 may include but is not limited to any hardware device that can perform computing. The server may be a single server, or a server cluster composed of multiple servers, or a cloud server. The above is only an example, and no limitation is made to this in this embodiment.

[0025] As an optional implementation, Figure 3As shown, the embodiment of the present application provides a multimodal data processing method, comprising the following steps:

[0026] S302, obtaining multimodal data to be identified, where the multimodal data includes image data and text data.

[0027] S304, obtaining an image embedding vector corresponding to the image data, and converting the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples for reducing the coding amount of image features.

[0028] S306, combining the pre-fused coding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

[0029] Specifically, in an embodiment of the present application, for the text data in the multimodal data to be identified, a common pre-trained text encoder (such as a BPE encoder) can be used to convert it into a text vector t; for the image data in the multimodal data to be identified, the VQGAN model is first used to pre-process it to obtain an initial encoding vector z, and then the pre-trained fusion vocabulary is used to replace the combination pattern in the vocabulary with the corresponding new image code (token ID) by table lookup replacement, thereby converting the initial encoding vector z to obtain a pre-fused encoding vector z m .

[0030] The text vector t is combined with the pre-fused encoding vector z obtained after processing the image m The multimodal data vector (target multimodal vector) is obtained by concatenation: y = [t; z m ].

[0031] In the embodiment of the present application, a method is adopted to obtain multimodal data to be identified, wherein the multimodal data includes image data and text data; obtain an image embedding vector corresponding to the image data, and convert the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained according to image training samples for reducing the amount of coding of image features; the pre-fused coding vector and the text embedding vector corresponding to the text data are combined to obtain a target multimodal vector; in the above method, based on the preset pre-trained fusion vocabulary, the model can learn the connection between visual features and training corpus more easily, and can also greatly reduce the number of codes required to represent each image, so that the model can process more image information or larger images within a limited context length. It can not only greatly save the computing resources occupied by the large language model, but also improve the performance of the multimodal large language model in understanding and fusing multimodal information; it solves the technical problem that the multimodal large language model in the related art occupies more computing resources, is difficult to effectively understand and fuse multimodal information, and has poor overall performance.

[0032] In one or more embodiments, the multimodal data processing method further includes:

[0033] Obtain the initial codebook corresponding to the preset image processing model and image training samples;

[0034] Acquire an image embedding vector corresponding to the image training sample based on the image processing model;

[0035] Based on the image embedding vector corresponding to the image training sample, the initial codebook is updated to obtain the pre-trained fusion vocabulary.

[0036] Specifically, in the embodiment of the present application, the above-mentioned image processing model may be a VQGAN model; taking the VQGAN model as an example, combined with Figure 4 and Figure 5 As shown, the VQGAN model is first used to quantize the input image. First, the input image is resized to h×h pixels, and then the pre-trained VQGAN model is used to encode the image. Through the initial codebook of length N in the VQGAN model, a quantized two-dimensional discrete codebook index array (image embedding vector) of size N×N is obtained; where h and N are positive integers;

[0037] In one example, the array may be flattened into a one-dimensional vector of length N×N. Then, the initial codebook is updated according to the one-dimensional vector of length N×N corresponding to the image training sample to obtain the pre-trained fusion vocabulary. For example, the length of the initial codebook is adjusted to N+K, where K is a positive integer. Here, each character in the initial codebook represents a coding mode.

[0038] In one or more embodiments, the preset image processing model includes a VQGAN model, and the updating of the initial codebook based on the image embedding vector corresponding to the image training sample to obtain the pre-trained fusion vocabulary includes the following steps:

[0039] Obtaining an image embedding vector corresponding to the image training sample obtained based on the VQGAN model;

[0040] For the image embedding vector corresponding to each image in the image training sample, the following codebook update operations are performed in sequence until the initial codebook reaches a preset code quantity:

[0041] Determine an image coding connection state of an image embedding vector corresponding to a current image, wherein the image coding connection state includes an image coding connection state in a horizontal direction and an image coding connection state in a vertical direction;

[0042] Determine, according to the image coding connection state, whether there is a first coding pair in the current image that appears more than a preset number of times; the first coding pair indicates two image codes that are adjacent in the horizontal direction or the vertical direction;

[0043] If the current image has a first code pair that appears more than a preset number of times, the first code pair is replaced with a first updated code value to obtain a first image embedding vector, and the first updated code value is added to the initial code book;

[0044] Determine whether there is a second coding pair in the updated image embedding vector that appears more than a preset number of times; the second coding pair indicates two image codes that are adjacent in the horizontal direction or vertical direction; if so, replace the second coding pair with a second updated coding value to obtain a second image embedding vector, and add the second updated coding value to the initial coding book.

[0045] Specifically, in the embodiment of the present application, the image embedding vector corresponding to the image training sample obtained based on the VQGAN model is first obtained;

[0046] Creating a first adjacency matrix H and a second adjacency matrix V based on the image embedding vector, wherein the first adjacency matrix H is used to record the image coding connection state in the horizontal direction corresponding to the image training sample, and the second adjacency matrix V is used to record the image coding connection state in the vertical direction corresponding to the image training sample;

[0047] Determine all code pairs (i, j) for which H[i][j] is greater than a preset number or V[i][j] is greater than the preset number; H[i][j] represents the number of times code i is adjacent to code j in the horizontal direction; V[i][j] represents the number of times code i is adjacent to code j in the vertical direction;

[0048] For each found code pair (i, j) combination, the size of the initial code book is calculated as N, and the code pair (i, j) is taken as a whole. Here, for example, the new code q can be used to replace the (i, j) code pair, and the original image embedding vector is updated to obtain the first image embedding vector. Then, the code q is added to the initial code book. At this time, the size of the initial code book becomes N+1;

[0049] Then, an extended search is continued in the vertical coding direction and the vertical coding direction to determine all code pairs (q, m) whose H[q][m] is greater than a preset number of times or V[q][m] is greater than the preset number of times; the H[q][m] represents the number of times code q is adjacent to code m in the horizontal direction; V[q][m] represents the number of times code q is adjacent to code k in the vertical direction; if it exists, a second code pair (q, m) is formed, and m is the code adjacent to code q; here, for example, a new code n can be used to replace the (q, m) code pair, update the original image embedding vector, obtain the second image embedding vector, and then add code n to the initial code book, at which time the size of the initial code book becomes N+2; if there is no code pair (q, m) whose H[q][m] is greater than the preset number of times or V[q][m] is greater than the preset number of times, the next image of the image training sample is used as the current image to continue to perform the code book update operation.

[0050] Then, the above steps are cyclically performed for a plurality of image training samples, and when the initial codebook reaches a preset code quantity, it is determined that the pre-trained fusion vocabulary is obtained.

[0051] Specifically, Figure 6 As shown, in an embodiment of the present application, the size of the first adjacency matrix H and the second adjacency matrix V can be N×N, where N is the size of the initial codebook. The present application can use a large number of image data sets to further cyclically train and expand the initial codebook. For example, the above steps are set to the number of cycles K, and finally the training obtains an expanded vocabulary of size N+K. The expanded vocabulary is the pre-trained fusion vocabulary mentioned above. Based on the pre-fusion vocabulary, any obtained image tensor of length N×N can be replaced with the K expanded combination patterns appearing in the vocabulary with the corresponding new image encoding, thereby realizing the pre-fusion of the spatial prior information of the one-dimensional image tensor.

[0052] In one or more embodiments, the multimodal data processing method further includes:

[0053] If the current image does not have a first coding pair that appears more than a preset number of times, taking the next image of the image training sample as the current image to continue to perform the codebook update operation;

[0054] When the number of codes in the initial codebook reaches a preset number of codes, it is determined that the pre-trained fusion vocabulary is obtained.

[0055] For example, if it is determined that all H[i][j] corresponding to the current image are less than a preset number of times, or V[i][j] is less than the preset number of coding pairs (i, j), then the next image of the image training sample is used as the current image to continue to perform the codebook update operation. After performing the above-mentioned codebook update operation multiple times, when the number of codes in the initial codebook reaches the preset number of codes, it is determined that the pre-trained fusion vocabulary is obtained.

[0056] In one or more embodiments, the multimodal data processing method further includes:

[0057] Mapping the image training sample to a latent space through the encoder of the VQGAN model to generate a latent vector corresponding to the image training sample;

[0058] Discretize the potential vector into a set of discrete vectors by a quantizer of the VQGAN model, and determine the code vector closest to the discrete vector by nearest neighbor search;

[0059] The decoder based on the VQGAN model maps the code vector back to the image space to generate a reconstructed image;

[0060] The reconstructed image is adjusted based on the generative adversarial network in the VQGAN model to obtain an image embedding vector corresponding to the image training sample.

[0061] like Figure 5 As shown in Figure 1, the VQGAN (Vector Quantized Generative Adversarial Network) model is a model that combines vector quantization technology with a generative adversarial network (GAN), and is mainly used for the generation and compression of high-quality images. The VQGAN model decomposes an image into a set of discrete vectors (or codes), which are then generated and reconstructed through GAN, thereby retaining the details and texture of the image. The VQGAN model consists of the following main parts:

[0062] 1. Encoder: Maps the input image to a latent space and generates a latent vector.

[0063] 2. Quantizer: discretizes the continuous latent vector into a finite set of codes (or vectors).

[0064] 3. Decoder: Maps the discretized code back to the image space to generate a reconstructed image.

[0065] 4. Generative Adversarial Network (GAN): It consists of a generator and a discriminator and is used to improve the quality of image generation.

[0066] In the encoder of VQGAN, the input image x is mapped to a latent space through a deep neural network to generate a latent vector z e (x), the mapping formula includes formula (1): z e (x) = Encoder (x) (1)

[0067] Afterwards, VQGAN uses vector quantization (VQ) technology to convert the continuous latent vector z e (x) is mapped to a set of discrete vectors, and the nearest code vector z is found through nearest neighbor search q (x). Specifically, configure e i is the i-th vector in the codebook, then the quantization formula includes formula (2):

[0068] z q (x) = e i ,where i=arg min j ||z e (x)-e j || (2);

[0069] Among them, z q (x) is the discretized code vector.

[0070] The decoder uses a convolutional neural network to discretize the code vector z q (x) Map back to the image space to generate a reconstructed image Among them, the reconstructed image The mapping formula includes formula (3):

[0071] In one or more embodiments, the multimodal data processing method further includes:

[0072] The VQGAN model is jointly trained based on a preset reconstruction loss function, a quantization loss function, and an adversarial loss function. When the VQGAN model reaches a convergence condition, a preset image processing model is determined to be obtained.

[0073] In order to ensure that the encoding after the image quantization and discrete processing can fully contain the original image information, the VQGAN model is jointly trained based on the preset reconstruction loss function, quantization loss function and adversarial loss function;

[0074] Regarding the reconstruction loss, it is necessary to measure the difference between the reconstructed image and the original image. Specifically, the corresponding reconstruction loss function includes formula (4):

[0075]

[0076] Regarding the quantization loss, it is necessary to minimize the difference between the latent vector and its quantized vector. Specifically, the corresponding loss function is shown in formula (5):

[0077]

[0078] Among them, β is an adjustable weight parameter, and sg represents the Stop-Gradient operation commonly used in deep learning, which is used to retain the calculated gradient to ensure that the gradient can be successfully returned.

[0079] Regarding adversarial loss, it is necessary to improve the reconstruction quality of the image and configure the loss function of the generator And the loss function of the discriminator As shown in formula (6) and formula (7):

[0080]

[0081] Among them, D(x) represents the discrimination result of the discriminator on the input image.

[0082] In one or more embodiments, the multimodal data processing method further includes:

[0083] Using the target multimodal vector as sample data, training a Transformer model;

[0084] When the accuracy of the Transformer model in identifying the data sequence in the sample data reaches a preset threshold, it is determined that a pre-trained multimodal data recognition model is obtained.

[0085] like Figure 7 As shown in Figure 2, the core of the Transformer model is the attention mechanism, which allows the model to dynamically focus on different parts of the input sequence when processing the current element. The calculation process is as follows: given the query matrix Key Matrix Sum Value Matrix Where n is the number of queries, m is the number of key-value pairs, and d k and dv are the dimensions of key and value respectively. The attention weight is calculated by formula (8):

[0086]

[0087] in, It is a scaling factor used to alleviate the problem that the inner product value becomes larger as the dimension increases.

[0088] In order to further enhance the expressiveness of the model, Transformer introduces a multi-head attention mechanism, including formula (9):

[0089] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (9)

[0090] The calculation formula for each attention head includes formula (10):

[0091]

[0092] Among them, W O , and are all trainable parameter matrices.

[0093] After calculating the dependencies between input sequences through the multi-head attention mechanism, a feed-forward neural network layer (FFN) is applied to each position. The feed-forward neural network used is composed of two linear transformations and a ReLU activation function, as shown in formula (11):

[0094] FFN(x)=max(0,xW1+b1)W2+b2 (11)

[0095] in, and is a trainable weight matrix, b1 and b2 are bias terms.

[0096] Each sub-layer will be followed by layer normalization and residual connection, including formula (12):

[0097] LayerNorm(x+Sublayer(x)) (12)

[0098] In addition, since Transformer does not contain convolution or recurrent structures, it is unable to capture the position information in the sequence. Therefore, position encoding is introduced. The position encoding formula includes formula (13):

[0099]

[0100] Among them, pos represents the position and i represents the dimension index. The position encoding is added to the input embeddings, allowing the model to utilize position information.

[0101] The training process of the Transformer model includes two steps: forward propagation and backward propagation. Using the autoregressive model architecture, the structure described above is iterated through multiple layers to calculate attention and finally generate the target sequence, and the model parameters are optimized through back propagation.

[0102] When the accuracy of the Transformer model in identifying the data sequence in the sample data reaches a preset threshold, it is determined that a pre-trained multimodal data recognition model is obtained.

[0103] In one or more embodiments, the multimodal data processing method further includes:

[0104] Obtain the target multimodal vector corresponding to the image and text input by the user;

[0105] Identify image features and text features corresponding to the target multimodal vector based on a pre-trained multimodal data recognition model;

[0106] Based on the image features and text features, semantic information corresponding to the image and text is generated.

[0107] Specifically, Figure 4 As shown, the image input by the user includes grass and animals in the image, and the text can be questions from the user about the image, such as text information such as what animals are in the picture. First, the image embedding vector corresponding to the image data is obtained (for example, the encoding matrix with the initial encoding of 3*3), and the image embedding vector is converted into a pre-fused encoding vector [111, 3, 22, 333] based on the pre-trained fusion vocabulary (pre-trained VQGAN codebook); the pre-fused encoding vector and the text embedding vector corresponding to the text data are combined to obtain the target multimodal vector. Based on the pre-trained Transformer model and the pre-trained fusion vocabulary, the original encoding matrix of size 3*3 represented by the above-mentioned pre-fused encoding vector is identified. Based on the image features and text features, the semantic information corresponding to the image and text is generated, for example, including: the image and the language features of the image (the picture depicts a dog on the grass, etc.).

[0108] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0109] According to another aspect of the embodiment of the present application, a multimodal data processing device for implementing the above multimodal data processing method is also provided. Figure 8 As shown, the device comprises:

[0110] A first acquisition unit 802 is used to acquire multimodal data to be identified, where the multimodal data includes image data and text data;

[0111] The second acquisition unit 804 is used to acquire an image embedding vector corresponding to the image data, and convert the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples and used to reduce the coding amount of image features;

[0112] The combining unit 806 is used to combine the pre-fused coding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

[0113] In the embodiment of the present application, a method is adopted to obtain multimodal data to be identified, wherein the multimodal data includes image data and text data; obtain an image embedding vector corresponding to the image data, and convert the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained according to image training samples for reducing the amount of coding of image features; the pre-fused coding vector and the text embedding vector corresponding to the text data are combined to obtain a target multimodal vector; in the above method, based on the preset pre-trained fusion vocabulary, the model can learn the connection between visual features and training corpus more easily, and can also greatly reduce the number of codes required to represent each image, so that the model can process more image information or larger images within a limited context length. It can not only greatly save the computing resources occupied by the large language model, but also improve the performance of the multimodal large language model in understanding and fusing multimodal information; it solves the technical problem that the multimodal large language model in the related art occupies more computing resources, is difficult to effectively understand and fuse multimodal information, and has poor overall performance.

[0114] In one or more embodiments, the multimodal data processing device further includes:

[0115] A third acquisition unit is used to acquire an initial codebook corresponding to a preset image processing model and an image training sample;

[0116] A fourth acquisition unit, configured to acquire an image embedding vector corresponding to the image training sample based on the image processing model;

[0117] An updating unit is used to update the initial codebook based on the image embedding vector corresponding to the image training sample to obtain the pre-trained fusion vocabulary.

[0118] In one or more embodiments, the updating unit includes:

[0119] A first acquisition module is used to obtain an image embedding vector corresponding to the image training sample obtained based on the VQGAN model;

[0120] The first determination module is used to sequentially perform the following codebook update operations on the image embedding vector corresponding to each image in the image training sample until the initial codebook reaches a preset code quantity:

[0121] Determine an image coding connection state of an image embedding vector corresponding to a current image, wherein the image coding connection state includes an image coding connection state in a horizontal direction and an image coding connection state in a vertical direction;

[0122] A second determination module is used to determine whether there is a first code pair in the current image that appears more than a preset number of times according to the image code connection state; the first code pair indicates two image codes that are adjacent in the horizontal direction or the vertical direction;

[0123] A replacement module, configured to replace the first code pair with a first updated code value if the current image has a first code pair that appears more than a preset number of times, obtain a first image embedding vector, and add the first updated code value to the initial code book;

[0124] The judgment update module is used to judge whether there is a second coding pair in the updated image embedding vector whose number of occurrences is greater than a preset number; the second coding pair indicates two image codes adjacent in the horizontal direction or the vertical direction; if it exists, the second coding pair is replaced by a second updated coding value to obtain a second image embedding vector, and the second updated coding value is added to the initial coding book.

[0125] In one or more embodiments, the multimodal data processing device further includes:

[0126] A judgment unit, configured to, if the current image does not have a first coding pair that appears more than a preset number of times, take the next image of the image training sample as the current image to continue to perform the codebook update operation;

[0127] The first determining unit is used to determine to obtain the pre-trained fusion vocabulary when the number of codes in the initial codebook reaches a preset number of codes.

[0128] In one or more embodiments, the multimodal data processing device further includes:

[0129] A first generating unit, configured to map the image training sample to a latent space through an encoder of the VQGAN model to generate a latent vector corresponding to the image training sample;

[0130] A discretization unit, configured to discretize the potential vector into a set of discrete vectors by a quantizer of the VQGAN model, and determine a code vector closest to the discrete vector by nearest neighbor search;

[0131] A mapping unit, configured to map the code vector back to the image space based on the decoder of the VQGAN model to generate a reconstructed image;

[0132] An adjustment unit is used to adjust the reconstructed image based on the generative adversarial network in the VQGAN model to obtain an image embedding vector corresponding to the image training sample.

[0133] In one or more embodiments, the multimodal data processing device further includes:

[0134] The first training unit is used to jointly train the VQGAN model based on a preset reconstruction loss function, a quantization loss function and an adversarial loss function, and when the VQGAN model reaches a convergence condition, determine to obtain a preset image processing model.

[0135] In one or more embodiments, the multimodal data processing device further includes:

[0136] A second training unit is used to train a Transformer model by using the target multimodal vector as sample data;

[0137] The second determination unit is used to determine the pre-trained multimodal data recognition model when the accuracy of the Transformer model in identifying the data sequence in the sample data reaches a preset threshold.

[0138] In one or more embodiments, the multimodal data processing device further includes:

[0139] A fifth acquisition unit, used to acquire a target multimodal vector corresponding to the image and text input by the user;

[0140] A recognition unit, configured to recognize image features and text features corresponding to the target multimodal vector based on a pre-trained multimodal data recognition model;

[0141] A generating unit is used to generate semantic information corresponding to the image and text based on the image features and text features.

[0142] According to another aspect of the embodiment of the present application, an electronic device for implementing the above multimodal data processing method is also provided. The electronic device may be Fig. 9 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a server as an example. Fig. 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0143] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0144] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0145] S1, obtaining multimodal data to be identified, wherein the multimodal data includes image data and text data;

[0146] S2, obtaining an image embedding vector corresponding to the image data, and converting the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples and used to reduce the coding amount of image features;

[0147] S3, combining the pre-fused encoding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

[0148] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig. 9 The electronic device and the electronic equipment described above are not limited in structure. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations shown.

[0149] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the multimodal data processing method and device in the embodiments of the present application. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, realizing the above-mentioned multimodal data processing method. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely located relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be specifically, but not limited to, used to store image data and text data. As an example, such as Fig. 9 As shown, the memory 902 may include, but is not limited to, the first acquisition unit 802, the second acquisition unit 804, and the combination unit 806 in the multimodal data processing device. In addition, other module units in the multimodal data processing device may also be included but are not limited to, which will not be described in detail in this example.

[0150] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the above-mentioned network may include wired networks and wireless networks. In one example, the transmission device 909 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0151] In addition, the electronic device further includes: a display 908 for displaying image data and text data input by a user; and a connection bus 910 for connecting various module components in the electronic device.

[0152] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes may form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, terminal and other electronic devices, may become a node in the blockchain system by joining the peer-to-peer network.

[0153] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned multimodal data processing method, wherein the computer program is configured to perform the steps of any one of the above-mentioned method embodiments when running.

[0154] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0155] S1, obtaining multimodal data to be identified, wherein the multimodal data includes image data and text data;

[0156] S2, obtaining an image embedding vector corresponding to the image data, and converting the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples and used to reduce the coding amount of image features;

[0157] S3, combining the pre-fused encoding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

[0158] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0159] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0160] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0161] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0162] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0165] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal data processing method, characterized in that: include: Acquire multimodal data to be identified, wherein the multimodal data includes image data and text data; Obtaining an image embedding vector corresponding to the image data, and converting the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples and used to reduce the coding amount of image features; The pre-fused encoding vector is combined with a text embedding vector corresponding to the text data to obtain a target multimodal vector.

2. The method according to claim 1, characterized in that The method further comprises: Obtain the initial codebook corresponding to the preset image processing model and image training samples; Acquire an image embedding vector corresponding to the image training sample based on the image processing model; Based on the image embedding vector corresponding to the image training sample, the initial codebook is updated to obtain the pre-trained fusion vocabulary.

3. The method according to claim 2, characterized in that The preset image processing model includes a VQGAN model, and the image embedding vector corresponding to the image training sample is updated based on the initial codebook to obtain the pre-trained fusion vocabulary, including: Obtaining an image embedding vector corresponding to the image training sample obtained based on the VQGAN model; For the image embedding vector corresponding to each image in the image training sample, the following codebook update operations are performed in sequence until the initial codebook reaches a preset code quantity: Determine an image coding connection state of an image embedding vector corresponding to a current image, wherein the image coding connection state includes an image coding connection state in a horizontal direction and an image coding connection state in a vertical direction; Determine, according to the image coding connection state, whether there is a first coding pair in the current image that appears more than a preset number of times; the first coding pair indicates two image codes that are adjacent in the horizontal direction or the vertical direction; If the current image has a first code pair that appears more than a preset number of times, the first code pair is replaced with a first updated code value to obtain a first image embedding vector, and the first updated code value is added to the initial code book; Determine whether there is a second coding pair in the updated image embedding vector that appears more than a preset number of times; the second coding pair indicates two image codes that are adjacent in the horizontal direction or vertical direction; if so, replace the second coding pair with a second updated coding value to obtain a second image embedding vector, and add the second updated coding value to the initial coding book.

4. The method according to claim 3, characterized in that The method further comprises: If the current image does not have a first coding pair that appears more than a preset number of times, taking the next image of the image training sample as the current image to continue to perform the codebook update operation; When the number of codes in the initial codebook reaches a preset number of codes, it is determined that the pre-trained fusion vocabulary is obtained.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: Mapping the image training sample to a latent space through the encoder of the VQGAN model to generate a latent vector corresponding to the image training sample; Discretize the potential vector into a set of discrete vectors by a quantizer of the VQGAN model, and determine the code vector closest to the discrete vector by nearest neighbor search; The decoder based on the VQGAN model maps the code vector back to the image space to generate a reconstructed image; The reconstructed image is adjusted based on the generative adversarial network in the VQGAN model to obtain an image embedding vector corresponding to the image training sample.

6. The method according to claim 5, characterized in that The method further comprises: The VQGAN model is jointly trained based on a preset reconstruction loss function, a quantization loss function, and an adversarial loss function. When the VQGAN model reaches a convergence condition, it is determined to obtain the preset image processing model.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Using the target multimodal vector as sample data, training a Transformer model; When the accuracy of the Transformer model in identifying the data sequence in the sample data reaches a preset threshold, it is determined that a pre-trained multimodal data recognition model is obtained.

8. The method according to claim 7, characterized in that The method further comprises: Obtain the target multimodal vector corresponding to the image and text input by the user; Identify image features and text features corresponding to the target multimodal vector based on a pre-trained multimodal data recognition model; Based on the image features and text features, semantic information corresponding to the image and text is generated.

9. A multimodal data processing device, characterized in that: include: A first acquisition unit, configured to acquire multimodal data to be identified, wherein the multimodal data includes image data and text data; A second acquisition unit is used to acquire an image embedding vector corresponding to the image data, and convert the image embedding vector into a pre-fused coding vector based on a pre-trained fusion vocabulary; the pre-trained fusion vocabulary is a coding book obtained based on image training samples and used to reduce the coding amount of image features; A combining unit is used to combine the pre-fused coding vector and the text embedding vector corresponding to the text data to obtain a target multimodal vector.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 8 when executed.

Images