A device and method for discrete semantic coding of multimodal data

Abstract

The present invention discloses a device and method for discrete semantic coding of multimodal data, the method comprising the following steps: S1, the input data of the data encoder is x, LSTM is used to encode the text data, and CNN model is used to encode the picture, and the output result is a multidimensional tensor h; S2, the dimension representing the real meaning in h is input into Transformer as the main coordinate, and then after several layers of Transformer calculation, the first M dimensions of Transformer are used to predict q(z|h). This scheme realizes the discretization of data encoding z through a new loss function, ensures that z contains rich information, and retains the information points in the original data to the greatest extent.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a device and method for discrete semantic coding of multimodal data. Background Art

[0002] Converting multimodal data into discrete codes is of great significance in many industries. First, because discrete variables take up less space than continuous analog data, they can greatly save data storage space, which is very useful in the fields of data lakes, the Internet of Things, and edge computing, greatly saving costs. Secondly, discrete codes can be retrieved more efficiently, so discrete codes have huge advantages for large-scale information retrieval (pictures, texts, videos, etc.). Finally, discrete codes have better readability than continuous vectors, which plays an important role in promoting researchers' understanding of artificial intelligence systems and optimizing artificial intelligence decisions. Summary of the invention

[0003] The present invention mainly provides a method and device for discrete semantic coding of multimodal data, which can better preserve the information points in the original data.

[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a device for discrete semantic coding of multimodal data, comprising:

[0005] Data encoder: encodes the input data x, and the output is a multidimensional tensor h;

[0006] Encoding generator: Generates the probability distribution q(z|h) of M K-dimensional discrete variables z based on the tensor h. There are M discrete variables, each of which is K-dimensional.

[0007] For a given tensor h, the encoder generator is a set of neural networks that generates a probability distribution based on M K-dimensional discrete variables as the encoding result q(z|h); at the same time, for each dimension, each different variable, the encoder includes discrete symbols and embedding vectors, the dimension of the embedding vector is D, and the embedding vector is an M×D matrix; the output generated by the encoder consists of two parts, the first part is M discrete symbols, and the other is M corresponding embedding vectors e(z), the size is e(z)∈R M×D ; The dimensions in h that represent real meaning are input into the Transfomer model as the main coordinates, and then after several layers of Transfomer calculations, the first M outputs of the Transfomer model are used as q(z|h).

[0008] A method for discrete semantic coding of multimodal data, based on the above-mentioned coding device, comprises the following steps:

[0009] S1. Encode the input data through the data encoder, and the output result is a multidimensional tensor h;

[0010] S2. The dimensions in h that represent real meaning are input into the Transformer model as the main coordinates. After several layers (generally 12 layers) of Transformer calculations, the first M outputs of the Transformer model are used as q(z|h).

[0011] Preferably, in step S1, LSTM is used to encode text data, and CNN model is used to encode image files.

[0012] Preferably, in step S2, the dimensions in h that represent real meanings include the time axis in the text and the orientation information in the picture.

[0013] Preferably, a method for discrete semantic encoding of multimodal data also includes a training method for the model, and the loss function used in the training method is:

[0014]

[0015] Where q(z) is simulated as follows:

[0016]

[0017] Where, L VAE is the ELBO loss function, I(Z,X) is the maximum mutual information loss, q qФ (z|x) is the encoder model, p(x) is the marginal distribution of x, and p pФ (x|z) is the decoder model, KL is the KL distance, q(z) is the marginal distribution of the pseudo-z, p(z) is the marginal distribution of z, and N is the number of data in a training set;

[0018] During training, the original data is reconstructed through the decoder after obtaining the encoded result.

[0019] The substantial effect brought by the present invention is that, by constructing a neural network model, data is encoded and decoded, and the discretization of the data encoding z is realized through the novel loss function we proposed, so as to ensure that z contains rich information and retains the information points in the original data to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of a data encoding neural network model architecture of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0022] Embodiment: The present invention proposes a discrete data encoding method for multimodal information such as text and pictures. This method encodes and decodes data by constructing a neural network model, and discretizes the data encoding z through the novel loss function we proposed, ensuring that z contains rich information and retains the information points in the original data to the greatest extent.

[0023] Our proposed model consists of three main modules:

[0024] 1. Data encoder: With the development of deep learning, text, image and other data have very mature encoding neural network modules. Assuming the input data is x, we use LSTM to encode the text data and CNN model to encode the image, and the output result is a multi-dimensional tensor h;

[0025] 2. Encoder generator: Given a tensor h, the encoder generator is a set of neural networks that generate the probability distribution q(z|h) of M K-dimensional discrete variables. At the same time, for each dimension, each different variable (a total of K×M), the encoder is equipped with their symbol embedding vectors, and the dimension of the vector is set to D; the output generated by the encoder consists of two parts. The first step is M discrete symbols z, and the other is M corresponding embedding vectors e(z), with a size of e(z)∈R M×D In order to better improve the prediction effect of discrete variables of the encoder, we use a Transformer model Q for prediction. Specifically, we input the dimensions in h that represent real meaning as the main coordinates (such as the time axis in text, the x and y axes in pictures) into Transformer, and then after several layers of Transformer calculations, use the first M dimensions of Transformer to predict q(z|h).

[0026] 3. Decoder: The decoder is another set of generative neural networks used to reconstruct the original data through discrete coding. Similarly, there are many mature architectures for the decoder to choose from, such as Transfomer or FFHQ models.

[0027] We use SGD to train the above model. Specifically, the training is divided into two parts:

[0028] Collection of training data: The method proposed in the present invention does not require manual annotation of data, but only requires a large amount of data itself, such as a large number of documents or a large number of image libraries.

[0029] Loss function: ELBO loss + maximum mutual information loss

[0030]

[0031] Among them, q(z) was difficult to simulate in the past. We innovatively proposed the following simulation method to simulate q(z).

[0032]

[0033] In addition, in some cases, when business experts have prior knowledge of data encoding, we can manually set p(z) and incorporate it into model training.

[0034] Finally, when we have the above model, we only need to use the data encoder and code generation. For any data, we can use them to encode the data and obtain M discrete codes. These codes are used for compressed storage of data or as indexes for retrieval, etc. When the data needs to be reconstructed, we can use the decoder to reconstruct the original data.

[0035] The main innovations of this solution are:

[0036] 1. Using Transformer as the encoding generator, the effect is much better than the traditional MLP-based encoding generator.

[0037] 2. Innovative loss function, using ELBO combined with maximum mutual information to train the model.

[0038] 3. Innovative mutual information simulation method.

[0039] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0040] Although this article uses more terms such as network upgrade model, coding, vector, etc., it does not exclude the possibility of using other terms. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A device for discrete semantic encoding of multimodal data, characterized in that: include: Data encoder: encodes the input data x, and the output is a multidimensional tensor h; LSTM is used to encode text data, and CNN model is used to encode image files; Encoding generator: Generates the probability distribution q(z|h) of M K-dimensional discrete variables based on the tensor h. There are M discrete variables, each of which is K-dimensional. For a given tensor h, the encoder generator is a set of neural networks that generates a probability distribution based on M K-dimensional discrete variables as the encoding result q(z|h); at the same time, for each dimension, each different variable, the encoder contains discrete symbols and embedding vectors, the dimension of the embedding vector is D, and the embedding vector is an M×D matrix; the output generated by the encoder consists of two parts, the first part is M discrete symbols, and the other is M corresponding embedding vectors e(z), the size is e(z)∈R M×D ; The dimensions in h that represent real meaning are input into the Transformer model as the main coordinates, and then after several layers of Transformer calculations, the first M outputs of the Transformer model are used as q(z|h); The dimensions in h that represent real meaning include the time axis in the text and the orientation information in the picture.

2. A method for discrete semantic coding of multimodal data, based on the coding device according to claim 1, characterized in that: The following steps are involved: S1. Encode the input data through the data encoder, and the output result is a multidimensional tensor h; S2. Input the dimensions in h that represent real meaning into the Transformer model as the main coordinates. Then, after several layers of Transformer calculations, use the first M outputs of the Transformer model as q(z|h). In step S1, LSTM is used to encode text data, and CNN model is used to encode image files; In step S2, the dimensions in h that represent real meaning include the time axis in the text and the orientation information in the picture.

3. The method for discrete semantic encoding of multimodal data according to claim 2, characterized in that: It also includes the training method of the model. The loss function used in the training method is: Where q(z) is simulated as follows: Where, L VAE is the ELBO loss function, I(Z,X) is the maximum mutual information loss, q qФ (z|x) is the encoder model, p(x) is the marginal distribution of x, and p pФ (x|z) is the decoder model, KL is the KL distance, q(z) is the marginal distribution of the pseudo-z, p(z) is the marginal distribution of z, and N is the number of data in a training set; During training, the original data is reconstructed through the decoder after obtaining the encoded result.

Images