Bimodal representation model training method, device, equipment and medium

Abstract

The present application discloses a method, apparatus, device and medium for training a dual-modal representation model, wherein the method includes: obtaining data to be processed, pre-processing the data to be processed, and obtaining pre-processed data. Feature encoding is performed on the pre-processed data to obtain multiple feature vectors. Multiple image block vectors are pooled to obtain image block pooling vectors; multiple word vectors are pooled to obtain word pooling vectors. A first loss function is constructed based on the image block pooling vectors, a second loss function is constructed based on the word pooling vectors, and a third loss function is constructed based on the image block pooling vectors and the word pooling vectors. The model to be trained is trained according to all loss functions to obtain a dual-modal representation model. Training based on three loss functions makes the dual-modal representation model more capable of feature fusion and feature expression of two different modalities, image and text.

Description

Technical Field

[0001] The present application relates to the field of model training technology, for example, to a dual-modal representation model training method, apparatus, equipment and medium. Background Art

[0002] Contrastive training can be applied in both supervised and unsupervised scenarios and is currently widely used in fields such as computer vision and natural language processing. In multimodal contrastive training, the industry trains multimodal representation models by calculating the noise contrast estimation function between the two modalities. This involves first selecting positive and negative sample pairs, inputting them into the model, and extracting features. Loss values ​​are then calculated using the features of the original sample and the positive sample, and the original sample and the negative sample, respectively. However, this multimodal contrastive training is not very accurate when mapping the two modalities to the same representation space, resulting in weak representational capabilities in the trained bimodal representation model. Summary of the Invention

[0003] The present application provides a bimodal representation model training method, apparatus, device and medium, aiming to solve the problem of weak representation capability of the bimodal representation model.

[0004] To solve the above problems, this application adopts the following technical solutions:

[0005] This paper provides a bimodal representation model training method, including:

[0006] Acquiring data to be processed, wherein the data to be processed includes image data and text data;

[0007] Preprocessing the data to be processed to obtain preprocessed data;

[0008] Performing feature encoding on the preprocessed data to obtain a plurality of feature vectors, wherein the plurality of feature vectors include a plurality of image block vectors and a plurality of word vectors;

[0009] Pooling the multiple image block vectors to obtain image block pooling vectors; pooling the multiple word vectors to obtain word pooling vectors;

[0010] Constructing a first loss function according to the image block pooling vector, constructing a second loss function according to the word pooling vector, and constructing a third loss function according to the multiple image block vectors and the multiple word vectors;

[0011] The model to be trained is trained according to the first loss function, the second loss function and the third loss function to obtain a bimodal representation model.

[0012] Preferably, constructing a first loss function according to the image block pooling vector includes:

[0013] Taking the image block pooling vector as an independent variable of a logarithmic function, and constructing the first loss function through the logarithmic function;

[0014] The formula of the first loss function is as follows:

[0015]

[0016] in, is the image block pooling vector, log represents the logarithmic function, and Loss1 is the first loss function.

[0017] Preferably, constructing a second loss function according to the word pooling vector includes:

[0018] Using the word pooling vector as the independent variable of the logarithmic function, constructing the second loss function through two logarithmic functions;

[0019] The formula of the second loss function is as follows:

[0020]

[0021] in, is the word pooling vector corresponding to the image block pooling vector, log is the logarithmic function, and Loss2 is the second loss function.

[0022] Preferably, constructing a third loss function according to the image block vector and the word pooling vector includes:

[0023] Using the current image block vector and the word vector as independent variables of a noise contrast estimation function, and constructing the third loss function through the noise contrast estimation function;

[0024] The formula of the third loss function is as follows:

[0025]

[0026] Wherein, q is the current image block vector, k + is the word vector corresponding to the current image block vector, k i is the i-th word vector, exp represents the exponential function, τ is the first parameter of the third loss function, Info_NCE is the third loss function, and log is the logarithmic function.

[0027] Preferably, the preprocessing of the data to be processed to obtain preprocessed data includes:

[0028] dividing the image data into a plurality of image blocks;

[0029] Segmenting the text data into a plurality of words;

[0030] The plurality of image blocks and the plurality of words are used as the pre-processing data.

[0031] Preferably, the feature encoding of the preprocessed data to obtain a plurality of feature vectors includes:

[0032] Inputting the multiple image blocks into a BERT model to perform feature encoding on the multiple image blocks to obtain the image block vectors;

[0033] The multiple words are input into the BERT model to perform feature encoding on the multiple words to obtain the word vectors.

[0034] Preferably, the training of the to-be-trained model according to the first loss function, the second loss function, and the third loss function to obtain a bimodal representation model includes:

[0035] Combining the first loss function, the second loss function and the third loss function into a final loss function;

[0036] Calculating a loss function value of the final loss function;

[0037] Perform back propagation according to the loss function value to update the model parameters of the model to be trained;

[0038] Determine whether a parameter error between the model parameter and the preset model parameter is less than a parameter error threshold; if so, stop training to obtain the bimodal representation model.

[0039] This application also provides a dual-modal representation model training device, comprising:

[0040] A module for acquiring data to be processed, used for acquiring data to be processed, wherein the data to be processed includes image data and text data;

[0041] A preprocessing module, configured to preprocess the data to be processed to obtain preprocessed data;

[0042] a feature encoding module, configured to perform feature encoding on the preprocessed data to obtain a plurality of feature vectors, wherein the plurality of feature vectors include a plurality of image block vectors and a plurality of word vectors;

[0043] A pooling module, configured to pool the plurality of image block vectors to obtain image block pooling vectors; and pool the plurality of word vectors to obtain word pooling vectors;

[0044] A loss function construction module, configured to construct a first loss function based on the image block pooling vector, a second loss function based on the word pooling vector, and a third loss function based on the multiple image block vectors and the multiple word vectors;

[0045] A training module is used to train the model to be trained according to the first loss function, the second loss function and the third loss function to obtain a bimodal representation model.

[0046] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned dual-modal representation model training methods.

[0047] The dual-modal representation model training method of the present application includes obtaining data to be processed, which includes image data and text data. Preprocessing the data to be processed to obtain preprocessed data. Feature encoding is performed on the preprocessed data to obtain multiple feature vectors, and the multiple feature vectors include multiple image block vectors and multiple word vectors. Pooling is performed on multiple image block vectors to obtain image block pooling vectors; pooling is performed on multiple word vectors to obtain word pooling vectors. A first loss function is constructed based on the image block pooling vector, a second loss function is constructed based on the word pooling vector, and a third loss function is constructed based on the image block pooling vector and the word pooling vector. The model to be trained is trained according to the first loss function, the second loss function and the third loss function to obtain a dual-modal representation model. The first loss function is used to train the image representation ability of the dual-modal representation model, the second loss function is used to train the text representation ability of the dual-modal representation model, and the third loss function is used to balance the representation ability of the dual-modal representation model for images and texts. Training based on the three loss functions makes the dual-modal representation model more capable of feature fusion and feature expression of two different modalities, image and text. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of a flow chart of a dual-modal representation model training method according to an embodiment;

[0049] Figure 2 A schematic diagram of a process for constructing multiple loss functions according to an embodiment;

[0050] Figure 3 A schematic diagram of a flow chart of preprocessing data to be processed according to an embodiment;

[0051] Figure 4 A schematic diagram of a process for performing feature encoding on pre-processed data according to an embodiment;

[0052] Figure 5 Schematic diagram of a process for training a model to be trained according to an embodiment;

[0053] Figure 6 1 is a schematic block diagram of the structure of a dual-modal representation model training device according to an embodiment;

[0054] Figure 7The figure is a schematic block diagram of the structure of a computer device according to an embodiment.

[0055] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] It will be understood by those skilled in the art that, unless otherwise stated, the singular forms "a", "an", "above", and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements, units, units and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, units, units, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0059] In one embodiment, referring to Figure 1 , is a flowchart of the dual-modal representation model training method applied in this proposal, including the following steps S1-S6:

[0060] S1: Acquire data to be processed, where the data to be processed includes image data and text data.

[0061] The image data includes multiple input images, and the text data includes multiple input texts. One input image corresponds to one input text.

[0062] Image data and text data represent two different modalities. After processing, the image data and text data are used to train a bimodal representation model that can simultaneously represent images and text.

[0063] S2: Preprocessing the data to be processed to obtain preprocessed data.

[0064] dividing the image data into a plurality of image blocks;

[0065] Segmenting the text data into a plurality of words;

[0066] The plurality of image blocks and the plurality of words are used as the pre-processing data.

[0067] An input image contains different image features. Each input image in the image data is divided into N image blocks, where N ranges from 4 to 8, and preferably, N is set to 4. The length and width of the image blocks can be equal or unequal, depending on the actual situation and are not limited here.

[0068] An input text contains different text features. Each input text in the text data is divided into multiple sentences. Each sentence is then segmented to obtain multiple words. The specific number of words is determined by the length of the input text.

[0069] S3: Perform feature encoding on the preprocessed data to obtain multiple feature vectors, where the multiple feature vectors include multiple image block vectors and multiple word vectors.

[0070] Inputting the multiple image blocks into a BERT model to perform feature encoding on the multiple image blocks to obtain the image block vectors;

[0071] The multiple words are input into the BERT model to perform feature encoding on the multiple words to obtain the word vectors.

[0072] The BERT model consists of an encoding module and a decoding module. The BERT model can encode features and obtain corresponding vectors.

[0073] S4: Pooling the multiple image block vectors to obtain image block pooling vectors; pooling the multiple word vectors to obtain word pooling vectors.

[0074] Perform max pooling on multiple image block vectors to obtain image block pooling vectors. Perform max pooling on multiple word vectors to obtain word pooling vectors. The dimensions of the image block pooling vector and the word pooling vector are the same.

[0075] For example, image data is divided into six image blocks and fed into the BERT model to obtain six image block vectors. Max pooling is performed by taking the maximum value of each dimension of the six image block vectors to obtain a vector consisting of the maximum values ​​of each dimension, i.e., the image block pooling vector. Text data is divided into eight words and fed into the BERT model to obtain eight word vectors. Max pooling is performed by taking the maximum value of each dimension of the eight word vectors to obtain a vector consisting of the maximum values ​​of each dimension, i.e., the word pooling vector.

[0076] S5: Construct a first loss function based on the image block pooling vector, construct a second loss function based on the word pooling vector, and construct a third loss function based on the multiple image block vectors and the multiple word vectors.

[0077] Taking the image block pooling vector as an independent variable of a logarithmic function, and constructing the first loss function through the logarithmic function;

[0078] The formula of the first loss function is as follows:

[0079]

[0080] in, is the image block pooling vector, log represents the logarithmic function, and Loss1 is the first loss function.

[0081] Using the word pooling vector as the independent variable of the logarithmic function, constructing the second loss function through two logarithmic functions;

[0082] The formula of the second loss function is as follows:

[0083]

[0084] in, is the word pooling vector, log is the logarithmic function, and Loss2 is the second loss function.

[0085] Using the current image block vector and the word vector as independent variables of a noise contrast estimation function, and constructing the third loss function through the noise contrast estimation function;

[0086] The formula of the third loss function is as follows:

[0087]

[0088] Among them, q is the current image block vector, k + is the word vector corresponding to the current image block vector, k iis the i-th word vector, exp represents the exponential function, τ is the first parameter of the third loss function, Info_NCE is the third loss function, and log is the logarithmic function.

[0089] The first loss function is used to train the image representation ability of the bimodal representation model, the second loss function is used to train the text representation ability of the bimodal representation model, and the third loss function is used to balance the representation ability of the bimodal representation model for images and text.

[0090] S6: Train the model to be trained according to the first loss function, the second loss function and the third loss function to obtain a bimodal representation model.

[0091] Combining the first loss function, the second loss function and the third loss function into a final loss function;

[0092] Calculating a loss function value of the final loss function;

[0093] Perform back propagation according to the loss function value to update the model parameters of the model to be trained;

[0094] Determine whether a parameter error between the model parameter and the preset model parameter is less than a parameter error threshold; if so, stop training to obtain the bimodal representation model.

[0095] Training based on three loss functions makes the bimodal representation model more capable of feature fusion and feature expression of two different modalities, image and text.

[0096] The dual-modal representation model training method of the embodiment of the present application includes obtaining data to be processed, which includes image data and text data. Preprocessing the data to be processed to obtain preprocessed data. Feature encoding is performed on the preprocessed data to obtain multiple feature vectors, and the multiple feature vectors include multiple image block vectors and multiple word vectors. Pooling is performed on the multiple image block vectors to obtain image block pooling vectors; pooling is performed on the multiple word vectors to obtain word pooling vectors. A first loss function is constructed based on the image block pooling vectors, a second loss function is constructed based on the word pooling vectors, and a third loss function is constructed based on the image block pooling vectors and the word pooling vectors. The model to be trained is trained according to the first loss function, the second loss function and the third loss function to obtain a dual-modal representation model. The first loss function is used to train the image representation ability of the dual-modal representation model, the second loss function is used to train the text representation ability of the dual-modal representation model, and the third loss function is used to balance the representation ability of the dual-modal representation model for images and texts. Training based on the three loss functions makes the dual-modal representation model more capable of feature fusion and feature expression of two different modalities, image and text.

[0097] In one embodiment, referring to Figure 2 The step S5 of constructing a first loss function based on the image block pooling vector, constructing a second loss function based on the word pooling vector, and constructing a third loss function based on the multiple image block vectors and the multiple word vectors includes the following steps S51-S53:

[0098] S51: Construct a first loss function according to the image block pooling vector.

[0099] The image block pooling vector is used as an independent variable of a logarithmic function, and the first loss function is constructed through the logarithmic function.

[0100] The formula of the first loss function is as follows:

[0101]

[0102] in, is the image block pooling vector, log represents the logarithmic function, and Loss1 is the first loss function.

[0103] The first loss function changes the cross entropy loss function to obtain the cross entropy loss function:

[0104]

[0105] When input Pooling vector for image patch When , y is set to 0, and the first loss function is obtained. A trained model with strong representation ability has the same accuracy for different input vectors. There are two types of input vectors, namely image block pooling vectors or word pooling vectors, so the target value of the cross entropy loss function is set to

[0106] The first loss function is used to train the image representation ability of the bimodal representation model.

[0107] S52: Construct a second loss function according to the word pooling vector.

[0108] Using the word pooling vector as the independent variable of the logarithmic function, constructing the second loss function through two logarithmic functions;

[0109] The formula of the second loss function is as follows:

[0110]

[0111] in, is the word pooling vector, log is the logarithmic function, and Loss2 is the second loss function.

[0112] When input is the word pooling vector When , y is set to 1 and the second loss function is obtained.

[0113] The second loss function is used to train the text representation ability of the bimodal representation model.

[0114] S53: Construct a third loss function according to the image block vector and the word pooling vector.

[0115] The current image block vector and the word vector are used as independent variables of a noise contrast estimation function, and the third loss function is constructed through the noise contrast estimation function.

[0116] The formula of the third loss function is as follows:

[0117]

[0118] Among them, q is the current image block vector, k + is the word vector corresponding to the current image block vector, k i is the i-th word vector, exp represents the exponential function, τ is the first parameter of the third loss function, Info_NCE is the third loss function, log is the logarithmic function, and “.” represents the dot multiplication operation.

[0119] The current image block vector q and the corresponding word vector k + A positive sample pair is formed, which consists of an image block vector and a word vector, where the image block vector and the word vector have a corresponding relationship.

[0120] If the i-th word vector k i Does not correspond to the current image block vector q, then the i-th word vector k i Together with the current image block vector q, a negative sample pair is formed. The negative sample pair consists of an image block vector and a word vector, where there is no corresponding relationship between the image block vector and the word vector.

[0121] Perform dot product of the current image block vector q with each word vector to obtain the dot product result. Calculate the ratio of the dot product result to the first parameter of the third loss function. Perform exponential operation on the ratio to obtain the distance between the current image block vector and the i-th word vector. The smaller the distance, the greater the similarity.

[0122] The numerator of the third loss function represents the similarity between the current image block and the corresponding word vector, and the denominator of the third loss function represents the sum of the similarities between the current image block and all word vectors. By calculating the ratio of the similarity between the current image block and the corresponding word vector to all similarities, the degree of matching between the current image block and the corresponding word vector can be measured.

[0123] The range of the first parameter of the third loss function is 0.1-0.3. Preferably, the first parameter of the third loss function is set to 0.2.

[0124] The third loss function is used to balance the representation capabilities of the bimodal representation model for images and text.

[0125] The embodiment of the present application constructs a first loss function based on the image block pooling vector, a second loss function based on the word pooling vector, and a third loss function based on multiple image block vectors and multiple word vectors, including constructing a first loss function based on the image block pooling vector, constructing a second loss function based on the word pooling vector, and constructing a third loss function based on the image block vector and the word pooling vector. The first loss function is used to train the image representation ability of the bimodal representation model, the second loss function is used to train the text representation ability of the bimodal representation model, and the third loss function is used to balance the representation ability of the bimodal representation model for images and text.

[0126] In one embodiment, referring to Figure 3 The above step S2 of preprocessing the data to be processed to obtain preprocessed data includes the following steps S21-S23:

[0127] S21: Divide the image data into multiple image blocks.

[0128] An input image contains different image features. Each input image in the image data is segmented into N image blocks. Image segmentation methods such as threshold segmentation, edge detection segmentation, region segmentation, or cluster segmentation can be used. Other image segmentation methods may also be used, depending on the specific situation and are not limited here. The value of N ranges from 4 to 8, and preferably, N is set to 4. The length and width of the image blocks may be equal or unequal, depending on the specific situation and are not limited here.

[0129] Each image block contains different image features, and different image blocks do not overlap.

[0130] S22: Segment the text data into multiple words.

[0131] An input text contains various text features. Using the N-gram model, each input text is segmented into multiple sentences. Each sentence is then segmented to produce multiple words. The specific number of words is determined by the length of the input text.

[0132] After obtaining multiple sentences, the word2vec model can be used to segment each sentence according to the word's part of speech, which includes nouns, verbs, adjectives, and adverbs. Segmentation can also be performed based on the relationship between different words. For example, triple relationships in the sentences after the input text is segmented can be found and segmented based on the triple relationships. The triple relationship refers to the relationship between the subject, predicate, and object in a sentence.

[0133] S23: Using the multiple image blocks and the multiple words as the pre-processing data.

[0134] The preprocessed data consists of two dimensions: image and text. The image dimension consists of multiple image blocks, and the text dimension consists of multiple words. Transforming the preprocessed data yields a feature vector, which is used to construct the loss function for the model to be trained. This allows the trained bimodal representation model to simultaneously represent both images and text.

[0135] In an embodiment of the present application, the data to be processed is preprocessed to obtain preprocessed data, including segmenting the image data into multiple image blocks and segmenting the text data into multiple words. The multiple image blocks and multiple words are used as preprocessed data. Each image block contains different image features, and different image blocks do not overlap. The specific number of words is determined by the length of the input text. The preprocessed data includes data in two dimensions: image and text. The data in the image dimension includes multiple image blocks, and the data in the text dimension includes multiple words.

[0136] In one embodiment, referring to Figure 4 The step S3 of performing feature encoding on the pre-processed data to obtain multiple feature vectors includes the following steps S31-S32:

[0137] S31: Input the multiple image blocks into the BERT model to perform feature encoding on the multiple image blocks to obtain the image block vectors.

[0138] The BERT model consists of an encoding module and a decoding module. The BERT model can encode features and obtain corresponding vectors.

[0139] Feature encoding includes image encoding and word encoding.

[0140] After image encoding, each image block obtains an image block vector, and different image block vectors represent different features of the corresponding input image.

[0141] S32: Input the multiple words into the BERT model to perform feature encoding on the multiple words to obtain the word vectors.

[0142] Each word is encoded to obtain a word vector, and different word vectors represent different features of the corresponding input text.

[0143] In an embodiment of the present application, feature encoding is performed on preprocessed data to obtain multiple feature vectors. This includes inputting multiple image blocks into a BERT model to perform feature encoding on the multiple image blocks to obtain image block vectors. Multiple words are input into a BERT model to perform feature encoding on the multiple words to obtain word vectors. Different image block vectors represent different features of the corresponding input image, and different word vectors represent different features of the corresponding input text.

[0144] In one embodiment, referring to Figure 5 The step S6 of training the model to be trained according to the first loss function, the second loss function, and the third loss function to obtain the bimodal representation model includes the following steps S61-S64:

[0145] S61: Combining the first loss function, the second loss function and the third loss function into a final loss function.

[0146] The sum of the first loss function, the second loss function and the third loss function is used as the final loss function, and training is performed based on the final loss function, so that the bimodal representation model has stronger feature fusion and feature expression capabilities of two different modalities, image and text.

[0147] S62: Calculate the loss function value of the final loss function.

[0148] In the process of training the model to be trained using the final loss function, the loss function values ​​of the first loss function, the second loss function and the third loss function are reduced simultaneously.

[0149] The final loss function value can reflect the error between the actual output result and the expected result. The smaller the loss function value, the smaller the error and the better the performance of the model in training.

[0150] S63: Perform back propagation according to the loss function value to update the model parameters of the model to be trained.

[0151] During the training process, the greater the difference in the loss function values ​​between two adjacent iterations, the faster the model parameters are updated.

[0152] S64: Determine whether a parameter error between the model parameter and the preset model parameter is less than a parameter error threshold; if so, stop training to obtain the bimodal representation model.

[0153] When the parameter error between the model parameters and the preset model parameters is less than the parameter error threshold, it indicates that the training model has met expectations, and training is stopped, and the trained model is used as the bimodal representation model. When the parameter error is greater than or equal to the parameter error threshold, it indicates that the training model has not met expectations, and training of the training model continues until the parameter error is less than the parameter error threshold.

[0154] The embodiment of the present application trains the model to be trained according to the first loss function, the second loss function and the third loss function to obtain a bimodal representation model, including forming the first loss function, the second loss function and the third loss function into a final loss function, and calculating the loss function value of the final loss function. Back propagation is performed according to the loss function value to update the model parameters of the model to be trained. It is judged whether the parameter error between the model parameters and the preset model parameters is less than the parameter error threshold. If so, the training is stopped to obtain a bimodal representation model. The sum of the first loss function, the second loss function and the third loss function is used as the final loss function, and training is performed based on the final loss function, so that the bimodal representation model has stronger feature fusion and feature expression capabilities for two different modalities of image and text. When the parameter error between the model parameters and the preset model parameters is less than the parameter error threshold, it means that the model in training has met expectations, and the training is stopped, and the trained model is used as the bimodal representation model.

[0155] Reference Figure 6 , is a schematic block diagram of the structure of a dual-modal representation model training device applied for in this scheme, the device includes:

[0156] The data to be processed acquisition module 10 is used to acquire the data to be processed, wherein the data to be processed includes image data and text data;

[0157] The image data includes multiple input images, and the text data includes multiple input texts. One input image corresponds to one input text.

[0158] Image data and text data represent two different modalities. After processing, the image data and text data are used to train a bimodal representation model that can simultaneously represent images and text.

[0159] A preprocessing module 20 is used to preprocess the data to be processed to obtain preprocessed data;

[0160] dividing the image data into a plurality of image blocks;

[0161] Segmenting the text data into a plurality of words;

[0162] The plurality of image blocks and the plurality of words are used as the pre-processing data.

[0163] An input image contains different image features. Each input image in the image data is divided into N image blocks, where N ranges from 4 to 8, and preferably, N is set to 4. The length and width of the image blocks can be equal or unequal, depending on the actual situation and are not limited here.

[0164] An input text contains different text features. Each input text in the text data is divided into multiple sentences. Each sentence is then segmented to obtain multiple words. The specific number of words is determined by the length of the input text.

[0165] A feature encoding module 30 is configured to perform feature encoding on the preprocessed data to obtain a plurality of feature vectors, wherein the plurality of feature vectors include a plurality of image block vectors and a plurality of word vectors;

[0166] Inputting the multiple image blocks into a BERT model to perform feature encoding on the multiple image blocks to obtain the image block vectors;

[0167] The multiple words are input into the BERT model to perform feature encoding on the multiple words to obtain the word vectors.

[0168] The BERT model consists of an encoding module and a decoding module. The BERT model can encode features and obtain corresponding vectors.

[0169] A pooling module 40 is configured to pool the plurality of image block vectors to obtain image block pooling vectors; and to pool the plurality of word vectors to obtain word pooling vectors;

[0170] Perform max pooling on multiple image block vectors to obtain image block pooling vectors. Perform max pooling on multiple word vectors to obtain word pooling vectors. The dimensions of the image block pooling vector and the word pooling vector are the same.

[0171] For example, image data is divided into six image blocks and fed into the BERT model to obtain six image block vectors. Max pooling is performed by taking the maximum value of each dimension of the six image block vectors to obtain a vector consisting of the maximum values ​​of each dimension, i.e., the image block pooling vector. Text data is divided into eight words and fed into the BERT model to obtain eight word vectors. Max pooling is performed by taking the maximum value of each dimension of the eight word vectors to obtain a vector consisting of the maximum values ​​of each dimension, i.e., the word pooling vector.

[0172] a loss function construction module 50, configured to construct a first loss function based on the image block pooling vector, a second loss function based on the word pooling vector, and a third loss function based on the multiple image block vectors and the multiple word vectors;

[0173] Taking the image block pooling vector as an independent variable of a logarithmic function, and constructing the first loss function through the logarithmic function;

[0174] The formula of the first loss function is as follows:

[0175]

[0176] in, is the image block pooling vector, log represents the logarithmic function, and Loss1 is the first loss function.

[0177] Using the word pooling vector as the independent variable of the logarithmic function, constructing the second loss function through two logarithmic functions;

[0178] The formula of the second loss function is as follows:

[0179]

[0180] in, is the word pooling vector, log is the logarithmic function, and Loss2 is the second loss function.

[0181] Using the current image block vector and the word vector as independent variables of a noise contrast estimation function, and constructing the third loss function through the noise contrast estimation function;

[0182] The formula of the third loss function is as follows:

[0183]

[0184] Among them, q is the current image block vector, k + is the word vector corresponding to the current image block vector, k i is the i-th word vector, exp represents the exponential function, τ is the first parameter of the third loss function, Info_NCE is the third loss function, and log is the logarithmic function.

[0185] The first loss function is used to train the image representation ability of the bimodal representation model, the second loss function is used to train the text representation ability of the bimodal representation model, and the third loss function is used to balance the representation ability of the bimodal representation model for images and text.

[0186] The training module 60 is used to train the model to be trained according to the first loss function, the second loss function and the third loss function to obtain a bimodal representation model.

[0187] Combining the first loss function, the second loss function and the third loss function into a final loss function;

[0188] Calculating a loss function value of the final loss function;

[0189] Perform back propagation according to the loss function value to update the model parameters of the model to be trained;

[0190] Determine whether a parameter error between the model parameter and the preset model parameter is less than a parameter error threshold; if so, stop training to obtain the bimodal representation model.

[0191] Training based on three loss functions makes the bimodal representation model more capable of feature fusion and feature expression of two different modalities, image and text.

[0192] The dual-modal representation model training device of the embodiment of the present application can implement the dual-modal representation model training method.

[0193] In one embodiment, the loss function construction module 50 includes:

[0194] A first loss function construction unit, configured to use the image block pooling vector as an independent variable of a logarithmic function and construct the first loss function through the logarithmic function;

[0195] The formula of the first loss function is as follows:

[0196]

[0197] in, is the image block pooling vector, log represents the logarithmic function, and Loss1 is the first loss function.

[0198] A second loss function construction unit, configured to use the word pooling vector as an independent variable of a logarithmic function and construct the second loss function through two logarithmic functions;

[0199] The formula of the second loss function is as follows:

[0200]

[0201] in, is the word pooling vector corresponding to the image block pooling vector, log is the logarithmic function, and Loss2 is the second loss function.

[0202] a third loss function construction unit, configured to use the current image block vector and the word vector as independent variables of a noise contrast estimation function, and construct the third loss function through the noise contrast estimation function;

[0203] The formula of the third loss function is as follows:

[0204]

[0205] Wherein, q is the current image block vector, k + is the word vector corresponding to the current image block vector, k i is the i-th word vector, exp represents the exponential function, τ is the first parameter of the third loss function, Info_NCE is the third loss function, and log is the logarithmic function.

[0206] In one embodiment, the pre-processing module 20 includes:

[0207] an image data segmentation unit, configured to segment the image data into a plurality of image blocks;

[0208] A text data segmentation unit, configured to segment the text data into a plurality of words;

[0209] A pre-processing data acquisition unit is configured to use the plurality of image blocks and the plurality of words as the pre-processing data.

[0210] In one embodiment, the feature encoding module 30 includes:

[0211] an image block feature encoding unit, configured to input the plurality of image blocks into a BERT model to perform feature encoding on the plurality of image blocks to obtain the image block vectors;

[0212] A word feature encoding unit is used to input the multiple words into the BERT model to perform feature encoding on the multiple words to obtain the word vectors.

[0213] In one embodiment, the training module 60 includes:

[0214] A final loss function composition unit, configured to combine the first loss function, the second loss function, and the third loss function into a final loss function;

[0215] A loss function value calculation unit, used to calculate the loss function value of the final loss function;

[0216] A model parameter updating unit, configured to perform back propagation according to the loss function value to update the model parameters of the to-be-trained model;

[0217] The bimodal representation model acquisition unit is used to determine whether the parameter error between the model parameters and the preset model parameters is less than the parameter error threshold. If so, the training is stopped to obtain the bimodal representation model.

[0218] As described above, it can be understood that the various components of the dual-modal representation model training device proposed in this application can realize the functions of any of the dual-modal representation model training methods described above, and the specific structure will not be repeated.

[0219] Reference Figure 7 In the embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image block pooling vectors and word pooling vectors, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen, etc. When the computer program is executed by the processor, a dual-modal representation model training method is implemented.

[0220] Those skilled in the art will understand that Figure 7 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0221] One embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for training a bimodal representation model. It is understood that the computer-readable storage medium in this embodiment can be either a volatile readable storage medium or a non-volatile readable storage medium.

[0222] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0223] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0224] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A dual-modal representation model training method, characterized in that: include: Acquiring data to be processed, wherein the data to be processed includes image data and text data; Preprocessing the data to be processed to obtain preprocessed data; Performing feature encoding on the preprocessed data to obtain a plurality of feature vectors, wherein the plurality of feature vectors include a plurality of image block vectors and a plurality of word vectors; Pooling the multiple image block vectors to obtain image block pooling vectors; pooling the multiple word vectors to obtain word pooling vectors; Constructing a first loss function according to the image block pooling vector, constructing a second loss function according to the word pooling vector, and constructing a third loss function according to the multiple image block vectors and the multiple word vectors; Training the to-be-trained model according to the first loss function, the second loss function, and the third loss function to obtain a bimodal representation model; The constructing a first loss function according to the image block pooling vector includes: Taking the image block pooling vector as an independent variable of a logarithmic function, and constructing the first loss function through the logarithmic function; The formula of the first loss function is as follows: in, is the image block pooling vector, log represents the logarithmic function, and Loss1 is the first loss function; The constructing a third loss function according to the multiple image block vectors and the multiple word vectors includes: Using the current image block vector and the word vector as independent variables of a noise contrast estimation function, and constructing the third loss function through the noise contrast estimation function; The formula of the third loss function is as follows: Wherein, q is the current image block vector, k + is the word vector corresponding to the current image block vector, k i is the i-th word vector, exp represents the exponential function, τ is the first parameter of the third loss function, Info_NCE is the third loss function, and log is the logarithmic function.

2. The dual-modal representation model training method according to claim 1, characterized in that: The constructing a second loss function according to the word pooling vector includes: Using the word pooling vector as the independent variable of the logarithmic function, constructing the second loss function through two logarithmic functions; The formula of the second loss function is as follows: in, is the word pooling vector corresponding to the image block pooling vector, log is the logarithmic function, and Loss2 is the second loss function.

3. The dual-modal representation model training method according to claim 1, characterized in that: The preprocessing of the data to be processed to obtain preprocessed data includes: dividing the image data into a plurality of image blocks; Segmenting the text data into a plurality of words; The plurality of image blocks and the plurality of words are used as the pre-processing data.

4. The dual-modal representation model training method according to claim 3, characterized in that: The feature encoding of the pre-processed data to obtain a plurality of feature vectors includes: Inputting the multiple image blocks into a BERT model to perform feature encoding on the multiple image blocks to obtain the image block vectors; The multiple words are input into the BERT model to perform feature encoding on the multiple words to obtain the word vectors.

5. The dual-modal representation model training method according to claim 1, characterized in that: The step of training the model to be trained according to the first loss function, the second loss function, and the third loss function to obtain a bimodal representation model includes: Combining the first loss function, the second loss function and the third loss function into a final loss function; Calculating a loss function value of the final loss function; Perform back propagation according to the loss function value to update the model parameters of the model to be trained; Determine whether a parameter error between the model parameter and the preset model parameter is less than a parameter error threshold; if so, stop training to obtain the bimodal representation model.

6. A dual-modal representation model training device, used to execute the dual-modal representation model training method according to any one of claims 1 to 5, characterized in that: include: A module for acquiring data to be processed, used for acquiring data to be processed, wherein the data to be processed includes image data and text data; A preprocessing module, configured to preprocess the data to be processed to obtain preprocessed data; a feature encoding module, configured to perform feature encoding on the preprocessed data to obtain a plurality of feature vectors, wherein the plurality of feature vectors include a plurality of image block vectors and a plurality of word vectors; A pooling module, configured to pool the plurality of image block vectors to obtain image block pooling vectors; and pool the plurality of word vectors to obtain word pooling vectors; A loss function construction module, configured to construct a first loss function based on the image block pooling vector, a second loss function based on the word pooling vector, and a third loss function based on the multiple image block vectors and the multiple word vectors; A training module is used to train the model to be trained according to the first loss function, the second loss function and the third loss function to obtain a bimodal representation model.

7. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the dual-modal representation model training method described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dual-modal representation model training method according to any one of claims 1 to 5 are implemented.

Images