Vector extraction model training method and device, electronic equipment and storage medium

Abstract

The embodiment of the present application provides a kind of training method and device of image-text vector extraction model, wherein the method comprises: extracting image representation vector and text representation vector from training sample data;According to single-modal alignment loss function and multiple image representation vectors of the same image-text content, single-modal alignment network model is trained;According to cross-modal alignment loss function, text representation vector and the image vector result that the single-modal alignment network model trained is output, cross-modal alignment network model is trained;The single-modal alignment network model and cross-modal alignment network model trained are used as the image-text vector extraction model trained finally.The embodiment of the present application is based on the extraction of image representation vector and text representation vector, and trains single-modal alignment network model and cross-modal alignment network model.Finally, the image-text vector extraction model obtained supports the vector extraction of multiple modal content of one text multiple images, and is more suitable for current content distribution system.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a training method, apparatus, electronic device, and computer-readable storage medium for a graph and text vector extraction model. Background Technology

[0002] In the era of deep learning, a crucial step in content-based distribution systems (including but not limited to search and recommendation scenarios) is extracting the representation vector of the content. Since content is often not limited to a single modality, it usually includes images, text, and videos. Therefore, it is essential to extract a unified representation vector that can effectively represent the content from different modalities.

[0003] Currently, pre-trained network models are typically used to extract content vectors. These current network models are mostly suitable for... Figure One Image-centric content, such as text or multiple images per article, lacks effective modeling for text with multiple images. However, text with multiple images is the main form of current content-based distribution systems, which typically use multiple images to describe different perspectives of the main text. Therefore, multimodal pre-trained models based on text with multiple images are more in line with the needs of actual production processes. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a training method, apparatus, electronic device, and computer-readable storage medium for a text-image vector extraction model that overcomes or at least partially solves the above problems.

[0005] To address the aforementioned problems, according to a first aspect of the present invention, a training method for a text-image vector extraction model is disclosed. The method includes: acquiring training sample data, the training sample data comprising: text sample data belonging to multiple text-image contents and image set sample data, the image set sample data comprising multiple image sample data; extracting an image representation vector from each of the image sample data and extracting a text representation vector from the text sample data; training a preset unimodal alignment network model based on a preset unimodal alignment loss function and multiple image representation vectors corresponding to the same text-image content; training a preset cross-modal alignment network model based on a preset cross-modal alignment loss function, the text representation vectors, and the image vector results output by the trained unimodal alignment network model; and using the trained unimodal alignment network model and the trained cross-modal alignment network model as the trained text-image vector extraction model.

[0006] Optionally, the training of the preset cross-modal alignment network model comprises: selecting positive sample data and negative sample data from the training sample data; the positive sample data comprises text sample data and image set sample data belonging to first image-text content, and the negative sample data is text sample data or image set sample data belonging to second image-text content, the first image-text content being different from the second image-text content; when the negative sample data is image set sample data, performing alignment processing on the image vector result and the text representation vector according to a preset first cross-modal alignment loss function; when the negative sample data is text sample data, performing alignment processing on the image vector result and the text representation vector according to a preset second cross-modal alignment loss function.

[0007] Optionally, the first cross-modal alignment loss function is a maximum value between a sum of a first parameter and a preset hyperparameter and a preset numerical value, the first parameter being wherein, the image vector result of the positive sample data, the text representation vector of the positive sample data, the image vector result of the negative sample data.

[0008] Optionally, the second cross-modal alignment loss function is a maximum value between a sum of a second parameter and a preset hyperparameter and a preset numerical value, the second parameter being wherein, the image vector result of the positive sample data, the text representation vector of the positive sample data, the text representation vector of the negative sample data.

[0009] Optionally, the training of the preset single-modal alignment network model comprises: performing alignment processing on multiple image representation vectors corresponding to the same image-text content by using an L2 regular loss function.

[0010] Optionally, the extraction of the image representation vector of each image sample data comprises: encoding each image sample data into an image token; selecting a corresponding embedding vector from a preset image token dictionary according to a discrete value of each image token; and taking the embedding vector as the image representation vector.

[0011] Optionally, the extraction of the text representation vector of the text sample data comprises: performing first encoding processing on the text sample data to obtain a text token sequence; and performing second encoding processing on the text token sequence to obtain the text representation vector.

[0012] According to a second aspect of the embodiments of the present application, a device for training a text-image vector extraction model is also disclosed. The device comprises: a sample acquisition module configured to acquire training sample data, wherein the training sample data comprises text sample data and image set sample data belonging to multiple text-image contents, and the image set sample data comprises multiple image sample data; a vector extraction module configured to extract an image representation vector of each of the image sample data and extract a text representation vector of the text sample data; a single-modal training module configured to train a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to a same text-image content; a cross-modal training module configured to train a preset cross-modal alignment network model according to a preset cross-modal alignment loss function, the text representation vector and an image vector result output by the trained single-modal alignment network model; and a model determination module configured to determine the trained single-modal alignment network model and the trained cross-modal alignment network model as a trained text-image vector extraction model.

[0013] Optionally, the cross-modal training module comprises: a sample selection module configured to select positive sample data and negative sample data from the training sample data, wherein the positive sample data comprises text sample data and image set sample data belonging to a first text-image content, the negative sample data is text sample data or image set sample data belonging to a second text-image content, and the first text-image content is different from the second text-image content; and a cross-modal alignment module configured to perform alignment processing on the image vector result and the text representation vector according to a preset first cross-modal alignment loss function when the negative sample data is image set sample data, and perform alignment processing on the image vector result and the text representation vector according to a preset second cross-modal alignment loss function when the negative sample data is text sample data.

[0014] Optionally, the first cross-modal alignment loss function is a maximum value between a sum of a first parameter and a preset hyperparameter and a preset numerical value, wherein the first parameter is wherein, is an image vector result of the positive sample data, is a text representation vector of the positive sample data, is an image vector result of the negative sample data.

[0015] Optionally, the second cross-modal alignment loss function is a maximum value between a sum of a second parameter and a preset hyperparameter and a preset numerical value, wherein the second parameter is wherein, is an image vector result of the positive sample data, is a text representation vector of the positive sample data, a text representation vector of the negative sample data.

[0016] Optionally, the single-modal training module is configured to align the multiple image representation vectors corresponding to the same image-text content by using an L2 regular loss function.

[0017] Optionally, the vector extraction module comprises: an image encoding module configured to encode each of the image sample data into an image label; a vector selection module configured to select a corresponding embedding vector from a preset image label dictionary according to a discrete value of each of the image labels; and a vector determination module configured to determine the embedding vector as the image representation vector.

[0018] Optionally, the vector extraction module comprises: a first text encoding module configured to perform first encoding processing on the text sample data to obtain a text label sequence; and a second text encoding module configured to perform second encoding processing on the text label sequence to obtain the text representation vector.

[0019] According to a third aspect of the embodiments of the present application, an electronic device is disclosed, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the training method of the image-text vector extraction model according to the first aspect when executing the computer program.

[0020] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is disclosed, which stores a computer program executable on a processor, and the computer program implements the training method of the image-text vector extraction model according to the first aspect when executed on the processor.

[0021] Compared with the prior art, the technical scheme provided by the embodiments of the present application has the following advantages:

[0022] The training scheme of the image-text vector extraction model provided by the embodiments of the present application comprises the following steps: obtaining training sample data, wherein the training sample data comprises text sample data belonging to multiple image-text contents and image set sample data, and the image set sample data comprises multiple image sample data; extracting an image representation vector of each image sample data and a text representation vector of the text sample data; training a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to the same image-text content; training a preset cross-modal alignment network model according to a preset cross-modal alignment loss function, the text representation vector, and an image vector result output by the trained single-modal alignment network model; and finally taking the trained single-modal alignment network model and the trained cross-modal alignment network model as a trained image-text vector extraction model.

[0023] The embodiment of the present application aims at a content in the form of one text and multiple images, and proposes a training method of a vector extraction model. The training method can extract an image representation vector of each image sample data and extract a text representation vector of text sample data. On the basis of the extracted image representation vector and text representation vector, a single-modal alignment network model and a cross-modal alignment network model are trained. The single-modal alignment network model is used for alignment processing of the image representation vectors of multiple image sample data belonging to the same image-text content. The cross-modal alignment network model is used for alignment processing of the text representation vectors of text sample data belonging to the same image-text content and the image vector result. Finally, the trained image-text vector extraction model can support vector extraction of the multi-modal content of one text and multiple images, and is more suitable for the current content distribution system. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a step flow chart of a training method of an image-text vector extraction model of the embodiment of the present application;

[0025] Figure 2 is a structural schematic diagram of a multi-modal vector extraction model of one text and multiple images based on modal alignment of the embodiment of the present application;

[0026] Figure 3 is a structural block diagram of a training device of an image-text vector extraction model of the embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of an electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0029] Referring to Figure 1 , a step flow chart of a training method of an image-text vector extraction model of the embodiment of the present application is shown. The training method of the image-text vector extraction model can be applied to a terminal or a server. The training method of the image-text vector extraction model can specifically include the following steps:

[0030] Step 101, obtaining training sample data.

[0031] In the embodiment of the present application, the training sample data can include text sample data and image set sample data, wherein the image set sample data includes a plurality of image sample data. In actual application, the training sample data can include text sample data and image set sample data belonging to a plurality of graphic-text contents. For example, the training sample data Y includes text sample data t1 and image set sample data I1 belonging to graphic-text content D1, and the image set sample data I1 includes image sample data i101, i102, i103 and i104. The training sample data Y also includes text sample data t2 and image set sample data I2 belonging to graphic-text content D2, and the image set sample data I2 includes image sample data i201, i202, i203 and i204.

[0032] In step 102, the image representation vector of each image sample data is extracted, and the text representation vector of the text sample data is extracted.

[0033] In the embodiment of the present application, the image representation vector of the image sample data can be extracted using a discrete variational auto encoder (dVAE). The text representation vector of the text sample data can be extracted using a deep self-attention network in the form of a transformer.

[0034] In step 103, the preset single-modal alignment network model is trained according to a preset single-modal alignment loss function and a plurality of image representation vectors corresponding to the same graphic-text content.

[0035] In the embodiment of the present application, the single-modal alignment network model is used to align the image representation vectors of a plurality of image sample data belonging to the same graphic-text content.

[0036] In step 104, the preset cross-modal alignment network model is trained according to a preset cross-modal alignment loss function, the text representation vector and the image vector result output by the trained single-modal alignment network model.

[0037] In the embodiment of the present application, the cross-modal alignment network model is used to align the text representation vector of the text sample data belonging to the same graphic-text content with the image vector result.

[0038] In step 105, the trained single-modal alignment network model and the trained cross-modal alignment network model are used as the trained graphic-text vector extraction model.

[0039] In the embodiment of the present application, the finally trained image-text vector extraction model can not only extract the image representation vector of the image sample data and the text representation vector of the text sample data, but also include the trained single-modal alignment network model and the trained cross-modal alignment network model. That is, the image-text vector extraction model can use the single-modal alignment network model to align the multiple image representation vectors, and can use the cross-modal alignment network model to align the text representation vector and the image vector result.

[0040] The training scheme of the image-text vector extraction model provided by the embodiment of the present application comprises the following steps: obtaining training sample data, wherein the training sample data comprises text sample data and image set sample data belonging to multiple image-text contents, and the image set sample data comprises multiple image sample data; extracting an image representation vector of each image sample data and a text representation vector of the text sample data; training a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to the same image-text content; training a preset cross-modal alignment network model according to a preset cross-modal alignment loss function and an image vector result output by the trained single-modal alignment network model; and finally taking the trained single-modal alignment network model and the trained cross-modal alignment network model as a trained image-text vector extraction model.

[0041] The training method of the vector extraction model provided by the embodiment of the present application is for the content in the form of one text and multiple images. The training method can extract an image representation vector of each image sample data and a text representation vector of the text sample data. On the basis of the extracted image representation vector and the text representation vector, a single-modal alignment network model and a cross-modal alignment network model are trained. The single-modal alignment network model is used to align the image representation vectors of the multiple image sample data belonging to the same image-text content. The cross-modal alignment network model is used to align the text representation vector of the text sample data belonging to the same image-text content and the image vector result. The finally trained image-text vector extraction model can support the vector extraction of the multi-modal content in the form of one text and multiple images, and is more suitable for the current content distribution system.

[0042] In a preferred embodiment of the present application, one implementation of training the preset cross-modal alignment network model is to select positive sample data and negative sample data from the training sample data. The positive sample data contains text sample data and image set sample data belonging to the first graphic-text content, and the negative sample data is text sample data or image set sample data belonging to the second graphic-text content, and the first graphic-text content is different from the second graphic-text content. When the negative sample data is image set sample data, the image vector result is aligned with the text representation vector according to the preset first cross-modal alignment loss function. When the negative sample data is text sample data, the image vector result is aligned with the text representation vector according to the preset second cross-modal alignment loss function.

[0043] In actual application, the cross-modal alignment network model can align vectors of different modalities, for example, align vectors of images and text.

[0044] The first cross-modal alignment loss function is the maximum value of the sum of the first parameter and the preset hyperparameter and the preset numerical value, the first parameter is

[0045] wherein, is the image vector result of the positive sample data, is the text representation vector of the positive sample data, is the image vector result of the negative sample data.

[0046] That is, the first cross-modal alignment loss function is:

[0047]

[0048] wherein, L cross_alignment (x + ,y + ,y - ) is the first cross-modal alignment loss function, x + is the text sample data in the positive sample data, y + is the image sample data in the positive sample data, y - is the image sample data in the negative sample data, 0 is the preset numerical value, and Ω is the preset hyperparameter.

[0049] The second cross-modal alignment loss function is the maximum value of the sum of the second parameter and the preset hyperparameter and the preset numerical value, the second parameter is

[0050] wherein, is the image vector result of the positive sample data, is the text representation vector of the positive sample data, is a text representation vector of the negative sample data.

[0051] The second cross-modal alignment loss function is:

[0052]

[0053] L cross_alighment (x + ,y + ,x - ) is the second cross-modal alignment loss function, x + is the text sample data in the positive sample data, y + is the image sample data in the positive sample data, x - is the text sample data in the negative sample data, 0 is a preset value, and Ω is a preset hyperparameter.

[0054] In a preferred embodiment of the present application, one implementation of training the preset single-modal alignment network model is to align multiple image representation vectors corresponding to the same image-text content by using an L2 regular loss function.

[0055] In actual application, the single-modal alignment network model can align image representation vectors of multiple image sample data belonging to the same image-text content. For example, N image sample data belonging to the same image-text content are {y1, y2, …, y N}, and image representation vectors of the N image sample data are The L2 regular loss function is represented as:

[0056]

[0057] L single_aligment is the L2 regular loss function, N is the number of image sample data or image representation vectors, i and j are both the number of image representation vectors, is the image representation vector numbered i, is the image representation vector numbered j.

[0058] In a preferred embodiment of the present invention, one method for extracting the image representation vector of each image sample data is to encode each image sample data into an image token. Based on the discrete values ​​of each image token, a corresponding embedding vector is selected from a preset image token dictionary. The embedding vector is then used as the image representation vector. In practical applications, each image sample data can be encoded into an image token of a uniform size (e.g., 32*32). Each image token can take 8192 possible discrete values, and each discrete value can correspond to a row of embedding vectors in the image token dictionary. This image token dictionary can contain 8192 rows and 768 columns of embedding vectors; that is, the size of the image token dictionary is 8192*768, and the image token dictionary represents the embedding vector corresponding to each image token.

[0059] In a preferred embodiment of the present invention, one method for extracting the text representation vector from text sample data is to perform a first encoding process on the text sample data to obtain a text token sequence. Then, a second encoding process is performed on the text token sequence to obtain the text representation vector. In practical applications, Byte Pair Encoding (BPE) can be used to perform the first encoding on the text sample data to obtain a text token sequence x′=[t1,t2,…,t…] of length L. L ], where x′ represents the text token sequence, t1, t2, ..., t L These represent the text tokens numbered 1, 2, ..., L in the text token sequence. The text token sequence x′ is then input into a Bidirectional Encoder Representation from Transformers (BERT) encoder for a second encoding process to obtain the text representation vector.

[0060] Based on the above description of an embodiment of a training method for a text-image vector extraction model, the following introduces a multimodal vector extraction model based on modal alignment, which involves extracting multiple images from a single text. (Refer to...) Figure 2 The diagram shows a schematic representation of a multimodal vector extraction model based on modal alignment for a single document with multiple images, according to an embodiment of the present invention.

[0061] The input to the multimodal vector extraction model is the text and image content (c = one article with multiple images). <x,{y1,y2,…,y N}>, where c represents the image and text content, x represents the text data in the image and text content, and {y1,y2,…,y Nrepresents image set data in the image-text content, and N is the number of image data in the image set data.

[0062] The multi-modal vector extraction model can include an image encoding module, a text encoding module, a single-modal contrast learning module (i.e., a single-modal alignment network model), a cross-modal contrast learning module (i.e., a cross-modal alignment network model), and a mask language model.

[0063] The image encoding module is configured to convert image data yi in the image-text content into an image representation vector i . The text encoding module is configured to convert text data x into a text representation vector h T . The single-modal contrast learning module is configured to align image representation vectors of multiple image data in the same image-text content. The cross-modal contrast learning module is configured to align the image representation vector and the text representation vector. It should be noted that the cross-modal contrast learning module can align the image representation vector and the text representation vector of the same image-text content, or can align the image representation vector and the text representation vector of different image-text contents. The mask language model is configured to improve the accuracy of the text representation vector.

[0064] In actual applications, the main purpose of the image encoding module is to effectively extract the vector representation of the image. Taking the structure of dVAE as an example (other network structures of auto-encoder can also be used), dVAE includes an encoder and a decoder, which are represented by Encoder image and Decoder Image , respectively: the encoder is composed of a ResNet network (the embodiments of the method are not limited to using ResNet), which can encode the input into 32*32 size picture tokens, each token can take 8192 possible discrete values, each discrete value can correspond to an embedding vector in the image label dictionary, and the size of the image label dictionary is 8192*768, representing the embedding vector corresponding to each image token. The decoder is a similar structure, the input is a 32*32 picture token, and the output is a reconstructed image.

[0065] The main purpose of the text encoding module is to effectively extract the vector representation of the text. The text encoding module uses two transformer structures as an encoder and a decoder to extract the text vector representation, which are represented by Encoder text and Decoder text , respectively. The original text data, after BPE encoding, obtains a text token sequence x′=[t1,t2,…,t L ] with a length of L, which is input into Encoder text (the structure of BERT is used as the encoder in this step) to obtain hT = Encoder text (x′). The decoder uses an autoregressive type of transformer (such as a GPT model) to regenerate a new text sequence x by inputting h T into the Decoder text .

[0066] The above image reconstruction process uses the structure of dVAE to learn an effective token sequence representing an image. Given the original input y of an image, after the image encoder Encoder image , h I is obtained, and after the decoder Decoder image , the reconstructed image is obtained. The loss function of the reconstructed image is where L reconstruct represents the loss function of the reconstructed image, y represents the image data, and y represents the reconstructed image data.

[0067] The text generation task is to generate the corresponding text by the file decoder in an autoregressive manner under the information condition of the given image. The corresponding loss function is where L represents the length of the text token sequence x′, i represents the number of each text token in the text token sequence x′, P(t i |x′ <i , h I ) represents the probability of generating the i-th text token t <i under the condition of the text representation vector x′ I and the image representation vector h i of the text token before the number i, t i represents the text token with the number i, x′ <i represents the text representation vector of the text token before the number i, and h I represents the image representation vector.

[0068] The mask language model randomly masks the input text token sequence x′ = [t1, t2, …, t L ] to form a new text token sequence x″ = [t1, <mask>..., t L ], and finally the concealed text <mask>The corresponding loss function is as follows

[0069]

[0070] wherein I is the set of masked text tokens, i is the number of the text token in the set of masked text tokens, P(t i represents the probability of generating the i-th text token t i under the condition of the text representation vector given the new text token sequence x". i represents the i-th text token, and x" represents the new text token sequence.

[0071] In the embodiments of the present application, the image representation vectors of the image sample data belonging to the same image-text content are subjected to single-modal alignment processing to obtain image vector results. Then, the image vector results and the text representation vectors of the text sample data belonging to the same image-text content are subjected to cross-modal alignment processing to obtain the image-text vector results of the image-text content.

[0072] For example, the training sample data Y contains the text sample data t1 and the image set sample data I1 belonging to the image-text content D1, and the image set sample data I1 contains the image sample data i101, i102, i103 and i104. The training sample data Y also contains the text sample data t2 and the image set sample data I2 belonging to the image-text content D2, and the image set sample data I2 contains the image sample data i201, i202, i203 and i204. The image representation vectors of the image sample data i101, i102, i103 and i104 are subjected to alignment processing to obtain the image vector results ix1, and the image vector results ix1 and the text representation vector of the text sample data t1 are subjected to alignment processing to obtain the image-text vector results itx1 of the image-text content D1. The image representation vectors of the image sample data i201, i202, i203 and i204 are subjected to alignment processing to obtain the image vector results ix2, and the image vector results ix2 and the text representation vector of the text sample data t2 are subjected to alignment processing to obtain the image-text vector results itx2 of the image-text content D2.

[0073] It should be noted that, for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0074] With reference to Figure 3 , a structural block diagram of a training device of a picture-text vector extraction model is shown, which can be applied to a terminal or a server. The training device of the picture-text vector extraction model can specifically include the following modules:

[0075] A sample acquisition module 31 is configured to acquire training sample data, which contains text sample data and image set sample data belonging to multiple picture-text contents, and the image set sample data contains multiple image sample data.

[0076] A vector extraction module 32 is configured to extract an image representation vector of each image sample data and extract a text representation vector of the text sample data.

[0077] A single-modal training module 33 is configured to train a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to the same picture-text content.

[0078] A cross-modal training module 34 is configured to train a preset cross-modal alignment network model according to a preset cross-modal alignment loss function, the text representation vector, and an image vector result output by the trained single-modal alignment network model.

[0079] A model determination module 35 is configured to determine the trained single-modal alignment network model and the trained cross-modal alignment network model as a trained picture-text vector extraction model.

[0080] In a preferred embodiment of the present application, the cross-modal training module 34 includes:

[0081] A sample selection module is configured to select positive sample data and negative sample data from the training sample data. The positive sample data contains text sample data and image set sample data belonging to a first picture-text content, and the negative sample data is text sample data or image set sample data belonging to a second picture-text content, and the first picture-text content is different from the second picture-text content.

[0082] A cross-modal alignment module is configured to perform alignment processing on the image vector result and the text representation vector according to a preset first cross-modal alignment loss function when the negative sample data is image set sample data, and perform alignment processing on the image vector result and the text representation vector according to a preset second cross-modal alignment loss function when the negative sample data is text sample data.

[0083] In a preferred embodiment of the present application, the first cross-modal alignment loss function is a maximum value between a sum of a first parameter and a preset hyperparameter and a preset numerical value, and the first parameter is wherein, is an image vector result of the positive sample data, is a text representation vector of the positive sample data, is an image vector result of the negative sample data.

[0084] In a preferred embodiment of the present application, the second cross-modal alignment loss function is a maximum value between a sum value of a second parameter and a preset hyperparameter, and a preset numerical value, the second parameter is wherein, is an image vector result of the positive sample data, is a text representation vector of the positive sample data, is a text representation vector of the negative sample data.

[0085] In a preferred embodiment of the present application, the single-modal training module 33 is configured to align the multiple image representation vectors corresponding to the same image-text content by using an L2 regularization loss function.

[0086] In a preferred embodiment of the present application, the vector extraction module 32 comprises: an image encoding module configured to encode each image sample data into an image label; a vector selection module configured to select a corresponding embedding vector from a preset image label dictionary according to a discrete value of each image label; and a vector determination module configured to determine the embedding vector as the image representation vector.

[0087] In a preferred embodiment of the present application, the vector extraction module 32 comprises:

[0088] a first text encoding module configured to perform first encoding processing on the text sample data to obtain a text label sequence;

[0089] a second text encoding module configured to perform second encoding processing on the text label sequence to obtain the text representation vector.

[0090] The present application also provides an electronic device, referring to Figure 4 , comprising a processor 401, a memory 402, and a computer program 4021 stored in the memory 402 and executable on the processor 401, wherein the processor 401 implements the training method of the image-text vector extraction model of the foregoing embodiments when executing the program 4021.

[0091] The present application also provides a readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the training method of the image-text vector extraction model of the foregoing embodiments.

[0092] For the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts are referred to the description of the method embodiments.

[0093] It should be noted that all the actions of acquiring signals, information or data in the embodiments of the present application are performed under the premise of complying with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the corresponding device owner.

[0094] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0095] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the embodiments of the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The embodiments of the present application are described with reference to flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure One one or more flows and / or blocks Figure One an apparatus that performs the functions specified in one or more flows and / or blocks.

[0097] These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure One one or more flows and / or blocks Figure One an apparatus that performs the functions specified in one or more flows and / or blocks.

[0098] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart Figure One one flow or a plurality of flows and / or the functions specified in the block Figure One one block or a plurality of blocks.

[0099] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art, once they know the basic creative concept, can make additional changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0100] Finally, it should also be noted that, in this paper, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or terminal device. Without more limitations, the element defined by the statement "including a…" does not exclude the presence of other identical elements in the process, method, article or terminal device including the element.

[0101] The above describes in detail the training method and device of a picture-text vector extraction model provided by the present application. The principles and implementation modes of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation mode and application range; in summary, the content of the specification should not be understood as a limitation of the present application.< / mask> < / mask>

Claims

1. A method for training a picture-text vector extraction model, characterized in that, The method comprises: obtaining training sample data, wherein the training sample data comprises text sample data and image set sample data belonging to multiple image-text contents, and the image set sample data comprises multiple image sample data; extracting an image representation vector of each image sample data and extracting a text representation vector of the text sample data; training a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to the same image-text content; training a preset cross-modal alignment network model according to a preset cross-modal alignment loss function, the text representation vector and an image vector result output by the trained single-modal alignment network model; training a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to the same image-text content; training a preset cross-modal alignment network model according to a preset cross-modal alignment loss function, the text representation vector and an image vector result output by the trained single-modal alignment network model; training a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to the same image-text content; 2. The method of claim 1, wherein, training a preset cross-modal alignment network model according to a preset cross-modal alignment loss function, the text representation vector and an image vector result output by the trained single-modal alignment network model; The method comprises: aligning multiple image representation vectors corresponding to the same image-text content by using an L2 regular loss function. The method comprises:

3. The method of claim 2, wherein, The first cross-modal alignment loss function is a maximum value between a sum value of a first parameter and a preset hyperparameter, and a preset numerical value, the first parameter is wherein, is an image vector result for the positive sample data, is a text representation vector for the positive sample data, is an image vector result for the negative sample data.

4. The method of claim 2, wherein, The second cross-modal alignment loss function is a maximum value between a sum value of a second parameter and a preset hyperparameter, and a preset numerical value, the second parameter is wherein, is an image vector result for the positive sample data, is a text representation vector for the positive sample data, is a text representation vector for the negative sample data.

5. The method of claim 1, wherein, selecting positive sample data and negative sample data from the training sample data; the positive sample data comprises text sample data and image set sample data belonging to a first image-text content, and the negative sample data is text sample data or image set sample data belonging to a second image-text content, wherein the first image-text content is different from the second image-text content; when the negative sample data is image set sample data, aligning the image vector result and the text representation vector according to a preset first cross-modal alignment loss function; when the negative sample data is text sample data, aligning the image vector result and the text representation vector according to a preset second cross-modal alignment loss function. The method comprises:

6. The method of claim 1, wherein, encoding each image sample data into an image label; selecting a corresponding embedding vector from a preset image label dictionary according to a discrete value of each image label; using the embedding vector as the image representation vector. 7.A device for training a picture-text vector extraction model, characterized in that, The method comprises: performing first encoding processing on the text sample data to obtain a text label sequence; performing second encoding processing on the text label sequence to obtain the text representation vector. The device comprises: a sample obtaining module configured to obtain training sample data, wherein the training sample data comprises text sample data and image set sample data belonging to multiple image-text contents, and the image set sample data comprises multiple image sample data; a vector extracting module configured to extract an image representation vector of each image sample data and extract a text representation vector of the text sample data; a single-modal training module configured to train a preset single-modal alignment network model according to a preset single-modal alignment loss function and multiple image representation vectors corresponding to the same image-text content, wherein the single-modal training module comprises: aligning multiple image representation vectors corresponding to the same image-text content by using an L2 regular loss function; a cross-modal training module configured to train a preset cross-modal alignment network model according to a preset cross-modal alignment loss function, the text representation vector, and an image vector result output by the trained single-modal alignment network model; a model determination module configured to determine the trained single-modal alignment network model and the trained cross-modal alignment network model as a trained image-text vector extraction model.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the training method of the image-text vector extraction model in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the training method of the image-text vector extraction model in any one of claims 1 to 6. The program is executed by the processor to implement the training method of the image-text vector extraction model in any one of claims 1 to 6.

Images