Zero-shot image-to-text conversion method and apparatus

Abstract

The application discloses a zero sample image text conversion method and device, and aims to solve the technical problem that the accuracy of the text description of the generated input image is low in the existing zero sample image text conversion method. The method comprises the following steps: performing entity representation training according to an obtained text corpus to generate a target entity representation matrix; adopting a preset image-text alignment representation model and a preset language processing model to perform training embedding extraction on the model training text in the text corpus to generate a to-be-trained hard-soft embedding; based on a preset cross-entropy loss function, performing model training on an initial decoder model according to the to-be-trained hard-soft embedding to determine a target decoder model; performing target embedding extraction on the to-be-converted image and the target entity representation matrix through the preset image-text alignment representation model and the preset language processing model to generate a target hard-soft embedding; and adopting the target decoder model to perform text conversion on the target hard-soft embedding to generate the text description corresponding to the to-be-converted image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a zero-shot image-text conversion method and device. BACKGROUND

[0002] Image-text conversion is a technology of converting image input into corresponding text description. Its implementation relies on model training on a large number of image-text pairs, and through cross-modal alignment information in the pre-trained model, the text representation is used to simulate the image representation, and noise is introduced to make up for the gap between modalities.

[0003] In the field of multi-modal, large-scale image-text alignment models such as Contrastive Language-Image Pre-training (CLIP) are emerging. These pre-training models are trained on a large number of image-text pairs, and achieve alignment between different modalities. These models perform well in discriminative tasks such as classification, segmentation and detection. However, it is still necessary to further explore the effective transfer of the perception ability of these models to zero-shot generation tasks (such as image-text conversion).

[0004] Existing zero-shot image-text conversion methods mostly focus on integrating detected concepts (such as entities) into the input to generate text descriptions of images. However, the models used by these methods may have biases in understanding concepts, which may introduce some concepts unrelated to the core content of the image, so that the generated text description of the input image contains more irrelevant content, resulting in lower accuracy of the generated text description of the input image. SUMMARY

[0005] The present application provides a zero-shot image-text conversion method and device, which solves the technical problem of low accuracy of the generated text description of the input image caused by the existing zero-shot image-text conversion method.

[0006] The first aspect of the present application provides a zero-shot image-text conversion method, comprising:

[0007] Obtaining a text corpus, performing entity representation training on the text corpus using a pre-set image-text alignment representation model and a pre-set contrastive learning loss function, and generating a target entity representation matrix;

[0008] Training and embedding extraction are performed on the model training text in the text corpus using the pre-set image-text alignment representation model and a pre-set language processing model, and a to-be-trained hard-soft embedding is generated;

[0009] Based on a pre-set cross-entropy loss function, a target decoder model is determined by performing model training on an initial decoder model according to the to-be-trained hard-soft embedding;

[0010] According to the preset image-text alignment representation model and the preset language processing model, target embedding extraction is performed according to the to-be-converted image and the target entity representation matrix, and target hard-soft embedding is generated;

[0011] According to the target decoder model, text conversion is performed according to the target hard-soft embedding, and a text description corresponding to the to-be-converted image is generated.

[0012] Optionally, the preset image-text alignment representation model and the preset contrastive learning loss function are used to perform entity representation training according to the text corpus to generate a target entity representation matrix, including:

[0013] The number of noun entities in the text corpus is counted to determine the number of occurrences of each noun entity;

[0014] Any noun entity corresponding to an occurrence number greater than a preset number threshold is taken as a to-be-trained noun entity, and an entity vector corresponding to each to-be-trained noun entity is initialized to determine an initial entity vector corresponding to each to-be-trained noun entity;

[0015] The plurality of initial entity vectors are used to construct an initial entity representation matrix;

[0016] The preset image-text alignment representation model is used to perform image representation simulation on the matrix training text in the text corpus to generate a matrix training image representation;

[0017] The preset syntax analysis tool is used to construct a positive-negative example entity set according to the noun entity set to determine the positive-negative example entity set;

[0018] The similarity of the matrix training image representation and the entity vector corresponding to the positive-negative example entity in the positive-negative example entity set is calculated to determine a positive-negative example similarity;

[0019] The positive-negative example similarity is substituted into the preset contrastive learning loss function and is derived to determine a contrastive learning gradient;

[0020] The contrastive learning gradient is used to update the initial entity representation matrix to determine an intermediate entity representation matrix, and the number of matrix updates is counted in real time;

[0021] It is determined whether the number of matrix updates reaches a preset first training number threshold;

[0022] If so, the intermediate entity representation matrix is taken as a target entity representation matrix.

[0023] Optionally, the to-be-trained hard-soft embedding comprises a to-be-trained hard embedding and a to-be-trained soft embedding; the preset language processing model comprises a forward perception machine module and a syntax parser; the generating the to-be-trained hard-soft embedding by using the preset image-text alignment representation model and the preset language processing model to perform embedding extraction on the model training text in the text corpus comprises:

[0024] performing image representation simulation on the model training text in the text corpus by using the preset image-text alignment representation model to generate a model training image representation;

[0025] performing vector projection on the model training image representation by using the forward perception machine module to generate the to-be-trained soft embedding;

[0026] taking a plurality of noun entities in the model training text as inputs of the syntax parser to output the to-be-trained hard embedding.

[0027] Optionally, the determining the target decoder model by performing model training on the initial decoder model according to the to-be-trained hard-soft embedding based on the preset cross-entropy loss function comprises:

[0028] splicing the to-be-trained hard embedding and the to-be-trained soft embedding to generate a to-be-trained spliced embedding;

[0029] substituting the to-be-trained spliced embedding into the preset cross-entropy loss function and deriving to determine a cross-entropy gradient;

[0030] updating model parameters of the initial decoder model by using the cross-entropy gradient to determine an intermediate decoder model and to statistically determine a model update number in real time;

[0031] determining whether the model update number reaches a preset second training number threshold;

[0032] if so, taking the intermediate decoder model as the target decoder model.

[0033] Optionally, the target hard-soft embedding comprises a target soft embedding and a target hard embedding; the target entity representation matrix comprises target entity vectors corresponding to a plurality of to-be-trained noun entities; the generating the target hard-soft embedding by using the preset image-text alignment representation model and the preset language processing model to perform target embedding extraction according to the to-be-converted image and the target entity representation matrix comprises:

[0034] inputting the to-be-converted image into the preset image-text alignment representation model for representation to generate a to-be-converted image representation;

[0035] performing vector projection on the to-be-converted image representation by using the forward perception machine module to generate the target soft embedding;

[0036] The target entity vectors corresponding to each to-be-trained noun entity are respectively subjected to similarity calculation with the to-be-converted image representation, and target similarities corresponding to each to-be-trained noun entity are determined.

[0037] The to-be-trained noun entity corresponding to any target similarity greater than a preset similarity threshold is taken as a target noun entity.

[0038] The target noun entities are taken as inputs of a grammar parser, and target hard embeddings are output.

[0039] Optionally, the text conversion according to the target hard-soft embedding by using the target decoder model to generate a text description corresponding to the to-be-converted image comprises the following steps.

[0040] The target hard embedding and the target soft embedding are spliced to generate a target spliced embedding.

[0041] The target spliced embedding is converted by using the target decoder model to generate a text description corresponding to the to-be-converted image.

[0042] The second aspect of the present application provides a zero-shot image text conversion device, comprising:

[0043] The acquisition module is configured to acquire a text corpus, perform entity representation training on the text corpus by using a preset image-text alignment representation model and a preset contrastive learning loss function, and generate a target entity representation matrix.

[0044] The first extraction module is configured to perform training embedding extraction on model training text in the text corpus by using the preset image-text alignment representation model and a preset language processing model, and generate to-be-trained hard-soft embeddings.

[0045] The training module is configured to perform model training on an initial decoder model based on a preset cross-entropy loss function and the to-be-trained hard-soft embeddings, and determine a target decoder model.

[0046] The second extraction module is configured to perform target embedding extraction on a to-be-converted image and the target entity representation matrix by using the preset image-text alignment representation model and the preset language processing model, and generate target hard-soft embeddings.

[0047] The conversion module is configured to perform text conversion according to the target hard-soft embeddings by using the target decoder model, and generate a text description corresponding to the to-be-converted image.

[0048] The third aspect of the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the zero-shot image text conversion method according to any one of the above aspects.

[0049] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the steps of the zero-shot image-to-text conversion method according to any one of the above.

[0050] The fifth aspect of the present application provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, wherein when the program instructions are executed by a computer, the computer is caused to perform the steps of the zero-shot image-to-text conversion method according to any one of the above.

[0051] From the above technical solutions, the present application has the following advantages:

[0052] The above technical solution of the present application provides a zero-shot image-to-text conversion method, first acquires a text corpus, and uses a preset image-text alignment representation model and a preset contrast learning loss function to perform entity representation training according to the text corpus to generate a target entity representation matrix; then, uses the preset image-text alignment representation model and a preset language processing model to perform training embedding extraction on model training text in the text corpus to generate a to-be-trained hard-soft embedding; based on a preset cross-entropy loss function, a target decoder model is determined by performing model training on an initial decoder model according to the to-be-trained hard-soft embedding; target embedding extraction is performed on the to-be-converted image and the target entity representation matrix by the preset image-text alignment representation model and the preset language processing model to generate a target hard-soft embedding; finally, text conversion is performed on the target hard-soft embedding by the target decoder model to generate a text description corresponding to the to-be-converted image; based on the above solution, the model training text in the text corpus is extracted by the preset image-text alignment representation model and the preset language processing model, and the initial decoder model is trained by combining the preset cross-entropy loss function to determine the target decoder model, and then the target hard-soft embedding is converted by the target decoder model to generate the text description corresponding to the to-be-converted image. The target decoder model trained by the zero-shot image can better capture the object information in the image, thereby improving the accuracy of the text description. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0054] Figure 1A step flow chart of a zero-shot image text conversion method provided for the first embodiment of the present application is provided.

[0055] Figure 2 A schematic diagram of a zero-shot image text conversion framework provided for the first embodiment of the present application is provided.

[0056] Figure 3 A structural block diagram of a zero-shot image text conversion device provided for the second embodiment of the present application is provided. DETAILED DESCRIPTION

[0057] The embodiments of the present application provide a zero-shot image text conversion method and device, and aim to solve the technical problem that the accuracy of the text description of the generated input image is low in the existing zero-shot image text conversion method.

[0058] In order to make the technical problems, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0059] Please refer to Figure 1 , Figure 1 A step flow chart of a zero-shot image text conversion method provided for the first embodiment of the present application is provided.

[0060] The zero-shot image text conversion method provided by the present application comprises:

[0061] In step 101, a text corpus is obtained, and a preset image-text alignment representation model and a preset contrastive learning loss function are used to perform entity representation training according to the text corpus to generate a target entity representation matrix.

[0062] The text corpus comprises a plurality of model training texts for initial decoder model training and a plurality of matrix training texts for initial entity representation matrix training, and the model training texts and the matrix training texts each comprise a plurality of noun entities.

[0063] Further, the process of generating the target entity representation matrix by using the preset image-text alignment representation model and the preset contrastive learning loss function to perform entity representation training according to the text corpus can be implemented by performing the following steps S11 to S110:

[0064] In step S11, the number of noun entities in the text corpus is counted to determine the number of occurrences of each noun entity.

[0065] Step S12, any occurrence of the noun entity greater than the preset number threshold is taken as a noun entity to be trained, and the entity vector corresponding to each noun entity to be trained is initialized to determine the initial entity vector corresponding to each noun entity to be trained;

[0066] Step S13, using the plurality of initial entity vectors, an initial entity representation matrix is constructed;

[0067] It should be noted that from all the matrix training texts and all the model training texts in the text corpus, all the noun entities (noun entities to be trained) with occurrence frequency exceeding the preset number threshold are extracted to construct a noun entity set, and the entity vectors corresponding to all the noun entities to be trained in the noun entity set are randomly initialized to obtain an initial entity representation matrix that can be trained; wherein the preset number threshold can be set as needed, and the present application is not limited thereto.

[0068] Step S14, using the preset image-text alignment representation model to perform image representation simulation on the matrix training texts in the text corpus to generate matrix training image representations;

[0069] It should be noted that for each matrix training text in the text corpus, the image-text alignment representation model CLIP (preset image-text alignment representation model) can be used for representation, because the input during inference is an image representation, and the image representation cannot be obtained during training under the zero-shot setting, therefore, the alignment feature of the preset image-text alignment representation model can be used to simulate the corresponding image representation by adding Gaussian noise to the text representation, specifically, the matrix training text input to the preset image-text alignment representation model is first represented to obtain the corresponding text representation, then Gaussian noise is added to the text representation to simulate the matrix training image representation, this process can be represented as:

[0070] ;

[0071] Wherein, is the i-th matrix training image representation; is the i-th matrix training text representation; is the selected Gaussian noise variance; is the Gaussian noise; is the preset image-text alignment representation model.

[0072] Step S15, using the preset syntax analysis tool to construct a positive and negative example entity set according to the noun entity set to determine the positive and negative example entity set;

[0073] The positive and negative example entity set includes a positive example entity set and a negative example entity set.

[0074] It should be noted that the noun phrases contained in the noun entity set are extracted by the syntax analysis tool NLTK to form the positive example entity set N' negative example entities are randomly selected from the set of positive example entities, and entities in the set of positive example entities are excluded to form a set of negative example entities ; specifically, a plurality of noun phrases are extracted from the set of named entities by using a syntax analysis tool NLTK to form the set of positive example entities , and N' named entities are randomly selected from the set of named entities and the named entities in the set of positive example entities are excluded to form the set of negative example entities .

[0075] Step S16, similarity calculation is performed on the entity vectors corresponding to the positive and negative example entities in the matrix training image representation and the set of positive and negative example entities to determine the positive and negative example similarity;

[0076] The positive and negative example similarity includes the positive example similarity and the negative example similarity.

[0077] It should be noted that, based on the set of positive and negative example entities obtained in the above steps, the similarity between the image representation I i (the matrix training image representation) and the positive example entities in the set of positive example entities and the negative example entities in the set of negative example entities can be calculated:

[0078] ;

[0079] ;

[0080] wherein, is the positive example similarity; is the i-th model training image representation; is the entity vector corresponding to the positive example entity; is the set of positive example entities; f is an entity in the set of positive example entities; is the negative example similarity; is the set of negative example entities; is the entity vector corresponding to the negative example entity; m is an entity in the set of negative example entities.

[0081] It is worth mentioning that, since an entity is assigned multiple types of representations, the highest similarity type is used when retrieving the similarity between the image and the entity, taking the similarity between the image representation I i (the matrix training image representation) and the negative example entities in the set of negative example entities as an example:

[0082] ;

[0083] wherein, is the i-th model training image representation; is the j-th type of entity representation; K is the number of entity types; is the entity vector corresponding to the negative example entity.

[0084] Step S17, the positive and negative example similarity is substituted into the preset contrast learning loss function and is derived to determine the contrast learning gradient;

[0085] Step S18, the initial entity representation matrix is updated using the contrast learning gradient to determine the intermediate entity representation matrix, and the matrix update times are counted in real time;

[0086] Step S19, it is judged whether the matrix update times reach the preset first training times threshold;

[0087] Step S110, if the threshold is reached, the intermediate entity representation matrix is taken as the target entity representation matrix.

[0088] It should be noted that if the matrix update times do not reach the preset first training times threshold, the intermediate entity representation matrix is taken as the new initial entity representation matrix, new matrix training texts are selected from the text corpus to simulate image representation, new matrix training image representations are generated, and the similarity between the entity vectors corresponding to the new positive and negative example entities and the new matrix training image representations is calculated to determine new positive and negative example similarities, and then step S17 is executed until the matrix update times reach the preset first training times threshold. The intermediate entity representation matrix determined when the matrix update times reach the preset first training times threshold is taken as the target entity representation matrix; wherein the preset first training times threshold can be set as needed, and the present application is not limited thereto.

[0089] Further, the preset contrast learning loss function is specific to:

[0090] ;

[0091] Wherein, L is the contrast learning loss value; margin is the interval threshold that can be tolerated during alignment learning; is the positive example similarity; is the negative example similarity.

[0092] It is worth mentioning that when the matrix is trained, the parameters of the image-text alignment representation model CLIP (preset image-text alignment representation model) are kept frozen, and only the training of the entity representation matrix is performed.

[0093] Step 102, the preset image-text alignment representation model and the preset language processing model are used to perform training embedding extraction according to the model training texts in the text corpus to generate a to-be-trained hard-soft embedding.

[0094] The to-be-trained hard-soft embedding includes a to-be-trained hard embedding and a to-be-trained soft embedding.

[0095] The preset language processing model comprises a forward perception machine module (MLP, Multilayer Perceptron) and a syntax parser.

[0096] Specifically, step 102 can comprise the following sub-steps S21-S23:

[0097] Step S21, using a preset image-text alignment representation model to simulate image representation of model training text in the text corpus, to generate model training image representation;

[0098] Step S22, projecting the model training image representation by a forward perception machine module to generate a to-be-trained soft embedding;

[0099] Step S23, taking a plurality of noun entities in the model training text as input of the syntax parser, and outputting a to-be-trained hard embedding.

[0100] It should be noted that for the input text, i.e., the model training text in the text corpus, the alignment capability of the CLIP pre-training model is used to simulate image representation by text representation, to obtain model training image representation, and a trained decoder GPT-2 model (target decoder model, Decoder of the Generative Pretrained Transformer 2) is used as a decoder to generate description in subsequent actual application. Since the representation is a continuous vector, and the input of GPT-2 is discrete text, the present application projects the continuous vector into the embedding layer of the decoder through a forward perception machine module (MLP) to realize information input. Define this part of input as soft embedding:

[0101] ;

[0102] Wherein, is the to-be-trained soft embedding; is the forward perception machine module; is the preset image-text alignment representation model; is the i-th model training text representation.

[0103] Further, to align the information of the image and the text in a more fine-grained manner, a syntax parser is used to extract a set of contained noun entities from the input text, and a hard embedding input with specific semantics is formed through the following template:

[0104] ;

[0105] Wherein, is the to-be-trained hard embedding.

[0106] Step 103, based on the preset cross-entropy loss function, model training is performed on the initial decoder model according to the to-be-trained hard-soft embedding, to determine a target decoder model.

[0107] Specifically, step 103 can include the following sub-steps S31-S35:

[0108] Step S31, the to-be-trained hard embedding and the to-be-trained soft embedding are spliced to generate a to-be-trained spliced embedding;

[0109] Step S32, the to-be-trained spliced embedding is substituted into the preset cross-entropy loss function and is derived to determine a cross-entropy gradient;

[0110] Step S33, the cross-entropy gradient is used to update the model parameters of the initial decoder model to determine an intermediate decoder model, and the number of model updates is counted in real time;

[0111] Step S34, it is judged whether the number of model updates reaches a preset second training number threshold;

[0112] Step S35, if yes, the intermediate decoder model is taken as the target decoder model.

[0113] It should be noted that the soft embedding and the hard embedding are spliced in the embedding layer of the GPT-2 decoder as the input of decoding, and the corresponding text (i.e., the model training text) is used as the label of self-decoding supervision, and the preset cross-entropy loss function is used to train the generated model (the initial decoder model); wherein the preset cross-entropy loss function is specifically:

[0114] ;

[0115] wherein, is a cross-entropy loss value; is a model training text; is a model parameter of the initial decoder model; is all words generated at the previous i time points, representing all words generated at all historical time points; is the i-th generated word; is a to-be-trained soft embedding; is a to-be-trained hard embedding; is splicing; is a to-be-trained spliced embedding.

[0116] Further, if the number of model updates does not reach the preset second training number threshold, the intermediate decoder model is taken as a new initial decoder model, and a new model training text is selected in the text corpus to generate a new to-be-trained spliced embedding, and then step S32 is executed again until the number of model updates reaches the preset second training number threshold, and the intermediate decoder model determined when the number of model updates reaches the preset second training number threshold is taken as the target decoder model.

[0117] Step 104: performing target embedding extraction on the to-be-converted image and the target entity representation matrix according to the preset image-text alignment representation model and the preset language processing model to generate target hard-soft embedding.

[0118] The target hard-soft embedding includes target soft embedding and target hard embedding.

[0119] The target entity representation matrix includes target entity vectors corresponding to the plurality of to-be-trained noun entities.

[0120] Specifically, step 104 can include the following sub-steps S41-S45:

[0121] Step S41: inputting the to-be-converted image into the preset image-text alignment representation model for representation to generate to-be-converted image representation;

[0122] It should be noted that the input image, i.e., the to-be-converted image, is directly input into the preset image-text alignment representation model for representation to generate to-be-converted image representation.

[0123] Step S42: projecting the to-be-converted image representation into a vector using a forward perception machine module to generate target soft embedding;

[0124] Step S43: performing similarity calculation on the to-be-converted image representation and the target entity vectors corresponding to each to-be-trained noun entity to determine target similarity corresponding to each to-be-trained noun entity;

[0125] It should be noted that based on the above steps, the target entity representation matrix is trained from the initial entity representation matrix, and the initial entity representation matrix includes initial entity vectors corresponding to the plurality of to-be-trained noun entities. When the initial entity representation matrix is trained, each initial entity vector in the initial entity representation matrix is taken as a corresponding target entity vector to form the target entity representation matrix. Therefore, the target entity representation matrix includes target entity vectors corresponding to the plurality of to-be-trained noun entities.

[0126] Further, the cosine similarity calculation is used to calculate the similarity between the input image (to-be-converted image representation) and each entity (to-be-trained noun entity) during reasoning:

[0127] ;

[0128] wherein, is a target similarity; is a representation of the i-th image to be converted; is a target entity vector corresponding to the noun entity to be trained; is a temperature coefficient for controlling randomness, the higher the temperature, the greater the randomness of entity retrieval; exp is an exponential function; j is the j-th entity.

[0129] Step S44, taking the noun entity to be trained corresponding to any target similarity greater than the preset similarity threshold as a target noun entity;

[0130] Further, a plurality of target noun entities matched with the input image are selected from the entity set by threshold screening, i.e., a plurality of target noun entities are obtained, and the process can be represented as:

[0131] ;

[0132] wherein, is an entity set composed of all target noun entities; is a target similarity; is a given screening probability, i.e., a preset similarity threshold; e is an entity; is a set composed of the top k entities with the largest similarity; k is a given number range selected from the highest similarity.

[0133] Step S45, taking the plurality of target noun entities as input of the syntax parser, and outputting a target hard embedding.

[0134] Step 105, using a target decoder model to convert the target hard and soft embeddings to generate a text description corresponding to the image to be converted.

[0135] Specifically, step 105 can include the following sub-steps S51-S52:

[0136] Step S51, splicing the target soft embedding and the target hard embedding to generate a target spliced embedding;

[0137] Step S52, using the target decoder model to convert the target spliced embedding to generate a text description corresponding to the image to be converted.

[0138] It should be noted that the generated model (target decoder model) trained and the retrieved entity set (set composed of all target noun entities) are comprehensively utilized to fill in the final generation template to generate the final text description (text description corresponding to the image to be converted), and the process can be represented as:

[0139]

[0140] ;

[0141] ;

[0142] wherein C is a text description corresponding to the image to be converted; is a target hard embedding; is a target soft embedding; is splicing; is a target spliced embedding; is a target decoder model; is an i-th image to be converted representation; is an entity set composed of all target noun entities.

[0143] As a technical effect, comparison can be made in combination with the prior art. Image text conversion refers to converting an input image into a corresponding text description. Traditional image text conversion methods usually need a large amount of image-text pair data for training, which is challenging in data acquisition. In recent years, zero-shot image text conversion methods have attracted attention. These methods use text representations to simulate image representations by utilizing pre-trained model cross-modal alignment information and introducing noise to make up for the modal gap. However, existing zero-shot image text conversion methods mainly focus on integrating detected concepts (such as entities) into the input, and less research is conducted on how to train the matching relationship between images and entities under a zero-shot setting through techniques such as contrastive learning. This results in generated text containing more irrelevant content and lower accuracy.

[0144] To solve the above problems, please refer to Figure 2 The present application proposes a zero-shot image text conversion method, which can be divided into three parts: entity representation training, text generation model training (decoder model training), and inference stage. Specifically, first, an entity dictionary in the field is constructed and each entity is assigned a multi-type learnable representation matrix to learn the relationship between entity representation and image representation, so that images can directly retrieve a number of noun entities associated with them according to the similarity between representations, i.e., an entity set is constructed and the representation of each entity is trained. Next, the present application uses a text representation of a text alignment model plus Gaussian noise to simulate image representation, further aligns the extracted noun entities from the text through a syntax parser, and then trains the text generation model for text reconstruction tasks. During inference, the trained text representation can be replaced by the aligned image representation, and the extracted entity concepts can be filled with retrieved entities to achieve zero-shot image text generation training independent of images, i.e., the ideal entities are extracted by the syntax parser and spliced with soft embedding for autoregressive training of the decoder. Finally, in the inference stage, the aligned entities are retrieved by the trained representation matrix, and the feature vector is input into the decoder to obtain the text description of the image.

[0145] In summary, the present application aims to improve the understanding and generation capabilities of the language model by simultaneously improving the image features and the entity information, explicitly assigning multiple types of representation vectors to the entities corresponding to the image through a multi-type entity representation framework, effectively improving the semantic representation completeness of the entities, retrieving more accurate entities that match the input image, aligning the image and text information in multiple granularities, and generating more text descriptions. The present embodiment enables the image-text conversion model trained by the zero-shot method to better capture the object information in the image, which can be used in the image description task and has certain reference significance for the same problems existing in other multi-modal zero-shot tasks.

[0146] In the embodiment of the present application, a zero-shot image-text conversion method is provided. First, a text corpus is obtained, and a pre-set image-text alignment representation model and a pre-set contrast learning loss function are used to perform entity representation training according to the text corpus to generate a target entity representation matrix. Then, the pre-set image-text alignment representation model and the pre-set language processing model are used to perform training embedding extraction on the model training text in the text corpus to generate a to-be-trained hard-soft embedding. Based on a pre-set cross-entropy loss function, an initial decoder model is trained according to the to-be-trained hard-soft embedding to determine a target decoder model. The pre-set image-text alignment representation model and the pre-set language processing model are used to perform target embedding extraction on the to-be-converted image and the target entity representation matrix to generate a target hard-soft embedding. Finally, the target decoder model is used to perform text conversion according to the target hard-soft embedding to generate a text description corresponding to the to-be-converted image. Based on the above scheme, the pre-set image-text alignment representation model and the pre-set language processing model are used to perform training embedding extraction on the model training text in the text corpus, and the pre-set cross-entropy loss function is combined to train the initial decoder model to determine the target decoder model. Then, the target decoder model is used to perform text conversion according to the target hard-soft embedding to generate a text description corresponding to the to-be-converted image. The target decoder model trained by the zero-shot image can better capture the object information in the image, thereby improving the accuracy of the text description.

[0147] Please refer to Figure 3 , Figure 3 The structure block diagram of a zero-shot image-text conversion device provided for the second embodiment of the present application is shown in FIG. 2.

[0148] The zero-shot image-text conversion device provided by the present application comprises:

[0149] The acquisition module 301 is configured to acquire a text corpus, perform entity representation training on the text corpus by using a pre-set image-text alignment representation model and a pre-set contrast learning loss function, and generate a target entity representation matrix.

[0150] The first extraction module 302 is configured to adopt the preset image-text alignment representation model and the preset language processing model to perform training embedding extraction on the model training text in the text corpus, and generate a to-be-trained hard-soft embedding;

[0151] The training module 303 is configured to perform model training on the initial decoder model based on the preset cross-entropy loss function and the to-be-trained hard-soft embedding, and determine a target decoder model;

[0152] The second extraction module 304 is configured to perform target embedding extraction on the to-be-converted image and the target entity representation matrix by the preset image-text alignment representation model and the preset language processing model, and generate a target hard-soft embedding;

[0153] The conversion module 305 is configured to perform text conversion on the target hard-soft embedding by using the target decoder model, and generate a text description corresponding to the to-be-converted image.

[0154] Further, the acquisition module 301 is specifically configured to:

[0155] count the number of noun entities in the text corpus, and determine the occurrence times corresponding to each noun entity;

[0156] take any noun entity corresponding to an occurrence time greater than a preset number threshold as a to-be-trained noun entity, and initialize the entity vector corresponding to each to-be-trained noun entity to determine an initial entity vector corresponding to each to-be-trained noun entity;

[0157] adopt the plurality of initial entity vectors to construct an initial entity representation matrix;

[0158] adopt the preset image-text alignment representation model to perform image representation simulation on the matrix training text in the text corpus, and generate a matrix training image representation;

[0159] adopt the preset syntax analysis tool to construct a positive-negative example entity set according to the noun entity set, and determine the positive-negative example entity set;

[0160] perform similarity calculation on the entity vectors corresponding to the positive-negative example entities in the matrix training image representation and the positive-negative example entity set, and determine a positive-negative example similarity;

[0161] substitute the positive-negative example similarity into a preset contrastive learning loss function and derive to determine a contrastive learning gradient;

[0162] adopt the contrastive learning gradient to update the initial entity representation matrix, determine an intermediate entity representation matrix, and statistically determine a matrix update number in real time;

[0163] determine whether the matrix update number reaches a preset first training number threshold;

[0164] If the threshold is reached, the intermediate entity representation matrix is taken as the target entity representation matrix.

[0165] Further, the to-be-trained hard-soft embedding includes a to-be-trained hard embedding and a to-be-trained soft embedding; the preset language processing model includes a forward perception machine module and a syntax parser; the first extraction module 302 is specifically configured to:

[0166] The preset image-text alignment representation model is used to perform image representation simulation on the model training text in the text corpus to generate a model training image representation;

[0167] The forward perception machine module is used to perform vector projection on the model training image representation to generate a to-be-trained soft embedding;

[0168] The plurality of noun entities in the model training text are taken as inputs of the syntax parser, and the to-be-trained hard embedding is output.

[0169] Further, the training module 303 is specifically configured to:

[0170] The to-be-trained hard embedding and the to-be-trained soft embedding are spliced to generate a to-be-trained spliced embedding;

[0171] The to-be-trained spliced embedding is substituted into the preset cross-entropy loss function and is derived to determine a cross-entropy gradient;

[0172] The cross-entropy gradient is used to update the model parameters of the initial decoder model to determine an intermediate decoder model, and the number of model updates is counted in real time;

[0173] It is determined whether the number of model updates reaches a preset second training number threshold;

[0174] If the threshold is reached, the intermediate decoder model is taken as the target decoder model.

[0175] Further, the target hard-soft embedding includes a target soft embedding and a target hard embedding; the second extraction module 304 is specifically configured to:

[0176] The to-be-converted image is input into the preset image-text alignment representation model for representation to generate a to-be-converted image representation;

[0177] The forward perception machine module is used to perform vector projection on the to-be-converted image representation to generate a target soft embedding;

[0178] The to-be-converted image representation and the target entity vector corresponding to each to-be-trained noun entity are subjected to similarity calculation to determine a target similarity corresponding to each to-be-trained noun entity;

[0179] Any to-be-trained noun entity corresponding to a target similarity greater than a preset similarity threshold is taken as a target noun entity;

[0180] The plurality of target noun entities are taken as inputs of a syntax parser, and target hard embeddings are output.

[0181] Further, the conversion module 305 is specifically configured to:

[0182] The target soft embedding and the target hard embedding are spliced to generate a target spliced embedding.

[0183] The target spliced embedding is converted by using a target decoder model to generate a text description corresponding to the to-be-converted image.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0185] The embodiment of the application further provides a computer device, including a memory and a processor, the memory stores a computer program; the computer program is executed by the processor, so that the processor executes the steps of the zero-shot image text conversion method of the above-mentioned embodiment one.

[0186] The embodiment of the application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the zero-shot image text conversion method of the above-mentioned embodiment one.

[0187] The embodiment of the application further provides a computer program product, including a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the zero-shot image text conversion method of the above-mentioned embodiment one.

[0188] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the above-described apparatus embodiments are merely schematic, and the division of units is merely a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, apparatuses or units, and can be electrical, mechanical or in other forms.

[0189] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place, or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0190] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A zero-shot image text-to-transformation method, characterized in that, The method comprises the following steps: acquiring a text corpus, performing entity representation training on the text corpus by using a preset image-text alignment representation model and a preset contrastive learning loss function, and generating a target entity representation matrix, which comprises the following steps: counting the number of noun entities in the text corpus to determine the number of occurrences of each noun entity; any noun entity corresponding to an occurrence number greater than a preset number threshold is regarded as a to-be-trained noun entity, and the entity vector corresponding to each to-be-trained noun entity is initialized to determine the initial entity vector corresponding to each to-be-trained noun entity; an initial entity representation matrix is constructed by using a plurality of initial entity vectors; the matrix training text in the text corpus is simulated by image representation by using the preset image-text alignment representation model to generate a matrix training image representation; a positive and negative example entity set is constructed according to the noun entity set by using a preset syntax analysis tool to determine the positive and negative example entity set; the similarity of the entity vectors corresponding to the positive and negative examples in the matrix training image representation and the positive and negative example entity set is calculated to determine the positive and negative example similarity; the positive and negative example similarity is substituted into the preset contrastive learning loss function and is derived to determine the contrastive learning gradient; the initial entity representation matrix is updated by using the contrastive learning gradient to determine an intermediate entity representation matrix, and the number of matrix updates is counted in real time; it is judged whether the number of matrix updates reaches a preset first training number threshold; if so, the intermediate entity representation matrix is regarded as the target entity representation matrix; the model training text in the text corpus is trained and embedded by using the preset image-text alignment representation model and a preset language processing model to generate a to-be-trained hard and soft embedding, which comprises the following steps: the to-be-trained hard and soft embedding comprises a to-be-trained hard embedding and a to-be-trained soft embedding; the model training text in the text corpus is simulated by image representation by using the preset image-text alignment representation model to generate a model training image representation; the model training image representation is projected by a forward perception machine module to generate a to-be-trained soft embedding; a plurality of noun entities in the model training text are taken as inputs of a syntax analyzer to output a to-be-trained hard embedding; an initial decoder model is trained based on a preset cross-entropy loss function according to the to-be-trained hard and soft embedding to determine a target decoder model; target hard and soft embedding is generated by the preset image-text alignment representation model and the preset language processing model according to the target entity representation matrix and the to-be-converted image; text conversion is performed by the target decoder model according to the target hard and soft embedding to generate a text description corresponding to the to-be-converted image.

2. The zero-shot image text transfer method of claim 1, wherein, the target decoder model is trained based on the preset cross-entropy loss function according to the to-be-trained hard and soft embedding, which comprises the following steps: the to-be-trained hard embedding and the to-be-trained soft embedding are spliced to generate a to-be-trained spliced embedding; the to-be-trained spliced embedding is substituted into the preset cross-entropy loss function and is derived to determine the cross-entropy gradient; The model parameters of the initial decoder model are updated by using the cross-entropy gradient, an intermediate decoder model is determined, and the number of model updates is counted in real time; It is judged whether the number of model updates reaches a preset second training number threshold; If so, the intermediate decoder model is taken as a target decoder model.

3. The zero-shot image text transfer method of claim 1, wherein, The target hard-soft embedding includes a target soft embedding and a target hard embedding; the target entity representation matrix includes target entity vectors corresponding to a plurality of to-be-trained noun entities; and the target embedding extraction is performed according to the to-be-converted image and the target entity representation matrix by using the preset image-text alignment representation model and the preset language processing model to generate the target hard-soft embedding, including: The to-be-converted image is input into the preset image-text alignment representation model for representation to generate a to-be-converted image representation; The to-be-converted image representation is projected by using a forward perception machine module to generate a target soft embedding; The to-be-converted image representation is respectively compared with the target entity vectors corresponding to each to-be-trained noun entity to determine the target similarity of each to-be-trained noun entity; Any to-be-trained noun entity corresponding to a target similarity greater than a preset similarity threshold is taken as a target noun entity; The plurality of target noun entities are taken as inputs of a grammar parser to output a target hard embedding.

4. The zero-shot image text transfer method of claim 3, wherein, The text conversion is performed according to the target hard-soft embedding by using the target decoder model to generate a text description corresponding to the to-be-converted image, including: The target soft embedding and the target hard embedding are spliced to generate a target spliced embedding; The target spliced embedding is converted by using the target decoder model to generate the text description corresponding to the to-be-converted image.

5. A zero-shot image-to-text conversion device, applied to the zero-shot image-to-text conversion method of claim 1, characterized in that, It includes: An acquisition module is configured to acquire a text corpus, perform entity representation training on the text corpus by using a preset image-text alignment representation model and a preset contrastive learning loss function to generate a target entity representation matrix; A first extraction module is configured to perform training embedding extraction on model training text in the text corpus by using the preset image-text alignment representation model and a preset language processing model to generate to-be-trained hard-soft embedding; A training module is configured to perform model training on an initial decoder model according to the to-be-trained hard-soft embedding based on a preset cross-entropy loss function to determine a target decoder model; A second extraction module is configured to perform target embedding extraction on a to-be-converted image and the target entity representation matrix by using the preset image-text alignment representation model and the preset language processing model to generate a target hard-soft embedding; A conversion module is configured to perform text conversion according to the target hard-soft embedding by using the target decoder model to generate a text description corresponding to the to-be-converted image.

6. A computer device, comprising: It includes a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the zero-shot image-text conversion method in any one of claims 1-4.

7. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the zero-shot image-text conversion method in any one of claims 1-4.

8. A computer program product, characterised in that, The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is caused to perform the zero-shot image-text transfer method according to any one of claims 1-4.

Images