Cross-modal bidirectional feature generation method for image and text bidirectional generation

Abstract

The invention discloses a cross-modal bidirectional feature generation method for bidirectional generation of images and texts. Comprising the following steps: 1) a novel cross-modal image and text bidirectional feature generation method, wherein a network model of the method is composed of an image feature generator, a text feature generator and a matching discriminator; (2) the CLIP ultrasonic visual language ability is used as a drive, a double-generation single-discriminator working mode is adopted, mapping of text and image features in a feature space can be learned, and feature conversion and generation between images and texts are achieved; 3) a self-attention text feature generator is provided, the generator combines image visual features and original image features of the CLIP, and the image visual features and the original image features are processed through a self-attention mechanism to generate text features, so that the image subtitle generation accuracy is improved; and 4) in order to enable the model to adapt to the updating gradient of the data set, introducing bidirectional matching loss, ensuring that the updating direction of the model is consistent with the direction of a real text-image pair, and introducing constraints between the real text-image pair to enable the model to learn more accurate feature representation. According to the method, the corresponding text or image can be generated by utilizing the existing image or text, and the modal data which is closer to the real image and text and has semantic consistency with the real image and text can be generated.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal bidirectional generation. By combining multiple generative adversarial networks and adopting a training strategy of dual generators and a single discriminator, a bidirectional matching loss is introduced to achieve cross-modal bidirectional generation of images and texts. Background Art

[0002] Intelligent image-text systems are eager to achieve bidirectional automatic generation of images and texts to improve user experience and creation efficiency. Ordinary users usually need to write captions when sharing life photos, and professional photographers also need to combine visual works with written descriptions during the creative process. In the traditional process, taking photos requires time for framing and composition, and writing descriptive texts requires careful consideration of word usage. In the absence of ready-made materials, automatically generating written descriptions from existing images or generating matching images according to text descriptions can significantly shorten the content creation cycle and lower the creation threshold for non-professionals. Thus, cross-modal bidirectional generation technology has important application value in the future field of visual content creation. Existing research mainly focuses on unidirectional image-text generation tasks and fails to fully explore the deep associations and bidirectional mapping potential between images and texts.

[0003] Currently existing bidirectional generation models mainly have three problems: (1) Traditional generation models adopt a unidirectional architecture design, and simple stacking of dual models is difficult to achieve bidirectional alignment at the semantic level; (2) Existing bidirectional models have limited generalization ability on small-scale vertical domain datasets (such as specific art styles) and have computational efficiency bottlenecks; (3) Natural images contain rich scene details, cultural symbols, and potential privacy information, and existing models are prone to semantic deviations during generation, making it difficult to ensure visual rationality and content security. Summary of the Invention

[0004] Based on these problems above, the present invention proposes a cross-modal bidirectional feature generation method for bidirectional generation of images and texts. The proposed training strategy of dual generators and a single discriminator can achieve the conversion between image and text features and further align the latent features between the two modalities; the proposed bidirectional matching loss can further enhance the semantic consistency between images and texts during the training process, ensuring that the gradient update direction of the model follows the data gradient direction of true pairings. The proposed cyclic text consistency enhances the feature alignment ability of the dual modality and achieves cross-modal bidirectional feature generation.

[0005] In view of this, the specific technical solution adopted by the present invention is as follows:

[0006] 1) Initialize the parameter θ in the image feature generator ti , the parameter θ in the text feature generator it , and the parameter θ in the matching discriminator d ;

[0007] 2) Use the batch pairing data sampler SA to sample paired data pairs (x i , t i ) from the paired images and texts;

[0008] 3) According to step 2), use the encoded original text features and original image features as the inputs of the image feature generator and the text feature generator respectively to obtain the generated image features or generated text features of the target modality;

[0009] 4) Output the generated image features obtained in step 3) to the text feature generator to obtain the reconstructed text features, and construct misaligned sample texts for the generated text features obtained in step 3) to obtain misaligned text features;

[0010] 5) Use the generated text features, misaligned text features, and reconstructed text features obtained in step 4) and the original text features as the inputs of the matching discriminator to calculate the losses to obtain the true matching loss, misaligned matching loss, and reconstructed matching loss;

[0011] 6) Calculate the two-way matching regularization for the original medical images and original text features obtained in step 2);

[0012] 7) Update the text feature generator, image feature generator, and matching discriminator according to the losses obtained in step 5) and step 6);

[0013] 8) Loop through steps 2) to 7) to train the model;

[0014] In summary, the advantages and beneficial effects of the present invention are as follows:

[0015] 1. The image and text two-way feature generation method of the present invention can generate corresponding texts according to images or generate corresponding images through texts, and the generated images have high quality, and the generated texts have higher semantic consistency and coherence.

[0016] 2. The proposed training strategy of dual generators and single discriminator can realize the feature transformation between images and texts by using two generators, further align the latent features between the two, and calculate the more efficient cyclic text consistency loss through a single discriminator, thus avoiding the discrimination of more complex image data.

[0017] 3. The proposed two-way matching regularization can use the original images and texts to calculate the gradient direction of the true paired data, so as to constrain the gradient of the model to update along the correct paired data gradient direction, further improving the generation quality and reducing the loss of context information. Description of the Drawings

[0018] Figure 1 This is the overall model framework diagram of the present invention;

[0019] Figure 2 This is the image feature generator diagram of the present invention;

[0020] Figure 3 This is the text feature generator diagram of the present invention;

[0021] Figure 4 This is the schematic diagram of bidirectional matching regularization of the present invention; Detailed implementation manners

[0022] Next, the technical solutions in the embodiments of the present invention will be clearly and detailedly described with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention.

[0023] The technical solution of the present invention to solve the above technical problems is as follows:

[0024] A cross-modal bidirectional feature generation method for bidirectional generation of images and texts, which includes the following steps:

[0025] Step 1: Initialize the parameters θ in the image feature generator G TI in, the parameters θ in the text feature generator G ti , and the parameters θ in the matching discriminator IT in; it ; d ;

[0026] Step 2: Use the batch paired data sampler SA to sample paired data pairs (x i , t i ) from the paired images and texts;

[0027] Step 3: Obtain the random Gaussian noise z, which is specifically expressed as follows:

[0028]

[0029] Step 4: The paired original images and texts (x i , t i ) are respectively used as the inputs of the text feature generator and the image feature generator, and the obtained outputs are expressed as:

[0030]

[0031] Step 5: The generated text features and generated image features obtained in Step 4 are respectively used as the inputs of the two generators again to obtain the reconstructed features, which are expressed as:

[0032]

[0033] Step Six: The reconstructed image features obtained in Step Five still need to be input into G IT to obtain tf″′ i , and misalign the original text features to construct misaligned text features, and the output is:

[0034] tf mis = T(tf i )

[0035] Step Seven: Use the text features and the original text features obtained in Steps Four, Five, and Six as the input of the matching discriminator, and the output is:

[0036]

[0037] Step Eight: Calculate the bidirectional matching regularization term, and calculate the true gradient direction using the original paired images and texts, which is specifically expressed as follows:

[0038]

[0039] Step Nine: Calculate the losses of the text feature generator and the matching discriminator, and the calculation formula is as follows:

[0040]

[0041] Step Ten: Calculate the loss of the image feature generator, and the calculation formula is as follows:

[0042]

[0043] Step Eleven: Calculate the parameter θ it and perform backpropagation to update the parameter, and the calculation formula is as follows:

[0044]

[0045] Step Twelve: Calculate the parameter θ d and perform backpropagation to update the parameter, and the calculation formula is as follows:

[0046]

[0047] Step Thirteen: Calculate the parameter θ ti and perform backpropagation to update the parameter, and the calculation formula is as follows:

[0048]

[0049] In summary, the innovations and advantages of the present invention:

[0050] The present invention proposes a novel cross-modal bidirectional feature generation network for image and text bidirectional generation method. The network model of this method consists of a text feature generator, an image feature generator, and a matching discriminator.

[0051] Driven by the visual language ability of CLIP, the present invention adopts a dual-generation single-discriminator working mode, which can learn the mapping between text and image features in the feature space and realize the feature conversion and generation between images and texts.

[0052] The present invention adopts a bidirectional matching loss to ensure that the direction of model update is consistent with the direction of real text-image pairs. Introducing constraints between real text-image pairs enables the model to learn more accurate feature representations.

[0053] The present invention uses two generators to generate image and text features respectively for transformation, and then the discriminator evaluates the difference between the generated features and the original text features, realizing the feature-level mapping learning in the dual-generation single-discriminator working mode and avoiding directly identifying more complex image features.

[0054] The above are the preferred examples of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, and improvements made by those skilled in the art within the principle of the present invention shall be regarded as being included in the protection scope of the present invention.

Claims

1. A cross-modal bidirectional feature generation method for bidirectional generation of images and texts, characterized in that: The following steps are involved: 1) Sample the paired image and text data, and encode the sampled data into the same latent feature space using an image or text encoder respectively; 2) according to step 1), the original text features and the original image features obtained by encoding are used as inputs of an image feature generator and a text feature generator respectively, to obtain generated image features or generated text features of the target modality; 3) The generated image features obtained in step 2) are output to a text feature generator to obtain reconstructed text features, and the generated text features obtained in step 2) are subjected to dislocation sample construction to obtain dislocation text features; 4) The generated text features, misaligned text features, and reconstructed text features obtained in step 2) are respectively used as inputs of the matching discriminator to calculate the loss to obtain the true matching loss, misaligned matching loss, and reconstructed matching loss; 5) performing a bidirectional matching regularization calculation based on the original image and original text features obtained in step 1); 6) updating the text feature generator, image feature generator and matching discriminator according to the losses obtained in step 4) and step 5); 7) Loop through steps 1) to 6) to train the model.

2. The cross-modal bidirectional feature generation method for bidirectional generation of image and text according to claim 1, characterized in that: Step 1) uses CLIP's text encoder ECTE and image encoder ECIE to encode the original text t i and the original image x i , the calculation formula is as follows:

3. The cross-modal bidirectional feature generation method for bidirectional generation of image and text according to claim 1, characterized in that: Step 2) is to generate image features through the image feature generator G IT and text feature generator G TI Implement input encoding text features and encoding image features to obtain generated image features if i And generate text features tf' i , z is Gaussian noise, and the calculation formula is as follows:

4. The cross-modal bidirectional feature generation method for bidirectional generation of image and text according to claim 1, characterized in that: Step 3) will generate image features if i Input to the text feature generator to generate the reconstructed text feature tf″ i , and construct the misaligned text feature tf mis , the calculation formula is as follows:

5. The cross-modal bidirectional feature generation method for bidirectional generation of image and text according to claim 1, characterized in that: Step 4) will calculate the loss of the text feature generator, image feature generator and matching discriminator, and the calculation formula is as follows: Among them, y i Represents the matching score between the generated, misplaced and reconstructed text features and the original text features, err i Represents the calculated loss value, True and False represent the positive and negative samples of the loss respectively.

6. The cross-modal bidirectional feature generation method for bidirectional generation of image and text according to claim 1, characterized in that: Step 5) Calculate the two-way matching loss of the real paired image and text, which is specifically expressed as follows: grad i =cat(grads[0],grad[1]) Among them, parameters a and b are hyperparameters. Increasing b will penalize larger gradients and help smooth the generator. J represents the gradient relative to the real image and text. grad_outputs has the same shape as J, but all the elements inside are 1.

7. The cross-modal bidirectional feature generation method for bidirectional generation of image and text according to claim 1, characterized in that: Step 6) back-propagates the loss obtained in step 4) and step 5) to update the parameters of the image feature generator, text feature generator and matching discriminator, which is specifically expressed as: Among them, err1 i ~err4 i These are the losses calculated by the formula in step 4), y3 i ~y4 i This is also the loss calculated in step 4).

Images