Text generation model training method and apparatus

Abstract

Embodiments of the present specification provide a text generation model training method and device, wherein the method comprises: determining an initial text sample, and extracting a text feature of the initial text sample; determining a multivariate Gaussian distribution noise for expressing a modal difference between the text feature and an image feature, obtaining an image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample; reflecting the image feature corresponding to the text feature to obtain a target text feature corresponding to the image feature according to the image feature corresponding to the text feature; determining a target text sample according to the target text feature, and training a text generation model according to the initial text sample and the target text sample. The multivariate Gaussian distribution noise for expressing the modal difference between the text feature and the image feature is used to map the text feature to the image feature, and the text generation model is trained without image-text paired data, thereby saving the data labeling link and reducing the cost.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, in particular to a text generation model training method. BACKGROUND

[0002] Image description task is a typical VL (Vision-Language, task involving both vision and text, such as image-text retrieval, text-based image generation, image description task, etc.) task, which aims to generate a text description sentence corresponding to an image for a given image.

[0003] In the past, an image description model can be obtained by a supervised training method to complete the image description task, but the supervised training method needs to rely on manually annotated image-text pair data sets for training. However, manual annotation of data sets is costly, and the manual annotation method is time-consuming and laborious and difficult to simply expand to large-scale data. Therefore, it is urgent to seek a training method that can drive the training of the image description model under low-cost image-text pair data or only text data. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a text generation model training method and a text generation method. One or more embodiments of the present specification also relate to a text generation model training device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a text generation model training method is provided, comprising:

[0006] determining an initial text sample, extracting a text feature of the initial text sample;

[0007] determining a multivariate Gaussian distribution noise for expressing the modal difference between the text feature and the image feature, obtaining an image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample;

[0008] According to the image feature corresponding to the text feature, the image feature corresponding to the target text feature is obtained by reflection mapping;

[0009] determining a target text sample according to the target text feature, and training a text generation model according to the initial text sample and the target text sample.

[0010] According to a second aspect of the embodiments of the present specification, a text generation model training device is provided, comprising:

[0011] The feature extraction module is configured to determine an initial text sample, and extract a text feature of the initial text sample;

[0012] The image feature determination module is configured to determine a multivariate Gaussian distribution noise for expressing a modal difference between the text feature and an image feature, and obtain the image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample;

[0013] The text feature obtaining module is configured to obtain a target text feature corresponding to the image feature corresponding to the text feature through reflection mapping;

[0014] The model obtaining module is configured to determine a target text sample according to the target text feature, and train a text generation model according to the initial text sample and the target text sample.

[0015] According to a third aspect of the embodiments of the present specification, a text generation model training method applied to a cloud end is provided, including:

[0016] receiving an initial text sample sent by a terminal, and extracting a text feature of the initial text sample;

[0017] determining a multivariate Gaussian distribution noise for expressing a modal difference between the text feature and an image feature, and obtaining the image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample;

[0018] obtaining a target text feature corresponding to the image feature corresponding to the text feature through reflection mapping;

[0019] determining a target text sample according to the target text feature, and training a text generation model according to the initial text sample and the target text sample;

[0020] sending the text generation model to the terminal.

[0021] According to a fourth aspect of the embodiments of the present specification, a text generation model training apparatus applied to a cloud end is provided, including:

[0022] The feature extraction module is configured to receive an initial text sample sent by a terminal, and extract a text feature of the initial text sample;

[0023] The image feature determination module is configured to determine a multivariate Gaussian distribution noise for expressing a modal difference between the text feature and an image feature, and obtain the image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample;

[0024] The text feature obtaining module is configured to reflectively map the image feature corresponding to the text feature to obtain a target text feature corresponding to the image feature according to the image feature corresponding to the text feature.

[0025] The model obtaining module is configured to determine a target text sample according to the target text feature, and train a text generation model according to the initial text sample and the target text sample.

[0026] The model sending module is configured to send the text generation model to the terminal.

[0027] According to a fifth aspect of an embodiment of the present specification, a text generation method is provided, comprising:

[0028] determining an image to be text generated, and inputting the image to be text generated into a text generation model;

[0029] generating target text corresponding to the image to be text generated through the text generation model, wherein the text generation model is trained according to the text generation model training method.

[0030] According to a sixth aspect of an embodiment of the present specification, a text generation apparatus is provided, comprising:

[0031] The image input module is configured to determine an image to be text generated, and input the image to be text generated into a text generation model;

[0032] The text generation module is configured to generate target text corresponding to the image to be text generated through the text generation model, wherein the text generation model is trained according to the text generation model training method.

[0033] According to a seventh aspect of an embodiment of the present specification, a text generation method is provided, applied to a cloud end, comprising:

[0034] receiving an image to be text generated sent by a terminal, and inputting the image to be text generated into a text generation model;

[0035] generating target text corresponding to the image to be text generated through the text generation model, wherein the text generation model is trained according to the text generation model training method;

[0036] sending the target text to the terminal.

[0037] According to an eighth aspect of an embodiment of the present specification, a text generation apparatus is provided, applied to a cloud end, comprising:

[0038] An image input module is configured to receive an image to be text generated sent by a terminal and input the image to be text generated into a text generation model.

[0039] A text generation module is configured to generate target text corresponding to the image to be text generated by the text generation model, wherein the text generation model is obtained by training according to the text generation model training method.

[0040] A text sending module is configured to send the target text to the terminal.

[0041] According to a ninth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0042] a memory and a processor;

[0043] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the text generation model training method and the text generation method.

[0044] According to a tenth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the instructions, when executed by a processor, implement the steps of the text generation model training method and the text generation method.

[0045] According to an eleventh aspect of an embodiment of the present specification, a computer program is provided, and when the computer program is executed in a computer, the computer program causes the computer to execute the steps of the text generation model training method and the text generation method.

[0046] The text generation model training method provided by one embodiment of the present specification comprises determining an initial text sample, extracting a text feature of the initial text sample, determining a multivariate Gaussian distribution noise for expressing a modal difference between the text feature and an image feature, obtaining an image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample, reflecting and mapping the image feature corresponding to the text feature to obtain a target text feature corresponding to the image feature, determining a target text sample according to the target text feature, and training a text generation model according to the initial text sample and the target text sample.

[0047] Specifically, the method determines an initial text sample, extracts a text feature of the initial text sample, and determines a multivariate Gaussian distribution noise; obtains an image feature corresponding to the text feature according to the text feature of the initial text sample and the multivariate Gaussian distribution noise, and obtains a target text feature corresponding to the image feature according to image feature reflection mapping, so as to avoid the problem of modal collapse in the case that the text feature does not correctly obtain the corresponding image feature; determines a target text sample according to the target text feature, and trains a text generation model according to the initial text sample and the target text sample. The method does not require image-text paired data, and can realize training of the model only by taking text data as a training sample, thereby saving a data labeling link and reducing cost. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a scene schematic diagram of a text generation model training method provided by an embodiment of the present specification;

[0049] Figure 2 is a flowchart of a text generation model training method provided by an embodiment of the present specification;

[0050] Figure 3 is a process flowchart of a text generation model training method provided by an embodiment of the present specification;

[0051] Figure 4 is a flowchart of a text generation method provided by an embodiment of the present specification;

[0052] Figure 5 is an application process schematic diagram of a text generation method provided by an embodiment of the present specification;

[0053] Figure 6 is a flowchart of an image generation model training method provided by an embodiment of the present specification;

[0054] Figure 7 is a flowchart of an image generation method provided by an embodiment of the present specification;

[0055] Figure 8 is a flowchart of a text generation model training method applied to a cloud provided by an embodiment of the present specification;

[0056] Figure 9 is a structural schematic diagram of a text generation model training device provided by an embodiment of the present specification;

[0057] Figure 10 is a structural schematic diagram of an image generation model training device provided by an embodiment of the present specification;

[0058] Figure 11is a structural block diagram of a computing device provided by one embodiment of the present specification. DETAILED DESCRIPTION

[0059] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples described herein, and it is understood that the scope of the present specification is not limited to the details below.

[0060] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0061] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. These terms are used only to distinguish one from another. For example, without departing from the scope of one or more embodiments of the present specification, first can be termed second, and similarly, second can be termed first. The term "if' as used herein can be interpreted as meaning "when" or "in response to determining" depending on the context.

[0062] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0063] Large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, billions or even tens of billions of model parameters. Large model can also be called foundation model. Through large-scale unlabeled corpus pre-training, a pre-trained model with hundreds of millions of parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language model (LLM) and multi-modal pre-training model.

[0064] In practical applications, large models only need a small amount of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in natural language processing (NLP) and computer vision fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), image generation, and natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0065] First, the technical terms related to one or more embodiments of the present specification are explained.

[0066] CLIP: Contrastive Language-Image Pre-Training, a model for matching images and text.

[0067] GPT-2: Generative Pre-trained Transformer, a generative language model.

[0068] Human-labeled image-text pair data: Taking the image description task as an example, it refers to giving an image and manually writing one or more text sentences that can correctly describe the image. The written text and the image have a close and accurate pair relationship.

[0069] Web image-text pair data: Data crawled from social networks. The text data obtained may not be closely associated with the image data. The text and image can be considered as a weakly associated pair.

[0070] The method for generating image description can be divided into supervised method, weakly supervised method or unsupervised method. The supervised method relies on manually annotated image-text pair data (i.e., manually annotated image-text pair data), uses a pre-trained convolutional neural network or object detection network to extract image features, and uses LSTM (Long Short Term Memory) or Transformer (a model using attention mechanism to improve model training speed) to decode image features to generate image description. However, the supervised method is limited by the annotation cost, and the size of the related dataset (image-text pair data) is not very large.

[0071] The weakly supervised method or the unsupervised method does not require image-text pair data, but there are still some problems in using existing weakly supervised methods or unsupervised methods. For example, the existing weakly supervised method or unsupervised method is only compatible with one or two specific data configurations containing only text data, but in actual application, users may have different data configurations, such as providing a small amount of high-quality paired data, paired network data (i.e., network image-text pair data) or network image data in addition to the text corpus containing only text data.

[0072] In the present specification, a text generation model training method and an image generation model training method are provided. The present specification also relates to a text generation model training device, an image generation model training device, a computing device, and a computer-readable storage medium. The embodiments are described in detail one by one below.

[0073] Referring to Figure 1 , Figure 1 A scene application diagram of a text generation model training method according to an embodiment of the present specification is shown.

[0074] Figure 1 The cloud-side device 102 and the end-side device 104 are included in the embodiment. The cloud-side device 102 can be understood as a cloud server. Of course, in another implementation, the cloud-side device 102 can also be replaced by a physical server. The end-side device 104 can be understood as various electronic devices, which can be a device with a screen or a device without a screen, including but not limited to a smart phone, a notebook computer, a smart speaker, a smart television, a PC (Personal Computer), a wearable device, etc. For ease of understanding, in the embodiments of the present specification, the cloud-side device 102 is taken as a cloud server and the end-side device 104 is taken as a notebook computer for detailed description.

[0075] In specific implementation, the cloud-side device 102 performs text generation model training. The text generation model can be understood as a model for inputting an image and outputting an image description.

[0076] The cloud-side device 102 determines training samples, for example, obtains the training samples from a database; in actual application, the database can include training samples of four data configurations, the training samples of data configuration 1 can be understood as annotated image-text paired data (i.e., image-text pairs), the training samples of data configuration 2 can be understood as network image-text paired data, the training samples of data configuration 3 can be understood as image data without correlation with text, and the training samples of data configuration 4 can be understood as text data.

[0077] Taking the text data of data configuration 4 as an example, the training samples are input into the text generation model, the text features of the training samples are extracted in the text generation model, and the multivariate Gaussian distribution noise used to express the modal difference between the text features and the image features is determined, the image features corresponding to the text features are obtained according to the multivariate Gaussian distribution noise and the text features of the training samples, and the target text features corresponding to the image features are obtained according to the reflection mapping of the image features, the target text samples are determined according to the target text features, and the text generation model is trained according to the target text samples and the training samples.

[0078] In a commonly used scenario, the user can interact with the text generation model arranged in the cloud-side device 102 by using the end-side device 104, and various applications such as voice interaction applications, web browser applications, communication applications, etc. can be installed on the end-side device 104.

[0079] The end-side device 104 receives an image to be text generated input by the user, for example, the image to be text generated is an image of a man riding a motorcycle; the end-side device 104 calls the text generation model in the cloud-side device 102, inputs the image to be text generated into the text generation model, obtains the target text corresponding to the image to be text generated in the text generation model, for example, obtains the target text as “a man rides a motorcycle on the road”, and can return the target text to the end-side device 104, so that the user obtains the target text.

[0080] In another commonly used scenario, the text generation model trained in the cloud-side device 102 can also be deployed to the end-side device 104, then the end-side device 104 receives an image to be text generated input by the user, and the end-side device 104 inputs the image to be text generated into the text generation model deployed in the end-side device 104, and obtains the target text corresponding to the image to be text generated in the text generation model.

[0081] The text generation model training method provided in the embodiments of the present specification can, after determining an initial text sample, extract text features of the initial text sample, and obtain image features based on multivariate Gaussian distribution noise for expressing modal differences between the text features and the image features, and the text features of the text sample, obtain target text features according to the image features, and then obtain a target text sample according to the target text features, and train the text generation model according to the initial text sample and the target text sample. The text generation model obtained through the training can realize the image description task without paired data, and the training of the model can be realized only by using text data as training samples, thereby saving the data labeling link and reducing the cost.

[0082] Referring to Figure 2 , Figure 2 A flowchart of a text generation model training method according to one embodiment of the present specification is shown, which specifically includes the following steps.

[0083] Step 202: determining an initial text sample and extracting text features of the initial text sample.

[0084] The initial text sample can be understood as a text of any type, any length, and any format, such as "a man riding a motorcycle on the road". The text features can be understood as converting the text into numerical features that can be used for machine learning.

[0085] In one or more embodiments of the present specification, the database contains multiple types of data, and the text in each type or a combination of several types can be used as an initial text sample. The training of the text generation model can be completed by using text data as training samples, thereby reducing the requirement and dependence on paired image text data. The specific implementation is as follows:

[0086] The determination of the initial text sample includes:

[0087] The first image text pair, the second image text pair, the third image text pair, and / or the text data are obtained from the database; the text in the first image text pair, the second image text pair, the third image text pair, and / or the text data is used as the initial text sample,

[0088] The quality of the image text pair in the first image text pair is greater than the quality of the image text pair in the second image text pair, and the quality of the image text pair in the second image text pair is greater than the quality of the image text pair in the third image text pair.

[0089] The database can be understood as a database containing text data and image text paired data of different qualities, and the database contains data of multiple data configurations.

[0090] The first image-text pair can be understood as annotated image-text pair data, and the pairing relationship between the image and the text in the image-text pair data is strong correlation, such as the image-text pair data in the COCO dataset and the Flickr dataset. The image-text pair data in the COCO dataset and the Flickr dataset has high quality, and the pairing relationship between the image and the text is strong correlation. The text in the image-text pair data is homologous to the text data in the database, and homologous can be understood as that the text in the image-text pair data is obtained from the same text acquisition channel as the text data.

[0091] The second image-text pair can be understood as that the text corresponding to the image is obtained from the network, that is, the text corresponding to the image does not necessarily have a strong pairing relationship with the image, and the text corresponding to the image is not necessarily a complete sentence. For example, the image-text pair data in the CC3M, CC12M, and YFCC datasets. The image-text pair data in the CC3M, CC12M, and YFCC datasets has low quality, and the text corresponding to the image does not necessarily have a strong pairing relationship with the image, and the text corresponding to the image is not necessarily a complete sentence. The text in the image-text pair data is heterologous to the text in the text corpus, and heterologous can be understood as that the text in the image-text pair data is obtained from a different text acquisition channel than the text data in the database.

[0092] The third image-text pair can be understood as containing images randomly sampled from other datasets, and the text in the image-text pair data and the image have no correlation.

[0093] The text data can be understood as text that appears in actual language use.

[0094] The quality of the image-text pair can be understood as the correlation between the image and the text in the image-text pair. The stronger the correlation between the image and the text, the higher the quality of the image-text pair.

[0095] In combination with the above embodiments, in the case where the database contains four kinds of data configuration training samples, the first image-text pair can be understood as data configuration 1, the second image-text pair can be understood as data configuration 2, the third image-text pair can be understood as data configuration 3, and the text data can be understood as data configuration 4.

[0096] In practical applications, the text in any one of the four kinds of data configurations can be used as a training sample for model training to obtain a text generation model.

[0097] The text generation model training method provided in the embodiments of the present specification includes a plurality of data configurations in the database, and the model training can be performed using the plurality of data configurations. The plurality of data configurations can all achieve model training and obtain a text generation model.

[0098] In one or more embodiments of the present specification, in order to convert the text in the plurality of data configurations into a language that can be recognized by machine learning, a text feature of the text is extracted, and the text feature can be used for machine learning. Specifically, the text generation model includes a text encoding network layer. In the text encoding network layer, a text feature of the text is extracted to obtain a text feature adapted to the text encoding network layer. The specific implementation is as follows:

[0099] The text generation model includes a text encoding network layer.

[0100] Correspondingly, the extracting the text feature of the initial text sample includes:

[0101] The initial text sample is input into the text generation model, and the text feature of the initial text sample is extracted through the text encoding network layer in the text generation model, wherein the text feature of the initial text sample is adapted to the text encoding network layer.

[0102] The text encoding network layer is responsible for converting the input into a feature. For example, in the case of input data being a text, the text encoding network layer extracts a text feature of the text.

[0103] In practical applications, the text encoding network layer can be implemented by using a CLIP model. The CLIP model can implement text encoding and image encoding. For example, in the CLIP model, inputting a text, the CLIP model can implement the function of extracting a text feature of the text. Or, in the CLIP model, inputting an image, the CLIP model can implement the function of extracting an image feature of the image.

[0104] In the embodiments of the present specification, taking the extraction of the text feature of the input initial text sample by the CLIP model as an example, the extraction of the text feature of the input initial text sample by the CLIP model is described in detail, and the specific implementation is as follows:

[0105] The initial text sample is input into the CLIP model, and the text feature of the input initial text sample is extracted by the CLIP model.

[0106] The text generation model training method provided in the embodiments of the present specification extracts the text feature of the initial text sample through the text encoding network layer in the text generation model, so as to convert the initial text sample into a language that can be recognized by machine learning.

[0107] Step 204: determining a multivariate Gaussian distribution noise for expressing the modal difference between the text feature and the image feature, obtaining the image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample.

[0108] The multivariate Gaussian distribution noise can be understood as a kind of noise whose probability density function obeys a multivariate Gaussian distribution (i.e., a normal distribution).

[0109] The image feature can be understood as converting the image into a numerical feature that can be used for machine learning. It should be noted that the text feature and the image feature in the present specification represent a text feature vector and an image feature vector.

[0110] The modal difference can be understood as the difference between data of different modalities (such as image data, text data, voice data, etc.) due to differences in data form, representation, semantics, etc.

[0111] Specifically, the multivariate Gaussian distribution noise can be used to express the modal difference between the text feature and the image feature, and the text feature of the initial text sample is mapped to the image feature corresponding to the text feature according to the determined multivariate Gaussian distribution noise and the text feature of the initial text sample.

[0112] In one or more embodiments of the present specification, there is a difference between the features of the two modalities of text and image, and the conversion from one modality to another modality can be achieved by injecting a multivariate Gaussian distribution noise. Specifically, the text generation model uses text data as input during the training process, extracts the text feature of the text, but the model is used for inference when the input is image data. Injecting a multivariate Gaussian distribution noise on the text feature extracted from the text data during model training can project the text feature to the image feature, so that in the case of applying the model, the image data can be directly used as the input, and the image feature of the image can be extracted. The specific implementation is as follows:

[0113] The method for determining the multivariate Gaussian distribution noise for expressing the modal difference between the text feature and the image feature comprises the following steps.

[0114] The multivariate Gaussian distribution noise for expressing the modal difference between the text feature and the image feature is determined through a noise sampling network layer in the text generation model.

[0115] The text generation model training method provided by the embodiments of the present specification contains a noise sampling network layer in the text generation model, and a multivariate Gaussian distribution noise can be determined in the noise sampling network layer. The multivariate Gaussian distribution noise is used to express the modal difference between the text feature and the image feature, and can achieve the conversion from the text feature space to the image feature space.

[0116] In one or more embodiments of the present specification, by sampling the text features and image features in the image-text pair data, calculating the difference between the text features and the image features, and then calculating the mean and covariance of the difference, the noise is sampled according to the calculated mean and covariance, and the multivariate Gaussian distribution noise is determined. The specific implementation is as follows:

[0117] The noise sampling network layer in the text generation model determines the multivariate Gaussian distribution noise for expressing the modal difference between the text features and the image features, comprising:

[0118] Determine the image-text pair to be sampled;

[0119] Sample the modal difference between the text features and the image features in the image-text pair to be sampled through the noise sampling network layer in the text generation model, and determine the multivariate Gaussian distribution noise for expressing the modal difference between the text features and the image features according to the sampling result.

[0120] Wherein, the image-text pair to be sampled can be understood as the image-text pair in the database, or the image-text pair randomly obtained from other data sets; sampling can be understood as calculating the difference between the text features and the image features; the sampling result can be understood as the mean and covariance of the calculated difference between the text features and the image features.

[0121] In practical applications, the image-text pair data to be sampled is randomly obtained, the image-text pair data is input into the noise sampling network layer in the text generation model, the modal difference between the text features and the image features in the image-text pair data to be sampled is sampled in the noise sampling network layer, the difference between the text features and the image features is calculated, and then the mean and covariance of the difference are calculated as the mean and covariance parameters of the multivariate Gaussian distribution. That is, a multivariate Gaussian distribution is determined by the mean and covariance parameters, and noise is sampled by the mean and covariance, so as to determine the multivariate Gaussian distribution noise for expressing the modal difference between the text features and the image features.

[0122] Following the above example, there are image-text pair data in the first image-text pair and the second image-text pair, and the mean and covariance can be directly calculated from the image-text pair data in the first image-text pair and the second image-text pair. For example, assuming that the first image-text pair has N images corresponding to N text data, the image features of the image are extracted using the text generation model, with a size of N*D1, and the text features of the text are N*D2. Calculate the difference between the image features and the text features, with a size of N*D3, and then calculate the mean and covariance of the difference as the mean and covariance parameters of the multivariate Gaussian distribution.

[0123] For the third image-text pair and the text data, the mean and covariance parameters are learned as part of the text generation model parameters through model training.

[0124] The text generation model training method provided by the embodiments of the present specification injects a multivariate Gaussian distribution noise on the extracted text features through the noise sampling network layer in the text generation model, so as to realize the cross-modal feature mapping from the text feature space to the image feature space.

[0125] In one or more embodiments of the present specification, in the training phase of the text generation model, the input of the text generation model is text data, and the extraction is text features, while in the application phase of the text generation model, the input of the text generation model is image data, and the extraction is image features. In order to identify the extracted image features in the application of the text generation model, the text features are mapped to image features in the training phase of the text generation model. The specific implementation is as follows:

[0126] The image features corresponding to the text features are obtained based on the multivariate Gaussian distribution noise and the text features of the initial text sample, and the image features corresponding to the text features are obtained based on the multivariate Gaussian distribution noise and the text features of the initial text sample.

[0127] The multivariate Gaussian distribution noise and the text features of the initial text sample are input into the mapping layer of the text generation model.

[0128] In the mapping layer, the text features of the initial text sample are mapped to the image features corresponding to the text features based on the multivariate Gaussian distribution noise.

[0129] Specifically, in the mapping layer, there is an Add&L2 Norm (residual connection and normalization), and the extracted text features of the initial text sample and the determined multivariate Gaussian distribution noise are input into the Add&L2 Norm in the mapping layer of the text generation model. The Add&L2 Norm in the mapping layer adds and normalizes the text features and the multivariate Gaussian distribution noise, merges the text features and the multivariate Gaussian distribution noise, and realizes the mapping of the text features to the corresponding image features.

[0130] The text generation model training method provided by the embodiments of the present specification injects a multivariate Gaussian distribution noise on the extracted text features of the initial text sample in the mapping layer of the text generation model, adds and normalizes the text features and the multivariate Gaussian distribution noise, and realizes the mapping of the text features of the initial text sample to the image features corresponding to the text features.

[0131] Step 206: According to the image features corresponding to the text features, the image features corresponding to the target text features are obtained by reflection mapping.

[0132] The target text feature can be understood as a text feature obtained by reflection mapping of the image feature.

[0133] In one or more embodiments of the present specification, in the case that the text feature of the initial text is adapted to the text encoding network layer, the target text feature corresponding to the image feature is obtained by reflection mapping according to the image feature corresponding to the text feature. The specific implementation is as follows:

[0134] The text generation model comprises a text encoding network layer;

[0135] Correspondingly, the target text feature corresponding to the image feature is obtained by reflection mapping according to the image feature corresponding to the text feature, comprising:

[0136] The target text feature corresponding to the image feature is obtained by reflection mapping according to the image feature corresponding to the text feature.

[0137] In actual application, the text feature of the initial text sample obtained by the CLIP model is located in the embedding space of the CLIP model, so in the case that the text feature of the initial text sample is located in the embedding space of the CLIP model, the target text feature corresponding to the image feature is obtained by reflection mapping according to the image feature corresponding to the text feature.

[0138] The text generation model training method provided by the embodiments of the present specification is that the text feature of the initial text is adapted to the text encoding network layer, and the target text feature is obtained by reflection mapping according to the image feature corresponding to the text feature of the initial text.

[0139] In one or more embodiments of the present specification, the image feature output by the mapping layer is input into the reflection mapping layer in the text generation model, and the image feature is mapped back to the text feature space through the reflection mapping layer to ensure the robustness of the model and avoid the modal collapse problem. The specific implementation is as follows:

[0140] The target text feature corresponding to the image feature is obtained by reflection mapping according to the image feature corresponding to the text feature, comprising:

[0141] The image feature corresponding to the text feature is input into the reflection mapping layer in the text generation model;

[0142] In the reflection mapping layer, the image feature corresponding to the text feature is reflected as the target text feature corresponding to the image feature, wherein the reflection mapping layer comprises a feedforward network layer, and the reflection mapping layer and the mapping layer are in an asymmetric structure.

[0143] In the feedforward network layer, the input image features are processed through a series of linear transformations and activation functions to obtain an intermediate result, and the intermediate result is connected with the input image features through residual connection and normalized to obtain the target text features corresponding to the image features.

[0144] In practical applications, the back-mapping layer in the text generation model is equivalent to adding an additional reconstruction constraint. In the case that the output of the mapping layer is not completely in the image feature space, the output of the mapping layer can also be mapped back to the text feature space through the back-mapping layer. That is, if the back-mapping layer does not exist, in the case that the mapping layer does not correctly map the text features to the image features, the text generation model may not generate the correct text description (i.e., modal collapse) when the image is input in the application stage of the text generation model. By using the back-mapping layer in the text generation model, the output of the back-mapping layer is constrained to be text features in the model training stage, so that even if there is a certain difference between the image and text domains between the model application stage and the model training stage, the text generation model can normally generate the text description corresponding to the image (i.e., the robustness of the model is ensured) when the image is input.

[0145] The text generation model training method provided by the embodiments of the present specification ensures the robustness of the model by using the back-mapping layer in the text generation model to map the image feature vector in the image feature space output by the mapping layer back to the text feature vector in the text feature space, and avoids the problem of modal collapse that the text generation model cannot generate the correct text description when the image is input in the application stage of the text generation model.

[0146] Step 208: determining a target text sample according to the target text features, and training a text generation model according to the initial text sample and the target text sample.

[0147] The target text sample can be understood as a text sample obtained according to the target text features.

[0148] Specifically, in the case of determining the target text sample, the loss function of the initial text sample and the target text sample is calculated, wherein the loss function is a function for measuring the gap between the model prediction and the true value, such as using the cross-entropy loss function to train the model.

[0149] In one or more embodiments of the present specification, the text generation model comprises a text encoding network layer and a text decoding network layer, wherein the text encoding network layer is responsible for extracting text features of an initial text sample; the text decoding network layer is responsible for decoding target text features to generate target text, and in the case that the embedding spaces of the text encoding network layer and the text decoding network layer are different, the features adapted to the text encoding network layer need to be projected into features that can be adapted to the text decoding network layer. The specific implementation is described as follows:

[0150] The text generation model comprises a text decoding network layer;

[0151] Correspondingly, the determining of the target text sample according to the target text features comprises:

[0152] projecting, by the feature projection layer of the text generation model, the target text features adapted to the text encoding network layer into projected text features adapted to the text decoding network layer;

[0153] inputting the projected text features into the text decoding network layer, and decoding the projected text features into a target text sample in the text decoding network layer.

[0154] Among them, the text decoding network layer can be understood as a network layer capable of decoding text features into text.

[0155] In practical applications, the text decoding network layer can be implemented by using a GPT-2 model; the GPT-2 model can realize the function of text decoding; taking the example of realizing the extraction of text features of an initial text sample by using a CLIP model and realizing the decoding of projected text features into a target text sample by using a GPT-2 model, the text generation model training method is described in detail.

[0156] Since the embedding spaces of the CLIP model and the GPT-2 model are inconsistent, the target text features in the embedding space of the CLIP model can be projected into features that can be processed by the GPT-2 model through the feature projection layer of the text generation model, so that the projected text features that can be processed by the GPT-2 model are input into the GPT-2 model to generate the target text corresponding to the projected text features.

[0157] In actual application, the feature projection layer includes a multi-layer structure, and each layer has a consistent structure, which is an Encoder layer of a Transformer. The target text feature obtained in the feature projection layer comes from an embedding space of the CLIP model, and the target text feature needs to be input into the GPT-2 model for text generation. However, there is a domain difference between the CLIP model and the GPT-2 model, and the GPT-2 model cannot directly process the target text feature obtained by the CLIP model. The feature projection layer plays a role of an intermediate transition, and projects the target text feature in the embedding space of the CLIP model into a projection text feature that can be processed by the GPT-2 model.

[0158] Specifically, in the case where there is no prompt information, the projection text feature obtained by the feature projection layer can be input into the GTP-2 model, the projection text feature is decoded by the GTP-2 model, and a target text sample corresponding to the projection text feature is obtained.

[0159] The text generation model training method provided by the embodiment of the present specification can project the target text feature adapted to the text encoding network layer into a projection text feature adapted to the text decoding network layer by inputting the target text feature into the feature projection layer, so that the projection text feature is decoded into a target text sample in the text decoding network layer.

[0160] In another embodiment of the present specification, in the case where there is additional prompt information, a prompt interaction layer is constructed, and the additional prompt information and the projection text feature obtained by the feature projection layer are fused in the prompt interaction layer to obtain a more accurate target text. The specific implementation is as follows:

[0161] The projection text feature is input into the text decoding network layer, and the projection text feature is decoded into a target text sample in the text decoding network layer.

[0162] The keyword text feature of the keyword is extracted, and the keyword text feature is determined;

[0163] The projection text feature and the keyword text feature are input into a prompt interaction layer of the text generation model;

[0164] In the prompt interaction layer, the keyword text feature and the projection text feature are fused to obtain a fusion feature;

[0165] The fusion feature is input into the text decoding network layer, and the fusion feature is decoded into a target text sample in the text decoding network layer.

[0166] The keyword can be understood as a word in the initial text sample that can reflect the key information of the initial text sample. For example, in the case of the initial text sample "a man rides a motorcycle on the road", the keywords can be "man" and "motorcycle".

[0167] Specifically, a keyword extractor can be used outside the text generation model to extract keywords of the initial text sample, or a keyword extraction layer can be constructed in the text generation model to extract keywords of the initial text sample, and the present specification does not limit this.

[0168] After extracting the keywords of the initial text sample, the keyword text features of the keywords are determined, and the keyword text features and the projected text features are input into the prompt interaction layer. In the prompt interaction layer, the keyword text features and the projected text features are fused to obtain fused features, and the fused features are decoded to obtain the target text sample.

[0169] In practical applications, the prompt interaction layer includes a Cross MSA (Cross Multi-head Self Attention) layer, an Add&Norm (residual connection and layer normalization) layer, a Feedforward (feedforward network) layer, and an Add&Norm (residual connection and layer normalization) layer.

[0170] The projected text features are input into the Cross MSA layer in the prompt interaction layer as Query (query), and the keyword text features are input into the Cross MSA layer as Key (index) and Value (value). The similarity between Query and Key is calculated, the calculated similarity is multiplied by the corresponding Value, and Value is weighted to obtain an output. The output and the projected text features are input into the Add&Norm layer, that is, the original data is added as a reference after each data passes through the attention layer, to avoid loss of original data information. The output of the Add&Norm layer is input into the Feedforward layer. In the Feedforward layer, the output of the attention mechanism is projected into a higher dimensional space, which means that the data is stretched into a higher dimension, so the details of the data are magnified, and the attention mechanism pays more attention to the places that need attention. The output of the Feedforward layer and the output of the Add&Norm layer are input into the Add&Norm layer. It needs to be noted that there is an Add&Norm operation after each attention mechanism and Feedforward feedforward network, and finally the fused features of the keyword text features and the projected text features are obtained.

[0171] The text generation model training method provided by the embodiments of the present specification decodes the projected text features after the feature projection layer to obtain the target text sample in the absence of prompt information (keywords); and in the presence of prompt information (keywords), the projected text features and the prompt information are fused to obtain a more accurate target text sample.

[0172] The following describes the embodiments of the present specification in conjunction with the accompanying Figure 3 The text generation model training method is further described. Wherein, Figure 3 A processing process flow diagram of a text generation model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0173] The embodiments of the present specification take a text generation model including a CLIP model with fixed parameters and a GPT-2 model as an example to describe the text generation model training method in detail. In addition, the text generation model further includes a mapping module (Mapping Module), a reverse mapping module (Reverse Mapping Module), a feature projection module (PrefixProjector Module), and a prompt interaction module (Prompt Interaction Module). The text feature extraction of the initial text sample can be realized through the CLIP model, and the target text features can be decoded into the target text sample through the GTP-2 model.

[0174] Step 302: Obtain an input text, input the input text into the text generation model, use the text encoder of the CLIP model to extract the text feature vector of the input text, and use the Mapping Module to map the text features to the image feature space of the CLIP model.

[0175] Wherein, the text encoder can be understood as the text encoding network layer in the above embodiments; the input text can be understood as the initial text sample in the above embodiments, such as the input text "a man riding a motorcycle on the road".

[0176] The mapping module includes a noise sampler and a residual connection normalization; wherein the noise sampler can be understood as the noise sampling network layer in the above embodiments; the residual connection normalization can be understood as the Add&L2 Norm in the above embodiments; the mapping module can be understood as the mapping layer in the above embodiments.

[0177] In practical applications, the text feature vector of the extracted input text is input into the mapping module, the multivariate Gaussian distribution noise is obtained in the noise sampler in the mapping module, the text feature vector and the multivariate Gaussian distribution noise are input into the residual connection normalization in the mapping module, the text feature vector and the multivariate Gaussian distribution noise are fused through the residual connection normalization, and the text feature is mapped to the image feature.

[0178] Specifically, the Mapping Module has a multivariate Gaussian distribution drive, which injects a multivariate Gaussian distribution noise on the text feature to perform cross-modal feature mapping from the text feature space to the image feature space. The core lies in calculating the mean and covariance parameters of the multivariate Gaussian distribution.

[0179] In practical applications, the input text is input into the CLIP model to obtain the text feature vector corresponding to the input text, and the text feature vector is input into the mapping module to obtain the image feature vector corresponding to the text feature vector.

[0180] For specific implementation, reference can be made to the above-mentioned embodiments, which will not be described here again.

[0181] Step 304: input the image feature vector obtained through the Mapping Module into the Reverse Mapping Module, and use the Reverse Mapping Module to map the image feature vector in the CLIP image feature space back to the CLIP text feature space again.

[0182] The reverse mapping module can be understood as the reverse mapping layer in the above-mentioned embodiments, and the reverse mapping module includes a feedforward network layer and a residual connection normalization, wherein the residual connection normalization is the Add&Norm in the above-mentioned embodiments.

[0183] In practical applications, the image feature vector obtained in step 302 is input into the reverse mapping module, in the feedforward network layer, the input image feature vector is processed through a series of linear transformations and activation functions to obtain an intermediate result, the intermediate result is subjected to residual connection and normalization processing with the input image feature, and a target text feature vector corresponding to the image feature is obtained.

[0184] For specific implementation, reference can be made to the above-mentioned embodiments, which will not be described here again.

[0185] Step 306: input the target text feature vector obtained through the Reverse Mapping Module into the Prefix Projector Module for feature projection.

[0186] The feature projection module can be understood as the feature projection layer in the above-mentioned embodiments.

[0187] In actual application, since the CLIP model embedding space is inconsistent with the GPT-2 model embedding space, a Prefix Projector Module can be constructed for feature projection, which is composed of multiple layers of Transformer Encoder (similar in structure to the prompt interaction module). Specifically, the target text feature vector output by step 304 is input into the feature projection module to obtain a projected text feature vector (i.e., the projected text feature in the above embodiment).

[0188] For specific implementation, reference can be made to the above embodiment, which will not be repeated here.

[0189] Step 308: Obtain the prompt information of the input text, and fuse the prompt information and the projected text feature vector obtained through the Prefix Projector Module.

[0190] Specifically, to further endow the text generation model with the ability to process optional prompt information, a Prompt Interaction Module is constructed, which extracts keywords of the input text in the model training stage, or the user can optionally provide additional prompt information in the model application stage. The prompt interaction module applies attention mechanism on the prompt information to realize information fusion.

[0191] Wherein, the prompt information can be understood as the keyword in the above embodiment, and in the case that the input text is "a man riding a motorcycle on the road", the prompt information is "man, motorcycle, etc." The prompt interaction module can be understood as the prompt interaction layer in the above embodiment.

[0192] In actual application, the projected text feature vector output by step 308 is taken as Q (i.e., Query in the above embodiment), and the prompt information is taken as K (i.e., Key in the above embodiment) and V (i.e., Value in the above embodiment) to input into the cross-modal multi-head attention layer in the prompt interaction module. The attention is calculated in the cross-modal multi-head attention layer to better capture the information of different aspects in the input. Q and the output of the cross-modal multi-head attention layer are input into the residual connection normalization layer, and the output of the residual connection normalization layer is input into the feedforward network layer and the residual connection normalization layer of the next level. The output of the feedforward network layer is also input into the residual connection normalization layer of the next level to obtain the fused feature.

[0193] For specific implementation, reference can be made to the above embodiment, which will not be repeated here.

[0194] Step 310: Generate the text description using the GPT-2 model.

[0195] In practical applications, the fusion features obtained in step 308 are input into a text decoder of a GPT-2 model to obtain the target text "a man rides a motorcycle on the road".

[0196] The text decoder can be understood as the text decoding network layer in the above embodiments, and the text description can be understood as the target text in the above embodiments.

[0197] For details, refer to the above embodiments, which will not be described here.

[0198] The text generation model training method provided by the embodiments of the present specification performs mapping from a text feature space to an image feature space through a multivariate Gaussian distribution noise, thereby realizing model training driven by a text corpus in a self-supervised form, greatly reducing the dependence on paired data, and further constructing a prompt interaction module in the text generation model to fuse optional prompt information to improve the performance of the text generation model in generating a text description.

[0199] Referring to Figure 4 , Figure 4 A flowchart of a text generation method provided by an embodiment of the present specification is shown.

[0200] Step 402: Determine an image to be text generated, and input the image to be text generated into a text generation model.

[0201] The image to be text generated can be understood as an image used to obtain an image description.

[0202] Specifically, an image used to obtain an image description input by a user is received and input into the text generation model. In addition, in the case that the user provides additional prompt information, the additional prompt information input by the user is received and input into the text generation model.

[0203] For example, an image input by a user is received, which is a man surfing, and the image is input into the text generation model. In addition, additional prompt information such as "person, surfboard, water surface" provided by the user can be input into the text generation model.

[0204] Step 404: Generate a target text corresponding to the image to be text generated through the text generation model, wherein the text generation model is obtained by training according to the text generation model training method.

[0205] The target text can be understood as a text corresponding to a relevant description of an image target or object in the image to be text generated.

[0206] In practical applications, the text generation model obtained through the above embodiments generates an image description related to the semantics of the image to be text generated.

[0207] In the above example, by using the text generation model, an image of a man surfing is input, and the output text is "a man stands on a surfboard and surfs on the water surface".

[0208] The text generation method provided by the embodiments of the present specification can generate an image description text corresponding to the input image in the text generation model to complete the image description task.

[0209] Referring to Figure 5 , Figure 5 The application process of the text generation method provided by an embodiment of the present specification is shown in the flowchart.

[0210] The image is input into the text generation model, and an image description text corresponding to the image is generated by the text generation model.

[0211] Wherein, the image can be understood as the image to be text generated in the above embodiments, and the image description text can be understood as the target text in the above embodiments.

[0212] In practical applications, an image of a man riding a motorcycle is input into the text generation model, and the output image description text of the text generation model is "a man rides a motorcycle on the road".

[0213] The above is a schematic scheme of the text generation method of the present embodiment. It should be noted that the technical scheme of the text generation method belongs to the same concept as the technical scheme of the text generation model training method described above, and the details of the technical scheme of the text generation method that are not described in detail can be referred to the description of the technical scheme of the text generation model training method.

[0214] Referring to Figure 6 , Figure 6 The flowchart of the image generation model training method provided by an embodiment of the present specification is shown.

[0215] Corresponding to the above text generation model training method, an image generation model training method can be implemented. In practical applications, the text generation model obtained by the text generation model training method can be adjusted to obtain an image generation model, for example, the text decoding network layer in the text generation model can be replaced by an image decoding network layer to determine the target image according to the image features.

[0216] Wherein, the image decoding network layer can be implemented by using a Diffusion model (a deep generative model), an AutoEncoder model (a deep generative model), etc., and is used to generate a target image according to image features.

[0217] Step 602: Determine the initial image sample and extract the image features of the initial image sample.

[0218] Step 604: Determine the multivariate Gaussian distribution noise used to represent the modal differences between text features and image features, and obtain the text features corresponding to the image features based on the multivariate Gaussian distribution noise and the image features of the initial image sample.

[0219] Step 606: Obtain the target image features corresponding to the text features by inverse mapping based on the text features corresponding to the image features.

[0220] Step 608: Determine target image samples based on the target image features, and train an image generation model based on the initial image samples and the target image samples.

[0221] Specifically, the implementation of steps 602-608 can be found in steps 202-208.

[0222] The above is an illustrative scheme of an image generation model training method according to this embodiment. It should be noted that the technical solution of this image generation model training method belongs to the same concept as the technical solution of the text generation model training method described above. For details not described in detail in the technical solution of the image generation model training method, please refer to the description of the technical solution of the text generation model training method described above.

[0223] See Figure 7 , Figure 7 A flowchart illustrating an image generation method provided in one embodiment of this specification is shown.

[0224] Step 702: Determine the text to be generated from the image and input the text to be generated into the image generation model.

[0225] Step 704: Generate a target image corresponding to the text to be generated using the image generation model, wherein the image generation model is trained according to the image generation model training method.

[0226] The above is an illustrative scheme of an image generation method according to this embodiment. It should be noted that the technical solution of this image generation method and the technical solution of the text generation model training method described above belong to the same concept. For details not described in detail in the technical solution of the image generation method, please refer to the description of the technical solution of the text generation model training method described above.

[0227] See Figure 8 , Figure 8 This document illustrates a flowchart of a text generation model training method for cloud-based applications, provided by one embodiment of this specification.

[0228] Step 802: receiving an initial text sample sent by a terminal, and extracting text features of the initial text sample.

[0229] Wherein, the terminal can be understood as various electronic devices, which can be a device with a screen or a device without a screen. It includes but is not limited to a smart phone, a tablet computer, a smart speaker, a smart TV, a PC (Personal Computer), a wearable device, etc.

[0230] Step 804: determining a multivariate Gaussian distribution noise for expressing the modal difference between the text features and the image features, and obtaining image features corresponding to the text features based on the multivariate Gaussian distribution noise and the text features of the initial text sample.

[0231] Step 806: obtaining target text features corresponding to the image features according to the reflection mapping of the text features corresponding to the image features.

[0232] Step 808: determining a target text sample according to the target text features, and training a text generation model according to the initial text sample and the target text sample.

[0233] Step 810: sending the text generation model to the terminal.

[0234] In practical applications, the trained text generation model can be sent to the terminal, so that the text generation model is deployed in the terminal.

[0235] The above is a text generation model training method of the embodiment, which is an illustrative scheme applied to the cloud. It should be noted that the text generation model training method applied to the cloud is the same concept as the technical scheme of the text generation model training method described above. The technical scheme of the text generation model training method applied to the cloud is not described in detail, and the description of the technical scheme of the text generation model training method can be referred to.

[0236] Corresponding to the above method embodiment, the present specification also provides a text generation model training device embodiment, Figure 9 A structural schematic diagram of a text generation model training device provided by an embodiment of the present specification is shown. As shown in the figure, Figure 9 The device includes:

[0237] The feature extraction module 902 is configured to determine an initial text sample, and extract text features of the initial text sample;

[0238] The image feature determination module 904 is configured to determine a multivariate Gaussian distribution noise for expressing a modal difference between a text feature and an image feature, obtain an image feature corresponding to the text feature based on the multivariate Gaussian distribution noise and the text feature of the initial text sample;

[0239] The text feature obtaining module 906 is configured to obtain a target text feature corresponding to the image feature according to the reflection mapping of the image feature corresponding to the text feature;

[0240] The model obtaining module 908 is configured to determine a target text sample according to the target text feature, and train a text generation model according to the initial text sample and the target text sample.

[0241] Optionally, the image feature determination module 904 is further configured to:

[0242] Determine the multivariate Gaussian distribution noise for expressing the modal difference between the text feature and the image feature through a noise sampling network layer in the text generation model.

[0243] Optionally, the image feature determination module 904 is further configured to:

[0244] Determine an image-text pair to be sampled;

[0245] Sample the modal difference between the text feature and the image feature in the image-text pair to be sampled through a noise sampling network layer in the text generation model, and determine the multivariate Gaussian distribution noise for expressing the modal difference between the text feature and the image feature according to a sampling result.

[0246] Optionally, the image feature determination module 904 is further configured to:

[0247] Input the multivariate Gaussian distribution noise and the text feature of the initial text sample into a mapping layer of the text generation model;

[0248] In the mapping layer, map the text feature of the initial text sample to the image feature corresponding to the text feature based on the multivariate Gaussian distribution noise.

[0249] Optionally, the text feature obtaining module 906 is further configured to:

[0250] Input the image feature corresponding to the text feature into a reflection mapping layer in the text generation model;

[0251] The image feature corresponding to the text feature is reflected to a target text feature corresponding to the image feature in the reflection mapping layer, wherein the reflection mapping layer comprises a feedforward network layer, and the reflection mapping layer and the mapping layer are in an asymmetric structure.

[0252] Optionally, the feature extraction module 902 is further configured to:

[0253] The initial text sample is input into a text generation model, and a text feature of the initial text sample is extracted through a text encoding network layer in the text generation model, wherein the text feature of the initial text sample is adapted to the text encoding network layer.

[0254] Optionally, the text feature obtaining module 906 is further configured to:

[0255] The target text feature corresponding to the image feature is obtained through reflection mapping according to the image feature corresponding to the text feature, and the target text feature is adapted to the text encoding network layer.

[0256] Optionally, the model obtaining module 908 is further configured to:

[0257] The target text feature adapted to the text encoding network layer is projected to a projected text feature adapted to the text decoding network layer through a feature projection layer of the text generation model;

[0258] The projected text feature is input into the text decoding network layer, and the projected text feature is decoded to a target text sample in the text decoding network layer.

[0259] Optionally, the model obtaining module 908 is further configured to:

[0260] A keyword of the initial text sample is extracted, and a keyword text feature of the keyword is determined;

[0261] The projected text feature and the keyword text feature are input into a prompt interaction layer of the text generation model;

[0262] The keyword text feature is fused with the projected text feature to obtain a fused feature in the prompt interaction layer;

[0263] The fused feature is input into the text decoding network layer, and the fused feature is decoded to a target text sample in the text decoding network layer.

[0264] Optionally, the feature extraction module 902 is further configured to:

[0265] obtain a first image-text pair, a second image-text pair, a third image-text pair and / or text data from a database; take the text in the first image-text pair, the second image-text pair, the third image-text pair and / or the text data as an initial text sample,

[0266] wherein the quality of the image-text pair in the first image-text pair is higher than the quality of the image-text pair in the second image-text pair, and the quality of the image-text pair in the second image-text pair is higher than the quality of the image-text pair in the third image-text pair.

[0267] The device determines the initial text sample, extracts the text features of the initial text sample, and determines the multivariate Gaussian distribution noise, which can better express the modal difference between the text features and the image features than the independent Gaussian distribution noise; according to the text features of the initial text sample and the multivariate Gaussian distribution noise, the image features corresponding to the text features are obtained, and the target text features corresponding to the image features are obtained according to the image feature reflection mapping, thereby avoiding the problem of modal collapse in the case that the text features do not correctly obtain corresponding image features; the target text sample is determined according to the target text features, and the text generation model is trained according to the initial text sample and the target text sample. The text generation model obtained by training does not require paired data to implement the image description task, and only the text data is used as the training sample to implement the training of the model, thereby saving the data labeling link and reducing the cost.

[0268] The above is a schematic scheme of the text generation model training device of the embodiment. It should be noted that the technical scheme of the text generation model training device belongs to the same concept as the technical scheme of the text generation model training method described above. The technical scheme of the text generation model training device is not described in detail, and can be referred to the description of the technical scheme of the text generation model training method.

[0269] Corresponding to the method embodiments described above, the present specification also provides image generation model training device embodiments, Figure 10 A structural schematic diagram of an image generation model training device provided by an embodiment of the present specification is shown. As shown in the figure, Figure 10 The device comprises:

[0270] The feature extraction module 1002 is configured to determine an initial image sample and extract image features of the initial image sample.

[0271] The text feature determination module 1004 is configured to determine multivariate Gaussian distribution noise for expressing the modal difference between the text features and the image features, and obtain text features corresponding to the image features based on the multivariate Gaussian distribution noise and the image features of the initial image sample.

[0272] The image feature obtaining module 1006 is configured to reflect map the target image feature corresponding to the text feature according to the text feature corresponding to the image feature.

[0273] The model obtaining module 1008 is configured to determine a target image sample according to the target image feature, and train an image generation model according to the initial image sample and the target image sample.

[0274] The device determines an initial image sample, extracts the image feature of the initial image sample, and determines a multivariate Gaussian distribution noise, which can better express the modal difference between the text feature and the image feature than the independent Gaussian distribution noise. According to the image feature of the initial image sample and the multivariate Gaussian distribution noise, the text feature corresponding to the image feature is obtained, and the target image feature corresponding to the text feature is obtained by reflection mapping according to the text feature, which avoids the problem of modal collapse in the case that the image feature does not correctly obtain the corresponding text feature. According to the target image feature, a target image sample is determined, and an image generation model is trained according to the initial image sample and the target image sample. The trained image generation model can realize the image generation task without paired data, and can realize the training of the model only by using image data as training samples, thereby saving the data labeling link and reducing the cost.

[0275] The above is a schematic scheme of an image generation model training device according to an embodiment of the present application. It should be noted that the technical scheme of the image generation model training device belongs to the same concept as the technical scheme of the text generation model training method described above. The technical scheme of the image generation model training device is not described in detail, and the description of the technical scheme of the text generation model training method can be referred to.

[0276] Figure 11 A structural block diagram of a computing device 1100 according to an embodiment of the present application is shown. The components of the computing device 1100 include but are not limited to a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 through a bus 1130, and a database 1150 is used to save data.

[0277] The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1160. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 1140 can include one or more of any type of network interface (for example, a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, Near Field Communication (NFC).

[0278] In one embodiment of the present specification, the above-mentioned components of the computing device 1100 and other components not shown in the Figure 11 may be connected to each other, such as through a bus. It should be understood that Figure 11 The computing device structure diagram shown is for the purpose of example only, and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed.

[0279] The computing device 1100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1100 can also be a mobile or stationary server.

[0280] The processor 1120 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the text generation model training method and the text generation method described above.

[0281] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the text generation model training method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be seen from the description of the technical scheme of the text generation model training method.

[0282] An embodiment of the present specification also provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the text generation model training method and the text generation method.

[0283] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the text generation model training method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be seen from the description of the technical scheme of the text generation model training method.

[0284] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer executes the steps of the text generation model training method and the text generation method.

[0285] The above is a schematic scheme of the computer program of the embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the text generation model training method described above belong to the same concept, and the details of the technical scheme of the computer program that are not described in detail can be seen from the description of the technical scheme of the text generation model training method.

[0286] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in an order other than that described in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0287] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or deletions according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0288] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present specification.

[0289] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0290] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited only by the claims and their entire scope and equivalents.

Claims

1. A method for training a text generation model, comprising: Determine an initial text sample and extract its text features; A multivariate Gaussian distribution noise is determined to represent the modal difference between the text features and the image features. Based on the multivariate Gaussian distribution noise and the text features of the initial text sample, the image features corresponding to the text features are obtained. Based on the image features corresponding to the text features, the target text features corresponding to the image features are obtained by inverse mapping. Target text samples are determined based on the target text features, and a text generation model is trained based on the initial text samples and the target text samples.

2. The text generation model training method according to claim 1, wherein determining the multivariate Gaussian distribution noise used to represent the modal differences between text features and image features includes: By using the noise sampling network layer in the text generation model, a multivariate Gaussian noise distribution is determined to represent the modal differences between text features and image features.

3. The text generation model training method according to claim 2, wherein determining the multivariate Gaussian distribution noise used to represent the modal differences between text features and image features through the noise sampling network layer in the text generation model includes: Identify the image-text pair to be sampled; The noise sampling network layer in the text generation model samples the modal differences between text features and image features in the image-text pair to be sampled, and determines the multivariate Gaussian distribution noise to represent the modal differences between text features and image features based on the sampling results.

4. The text generation model training method according to claim 1, wherein obtaining the image features corresponding to the text features based on the multivariate Gaussian distribution noise and the text features of the initial text sample includes: The multivariate Gaussian noise and the text features of the initial text sample are input into the mapping layer of the text generation model; In the mapping layer, based on the multivariate Gaussian distribution noise, the text features of the initial text sample are mapped to the image features corresponding to the text features.

5. The text generation model training method according to claim 1, wherein obtaining the target text features corresponding to the image features by inverse mapping based on the image features corresponding to the text features includes: The image features corresponding to the text features are input into the inverse mapping layer of the text generation model; In the inverse mapping layer, the image features corresponding to the text features are inversely mapped to the target text features corresponding to the image features. The inverse mapping layer includes a feedforward network layer, and the inverse mapping layer and the mapping layer have an asymmetric structure.

6. The text generation model training method according to claim 1, wherein the text generation model includes a text encoding network layer; Accordingly, the extraction of text features from the initial text sample includes: The initial text sample is input into the text generation model, and the text features of the initial text sample are extracted through the text encoding network layer in the text generation model, wherein the text features of the initial text sample are adapted to the text encoding network layer; Accordingly, the step of obtaining the target text features corresponding to the image features based on the image features corresponding to the text features through inverse mapping includes: Based on the image features corresponding to the text features, the target text features corresponding to the image features and adapted to the text encoding network layer are obtained by inverse mapping.

7. The text generation model training method according to claim 6, wherein the text generation model includes a text decoding network layer; Accordingly, determining the target text sample based on the target text features includes: The target text features adapted to the text encoding network layer are projected into projected text features adapted to the text decoding network layer through the feature projection layer of the text generation model. The projected text features are input into the text decoding network layer, where the projected text features are decoded into target text samples.

8. The text generation model training method according to claim 7, wherein inputting the projected text features into the text decoding network layer, and decoding the projected text features into target text samples in the text decoding network layer, comprises: Extract keywords from the initial text sample and determine the keyword text features of the keywords; The projected text features and the keyword text features are input into the prompt interaction layer of the text generation model; In the prompt interaction layer, the keyword text features and the projected text features are fused to obtain fused features; The fused features are input into the text decoding network layer, where they are decoded into target text samples.

9. The text generation model training method according to claim 1, wherein determining the initial text sample includes: Retrieve the first image-text pair, the second image-text pair, the third image-text pair, and / or text data from the database; use the text from the first image-text pair, the second image-text pair, the third image-text pair, and / or text data as the initial text sample. The quality of the image-text pairs in the first image-text pair is higher than that in the second image-text pair, and the quality of the image-text pairs in the second image-text pair is higher than that in the third image-text pair.

10. A text generation model training method, applied in the cloud, comprising: The initial text sample sent by the receiving terminal is used to extract the text features of the initial text sample; A multivariate Gaussian distribution noise is determined to represent the modal difference between the text features and the image features. Based on the multivariate Gaussian distribution noise and the text features of the initial text sample, the image features corresponding to the text features are obtained. Based on the image features corresponding to the text features, the target text features corresponding to the image features are obtained by inverse mapping. Target text samples are determined based on the target text features, and a text generation model is trained based on the initial text samples and the target text samples. The text generation model is sent to the terminal.

11. A text generation method, comprising: Determine the image to be generated from the text, and input the image to be generated from the text into the text generation model; The text generation model generates target text corresponding to the image to be generated, wherein the text generation model is trained using the text generation model training method according to any one of claims 1-10.

12. A text generation method, applied in the cloud, comprising: The receiving terminal sends an image to be generated from text, and the image to be generated from text is input into the text generation model; The text generation model generates target text corresponding to the image to be generated, wherein the text generation model is trained by the text generation model training method according to any one of claims 1-10; The target text is sent to the terminal.

13. A text generation model training device, comprising: The feature extraction module is configured to determine an initial text sample and extract text features from the initial text sample; The image feature determination module is configured to determine a multivariate Gaussian distribution noise used to represent the modal difference between the text features and the image features, and to obtain the image features corresponding to the text features based on the multivariate Gaussian distribution noise and the text features of the initial text sample. The text feature acquisition module is configured to obtain the target text feature corresponding to the image feature by inverse mapping based on the image feature corresponding to the text feature; The model acquisition module is configured to determine target text samples based on the target text features, and to train a text generation model based on the initial text samples and the target text samples.

14. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the text generation model training method according to any one of claims 1 to 10, or implement the steps of the text generation method according to any one of claims 11 to 12.

Images