Model training method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a model training method, apparatus, device, storage medium and program product, which relate to the field of computer vision and natural language processing in the field of artificial intelligence. The method includes: obtaining image-text pair data as training data; constructing a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, training the image-to-text model based on the image data and the first adversarial network to obtain a first training result; constructing a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, training the text-to-image data based on the text data and the second adversarial network to obtain a second training result; adjusting the model parameters of the image-to-text model based on the first training result, and adjusting the model parameters of the text-to-image model based on the second training result.

Description

Technical Field

[0001] The present disclosure relates to the fields of computer vision and natural language processing in the field of artificial intelligence, and in particular to a model training method, apparatus, device, storage medium, and program product. Background Art

[0002] In recent years, multimodal understanding models have made significant progress and development. Large Vision-Language Models (VLMs) are a new type of model that has gained increasing attention with the rise of Large Language Models (LLMs). These models aim to integrate information from multiple modalities (such as text, images, and videos) to achieve more efficient information understanding and intelligent recognition.

[0003] In addition, generative models have also made rapid progress, such as diffusion models, which can generate new samples that are similar but not identical to the training data by learning the distribution of training data. Generative models can generate images based on input text.

[0004] Currently, expert models are used to assist multimodal understanding models in generating text, or expert models are used to assist generative models in generating images, so as to improve the text generation effect of multimodal understanding models and the image generation effect of generative models.

[0005] However, in the above method, the model generation effect is limited by the upper limit of the effect of the corresponding expert model. If the expert model itself is not effective, the generation effect of the corresponding model will also be poor. Summary of the Invention

[0006] The present disclosure provides a model training method, apparatus, device and storage medium, and on this basis, provides a text generation method, apparatus, device and storage medium, and a video generation method, apparatus, device and storage medium.

[0007] According to a first aspect of the present disclosure, a model training method is provided, comprising:

[0008] Acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data;

[0009] constructing a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, and training the image-to-text model based on the image data and the first adversarial network to obtain a first training result;

[0010] constructing a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and training the text-to-image data based on the text data and the second adversarial network to obtain a second training result;

[0011] The model parameters of the image-to-text model are adjusted based on the first training result, and the model parameters of the text-to-image model are adjusted based on the second training result to obtain a trained image-to-text model and a trained text-to-image model.

[0012] According to a second aspect of the present disclosure, a text generation method is provided, comprising:

[0013] Get video data;

[0014] extracting key image frames from the video data;

[0015] The key image frame is input into the text model for text generation processing to obtain the description text of the key image frame. The image-to-text model is trained based on the method of the first aspect as described above.

[0016] According to a third aspect of the present disclosure, a video generation method is provided, comprising:

[0017] Get the video generation request;

[0018] Generate a request based on the video to obtain multiple text information;

[0019] Inputting the plurality of text information into an image model for image generation processing to obtain a plurality of generated images corresponding to the plurality of text information, wherein the text-to-image model is trained based on the method of the first method described above;

[0020] selecting a target generated image from the plurality of generated images;

[0021] A target video is generated based on the target generation image.

[0022] According to a fourth aspect of the present disclosure, a model training device is provided, comprising:

[0023] An acquisition module, configured to acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data;

[0024] A first training module is configured to construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, and to train the image-to-text model based on the image data and the first adversarial network to obtain a first training result;

[0025] a second training module, configured to construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and train the text-to-image data based on the text data and the second adversarial network to obtain a second training result;

[0026] An adjustment module is used to adjust the model parameters of the image-to-text model based on the first training result, and to adjust the model parameters of the text-to-image model based on the second training result, to obtain a trained image-to-text model and a trained text-to-image model.

[0027] According to a fifth aspect of the present disclosure, there is provided a text generation device, comprising:

[0028] An acquisition module, used to acquire video data;

[0029] An extraction module, configured to extract key image frames from the video data;

[0030] A text generation module is used to input the key image frame into a text model for text generation processing to obtain a description text of the key image frame. The image-to-text model is trained based on the device described in the fourth aspect above.

[0031] According to a sixth aspect of the present disclosure, there is provided a video generating apparatus, comprising:

[0032] A first acquisition module is used to obtain a video generation request;

[0033] A second acquisition module is used to obtain multiple text information based on the video generation request;

[0034] an image generation module, configured to input the plurality of text information into an image model for image generation processing, thereby obtaining a plurality of generated images corresponding to the plurality of text information, wherein the text-to-image model is trained based on the apparatus described in the fourth aspect;

[0035] A selection module, configured to select a target generated image from the plurality of generated images;

[0036] A video generation module is used to generate a target video based on the target generation image.

[0037] According to a seventh aspect of the present disclosure, there is provided an electronic device, including:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method as described in the first aspect above, or to enable the at least one processor to execute the method as described in the second aspect above, or to enable the at least one processor to execute the method as described in the third aspect above.

[0041] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect above, or the computer instructions are used to cause the computer to execute the method according to the second aspect above, or the computer instructions are used to cause the computer to execute the method according to the third aspect above.

[0042] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method described in the first aspect above, or, when the computer program is executed by a processor, the computer program implements the steps of the method described in the second aspect above, or, when the computer program is executed by a processor, the computer program implements the steps of the method described in the third aspect above.

[0043] The technology disclosed herein solves the problem that the model generation effects of multimodal understanding models and generative models are limited by the upper limit of the effects of corresponding expert models, improves the accuracy of text generation by image-to-text models, and makes the images generated by text-to-image models more consistent with the content described in the text.

[0044] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0046] Figure 1 is a scene graph that can implement the disclosed model training method;

[0047] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0048] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0049] Figure 4 is a schematic diagram of a framework of a model training method according to the present disclosure;

[0050] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0051] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0052] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0053] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0054] Figure 9 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0055] Figure 10 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0057] Multimodal understanding models have made significant progress and development in recent years. Multimodal understanding models can generate descriptive text based on input images. However, in applications, the descriptive text generated by the multimodal understanding model may not be consistent with the input image, or may even be completely fictitious or incorrect. This phenomenon is called the hallucination problem. Currently, expert models such as SAM (Segment Anything Model) and OCR (Optical Character Recognition) are used to assist multimodal understanding models. This helps them better understand the structure and content of images, thereby generating more accurate descriptive text and alleviating the hallucination problem.

[0058] At the same time, generative models have also made rapid progress, such as diffusion models, which can generate images based on input text. However, the current generative models are not capable of generating images based on text. The GPT-4 (Generative Pre-trained Transformer 4) model is currently used to describe the input text in a fine-grained manner to obtain detailed descriptive text. This detailed descriptive text is then converted into an embedding vector (embedding). Then, ControlNet (control network) is used to guide the generation of images that meet the requirements based on the embedding vector and other control signals. Through the guidance of ControlNet, the generated images are ensured to be not only of high quality but also highly consistent with the fine-grained description of the input.

[0059] In the above scheme, the multimodal understanding model is an image-to-text model that generates text from images, and the generation model is a text-to-image model that generates images from text. The generation performance of the image-to-text model and the text-to-image model is limited by the upper limit of the performance of the corresponding expert model. If the expert model itself is not effective, the generation performance of the image-to-text model and the text-to-image model will also be poor. Therefore, there is an urgent need to provide an image-to-text model that can better understand the structure and content of images and generate accurate descriptive text, and there is an urgent need to provide a text-to-image model that can generate images that are more visually consistent with the text description, and the generation performance of the model is not limited by the performance of the expert model.

[0060] The present disclosure provides a model training method, which is applied to the fields of computer vision and natural language processing in the field of artificial intelligence. The image-to-text model trained using the model training method generates descriptive text that is closer to the meaning expressed by the image, and the text-to-image model trained using the model training method generates an image that is more consistent with the content described in the text.

[0061] It should be noted that the image data, text data, and video data in this disclosure are all sourced from public datasets. The collection, storage, use, processing, transmission, provision, and disclosure of the information involved in this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

[0062] The application scenario of the present application is specifically a scenario for model training, in which the training data can be processed in the server to train the image-to-text model and the text-to-image model. In addition, the model training can also be performed in the terminal device, which can be a desktop computer, laptop computer, tablet computer and smart phone, etc.

[0063] Figure 1 It is a scene graph that can implement the model training method disclosed in this disclosure. Figure 1As shown, the training data can be stored in the server 101, and the terminal device 102 can obtain the training data from the server 101, and train the image-to-text model and the text-to-image model based on the training data. The trained image-to-text model and text-to-image model can then be stored in the server 103. The image-to-text model and text-to-image model in the server 103 can be called by different entities or systems.

[0064] The following specific embodiments describe in detail the technical solutions of the present disclosure and how the technical solutions of the present disclosure solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. To enable readers to more deeply understand the implementation principles of the present disclosure, the following embodiments of the present disclosure will be described in conjunction with the accompanying drawings.

[0065] Figure 2 is a schematic diagram according to the first embodiment of the present disclosure, as shown in Figure 2 As shown, the model training method of this embodiment specifically includes:

[0066] S201. Acquire image-text pair data as training data, where the image-text pair data includes image data and text data matching the image data.

[0067] Image-text pair data consists of image data and text data describing the image data.

[0068] In the embodiment of the present disclosure, image-text pair data can be obtained from different databases and used as training data for model training.

[0069] S202. Build a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, train the image-to-text model based on the image data and the first adversarial network, and obtain a first training result.

[0070] In the embodiment of the present disclosure, two models need to be trained, one is a text-to-image model, which is used to generate descriptive text based on the input image, and the other is an image-to-text model, which is used to generate a corresponding image based on the input descriptive text.

[0071] In the process of training the image-to-text model, the image-to-text model to be trained is used as a generator, and the text-to-image model to be trained is used as a discriminator to construct a first adversarial network, and the model is trained based on the image data and the first adversarial network.

[0072] S203: construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and train the text-to-image data based on the text data and the second adversarial network to obtain a second training result.

[0073] In the process of training the text-to-image model, the text-to-image model to be trained is used as a generator, and the image-to-text model to be trained is used as a discriminator to construct a second adversarial network, and the model is trained based on the text data and the second adversarial network.

[0074] It can be understood that the first adversarial network or the second adversarial network of the embodiment of the present disclosure is different from the traditional adversarial network. In the traditional adversarial network, the generator is usually responsible for generating data, and the discriminator is responsible for distinguishing the generated data from the real data. In the first adversarial network of the embodiment of the present disclosure, the generator is responsible for generating descriptive text of the image, and the discriminator is responsible for generating the image. In the second adversarial network of the embodiment of the present disclosure, the generator is responsible for generating the image, and the discriminator is responsible for generating descriptive text of the image.

[0075] The adversarial network of the embodiment of the present disclosure is composed of a generator and a discriminator. When training the image-to-text model, the model parameters of the text-to-image model are fixed, and the model parameters of the image-to-text model are trained and updated; when training the text-to-image model, the model parameters of the image-to-text model are fixed, and the model parameters of the text-to-image model are trained and updated. The image-to-text model and the text-to-image model serve as generators and discriminators for each other, thereby achieving joint optimization of the image-to-text model and the text-to-image model.

[0076] S204: Adjust the model parameters of the image-to-text model based on the first training result, and adjust the model parameters of the text-to-image model based on the second training result to obtain a trained image-to-text model and a trained text-to-image model.

[0077] In an embodiment of the present disclosure, the model parameters of the image-to-text model can be adjusted according to the first training result, and the model parameters of the text-to-image model can be adjusted according to the second training result to obtain a trained image-to-text model and a trained text-to-image model.

[0078] The model training method provided by the embodiments of the present disclosure trains the image-to-text model and the text-to-image model together. During the training process, the image-to-text model and the text-to-image model serve as generators and discriminators for each other. The image-to-text model gradually learns to generate descriptive text that is closer to the image content, and the text-to-image model gradually learns to generate images that are more consistent with the text description content. The image-to-text model trained by this method can better understand the structure and content of the image and generate accurate descriptive text. The text-to-image model trained by this method can generate images that are more visually consistent with the text description. Moreover, the trained image-to-text model and text-to-image model do not rely on other expert models, and their generation effects are not limited by the effects of the expert models.

[0079] Figure 3 is a schematic diagram according to the second embodiment of the present disclosure, as shown in Figure 3 As shown, this embodiment Figure 2 Based on the embodiment, the model training method is described in detail. The model training method of this embodiment specifically includes:

[0080] S301: Acquire image-text pair data as training data, where the image-text pair data includes image data and text data matching the image data.

[0081] Among them, a generation process of image-text pair data may include: obtaining a sample image, generating descriptive text for the sample image, rewriting the preliminary descriptive text to obtain detailed descriptive text, the detailed descriptive text contains richer details and contextual information, and thus obtaining preliminary first image-text pair data (sample image-detailed descriptive text), manually cleaning and optimizing the preliminary first image-text pair data to obtain high-quality image-text pair data that can be used as training data.

[0082] Another generation process of image-text pair data may include: obtaining sample descriptive text, which may be extracted from a novel text. Specifically, a multimodal understanding model may be used to segment the novel text into scene descriptions to obtain descriptive texts corresponding to different shots or scenes. Sample descriptive text may be selected from these descriptive texts. The sample descriptive text may be expanded to obtain expanded text. A pre-trained model may then be used to generate a corresponding text image based on the expanded text, thereby obtaining preliminary second image-text pair data (sample descriptive text-text image). The preliminary second image-text pair data may be manually cleaned and optimized to obtain high-quality image-text pair data that can be used as training data.

[0083] The image-text pair data in the embodiment of the present disclosure may be part or all of the cleaned first image-text pair data and the cleaned second image-text pair data.

[0084] S302: construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, train the image-to-text model based on the image data and the first adversarial network, and obtain a first training result.

[0085] In one possible implementation, training the image-to-text model based on the image data and the first adversarial network to obtain a first training result may specifically include:

[0086] Inputting the image data into the image-to-text model to obtain first generated text data;

[0087] Performing expansion processing on the first generated text data to obtain second generated text data;

[0088] inputting the second generated text data into a text-to-image model to obtain first generated image data;

[0089] The first generated text data and the first generated image data are used as a first training result obtained through training.

[0090] The first generated text data is expanded, specifically by inputting the first generated text data into a pre-trained large language model to obtain expanded second generated text data.

[0091] The process of training an image-to-text model involves a data processing process of generating text based on an image, rewriting the text, and then generating an image based on the rewritten text. The first generated text data and first generated image data obtained during this data processing process serve as the first training result. In this embodiment, both the first generated text data and the first generated image data are used as the first training result in the subsequent adjustment of model parameters of the image-to-text model. Adjusting model parameters based on multiple types of data can improve the performance of the trained model.

[0092] S303: Build a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and train the text-to-image data based on the text data and the second adversarial network to obtain a second training result.

[0093] In a possible implementation, training the text-to-image data based on the text data and the second adversarial network to obtain a second training result may specifically include:

[0094] Performing expansion processing on the text data to obtain third generated text data;

[0095] inputting the third generated text data into a text-to-image model to obtain second generated image data;

[0096] inputting the second generated image data into an image-to-text model to obtain fourth generated text data;

[0097] The second generated image data and the fourth generated text data are used as a second training result obtained through training.

[0098] The text data is expanded, specifically by inputting the text data into a pre-trained large language model to obtain expanded third generated text data.

[0099] The process of training a text-to-image model involves a data processing process of rewriting text, generating an image based on the rewritten text, and then generating text based on the image. The second generated image data and the fourth generated text data obtained in this data processing process are used as the second training result. In this embodiment, both the second generated image data and the fourth generated text data are used as the second training result to subsequently adjust the model parameters of the text-to-image model. Adjusting the model parameters based on multiple types of data can improve the performance of the trained model.

[0100] S304. Calculate a model loss value of the image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value.

[0101] In an embodiment of the present disclosure, the model loss value of the image-to-text model is calculated based on the first generated text data and the first generated image data.

[0102] In one possible implementation, calculating a model loss value of an image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value may specifically include:

[0103] Determining a text difference between the text data and the first generated text data to obtain first difference information;

[0104] determining a pixel difference between the image data and the first generated image data to obtain second difference information;

[0105] Extracting image features of the image data to obtain first image features;

[0106] Extracting text features of the first generated text data to obtain first text features;

[0107] determining a feature difference between the first image feature and the first text feature to obtain third difference information;

[0108] Based on the first difference information, the second difference information, and the third difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

[0109] The model loss value of the image-to-text model, i.e., the first loss value, can be determined based on the first difference information, the second difference information, and the third difference information, wherein the first difference information is used to measure the text difference between the original text data and the generated first generated text data, such as comparing character differences, word and phrase differences, etc.; the second difference information is used to measure the pixel-level difference between the original image data and the generated first generated image data, such as comparing pixel value differences; and the third difference information is used to measure the feature features between the image data and the generated first generated text data.

[0110] Exemplarily, a multimodal twin-tower model can be used to extract image features and text features. The multimodal twin-tower model can extract features of different data modalities (such as text, images, etc.) through two independent neural networks (usually referred to as "towers"), and then fuse or compare the feature representations of the two towers at the interoperability layer to complete a specific task. Specifically, the image data is input into the multimodal twin-tower model, and after being processed by the image tower, an image feature vector (i.e., a first image feature) is obtained. The first generated text data is input into the multimodal twin-tower model, and after being processed by the text tower, a text feature vector (i.e., a first text feature) is obtained. The image feature vector and the text feature vector can be compared by cosine similarity to obtain third difference information.

[0111] In this embodiment, by integrating multiple types of difference information to calculate the loss value of the model, the model performance can be accurately evaluated and the model optimization can be guided.

[0112] In one possible implementation, calculating a model loss value of the image-to-text model based on the first difference information, the second difference information, and the third difference information to obtain a first loss value may specifically include:

[0113] extracting the image features of the image data again to obtain a second image feature;

[0114] extracting image features of the first generated image data to obtain third image features;

[0115] determining a feature difference between the second image feature and the third image feature to obtain fourth difference information;

[0116] Based on the first difference information, the second difference information, the third difference information, and the fourth difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

[0117] In this embodiment, based on the first difference information, the second difference information, and the third difference information, the model loss value of the image-to-text model may be determined in combination with the fourth difference information. The fourth difference information is used to measure the feature difference between the image data and the generated first generated image data.

[0118] Exemplarily, the image features of the image data can be extracted again. This time, the image features can be extracted using another model, such as a ResNet (Residual Network) model. The ResNet model can be used to extract the image features of the image data and the first generated image data to obtain the corresponding image feature vectors. By comparing these two image feature vectors, the fourth difference information is obtained.

[0119] The loss function (LossA) of the image-to-text model can be expressed as follows:

[0120] LossA=w1×cross_entropy loss(T,Tp1)+w2×L2 loss(I,Ip1)+w3×cosinesimilarity loss(f1(I),f1(Ip1))+w4×cosine similarity loss(f2(I),f2(Tp1));

[0121] Among them, w1, w2, w3, and w4 are weights between 0 and 1; T represents the original text data; Tp1 represents the first generated text data; cross-entropy loss() represents the cross-entropy loss function, which measures the text difference between the original text data and the generated first generated text data; I represents the original image data; Ip1 represents the first generated image data; L2loss() represents the calculation of mean square error, which measures the pixel-level difference between the original image data and the generated first generated image data; f1() represents the use of the ResNet model to extract image features; cosine similarity loss() represents the calculation of cosine similarity to measure feature differences; f2() represents the use of the multimodal dual-tower model to extract image features or text features.

[0122] S305. Based on the second generated image data and the fourth generated text data, calculate the model loss value of the text-to-image model to obtain a second loss value.

[0123] In an embodiment of the present disclosure, the model loss value of the text-to-image model is calculated based on the second generated image data and the fourth generated text data.

[0124] In one possible implementation, calculating a model loss value of the text-to-image model based on the second generated image data and the fourth generated text data to obtain a second loss value may specifically include:

[0125] determining a text difference between the text data and the fourth generated text data to obtain fifth difference information;

[0126] determining a pixel difference between the image data and the second generated image data to obtain sixth difference information;

[0127] Extracting image features of the image data and extracting text features of the fourth generated text data to obtain fourth image features and second text features; determining a feature difference between the fourth image features and the second text features to obtain seventh difference information;

[0128] Extracting image features of the image data again to obtain a fifth image feature; extracting image features of the second generated image data to obtain a sixth image feature; determining a feature difference between the fifth image feature and the sixth image feature to obtain eighth difference information;

[0129] Based on the fifth difference information, the sixth difference information, the seventh difference information, and the eighth difference information, a model loss value of the text-to-image model is calculated to obtain a second loss value.

[0130] The loss function (LossB) of the text-to-image model can be shown as follows:

[0131] LossB=w1×cross_entropy loss(T,Tp2)+w2×L2 loss(I,Ip2)+w3×cosinesimilarity loss(f1(I),f1(Ip2))+w4×cosine similarity loss(f2(T),f2(Ip2));

[0132] Where w1, w2, w3, and w4 are weights between 0 and 1. T represents the original text data; Tp2 represents the fourth generated text data; cross-entropy loss() represents the cross-entropy loss function, which measures the textual difference between the original text data and the fourth generated text data; I represents the original image data; Ip2 represents the second generated image data; L2loss() represents the calculation of mean squared error, which measures the pixel-level difference between the original image data and the second generated image data; f1() represents the use of the ResNet model to extract image features; cosine similarity loss() represents the calculation of cosine similarity, which measures feature differences; and f2() represents the use of the multimodal dual-tower model to extract image features or text features.

[0133] S306. Adjust the model parameters of the image-to-text model based on the first loss value, and adjust the model parameters of the text-to-image model based on the second loss value to obtain a trained image-to-text model and a trained text-to-image model.

[0134] In a possible implementation, the model training method of the embodiment of the present disclosure may further include:

[0135] Alternately train image-to-text models and text-to-image models;

[0136] For the alternately trained image-to-text model and text-to-image model, if the first loss value corresponding to the image-to-text model meets the preset first loss condition, and the second loss value corresponding to the text-to-image model meets the preset second loss condition, the training is stopped to obtain the target image-to-text model and the target text-to-image model.

[0137] In this embodiment, the image-to-text model and the text-to-image model can be trained alternately, that is, after training the image-to-text model, the text-to-image model is trained, and after training the text-to-image model, the image-to-text model is trained again, and the cycle is alternated until the training stop condition is met.

[0138] In a specific implementation, the image-to-text model is trained, the current image-to-text model is used as a generator, the current text-to-image model is used as a discriminator, and the first loss value of this training is calculated. If the first loss value meets the preset first loss condition, the training of the subsequent text-to-image model is started directly; if the first loss value does not meet the preset first loss condition, the model parameters of the image-to-text model are adjusted based on the first loss value to obtain the trained image-to-text model, and then the training of the text-to-image model is started.

[0139] When training the text-to-image model, the current text-to-image model is used as the generator, and the current image-to-text model is used as the discriminator. The second loss value of this training is calculated. If the second loss value meets the preset second loss condition, it is necessary to determine whether the first loss value calculated in the previous training image-to-text model process meets the preset first loss condition. If so, it can be determined that the training stop condition is met, and the training is stopped to output the current image-to-text model and the current text-to-image model to obtain the target image-to-text model and the target text-to-image model; if the first loss value calculated in the previous training image-to-text model process does not meet the preset first loss condition, then continue training and perform subsequent image-to-text model training; if the second loss value does not meet the preset second loss condition, then adjust the model parameters of the text-to-image model based on the second loss value to obtain the trained text-to-image model, and then start training the image-to-text model.

[0140] It should be noted that in the embodiments of the present disclosure, there is no specific limitation on the order of training the image-to-text model and the text-to-image model, that is, the image-to-text model can be trained first, or the text-to-image model can be trained first.

[0141] In this embodiment, the image-to-text model and the text-to-image model form a dynamic "game" during the alternating training process. This game mechanism encourages each model to improve its own capabilities in the continuous competition.

[0142] Reference Figure 4 As shown, it is a framework diagram of the model training method according to the present invention. In the process of training the image-to-text model, the image-to-text model is used as the generator to generate the corresponding text based on the image, and the text-to-image model is used as the discriminator in this process; in the process of training the text-to-image model, the text-to-image model is used as the generator to generate the corresponding image based on the text, and the image-to-text model is used as the discriminator in this process.

[0143] The model training method provided by the embodiments of the present disclosure trains the image-to-text model and the text-to-image model together. During the training process, the image-to-text model and the text-to-image model serve as generators and discriminators for each other. The image-to-text model gradually learns to generate descriptive text that is closer to the image content, and the text-to-image model gradually learns to generate images that are more consistent with the text description content. The image-to-text model trained by this method can better understand the structure and content of the image and generate accurate descriptive text. The text-to-image model trained by this method can generate images that are more visually consistent with the text description. Moreover, the trained image-to-text model and text-to-image model do not rely on other expert models, and their generation effects are not limited by the effects of the expert models.

[0144] The above examples describe the training process for image-to-text and text-to-image models. The following examples illustrate the application of these models. For example, the image-to-text model can be applied to scenarios requiring a textual description of video content, while the text-to-image model can be applied to scenarios where a corresponding video is generated based on the textual content.

[0145] Figure 5 is a schematic diagram according to the third embodiment of the present disclosure, as shown in Figure 5 As shown, the text generation method of this embodiment specifically includes:

[0146] S501: Obtain video data.

[0147] In the embodiments of the present disclosure, various types of video data may be acquired, such as film and television video data, teaching video data, live video data, and game video data.

[0148] S502: Extract key image frames from video data.

[0149] After acquiring the video data, key image frames can be extracted from the video data. The key image frames contain key picture information. By extracting the key image frames and analyzing only the key image frames, the amount of data required for calculation can be reduced.

[0150] S503: Input the key image frame into the text model for text generation processing to obtain the description text of the key image frame. The image-to-text model is trained based on the model training method of the above embodiment.

[0151] In an embodiment of the present disclosure, a key image frame may be input into a pre-trained image-to-text model to obtain a description text of the key image frame.

[0152] The text generation method provided by the embodiment of the present disclosure can quickly generate description text for video data, so as to facilitate rapid understanding of the video content.

[0153] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure, as shown in Figure 6 As shown, the video generation method of this embodiment specifically includes:

[0154] S601: Obtain a video generation request.

[0155] The user performs a video generation operation on the terminal device, such as clicking a video generation button. In response to the video generation operation, a video generation request is generated.

[0156] S602: Generate a request based on the video to obtain multiple text information.

[0157] In the disclosed embodiment, video generation information can be parsed from a video generation request. The video generation information indicates requirements that the generated video must meet. The video generation information may include style information and theme information of the generated video. Subsequently, multiple text information can be obtained based on the video generation information. The text information may be pre-stored in a designated database.

[0158] S603: Input multiple text information into the image model for image generation processing to obtain multiple generated images corresponding to the multiple text information. The text-to-image model is trained based on the model training method of the above embodiment.

[0159] By inputting multiple text information into the image model, multiple generated images corresponding to the multiple text information can be quickly obtained.

[0160] S604: Select a target generated image from the multiple generated images.

[0161] S605: Generate a target video based on the target generation image.

[0162] In the embodiment of the present disclosure, a target generated image may be selected from a plurality of generated images, and the target generated image may be used for video generation.

[0163] The video generation method provided by the embodiments of the present disclosure enables rapid generation of video content.

[0164] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure, as shown in Figure 7 As shown, the model training device 700 of this embodiment specifically includes:

[0165] An acquisition module 701 is configured to acquire image-text pair data as training data, where the image-text pair data includes image data and text data that matches the image data.

[0166] A first training module 702 is configured to construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, and train the image-to-text model based on the image data and the first adversarial network to obtain a first training result;

[0167] A second training module 703 is configured to construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and train the text-to-image data based on the text data and the second adversarial network to obtain a second training result;

[0168] The adjustment module 704 is used to adjust the model parameters of the image-to-text model based on the first training result and adjust the model parameters of the text-to-image model based on the second training result to obtain a trained image-to-text model and a trained text-to-image model.

[0169] In a possible implementation, the first training module includes:

[0170] A first input submodule, configured to input image data into an image-to-text model to obtain first generated text data;

[0171] A first expansion submodule, configured to expand the first generated text data to obtain second generated text data;

[0172] a second input submodule, configured to input the second generated text data into the text-to-image model to obtain the first generated image data;

[0173] The first determining submodule is configured to use the first generated text data and the first generated image data as a first training result obtained through training.

[0174] In a possible implementation, the second training module includes:

[0175] A second expansion submodule is used to expand the text data to obtain third generated text data;

[0176] a third input submodule, configured to input the third generated text data into the text-to-image model to obtain second generated image data;

[0177] a fourth input submodule, configured to input the second generated image data into the image-to-text model to obtain fourth generated text data;

[0178] The second determining submodule is configured to use the second generated image data and the fourth generated text data as a second training result obtained through training.

[0179] In a possible implementation, the adjustment module includes:

[0180] A first calculation submodule is configured to calculate a model loss value of an image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value;

[0181] A second calculation submodule is used to calculate a model loss value of the text-to-image model based on the second generated image data and the fourth generated text data to obtain a second loss value;

[0182] An adjustment submodule is used to adjust the model parameters of the image-to-text model based on the first loss value, and to adjust the model parameters of the text-to-image model based on the second loss value, to obtain a trained image-to-text model and a trained text-to-image model.

[0183] In a possible implementation, the method further includes:

[0184] Alternating training module, used to alternately train image-to-text model and text-to-image model;

[0185] The training stop judgment module is used to stop training the alternately trained image-to-text model and text-to-image model to obtain the target image-to-text model and target text-to-image model if the first loss value corresponding to the image-to-text model meets the preset first loss condition and the second loss value corresponding to the text-to-image model meets the preset second loss condition.

[0186] In a possible implementation, the first computing submodule includes:

[0187] a first determining unit, configured to determine a text difference between the text data and the first generated text data, and obtain first difference information;

[0188] a second determining unit, configured to determine a pixel difference between the image data and the first generated image data to obtain second difference information;

[0189] A first extraction unit is used to extract image features of the image data to obtain first image features;

[0190] a second extraction unit, configured to extract text features of the first generated text data to obtain first text features;

[0191] a third determining unit, configured to determine a feature difference between the first image feature and the first text feature to obtain third difference information;

[0192] A calculation unit is used to calculate a model loss value of the image-to-text model based on the first difference information, the second difference information, and the third difference information to obtain a first loss value.

[0193] In a possible implementation, the computing unit includes:

[0194] A first extraction subunit is used to extract image features of the image data again to obtain second image features;

[0195] a second extraction subunit, configured to extract image features of the first generated image data to obtain third image features;

[0196] a determining subunit, configured to determine a feature difference between the second image feature and the third image feature to obtain fourth difference information;

[0197] The calculation subunit is used to calculate the model loss value of the image-to-text model based on the first difference information, the second difference information, the third difference information and the fourth difference information to obtain a first loss value.

[0198] The model training device provided in the embodiment of the present disclosure can execute the model training method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0199] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure, as shown in Figure 8 As shown, the text generation device 800 of this embodiment specifically includes:

[0200] Acquisition module 801, used to acquire video data;

[0201] Extraction module 802, for extracting key image frames from video data;

[0202] The text generation module 803 is used to input the key image frame into the text model for text generation processing to obtain the description text of the key image frame. The image-to-text model is trained based on the model training device of the above embodiment.

[0203] The text generation device provided in the embodiment of the present disclosure can execute the text generation method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0204] Figure 9 is a schematic diagram according to the seventh embodiment of the present disclosure, as shown in Figure 9 As shown, the video generating device 900 of this embodiment specifically includes:

[0205] A first acquisition module 901 is used to obtain a video generation request;

[0206] A second acquisition module 902 is configured to acquire multiple text information based on the video generation request;

[0207] An image generation module 903 is configured to input multiple text information into an image model for image generation processing to obtain multiple generated images corresponding to the multiple text information. The text-to-image model is trained based on the model training device of the above embodiment;

[0208] A selection module 904 is configured to select a target generated image from a plurality of generated images;

[0209] The video generation module 905 is configured to generate a target video based on the target generation image.

[0210] The video generation device provided in the embodiment of the present disclosure can execute the video generation method provided in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0211] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0212] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the method provided in any of the above embodiments.

[0213] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0214] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0215] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0216] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the model training method or the text generation method or the video generation method. For example, in some embodiments, the model training method or the text generation method or the video generation method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the model training method or the text generation method or the video generation method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the model training method or the text generation method or the video generation method in any other appropriate manner (e.g., by means of firmware).

[0217] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0218] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0219] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0220] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0221] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0222] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0223] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0224] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A model training method, characterized in that: include: Acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data; Constructing a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator; Inputting the image data into the text model to obtain first generated text data; performing expansion processing on the first generated text data to obtain second generated text data; inputting the second generated text data into the text-to-image model to obtain first generated image data; using the first generated text data and the first generated image data as a first training result obtained through training; A second adversarial network is constructed using the text-to-image model as a generator and the image-to-text model as a discriminator. performing expansion processing on the text data to obtain third generated text data; inputting the third generated text data into the text-to-image model to obtain second generated image data; inputting the second generated image data into the image-to-text model to obtain fourth generated text data; using the second generated image data and the fourth generated text data as a second training result obtained through training; The model parameters of the image-to-text model are adjusted based on the first training result, and the model parameters of the text-to-image model are adjusted based on the second training result to obtain a trained image-to-text model and a trained text-to-image model.

2. The method according to claim 1, characterized in that Also includes: Alternatingly training the image-to-text model and the text-to-image model; For the alternately trained image-to-text model and the text-to-image model, if the first loss value corresponding to the image-to-text model satisfies the preset first loss condition, and the second loss value corresponding to the text-to-image model satisfies the preset second loss condition, the training is stopped to obtain the target image-to-text model and the target text-to-image model.

3. The method according to claim 1, characterized in that The adjusting the model parameters of the image-to-text model based on the first training result, and adjusting the model parameters of the text-to-image model based on the second training result to obtain a trained image-to-text model and a trained text-to-image model, includes: Calculating a model loss value of the image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value; Calculating a model loss value of the text-to-image model based on the second generated image data and the fourth generated text data to obtain a second loss value; The model parameters of the image-to-text model are adjusted based on the first loss value, and the model parameters of the text-to-image model are adjusted based on the second loss value to obtain a trained image-to-text model and a trained text-to-image model.

4. The method according to claim 3, characterized in that The step of calculating a model loss value of the image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value includes: determining a text difference between the text data and the first generated text data to obtain first difference information; determining a pixel difference between the image data and the first generated image data to obtain second difference information; Extracting image features of the image data to obtain first image features; extracting text features of the first generated text data to obtain first text features; determining a feature difference between the first image feature and the first text feature to obtain third difference information; Based on the first difference information, the second difference information, and the third difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

5. The method according to claim 4, characterized in that The calculating the model loss value of the image-to-text model based on the first difference information, the second difference information, and the third difference information to obtain a first loss value includes: extracting the image feature of the image data again to obtain a second image feature; extracting image features of the first generated image data to obtain third image features; determining a feature difference between the second image feature and the third image feature to obtain fourth difference information; Based on the first difference information, the second difference information, the third difference information and the fourth difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

6. A text generation method, characterized in that: include: Get video data; extracting key image frames from the video data; The key image frame is input into the text model for text generation processing to obtain a description text of the key image frame, and the image-to-text model is trained based on the method according to any one of claims 1-5.

7. A video generation method, characterized in that: include: Get the video generation request; Generate a request based on the video to obtain multiple text information; Inputting the multiple text information into an image model for image generation processing to obtain multiple generated images corresponding to the multiple text information, wherein the text-to-image model is trained based on the method according to any one of claims 1 to 5; selecting a target generated image from the plurality of generated images; A target video is generated based on the target generation image.

8. A model training device, characterized in that: include: An acquisition module, configured to acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data; A first training module is configured to construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator; Inputting the image data into the image-to-text model to obtain first generated text data; performing expansion processing on the first generated text data to obtain second generated text data; inputting the second generated text data into the text-to-image model to obtain first generated image data; and using the first generated text data and the first generated image data as a first training result obtained by training; A second training module is configured to construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator; perform expansion processing on the text data to obtain third generated text data; input the third generated text data into the text-to-image model to obtain second generated image data; input the second generated image data into the image-to-text model to obtain fourth generated text data; and use the second generated image data and the fourth generated text data as a second training result obtained through training; An adjustment module is used to adjust the model parameters of the image-to-text model based on the first training result, and to adjust the model parameters of the text-to-image model based on the second training result, to obtain a trained image-to-text model and a trained text-to-image model.

9. A text generation device, characterized in that: include: An acquisition module, used to acquire video data; An extraction module, configured to extract key image frames from the video data; A text generation module is used to input the key image frame into a text model for text generation processing to obtain a description text of the key image frame, and the image-to-text model is trained based on the device according to claim 8.

10. A video generating device, characterized in that: include: A first acquisition module is used to obtain a video generation request; A second acquisition module is used to obtain multiple text information based on the video generation request; an image generation module, configured to input the plurality of text information into an image model for image generation processing to obtain a plurality of generated images corresponding to the plurality of text information, wherein the text-to-image model is trained based on the apparatus according to claim 8; A selection module, configured to select a target generated image from the plurality of generated images; A video generation module is used to generate a target video based on the target generation image.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5, or to enable the at least one processor to perform the method of claim 6, or to enable the at least one processor to perform the method of claim 7.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5, or the computer instructions are used to cause the computer to execute the method according to claim 6, or the computer instructions are used to cause the computer to execute the method according to claim 7.

13. A computer program product, characterized in that The method comprises a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to claim 6, or implements the steps of the method according to claim 7.

Images