Model training method and device, equipment, storage medium and program product

Abstract

The invention provides a model training method and device, equipment, a storage medium and a program product, and relates to the field of computer vision and natural language processing in the field of artificial intelligence, and the method comprises the steps: obtaining image text pair data as training data; constructing a first adversarial network by taking the to-be-trained image-to-text model as a generator and taking the to-be-trained text-to-image model as a discriminator, and training the image-to-text model based on the image data and the first adversarial network to obtain a first training result; constructing a second adversarial network by taking the text-to-image model as a generator and the image-to-text model as a discriminator, and training the text-to-image data based on the text data and the second adversarial network to obtain a second training result; and adjusting model parameters of the image-to-text model based on the first training result, and adjusting model parameters of the text-to-image model based on the second training result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision and natural language processing in the field of artificial intelligence, and in particular to a model training method, device, equipment, storage medium and program product. Background Art

[0002] In recent years, multimodal understanding models have made significant progress and development. The multimodal understanding model (Large Vision-Language Model, VLM) is a new type of model that has gradually attracted attention with the rise of large language models (LLM). This type of model aims to integrate information from multiple modalities (such as text, images, videos, etc.) to achieve more efficient information understanding and intelligent recognition.

[0003] In addition, generative models have also made rapid progress, such as diffusion models, which can generate new samples that are similar but not exactly the same as the training data by learning the distribution of training data. Generative models can generate images based on input text.

[0004] Currently, expert models are used to assist multimodal understanding models in generating text, or expert models are used to assist generative models in generating images, so as to improve the text generation effect of multimodal understanding models and improve the image generation effect of generative models.

[0005] However, in the above method, the model generation effect is limited by the upper limit of the effect of the corresponding expert model. If the expert model itself is not effective, the generation effect of the corresponding model will also be poor. Summary of the invention

[0006] The present disclosure provides a model training method, apparatus, device and storage medium, and on this basis, provides a text generation method, apparatus, device and storage medium, and a video generation method, apparatus, device and storage medium.

[0007] According to a first aspect of the present disclosure, a model training method is provided, comprising:

[0008] Acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data;

[0009] Using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator to construct a first adversarial network, and training the image-to-text model based on the image data and the first adversarial network to obtain a first training result;

[0010] Using the text-to-image model as a generator and the image-to-text model as a discriminator to construct a second adversarial network, and training the text-to-image data based on the text data and the second adversarial network to obtain a second training result;

[0011] The model parameters of the image-to-text model are adjusted based on the first training result, and the model parameters of the text-to-image model are adjusted based on the second training result to obtain a trained image-to-text model and a trained text-to-image model.

[0012] According to a second aspect of the present disclosure, there is provided a text generation method, comprising:

[0013] Get video data;

[0014] extracting key image frames from the video data;

[0015] The key image frame is input into the image into the text model for text generation processing to obtain the description text of the key image frame. The image-to-text model is trained based on the method of the first aspect as described above.

[0016] According to a third aspect of the present disclosure, a video generation method is provided, comprising:

[0017] Get the video generation request;

[0018] Generate a request to obtain multiple text information based on the video;

[0019] Input the multiple text information into an image model for image generation processing to obtain multiple generated images corresponding to the multiple text information, wherein the text-to-image model is trained based on the method of the first method above;

[0020] selecting a target generated image from the plurality of generated images;

[0021] A target video is generated based on the target generation image.

[0022] According to a fourth aspect of the present disclosure, a model training device is provided, comprising:

[0023] An acquisition module, used to acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data;

[0024] A first training module, configured to construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, and to train the image-to-text model based on the image data and the first adversarial network to obtain a first training result;

[0025] A second training module, configured to construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and to train the text-to-image data based on the text data and the second adversarial network to obtain a second training result;

[0026] An adjustment module is used to adjust the model parameters of the image-to-text model based on the first training result, and to adjust the model parameters of the text-to-image model based on the second training result, so as to obtain a trained image-to-text model and a trained text-to-image model.

[0027] According to a fifth aspect of the present disclosure, there is provided a text generation device, comprising:

[0028] An acquisition module, used for acquiring video data;

[0029] An extraction module, used for extracting key image frames from the video data;

[0030] A text generation module is used to input the key image frame into a text model for text generation processing to obtain a description text of the key image frame. The image-to-text model is trained based on the device described in the fourth aspect above.

[0031] According to a sixth aspect of the present disclosure, there is provided a video generating device, comprising:

[0032] A first acquisition module, used to acquire a video generation request;

[0033] A second acquisition module, used for acquiring multiple text information based on the video generation request;

[0034] An image generation module, used for inputting the multiple text information into an image model for image generation processing to obtain multiple generated images corresponding to the multiple text information, wherein the text-to-image model is trained based on the apparatus described in the fourth aspect above;

[0035] A selection module, configured to select a target generated image from the plurality of generated images;

[0036] A video generation module is used to generate a target video based on the target generation image.

[0037] According to a seventh aspect of the present disclosure, there is provided an electronic device, including:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in the first aspect above, or so that the at least one processor can execute the method as described in the second aspect above, or so that the at least one processor can execute the method as described in the third aspect above.

[0041] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect above, or the computer instructions are used to cause the computer to execute the method according to the second aspect above, or the computer instructions are used to cause the computer to execute the method according to the third aspect above.

[0042] According to a ninth aspect of the present disclosure, a computer program product is provided, the computer program product comprising: a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect above, or, when executed by a processor, implements the steps of the method described in the second aspect above, or, when executed by a processor, implements the steps of the method described in the third aspect above.

[0043] The technology disclosed in the present invention solves the problem that the model generation effect of the multimodal understanding model and the generative model is limited by the upper limit of the effect of the corresponding expert model, improves the accuracy of text generation by the image-to-text model, and makes the generated image of the text-to-image model more consistent with the content described in the text.

[0044] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0046] Figure 1 is a scene graph that can implement the disclosed model training method;

[0047] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0048] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0049] Figure 4 is a schematic diagram of a framework of a model training method according to the present disclosure;

[0050] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0051] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0052] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0053] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0054] Fig. 9 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0055] Fig.10 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0057] Multimodal understanding models have made significant progress and development in recent years. Multimodal understanding models can generate descriptive text based on input images. However, in the application of multimodal understanding models, the descriptive text generated by the model may be inconsistent with the input image, or even completely fictitious or wrong. This phenomenon is called the hallucination problem. At present, expert models such as SAM (Segment Anything Model) and OCR (Optical Character Recognition) are used to assist multimodal understanding models, helping them to better understand the structure and content of images, thereby generating more accurate descriptive text and alleviating hallucination problems.

[0058] At the same time, generative models have also made rapid progress, such as diffusion models, which can generate images based on input text. However, the current generative models are not capable of generating images based on text. The GPT-4 (Generative Pre-trained Transformer 4) model is currently used to describe the input text in a fine-grained manner to obtain detailed descriptive text, which is then converted into an embedding vector (embedding). Then, ControlNet (control network) is used to guide the generation of images that meet the requirements based on the embedding vector and other control signals. Through the guidance of ControlNet, it is ensured that the generated images are not only of high quality, but also highly consistent with the fine-grained description of the input.

[0059] In the above scheme, the multimodal understanding model is an image-to-text model that generates text from images, and the generation model is a text-to-image model that generates images from text. The generation effects of the image-to-text model and the text-to-image model are limited by the upper limit of the effect of the corresponding expert model. If the expert model itself is not effective, the generation effects of the image-to-text model and the text-to-image model will not be good. Therefore, it is urgent to provide an image-to-text model that can better understand the structure and content of the image and generate accurate descriptive text, and it is urgent to provide a text-to-image model that can generate images that are more visually consistent with the text description, and the generation effect of the model is not limited by the effect of the expert model.

[0060] The present disclosure provides a model training method, which is applied to the fields of computer vision and natural language processing in the field of artificial intelligence. The image-to-text model trained by the model training method can generate a descriptive text that is closer to the meaning expressed by the image, and the text-to-image model trained by the model training method can generate an image that is more consistent with the content described in the text.

[0061] It should be noted that the image data, text data and video data in the technical solution disclosed in this disclosure are all from public data sets. The collection, storage, use, processing, transmission, provision and disclosure of the information involved in the technical solution disclosed in this disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0062] The application scenario of the present application is specifically a scenario for model training, in which the training data can be processed in the server to train the image-to-text model and the text-to-image model. In addition, the model training can also be performed in the terminal device, which can be a desktop computer, a laptop computer, a tablet computer, a smart phone, etc.

[0063] Figure 1 is a scene graph that can implement the model training method disclosed in the present invention. Figure 1As shown, the training data can be stored in the server 101, and the terminal device 102 can obtain the training data from the server 101, and train the image-to-text model and the text-to-image model based on the training data. The trained image-to-text model and the text-to-image model can then be stored in the server 103, and the image-to-text model and the text-to-image model in the server 103 can be called by different entities or systems.

[0064] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In order to enable readers to have a deeper understanding of the implementation principle of the present invention, the following embodiments of the present invention will be described in conjunction with the accompanying drawings.

[0065] Figure 2 is a schematic diagram according to the first embodiment of the present disclosure, such as Figure 2 As shown, the model training method of this embodiment specifically includes:

[0066] S201. Acquire image-text pair data as training data, where the image-text pair data includes image data and text data matching the image data.

[0067] The image-text pair data consists of image data and text data describing the image data.

[0068] In the disclosed embodiments, image-text pair data may be obtained from different databases, and the image-text pair data may be used as training data for model training.

[0069] S202, using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator to construct a first adversarial network, training the image-to-text model based on the image data and the first adversarial network, and obtaining a first training result.

[0070] In the disclosed embodiment, two models need to be trained, one is a text-to-image model, which is used to generate descriptive text based on an input image, and the other is an image-to-text model, which is used to generate a corresponding image based on the input descriptive text.

[0071] In the process of training the image-to-text model, the image-to-text model to be trained is used as a generator, and the text-to-image model to be trained is used as a discriminator to construct a first adversarial network, and the model is trained according to the image data and the first adversarial network.

[0072] S203, constructing a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and training the text-to-image data based on the text data and the second adversarial network to obtain a second training result.

[0073] In the process of training the text-to-image model, the text-to-image model to be trained is used as a generator, and the image-to-text model to be trained is used as a discriminator to construct a second adversarial network, and the model is trained according to the text data and the second adversarial network.

[0074] It can be understood that the first adversarial network or the second adversarial network of the embodiment of the present disclosure is different from the traditional adversarial network. In the traditional adversarial network, the generator is usually responsible for generating data, and the discriminator is responsible for distinguishing the generated data from the real data. In the first adversarial network of the embodiment of the present disclosure, the generator is responsible for generating descriptive text of the image, and the discriminator is responsible for generating the image. In the second adversarial network of the embodiment of the present disclosure, the generator is responsible for generating the image, and the discriminator is responsible for generating descriptive text of the image.

[0075] The adversarial network of the disclosed embodiment is composed of a generator and a discriminator. When training an image-to-text model, the model parameters of the text-to-image model are fixed, and the model parameters of the image-to-text model are updated during training; and when training a text-to-image model, the model parameters of the image-to-text model are fixed, and the model parameters of the text-to-image model are updated during training. The image-to-text model and the text-to-image model serve as generators and discriminators for each other, thereby achieving joint optimization of the image-to-text model and the text-to-image model.

[0076] S204: Adjust the model parameters of the image-to-text model based on the first training result, and adjust the model parameters of the text-to-image model based on the second training result, to obtain a trained image-to-text model and a trained text-to-image model.

[0077] In the disclosed embodiment, the model parameters of the image-to-text model can be adjusted according to the first training result, and the model parameters of the text-to-image model can be adjusted according to the second training result to obtain a trained image-to-text model and a trained text-to-image model.

[0078] The model training method provided by the embodiment of the present disclosure trains the image-to-text model and the text-to-image model together. During the training process, the image-to-text model and the text-to-image model serve as generators and discriminators for each other. The image-to-text model gradually learns to generate descriptive text that is closer to the image content, and the text-to-image model gradually learns to generate images that are more consistent with the text description content. The image-to-text model trained by this method can better understand the structure and content of the image and generate accurate descriptive text. The text-to-image model trained by this method can generate images that are more visually consistent with the text description. The trained image-to-text model and text-to-image model do not rely on other expert models, and their generation effects are not limited by the effects of the expert models.

[0079] Figure 3 is a schematic diagram according to the second embodiment of the present disclosure, such as Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the model training method is described in detail. The model training method of this embodiment specifically includes:

[0080] S301. Acquire image-text pair data as training data, where the image-text pair data includes image data and text data matching the image data.

[0081] Among them, a generation process of image-text pair data may include: obtaining a sample image, generating a descriptive text for the sample image, rewriting the preliminary descriptive text to obtain a detailed description text, the detailed description text contains richer details and contextual information, and thus obtaining preliminary first image-text pair data (sample image-detailed description text), manually cleaning and optimizing the preliminary first image-text pair data, and obtaining high-quality image-text pair data that can be used as training data.

[0082] Another generation process of image-text pair data may include: obtaining sample descriptive text, which may be extracted from a novel text. Specifically, a multimodal understanding model may be used to segment the novel text into scene descriptions to obtain descriptive texts corresponding to different shots or scenes. Sample descriptive text may be selected from these descriptive texts. The sample descriptive text may be expanded to obtain expanded text. A pre-trained model may then be used to generate a corresponding text image based on the expanded text, thereby obtaining preliminary second image-text pair data (sample descriptive text-text image). The preliminary second image-text pair data may be manually cleaned and optimized to obtain high-quality image-text pair data that can be used as training data.

[0083] The image-text pair data in the embodiment of the present disclosure may be part or all of the cleaned first image-text pair data and the cleaned second image-text pair data.

[0084] S302: construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, train the image-to-text model based on the image data and the first adversarial network, and obtain a first training result.

[0085] In a possible implementation, the image-to-text model is trained based on the image data and the first adversarial network to obtain a first training result, which may specifically include:

[0086] Inputting the image data into the image-to-text model to obtain first generated text data;

[0087] Performing expansion processing on the first generated text data to obtain second generated text data;

[0088] inputting the second generated text data into a text-to-image model to obtain first generated image data;

[0089] The first generated text data and the first generated image data are used as first training results obtained through training.

[0090] The first generated text data is expanded, and specifically the first generated text data is input into a pre-trained large language model to obtain expanded second generated text data.

[0091] In the process of training the image-to-text model, a data processing process is involved in which text is generated based on an image, the text is rewritten, and then an image is generated based on the rewritten text, and the first generated text data and the first generated image data obtained in the data processing process are used as the first training result. In this embodiment, the first generated text data and the first generated image data are both used as the first training result to participate in the subsequent adjustment of the model parameters of the image-to-text model, and the model parameters are adjusted based on multiple types of data, which can improve the performance of the trained model.

[0092] S303: construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and train the text-to-image data based on the text data and the second adversarial network to obtain a second training result.

[0093] In a possible implementation, training the text-to-image data based on the text data and the second adversarial network to obtain a second training result may specifically include:

[0094] Performing expansion processing on the text data to obtain third generated text data;

[0095] inputting the third generated text data into a text-to-image model to obtain second generated image data;

[0096] inputting the second generated image data into an image-to-text model to obtain fourth generated text data;

[0097] The second generated image data and the fourth generated text data are used as the second training result obtained through training.

[0098] The text data is expanded, and specifically the text data is input into a pre-trained large language model to obtain expanded third generated text data.

[0099] In the process of training the text-to-image model, a data processing process is involved in rewriting text, generating an image based on the rewritten text, and then generating text based on the image, and the second generated image data and the fourth generated text data obtained in the data processing process are used as the second training result. In this embodiment, the second generated image data and the fourth generated text data are both used as the second training result to participate in the subsequent adjustment of the model parameters of the text-to-image model, and the model parameters are adjusted based on multiple types of data, which can improve the performance of the trained model.

[0100] S304. Based on the first generated text data and the first generated image data, calculate the model loss value of the image-to-text model to obtain a first loss value.

[0101] In an embodiment of the present disclosure, a model loss value of an image-to-text model is calculated based on the first generated text data and the first generated image data.

[0102] In a possible implementation, based on the first generated text data and the first generated image data, calculating the model loss value of the image-to-text model to obtain the first loss value may specifically include:

[0103] Determine a text difference between the text data and the first generated text data to obtain first difference information;

[0104] determining a pixel difference between the image data and the first generated image data to obtain second difference information;

[0105] Extracting image features of the image data to obtain first image features;

[0106] Extracting text features of the first generated text data to obtain first text features;

[0107] Determine a feature difference between the first image feature and the first text feature to obtain third difference information;

[0108] Based on the first difference information, the second difference information, and the third difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

[0109] The model loss value of the image-to-text model, i.e., the first loss value, can be determined based on the first difference information, the second difference information, and the third difference information, wherein the first difference information is used to measure the text difference between the original text data and the generated first generated text data, such as comparing character differences, word and phrase differences, etc.; the second difference information is used to measure the pixel-level difference between the original image data and the generated first generated image data, such as comparing pixel value differences; and the third difference information is used to measure the feature features between the image data and the generated first generated text data.

[0110] Exemplarily, the multimodal twin-tower model can be used to extract image features and text features. The multimodal twin-tower model can extract features of different data modalities (such as text, images, etc.) through two independent neural networks (usually referred to as "towers"), and then fuse or compare the feature representations of the two towers at the interoperability layer to complete a specific task. Specifically, the image data is input into the multimodal twin-tower model, and after being processed by the image tower, an image feature vector (i.e., a first image feature) is obtained. The first generated text data is input into the multimodal twin-tower model, and after being processed by the text tower, a text feature vector (i.e., a first text feature) is obtained. The image feature vector and the text feature vector can be compared by means of cosine similarity to obtain third difference information.

[0111] In this embodiment, by comprehensively calculating the loss value of the model by combining various difference information, the model performance can be accurately evaluated and the model optimization can be guided.

[0112] In a possible implementation, based on the first difference information, the second difference information, and the third difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value, which may specifically include:

[0113] Extracting the image feature of the image data again to obtain a second image feature;

[0114] Extracting image features of the first generated image data to obtain third image features;

[0115] determining a feature difference between the second image feature and the third image feature to obtain fourth difference information;

[0116] Based on the first difference information, the second difference information, the third difference information and the fourth difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

[0117] In this embodiment, based on the first difference information, the second difference information and the third difference information, the model loss value of the image-to-text model can also be determined in combination with the fourth difference information. The fourth difference information is used to measure the feature difference between the image data and the generated first generated image data.

[0118] Exemplarily, image features of the image data can be extracted again. This time, another model, such as a ResNet (Residual Network) model, can be used to extract image features of the image data and the first generated image data to obtain corresponding image feature vectors. By comparing the two image feature vectors, fourth difference information can be obtained.

[0119] The loss function (LossA) of the image-to-text model can be shown as follows:

[0120] LossA=w1×cross_entropy loss(T,Tp1)+w2×L2 loss(I,Ip1)+w3×cosinesimilarity loss(f1(I),f1(Ip1))+w4×cosine similarity loss(f2(I),f2(Tp1));

[0121] Among them, w1, w2, w3, w4 are weights between 0 and 1; T represents the original text data; Tp1 represents the first generated text data; cross-entropy loss() represents the cross-entropy loss function, which measures the text difference between the original text data and the generated first generated text data; I represents the original image data; Ip1 represents the first generated image data; L2loss() represents the calculation of mean square error, which measures the pixel-level difference between the original image data and the generated first generated image data; f1() represents the use of ResNet model to extract image features; cosine similarity loss() represents the calculation of cosine similarity to measure feature differences; f2() represents the use of multimodal dual-tower model to extract image features or text features.

[0122] S305. Based on the second generated image data and the fourth generated text data, calculate the model loss value of the text-to-image model to obtain a second loss value.

[0123] In an embodiment of the present disclosure, the model loss value of the text-to-image model is calculated based on the second generated image data and the fourth generated text data.

[0124] In a possible implementation, based on the second generated image data and the fourth generated text data, calculating the model loss value of the text-to-image model to obtain the second loss value may specifically include:

[0125] determining a text difference between the text data and the fourth generated text data to obtain fifth difference information;

[0126] determining a pixel difference between the image data and the second generated image data to obtain sixth difference information;

[0127] Extracting image features of the image data and extracting text features of the fourth generated text data to obtain fourth image features and second text features; determining feature differences between the fourth image features and the second text features to obtain seventh difference information;

[0128] Extracting the image feature of the image data again to obtain a fifth image feature; extracting the image feature of the second generated image data to obtain a sixth image feature; determining the feature difference between the fifth image feature and the sixth image feature to obtain eighth difference information;

[0129] Based on the fifth difference information, the sixth difference information, the seventh difference information and the eighth difference information, a model loss value of the text-to-image model is calculated to obtain a second loss value.

[0130] The loss function (LossB) of the text-to-image model can be shown as follows:

[0131] LossB=w1×cross_entropy loss(T,Tp2)+w2×L2 loss(I,Ip2)+w3×cosinesimilarity loss(f1(I),f1(Ip2))+w4×cosine similarity loss(f2(T),f2(Ip2));

[0132] Among them, w1, w2, w3, and w4 are weights between 0 and 1. T represents the original text data; Tp2 represents the fourth generated text data; cross-entropy loss() represents the cross-entropy loss function, which measures the text difference between the original text data and the generated fourth generated text data; I represents the original image data; Ip2 represents the second generated image data; L2loss() represents the calculation of the mean square error, which measures the pixel-level difference between the original image data and the generated second generated image data; f1() represents the use of the ResNet model to extract image features; cosine similarity loss() represents the calculation of cosine similarity to measure feature differences; f2() represents the use of the multimodal dual-tower model to extract image features or text features.

[0133] S306. Adjust the model parameters of the image-to-text model based on the first loss value, and adjust the model parameters of the text-to-image model based on the second loss value to obtain a trained image-to-text model and a trained text-to-image model.

[0134] In a possible implementation, the model training method of the embodiment of the present disclosure may further include:

[0135] Alternately train image-to-text models and text-to-image models;

[0136] For the alternately trained image-to-text model and text-to-image model, if the first loss value corresponding to the image-to-text model satisfies the preset first loss condition, and the second loss value corresponding to the text-to-image model satisfies the preset second loss condition, the training is stopped to obtain the target image-to-text model and the target text-to-image model.

[0137] In this embodiment, the image-to-text model and the text-to-image model can be trained alternately, that is, after training the image-to-text model, the text-to-image model is trained, and after training the text-to-image model, the image-to-text model is trained again, and the cycle is alternated until the training stop condition is met.

[0138] In a specific implementation, the image-to-text model is trained, the current image-to-text model is used as a generator, the current text-to-image model is used as a discriminator, and the first loss value of this training is calculated. If the first loss value meets the preset first loss condition, the subsequent text-to-image model training is started directly; if the first loss value does not meet the preset first loss condition, the model parameters of the image-to-text model are adjusted based on the first loss value to obtain the trained image-to-text model, and then the text-to-image model training is started.

[0139] When training the text-to-image model, the current text-to-image model is used as a generator, and the current image-to-text model is used as a discriminator. The second loss value of this training is calculated. If the second loss value satisfies the preset second loss condition, it is necessary to determine whether the first loss value calculated in the previous training image-to-text model process satisfies the preset first loss condition. If so, it can be determined that the training stop condition is met, and the training is stopped to output the current image-to-text model and the current text-to-image model to obtain the target image-to-text model and the target text-to-image model; and if the first loss value calculated in the previous training image-to-text model process does not meet the preset first loss condition, then continue training and perform subsequent image-to-text model training; if the second loss value does not meet the preset second loss condition, then adjust the model parameters of the text-to-image model based on the second loss value to obtain the trained text-to-image model, and then start training the image-to-text model.

[0140] It should be noted that in the embodiments of the present disclosure, there is no specific limitation on the order of training the image-to-text model and the text-to-image model, that is, the image-to-text model can be trained first, or the text-to-image model can be trained first.

[0141] In this embodiment, the image-to-text model and the text-to-image model form a dynamic "game" during the alternating training process. This game mechanism encourages each model to improve its own capabilities in the continuous confrontation.

[0142] Reference Figure 4 As shown, it is a framework diagram of the model training method according to the present invention. In the process of training the image-to-text model, the image-to-text model is used as a generator to generate corresponding text based on the image, and the text-to-image model is used as a discriminator in this process; in the process of training the text-to-image model, the text-to-image model is used as a generator to generate corresponding images based on the text, and the image-to-text model is used as a discriminator in this process.

[0143] The model training method provided by the embodiment of the present disclosure trains the image-to-text model and the text-to-image model together. During the training process, the image-to-text model and the text-to-image model serve as generators and discriminators for each other. The image-to-text model gradually learns to generate descriptive text that is closer to the image content, and the text-to-image model gradually learns to generate images that are more consistent with the text description content. The image-to-text model trained by this method can better understand the structure and content of the image and generate accurate descriptive text. The text-to-image model trained by this method can generate images that are more visually consistent with the text description. The trained image-to-text model and text-to-image model do not rely on other expert models, and their generation effects are not limited by the effects of the expert models.

[0144] The above embodiments introduce the training process of the image-to-text model and the text-to-image model. The following embodiments illustrate the application process of the image-to-text model and the text-to-image model. For example, the image-to-text model can be applied to a scenario where a text description of the video content is required, and the text-to-image model can be applied to a scenario where a corresponding video is generated based on the text content.

[0145] Figure 5 is a schematic diagram according to the third embodiment of the present disclosure, Figure 5 As shown, the text generation method of this embodiment specifically includes:

[0146] S501: Obtain video data.

[0147] In the embodiments of the present disclosure, various types of video data may be acquired, such as film and television video data, teaching video data, live video data, and game video data.

[0148] S502: Extract key image frames from video data.

[0149] After acquiring the video data, key image frames can be extracted from the video data. The key image frames contain key picture information. By extracting the key image frames and analyzing only the key image frames, the amount of data that needs to be calculated can be reduced.

[0150] S503, input the key image frame into the text model for text generation processing to obtain the description text of the key image frame. The image-to-text model is trained based on the model training method of the above embodiment.

[0151] In the disclosed embodiment, the key image frame may be input into a pre-trained image-to-text model to obtain a description text of the key image frame.

[0152] The text generation method provided by the embodiment of the present disclosure can quickly generate description text for video data to facilitate rapid understanding of the video content.

[0153] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure, Figure 6 As shown, the video generation method of this embodiment specifically includes:

[0154] S601: Obtain a video generation request.

[0155] The user performs a video generation operation on the terminal device, such as clicking a video generation button, and a video generation request is generated in response to the video generation operation.

[0156] S602: Generate a request based on the video to obtain multiple text information.

[0157] In the disclosed embodiment, video generation information can be parsed from the video generation request, the video generation information indicates the requirements that the video to be generated must meet, and the video generation information can include style information and theme information of the generated video, etc., and then multiple text information can be obtained based on the video generation information. The text information can be pre-stored in a specified database.

[0158] S603, input multiple text information into the text to the image model for image generation processing, and obtain multiple generated images corresponding to the multiple text information. The text-to-image model is trained based on the model training method of the above embodiment.

[0159] By inputting multiple text information into the image model, multiple generated images corresponding to the multiple text information can be quickly obtained.

[0160] S604: Select a target generated image from multiple generated images.

[0161] S605: Generate a target video based on the target generated image.

[0162] In the disclosed embodiment, a target generated image may be selected from a plurality of generated images, and the target generated image may be used to perform video generation.

[0163] The video generation method provided by the embodiment of the present disclosure realizes the rapid generation of video content.

[0164] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 7 As shown, the model training device 700 of this embodiment specifically includes:

[0165] An acquisition module 701 is used to acquire image-text pair data as training data, where the image-text pair data includes image data and text data matching the image data;

[0166] A first training module 702 is used to construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, and train the image-to-text model based on the image data and the first adversarial network to obtain a first training result;

[0167] A second training module 703 is used to construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and train the text-to-image data based on the text data and the second adversarial network to obtain a second training result;

[0168] The adjustment module 704 is used to adjust the model parameters of the image-to-text model based on the first training result, and adjust the model parameters of the text-to-image model based on the second training result, to obtain the trained image-to-text model and the trained text-to-image model.

[0169] In a possible implementation, the first training module includes:

[0170] A first input submodule, used for inputting image data into an image-to-text model to obtain first generated text data;

[0171] A first expansion submodule, used for expanding the first generated text data to obtain second generated text data;

[0172] A second input submodule, used for inputting the second generated text data into the text-to-image model to obtain the first generated image data;

[0173] The first determination submodule is used to use the first generated text data and the first generated image data as the first training result obtained by training.

[0174] In a possible implementation, the second training module includes:

[0175] A second expansion submodule is used to expand the text data to obtain third generated text data;

[0176] A third input submodule, used for inputting the third generated text data into the text-to-image model to obtain second generated image data;

[0177] a fourth input submodule, used to input the second generated image data into the image-to-text model to obtain fourth generated text data;

[0178] The second determining submodule is used to use the second generated image data and the fourth generated text data as the second training result obtained by training.

[0179] In a possible implementation, the adjustment module includes:

[0180] A first calculation submodule, configured to calculate a model loss value of an image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value;

[0181] A second calculation submodule, used to calculate a model loss value of the text-to-image model based on the second generated image data and the fourth generated text data to obtain a second loss value;

[0182] The adjustment submodule is used to adjust the model parameters of the image-to-text model based on the first loss value, and to adjust the model parameters of the text-to-image model based on the second loss value, so as to obtain the trained image-to-text model and the trained text-to-image model.

[0183] In a possible implementation, the method further includes:

[0184] An alternating training module, used to alternately train the image-to-text model and the text-to-image model;

[0185] The training stop judgment module is used to stop training the alternately trained image-to-text model and text-to-image model to obtain the target image-to-text model and the target text-to-image model if the first loss value corresponding to the image-to-text model meets the preset first loss condition and the second loss value corresponding to the text-to-image model meets the preset second loss condition.

[0186] In a possible implementation, the first computing submodule includes:

[0187] A first determining unit, used to determine a text difference between the text data and the first generated text data to obtain first difference information;

[0188] a second determining unit, configured to determine a pixel difference between the image data and the first generated image data to obtain second difference information;

[0189] A first extraction unit, used to extract image features of image data to obtain first image features;

[0190] A second extraction unit, used to extract text features of the first generated text data to obtain first text features;

[0191] A third determining unit, configured to determine a feature difference between the first image feature and the first text feature to obtain third difference information;

[0192] The calculation unit is used to calculate the model loss value of the image-to-text model based on the first difference information, the second difference information and the third difference information to obtain a first loss value.

[0193] In a possible implementation, the computing unit includes:

[0194] A first extraction subunit is used to extract the image feature of the image data again to obtain a second image feature;

[0195] A second extraction subunit is used to extract the image feature of the first generated image data to obtain a third image feature;

[0196] a determination subunit, configured to determine a feature difference between the second image feature and the third image feature to obtain fourth difference information;

[0197] The calculation subunit is used to calculate the model loss value of the image-to-text model based on the first difference information, the second difference information, the third difference information and the fourth difference information to obtain a first loss value.

[0198] The model training device provided in the embodiment of the present disclosure can execute the model training method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0199] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure, Figure 8 As shown, the text generation device 800 of this embodiment specifically includes:

[0200] An acquisition module 801 is used to acquire video data;

[0201] An extraction module 802 is used to extract key image frames from video data;

[0202] The text generation module 803 is used to input the key image frame into the text model for text generation processing to obtain the description text of the key image frame. The image-to-text model is trained based on the model training device in the above embodiment.

[0203] The text generation device provided in the embodiment of the present disclosure can execute the text generation method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail in this embodiment.

[0204] Fig. 9 is a schematic diagram according to the seventh embodiment of the present disclosure, Fig. 9 As shown, the video generating device 900 of this embodiment specifically includes:

[0205] A first acquisition module 901 is used to acquire a video generation request;

[0206] A second acquisition module 902, configured to acquire multiple text information based on the video generation request;

[0207] An image generation module 903 is used to input multiple text information into an image model for image generation processing to obtain multiple generated images corresponding to the multiple text information, wherein the text-to-image model is trained based on the model training device of the above embodiment;

[0208] A selection module 904, configured to select a target generated image from a plurality of generated images;

[0209] The video generation module 905 is used to generate a target video based on the target generation image.

[0210] The video generating device provided in the embodiment of the present disclosure can execute the video generating method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0211] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0212] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the method provided in any of the above embodiments.

[0213] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0214] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0215] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0216] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a model training method or a text generation method or a video generation method. For example, in some embodiments, the model training method or the text generation method or the video generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the model training method or the text generation method or the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the model training method or the text generation method or the video generation method in any other suitable manner (e.g., by means of firmware).

[0217] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0218] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0219] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0220] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0221] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0222] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0223] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0224] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method, characterized in that: include: Acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data; Using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator to construct a first adversarial network, and training the image-to-text model based on the image data and the first adversarial network to obtain a first training result; Using the text-to-image model as a generator and the image-to-text model as a discriminator to construct a second adversarial network, and training the text-to-image data based on the text data and the second adversarial network to obtain a second training result; The model parameters of the image-to-text model are adjusted based on the first training result, and the model parameters of the text-to-image model are adjusted based on the second training result to obtain a trained image-to-text model and a trained text-to-image model.

2. The method according to claim 1, characterized in that Also includes: Alternatingly training the image-to-text model and the text-to-image model; For the alternately trained image-to-text model and the text-to-image model, if the first loss value corresponding to the image-to-text model satisfies the preset first loss condition, and the second loss value corresponding to the text-to-image model satisfies the preset second loss condition, the training is stopped to obtain the target image-to-text model and the target text-to-image model.

3. The method according to claim 1 or 2, characterized in that: The step of training the image-to-text model based on the image data and the first adversarial network to obtain a first training result includes: Inputting the image data into the image-to-text model to obtain first generated text data; Performing expansion processing on the first generated text data to obtain second generated text data; inputting the second generated text data into the text-to-image model to obtain first generated image data; The first generated text data and the first generated image data are used as first training results obtained through training.

4. The method according to claim 3, characterized in that The step of training the text-to-image data based on the text data and the second adversarial network to obtain a second training result includes: Performing expansion processing on the text data to obtain third generated text data; inputting the third generated text data into the text-to-image model to obtain second generated image data; inputting the second generated image data into the image-to-text model to obtain fourth generated text data; The second generated image data and the fourth generated text data are used as a second training result obtained through training.

5. The method according to claim 4, characterized in that The adjusting the model parameters of the image-to-text model based on the first training result, and adjusting the model parameters of the text-to-image model based on the second training result to obtain a trained image-to-text model and a trained text-to-image model, comprises: Calculating a model loss value of the image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value; Calculating a model loss value of the text-to-image model based on the second generated image data and the fourth generated text data to obtain a second loss value; The model parameters of the image-to-text model are adjusted based on the first loss value, and the model parameters of the text-to-image model are adjusted based on the second loss value to obtain a trained image-to-text model and a trained text-to-image model.

6. The method according to claim 5, characterized in that The step of calculating the model loss value of the image-to-text model based on the first generated text data and the first generated image data to obtain a first loss value includes: Determine a text difference between the text data and the first generated text data to obtain first difference information; determining a pixel difference between the image data and the first generated image data to obtain second difference information; Extracting image features of the image data to obtain first image features; Extracting text features of the first generated text data to obtain first text features; determining a feature difference between the first image feature and the first text feature to obtain third difference information; Based on the first difference information, the second difference information and the third difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

7. The method according to claim 6, characterized in that The step of calculating the model loss value of the image-to-text model based on the first difference information, the second difference information, and the third difference information to obtain a first loss value includes: extracting the image feature of the image data again to obtain a second image feature; extracting image features of the first generated image data to obtain third image features; determining a feature difference between the second image feature and the third image feature to obtain fourth difference information; Based on the first difference information, the second difference information, the third difference information and the fourth difference information, a model loss value of the image-to-text model is calculated to obtain a first loss value.

8. A text generation method, characterized in that: include: Get video data; extracting key image frames from the video data; The key image frame is input into the image into a text model for text generation processing to obtain a description text of the key image frame, and the image-to-text model is trained based on the method described in any one of claims 1-7.

9. A video generation method, characterized in that: include: Get the video generation request; Generate a request to obtain multiple text information based on the video; Inputting the multiple text information into an image model for image generation processing to obtain multiple generated images corresponding to the multiple text information, wherein the text-to-image model is trained based on the method according to any one of claims 1 to 7; selecting a target generated image from the plurality of generated images; A target video is generated based on the target generation image.

10. A model training device, characterized in that: include: An acquisition module, used to acquire image-text pair data as training data, wherein the image-text pair data includes image data and text data matching the image data; A first training module, configured to construct a first adversarial network using the image-to-text model to be trained as a generator and the text-to-image model to be trained as a discriminator, and to train the image-to-text model based on the image data and the first adversarial network to obtain a first training result; A second training module, configured to construct a second adversarial network using the text-to-image model as a generator and the image-to-text model as a discriminator, and to train the text-to-image data based on the text data and the second adversarial network to obtain a second training result; An adjustment module is used to adjust the model parameters of the image-to-text model based on the first training result, and to adjust the model parameters of the text-to-image model based on the second training result, so as to obtain a trained image-to-text model and a trained text-to-image model.

11. A text generation device, characterized in that: include: An acquisition module, used for acquiring video data; An extraction module, used for extracting key image frames from the video data; A text generation module is used to input the key image frame into a text model for text generation processing to obtain a description text of the key image frame, and the image-to-text model is trained based on the device as described in claim 10.

12. A video generating device, characterized in that: include: A first acquisition module, used to acquire a video generation request; A second acquisition module, used for acquiring multiple text information based on the video generation request; An image generation module, used for inputting the plurality of text information into an image model for image generation processing to obtain a plurality of generated images corresponding to the plurality of text information, wherein the text-to-image model is trained based on the apparatus according to claim 10; A selection module, configured to select a target generated image from the plurality of generated images; A video generation module is used to generate a target video based on the target generation image.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 7, or to enable the at least one processor to execute the method of claim 8, or to enable the at least one processor to execute the method of claim 9.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7, or the computer instructions are used to cause the computer to execute the method according to claim 8, or the computer instructions are used to cause the computer to execute the method according to claim 9.

15. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1 to 7, or, when executed by a processor, implements the steps of the method described in claim 8, or, when executed by a processor, implements the steps of the method described in claim 9.

Images