Multi-modal generative model training method and device, and image generation method and device

Abstract

Embodiments of the invention provide a multi-modal generative model training method and apparatus, and an image generation method and apparatus. The method comprises the steps of obtaining a first dialogue sample comprising a first sample description, a second sample description and a sample instruction; obtaining a first generated image through an image generation network based on the first sample description; based on the second sample description and the sample instruction, a first generation description is obtained through a text generation network; based on the first generated image and the first generated description, obtaining a second generated description related to the output of the multi-modal generative network through the multi-modal generative network; generating a second generated image based on the second generated description; determining a target generation loss based on the first sample description, the output of the multi-modal generation network and the second generation image; and with minimization of target generation loss as a target, adjusting parameters of a multi-modal generation model containing the three networks, so as to obtain a model capable of generating better image description through training.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for training a multimodal generation model and generating an image. Background Art

[0002] In the field of image generation, image descriptions (also called text prompts) can generally be used to generate images corresponding to the image descriptions through image generation networks based on content generation technology (for example, image generation networks based on diffusion models). In the process of image generation, in order to improve the quality of the generated images, it is crucial to improve the quality of the image descriptions.

[0003] Therefore, how to obtain better quality image description becomes an urgent problem to be solved. Summary of the invention

[0004] One or more embodiments of the present disclosure provide a method and apparatus for training a multimodal generative model, and generating an image, so as to train a model that can generate a better image description, so as to obtain a better quality image description, and further obtain a better quality image.

[0005] According to a first aspect, a training method for a multimodal generation model is provided, wherein the multimodal generation model includes an image generation network, a text generation network, and a multimodal generation network, and the method includes:

[0006] Obtaining any first dialogue sample in a training sample set, where the first dialogue sample includes a first sample description and a corresponding second sample description and a sample instruction, where the sample instruction is used to indicate a generation direction of the second sample description;

[0007] Based on the first sample description, obtaining a first generated image through the image generation network;

[0008] Based on the second sample description and the sample instruction, obtaining a first generated description through the text generation network;

[0009] Based on the first generated image and the first generated description, obtaining a second generated description through the multimodal generation network, wherein the second generated description is related to an output of the multimodal generation network;

[0010] generating a second generated image based on the second generated description;

[0011] Determining a target generation loss based on the first sample description, the output of the multimodal generation network, and the second generated image;

[0012] The parameters of the multimodal generation model are adjusted with the goal of minimizing the target generation loss.

[0013] According to a second aspect, there is provided an image generation method, comprising:

[0014] Acquire data to be processed and a first instruction for indicating a generation direction of the data to be processed;

[0015] Based on the first instruction, obtaining a third generated description through a target text generation network of a target multimodal generation model;

[0016] Based on the data to be processed and the third generation description, a fourth generation description is obtained through a target multimodal generation network of the target multimodal generation model;

[0017] Based on the fourth generation description, a generated image corresponding to the data to be processed and the first instruction is obtained through a target image generation network of the target multimodal generation model.

[0018] According to a third aspect, a training device for a multimodal generation model is provided, wherein the multimodal generation model includes an image generation network, a text generation network, and a multimodal generation network, and the device includes:

[0019] A first acquisition module is configured to acquire any first dialogue sample in a training sample set, wherein the first dialogue sample includes a first sample description and a corresponding second sample description and a sample instruction, wherein the sample instruction is used to indicate a generation direction of the second sample description;

[0020] A first obtaining module is configured to obtain a first generated image through the image generation network based on the first sample description;

[0021] A second obtaining module is configured to obtain a first generated description through the text generation network based on the second sample description and the sample instruction;

[0022] a third obtaining module, configured to obtain a second generated description through the multimodal generation network based on the first generated image and the first generated description, wherein the second generated description is related to an output of the multimodal generation network;

[0023] A generating module, configured to generate a second generated image based on the second generated description;

[0024] A first determination module configured to determine a target generation loss based on the first sample description, the output of the multimodal generation network, and the second generated image;

[0025] The first adjustment module is configured to adjust the parameters of the multimodal generation model with the goal of minimizing the target generation loss.

[0026] According to a fourth aspect, there is provided an image generating device, comprising:

[0027] A second acquisition module is configured to acquire data to be processed and a first instruction for indicating a generation direction of the data to be processed;

[0028] a fourth obtaining module configured to obtain a third generated description based on the first instruction through a target text generation network of a target multimodal generation model;

[0029] A fifth obtaining module, based on the data to be processed and the third generation description, obtains a fourth generation description through a target multimodal generation network of the target multimodal generation model;

[0030] The sixth obtaining module obtains, based on the fourth generation description, a generated image corresponding to the data to be processed and the first instruction through a target image generation network of the target multimodal generation model.

[0031] According to a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect or the second aspect.

[0032] According to a sixth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect or the second aspect is implemented.

[0033] According to the training and image generation method and device of the multimodal generative model provided by the embodiment of the present disclosure, the first dialogue sample includes a first sample description, a second sample description corresponding to the first sample description, and a sample instruction for indicating the generation direction of the second sample description. Based on the first sample description, the image generation network of the multimodal generative model is used to obtain a first generated image. Then, based on the second sample description and the sample instruction, the text generation network of the multimodal generative model is used to obtain a first generated description generated by the second sample description according to the generation direction indicated by the sample instruction. Then, based on the first generated image and the first generated description, the multimodal generative network of the multimodal generative model is used to obtain a fusion of multimodal A second generated description of the data (i.e., features of the first generated image and features of the first generated description), the second generated description being related to the output of the multimodal generative network; based on the second generated description, a second generated image is generated; based on the first sample description, the output of the multimodal generative network and the second generated image, a target generation loss is determined, and the parameters of the multimodal generative model are adjusted with the goal of minimizing the target generation loss, so as to utilize the idea of ​​multimodal learning to adjust the multimodal generative model so as to enhance the ability of the multimodal generative model to generate higher quality image descriptions, and then the multimodal generative model can be utilized to obtain higher quality image descriptions, thereby improving the quality of images generated based on the image descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in the present disclosure;

[0036] Figure 2 A schematic diagram of a flow chart of a training method for a multimodal generative model provided in an embodiment;

[0037] Figure 3 A schematic diagram of a flow chart of an image generation method provided in an embodiment;

[0038] Figure 4 A schematic block diagram of a training device for a multimodal generative model provided in an embodiment;

[0039] Figure 5 A schematic block diagram of an image generating device provided in an embodiment;

[0040] Figure 6 A schematic block diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0041] The technical solution of the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0042] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0043] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0044] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0045] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0046] The embodiments of the present disclosure disclose a method and device for training a multimodal generation model, and a method and device for generating an image. The application scenarios and technical concepts of the method are first introduced as follows:

[0047] As mentioned above, in the field of image generation, image descriptions (also called text prompts) can generally be used to generate images corresponding to image descriptions through image generation networks based on content generation technology (for example, image generation networks based on diffusion models). In the process of generating images, in order to improve the quality of the generated images, it is crucial to improve the quality of image descriptions. Therefore, how to obtain better quality image descriptions has become a problem that needs to be solved urgently.

[0048] In view of this, the embodiments of the present disclosure provide a method and device for training a multimodal generation model, generating an image, so as to obtain a better quality image description, and then obtain a better quality image based on the image description. Figure 1A schematic diagram of an implementation scenario according to an embodiment disclosed in the present disclosure is shown. In the implementation scenario, a multimodal generation model is exemplarily shown. The multimodal generation model is a conversational multimodal generation model, which can perform extended optimization on the image description (or image) based on the instruction input by the user for indicating the generation direction of the image description (or image), obtain the corresponding extended optimized image description, and further obtain the corresponding image based on the extended optimized image description. Specifically, as Figure 1 As shown, the multimodal generation model may include an image generation network, a text generation network, and a multimodal generation network.

[0049] Among them, the image generation network can be an image generation network that has been pre-trained using several image-text pairs. Each image-text pair includes an image and its corresponding image description. For the sake of clear description, the image in the image-text pair can be called the second sample image, and the image description in the image-text pair can be called the third sample description. In one implementation, the image-text pair can be obtained from any specified public data set that can provide a large number of image-text pairs or a specified artificially produced data set, and the artificially produced data set includes: images obtained from a specified channel and image descriptions that are manually annotated for the images. The image generation network is trained using image-text pairs so that the trained image generation network has the ability to generate corresponding images based on image descriptions.

[0050] The text generation network may be a text generation network pre-trained using the occluded text generated by the texts and the occluded parts of the texts. The texts are image descriptions used to generate images. For the sake of clarity, the texts may be referred to as fourth sample descriptions. The text generation network may be used to perform text prediction, expansion and completion on the input image description.

[0051] In one implementation, the aforementioned image description may also be referred to as a text prompt, that is, a text prompt used to generate an image.

[0052] The training process of the multimodal generative model is introduced below. Specifically, a training sample set for training the multimodal generative model is obtained. The training sample set may include multiple dialogue samples, wherein any first dialogue sample may include a first sample description and its corresponding second sample description and a sample instruction for indicating the generation direction of the second sample description. The second sample description may include part of the content of the corresponding first sample description, for example, the second sample description is a description generated by the occluded part of the content of the corresponding first sample description, or the second sample description includes part of the content disassembled from the corresponding first sample description. For example, the first sample description is "a highly detailed portrait of a man with dark green hair and green glowing eyes, high detail clothing, concept art, anime, art station, professional", and accordingly, the second sample description may be "a highly detailed portrait of a man", or may be "a highly detailed portrait of a man with dark green hair and green glowing eyes, professiona".

[0053] In one implementation, the first sample description may be a description that satisfies specified conditions. Specifically, the first sample description and its corresponding image (i.e., an image-text pair) may be an image-text pair in a specified public data set whose quality exceeds a preset screening condition. The preset screening condition may include, but is not limited to, at least one of the following: an aesthetic score corresponding to the image in the image-text pair is not lower than a preset aesthetic score threshold (for example, when the aesthetic score range is [0, 1], the preset aesthetic score threshold is, for example, 0.75), a clarity score corresponding to the image in the image-text pair is not lower than a preset clarity threshold (for example, when the clarity score range is [0, 1], the preset clarity threshold is, for example, 0.80), and a matching value corresponding to the image-text pair is not lower than a preset matching threshold (for example, when the matching value range is [0, 1], the preset matching threshold is, for example, 0.80).

[0054] By using sample descriptions that meet specified conditions, the image description generation capability of the trained multimodal generative model can be better guaranteed, so that it can generate image descriptions with better quality, and thus obtain images with better quality.

[0055] An arbitrary dialogue sample is obtained from the training sample set as the first dialogue sample. Based on the first sample description in the first dialogue sample, a first generated image is obtained through an image generation network. Based on the second sample description and the sample instruction, a first generated description generated by the second sample description according to the generation direction indicated by the sample instruction is obtained through a text generation network. Then, based on the first generated image and the first generated description, a second generated description is obtained through a multimodal generation network, and the second generated model is related to the output of the multimodal generation model. Then, a second generated image is generated based on the second generated description. Then, the target generation loss is determined by combining the first sample description, the output of the multimodal generation model and the second generated image, and the parameters of the multimodal generation model are adjusted with the goal of minimizing the target generation loss.

[0056] Specifically, the first generation loss can be determined based on the output of the multimodal generation model and the first sample description; and the first matching loss can be determined through the text-image association model based on the second generated image and the first sample description; and then the target generation loss is determined by at least combining the first generation loss and the first matching loss, wherein the target generation loss is positively correlated with the first generation loss and the first matching loss, respectively. With the goal of minimizing the target generation loss, that is, with the purpose of minimizing the first matching loss and minimizing the first generation loss, the parameters of the multimodal generation model are adjusted to improve the quality of the image description generated by the multimodal generation model.

[0057] Among them, minimizing the first generation loss can make the text generation network generate better quality image descriptions; minimizing the constructed first matching loss can make the second generated image that integrates the features of the first generated image and the features of the first generated description match the first sample description better, so that the image generation ability of the image generation network and the text generation ability of the text generation network can be better integrated, and then the image domain data and the text (language) domain data can be better integrated to better improve the image description generation ability of the multimodal generation model and improve the quality of the generated image description (text). To better improve the image description generation ability of the multimodal generation model.

[0058] In the above process, the idea of ​​multimodal learning is utilized to construct fine-grained losses and adjust the multimodal generation model to enhance the ability of the multimodal generation model to generate higher quality image descriptions. The multimodal generation model can then be used to obtain higher quality image descriptions, thereby improving the quality of images generated based on the image descriptions.

[0059] Among them, the multimodal generation model can be a conversational multimodal generation model. For the trained multimodal generation model, based on the data to be processed (the image to be processed and / or the image description to be processed) input by the user and the instructions (conversation content) input by the user indicating the generation direction of the data to be processed, a higher-quality image description can be generated through the trained multimodal generation model, and then a higher-quality image can be generated based on the higher-quality image description.

[0060] In combination with specific embodiments, the training method and device of the multimodal generation model and the image generation method and device provided by the present disclosure are described in detail below. In order to make the layout clear, the process of training the image generation network and the text generation network is first introduced respectively.

[0061] Among them, the process of training an image generation network may include: obtaining an image generation network to be trained, obtaining a training data set S for pre-training the image generation network, and the training data set S may include multiple pairs of image-text pairs, and an image-text pair includes a sample image (for the sake of clarity, referred to as the second sample image) T and its corresponding image description (for the sake of clarity, referred to as the third sample description) X. In one case, the training data set S may be obtained from any public data set that can provide a large number of image-text pairs or a designated artificially produced data set, and the artificially produced data set includes: images obtained from a designated channel and image descriptions that are manually annotated for the images. In one case, the image description may also be called a text prompt, that is, a text prompt used to generate an image.

[0062] From the training data set S, obtain any image-text pair M including a third sample description X and its corresponding second sample image T, use the third sample description X, and obtain the corresponding generated image through the image generation network to be trained, which is called the third generated image T'; then use the difference between the second sample image T in the image-text pair M and the third generated image T' to determine the generation loss (called the fourth generation loss) corresponding to the image generation network to be trained. In one case, a preset loss function can be used to determine the fourth generation loss based on the difference between the second sample image T and the third generated image T'. The preset loss function may include, but is not limited to: a cross entropy loss function and an MSE (mean square error) loss function, etc.

[0063] Afterwards, with the goal of minimizing the fourth generation loss, that is, reducing the difference between the second sample image and the corresponding third generation image, the parameters of the image generation network to be trained are adjusted until the image generation network reaches the first convergence condition, and the training of the image generation network to be trained is determined to be completed, and accordingly, the image generation network in the aforementioned multimodal generation model is obtained. The first convergence condition may include, but is not limited to, the number of parameter adjustments is not less than the first adjustment threshold, or the calculated fourth generation loss is less than the first loss threshold.

[0064] Let's review the process of obtaining the image generation network through the training mentioned above. The above embodiment is explained by taking an image-text pair (i.e., a pair of third sample description and second sample image) as an example. In another embodiment, the above process can also be performed for a batch of samples, i.e., multiple image-text pairs, to obtain the third generated image corresponding to the third sample description in each image-text pair, and then based on the difference between the third generated image corresponding to the third sample description in each image-text pair and the second sample image in the image-text pair, determine the fourth generation loss (the fourth generation loss can be the first sum of the generation losses determined by the second sample image and the third generated image corresponding to the third sample description in all image-text pairs, or the average of the first sum), and adjust the parameters of the image generation network to be trained with the goal of minimizing the fourth generation loss. In this embodiment, the fourth generation loss is determined for a batch of samples, and then the parameters of the image generation network to be trained are adjusted, so that the number of parameter adjustments of the image generation network can be reduced, making it easier to implement the above process.

[0065] In one implementation, the image generation network may be an image generation network based on a Diffusion Model.

[0066] The process of training a text generation network may include: constructing a sample description pair set D for pre-training the text generation network to be trained, wherein the sample description pair set D may include a plurality of sample description pairs, wherein the construction process of a sample description pair Di may be as follows: obtaining a fourth sample description yi (for example, it may be expressed as {(a)(sunny)(windless)(of)(morning)}, or it may be expressed as {(a)(a)(sunny)(bright)(without)(wind)(of)(morning)(up)}, wherein the content in each bracket may represent a word in the fourth sample description yi, and the first expression is used as an example below for explanation), and including it in the sample description pair Di; using a preset occlusion algorithm, partially occluding the content of the fourth sample description yi, and obtaining the corresponding occluded fourth sample description as the target occluded description xi (for example, it may be expressed as {(a)(?)(?)(of)(morning)}, wherein the question mark “?” represents the occluded word), and including it in the sample description pair Di, that is, a sample description pair Di includes a fourth sample description yi and its corresponding target occluded description xi.

[0067] The preset masking algorithm may be any algorithm in the related art that can implement text masking. In one implementation, the preset masking algorithm may be used to randomly mask any content (or words) at any position other than the first sentence (e.g., the sentence before the first punctuation mark) of the fourth sample description yi; or randomly mask any words at any position other than the first a words of the fourth sample description yi, where a is a positive integer.

[0068] Afterwards, obtain the text generation network to be trained, obtain any sample description pair Di from the sample description pair set D, use the target occlusion description xi in the sample description pair Di, and pass the text generation network to be trained to obtain the output of the text generation network to be trained (hereinafter referred to as the first output for the sake of clear description), and then use the first output and the fourth sample description yi in the sample description pair Di to determine the generation loss corresponding to the text generation network to be trained (called the fifth generation loss).

[0069] In one implementation, the first output may include a first word probability set corresponding to each position in a plurality of positions, the first word probability set corresponding to each position including, for the target occlusion description xi, the first probability of each word (token) in the preset word set determined by the text generation network to be trained to appear at the position, the preset word set including a set of words that can be predicted by the text generation network to be trained, and each fourth sample description yi may include a plurality of words in the preset word set. The specific number of the plurality of positions may be determined based on the specified parameters set for the text generation network to be trained, indicating the longest length of the output.

[0070] Afterwards, based on the fourth sample description yi, from the first output, the first probability of the word at the corresponding position of the fourth sample description yi appearing in each occluded position in the target occlusion description xi (or each position in the target occlusion description xi) can be determined for the text generation network to be trained (for example, in the aforementioned example, the probability of the word "clear" appearing at the first occluded position in the target occlusion description xi, that is, the position of the second word, and the probability of the word "windless" appearing at the second occluded position in the target occlusion description xi, that is, the position of the third word).

[0071] Then, the cross entropy loss function is used to determine the fifth generation loss based on the first probability of the words at the corresponding positions of the fourth sample description yi appearing at each occluded position in the target occlusion description xi (or each position in the target occlusion description xi). The greater the first probability of the words at the corresponding positions of the fourth sample description yi appearing at each occluded position in the target occlusion description xi (or each position in the target occlusion description xi), the smaller the fifth generation loss, indicating that the image description generation ability of the text generation network to be trained is better. Correspondingly, in one case, the fifth generation loss l can be expressed by the following formula:

[0072]

[0073] Wherein, p(yi|xi) represents the probability that the text generation network to be trained outputs the fourth sample description yi for the target occluded description xi, and the p(yi|xi) can be determined based on the first probability that the words at the corresponding positions of the fourth sample description yi appear at each occluded position in the target occluded description xi (or each position in the target occluded description xi) (for example: based on the average value of the sum of the first probabilities of the words at the corresponding positions of the fourth sample description yi appearing at each occluded position in the target occluded description xi, or the result of the multiplication); D represents the aforementioned sample description pair set D to which the fourth sample description yi and the target occluded description xi belong; Indicates the calculation of expected value.

[0074] In another implementation, the first output includes a first word probability set corresponding to each of the several positions, and then the target program can determine the predicted description corresponding to the target occlusion description xi based on the first output, for example, by combining the specified first word determination rule and the first output to determine the predicted description. Then, based on the difference between the predicted description and the fourth sample description (for example, the difference between the two is represented by the Euclidean distance between the two, and for example, the difference between the predicted description and the fourth sample description is determined by using the MSE loss function), the fifth generation loss is determined, wherein the smaller the difference between the predicted description and the fourth sample description, the smaller the fifth generation loss. The first word determination rule can be any rule in the related art that can determine the corresponding word according to the word probability set corresponding to each of the several positions mentioned above (the word at one / some positions can be empty). In one implementation, the parameters in the first word determination rule can include pre-set parameters, and can also include parameters adjusted following the training process of the text generation network to be trained.

[0075] In yet another implementation, the first output may directly include the predicted description corresponding to the aforementioned target occlusion description xi, and accordingly, the fifth generation loss may be determined directly based on the difference between the predicted description and the fourth sample description (eg, the Euclidean distance).

[0076] Then, in order to minimize the fifth generation loss, that is, to maximize the probability that the text generation network to be trained outputs the fourth sample description yi for the target occlusion description xi, that is, to reduce the difference between the aforementioned predicted description and the fourth sample description, the parameters of the text generation network to be trained are adjusted until the text generation network reaches the second convergence condition, thereby obtaining the text generation network in the aforementioned multimodal generation model. The second convergence condition may include, but is not limited to, the number of parameter adjustments is not less than the second adjustment threshold, or the calculated fifth generation loss is less than the second loss threshold. The second adjustment threshold may be the same as or different from the aforementioned first adjustment threshold, and the second loss threshold may be the same as or different from the aforementioned first loss threshold.

[0077] Let's review the process of obtaining the text generation network through the aforementioned training. The above embodiment is explained by taking a sample description pair (i.e., a pair of fourth sample descriptions and their corresponding target occlusion descriptions) as an example. In another embodiment, the above process can also be performed on a batch of samples, i.e., multiple sample description pairs, to obtain the outputs of the text generation network to be trained for processing the target occlusion description in each sample description pair, and then determine the fifth generation loss based on the output corresponding to the target occlusion description in each sample description pair and the fourth sample description in each sample description pair, and adjust the parameters of the text generation network to be trained with the goal of minimizing the fifth generation loss. In this embodiment, the fifth generation loss is determined for a batch of samples, and then the parameters of the text generation network are adjusted. This can reduce the number of parameter adjustments to the text generation network and make it easier to implement the above process.

[0078] In one implementation, the text generation network may be a text generation network implemented based on a Transformer network.

[0079] After obtaining the image generation network and the text generation network, the image generation network and the text generation network, as well as the multimodal generation network, are used to construct a multimodal generation model, and then the multimodal generation model is trained. Figure 2 A flow chart of a method for training a multimodal generative model in one embodiment of the present disclosure is shown. The method can be implemented by a target program. The target program can be installed in an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 2 As shown, the multimodal generation model includes an image generation network, a text generation network and a multimodal generation network, and the method includes the following steps S210-S270:

[0080] In step S210, an arbitrary first dialogue sample in the training sample set is obtained, where the first dialogue sample includes a first sample description and its corresponding second sample description and a sample instruction, where the sample instruction is used to indicate a generation direction of the second sample description.

[0081] In one implementation, the target program may first construct a training sample set before training the multimodal generation model. The training sample set may include multiple dialogue samples, and the acquisition process of any first dialogue sample in the training sample set may specifically include the following steps 11-13: In step 11, a first sample description is acquired, and the first sample description is included in the first dialogue sample.

[0082] In one implementation, the first sample description may be a description that satisfies a specified condition. Specifically, the first sample description and its corresponding image (i.e., image-text pair) may be an image-text pair in a specified public data set whose quality exceeds a preset screening condition. The preset screening condition may include, but is not limited to, at least one of the following: the aesthetic score corresponding to the image in the image-text pair is not less than a preset aesthetic score threshold (for example, when the aesthetic score range is [0, 1], the preset aesthetic score threshold is, for example, 0.75), the clarity score corresponding to the image in the image-text pair is not less than a preset clarity threshold (for example, when the clarity score range is [0, 1], the preset clarity threshold is, for example, 0.80), and the matching value corresponding to the image-text pair is not less than a preset matching threshold (for example, when the matching value range is [0, 1], the preset matching threshold is, for example, 0.80). Accordingly, the target program may determine the image description in the image-text pair in the specified public data set whose quality exceeds the preset screening condition as the first sample description.

[0083] In one implementation, the aesthetic score corresponding to the image in the image-text pair can be determined by a trained aesthetic prediction model based on the image in the image-text pair. The aesthetic prediction model can be any model in the related art that can evaluate the aesthetics of an image, i.e., the aesthetic score. The larger the aesthetic score corresponding to the image, the more beautiful the image is and the better the image quality. In one implementation, the aesthetic prediction model can be a model pre-trained based on the image and its corresponding aesthetic score label, and the aesthetic prediction model can output the aesthetic score of the image for the input image. The images and their corresponding aesthetic score labels used to train the aesthetic prediction model can be selected from any data set including a large number of images and their corresponding aesthetic score labels that characterize the aesthetics of the image.

[0084] Based on the image in the image-text pair, a gradient value corresponding to the image in the image-text pair can be determined by a preset image gradient algorithm (for example, the Tenengrad gradient algorithm), and then based on the gradient value corresponding to the image, a clarity score corresponding to the image can be determined (for example, based on the correspondence between the preset gradient value and the clarity score, and the gradient value corresponding to the image, the clarity score corresponding to the image is determined), wherein the larger the gradient value corresponding to the image, the larger the clarity score corresponding to the image, and the better the image quality.

[0085] Based on the image-text pair, the matching value between the image in the image-text pair and its corresponding image description can be determined through a trained text-image association model, wherein the matching value can represent the matching degree between the image in the corresponding image-text pair and its corresponding image description, and the larger the matching value, the stronger the matching degree between the image in the corresponding image-text pair and its corresponding image description.

[0086] The better the quality of the image in the image-text pair, the stronger the match between the image and its corresponding image description, the better the quality of the image-text pair, and the more conducive it is to the training of multimodal image generation.

[0087] In one case, the text-image association model can be any model in the related art that can determine the matching value between the input image and its corresponding image description. The text-image association model can be, for example, a CLIP model (Contrastive Language-Image Pre-Training).

[0088] Next, in step 12, based on the first sample description, a second sample description is determined, the second sample description includes part of the content of the first sample description, and the second sample description is included in the first conversation sample.

[0089] In one case, the second sample description can be obtained by masking part of the content of the first sample description based on a preset masking algorithm, wherein the preset masking algorithm can be any algorithm in the related art that can implement text masking. In one implementation, using the preset masking algorithm, any content at any position other than the first sentence (for example, the sentence before the first punctuation mark) of the first sample description can be randomly masked, and words at any position other than the first a words of the fourth sample description yi can also be randomly masked, where a is a positive integer.

[0090] In another case, the second sample description corresponding to the first sample description can be obtained by semantically decomposing the first sample description. The first sample description can be semantically decomposed by a specific program or manually to obtain the decomposed second sample description. For example, the first sample description is "a beautiful girl wearing a red skirt and holding a yellow sunflower in her hand", and the second sample description obtained by semantic decomposition of the first sample description can be, for example, "a girl" or "a beautiful girl wearing a red skirt" or "a girl holding a sunflower in her hand". It can be understood that a first sample description can be decomposed into multiple second sample descriptions, and the first sample description can form multiple dialogue samples with the multiple second sample descriptions.

[0091] Afterwards, in step 13, for the second sample description, a sample instruction for indicating the generation direction of the second sample description is obtained. In one implementation, a preset storage space corresponding to the target program may pre-store a dialogue instruction set, wherein the dialogue instruction set includes a number of dialogue instructions, and the dialogue instruction may be, for example, "please add more details for this sentence" or "please add more details for this sentence" or "please modify "XX (target, such as a dog)" in the sentence to "YY (for example, a cat)"". The target program may select a dialogue instruction from the dialogue instruction set for the second sample description as a sample instruction for indicating the generation direction of the second sample description. In an exemplary scenario, when the selected sample instruction is "please modify "XX (target, such as a dog)" in the sentence to "YY (for example, a cat)", the second sample description may continue to be modified, for example, "YY" in the second sample description may be modified to "XX" to obtain a new second sample description that can be classified as the first dialogue sample.

[0092] In yet another implementation, a sample instruction corresponding to the second sample description may be constructed based on the second sample description and its corresponding first sample description through a specific program or manual operation, and the generation direction of the second sample description indicated by the sample instruction may be a direction indicating the generation of a better image description (a higher quality image may be generated) that better matches the first sample description. For example, a sample instruction corresponding to the second sample description is constructed based on the missing content of the second sample description relative to its corresponding first sample description. For example, the first sample description lacks the content describing the image details and the second sample description is obtained. The constructor may construct a sample instruction for the second sample description that indicates the generation of more details. For another example, the first sample description lacks the content describing the weather and the second sample description is obtained. The constructor may construct a sample instruction for the second sample description that indicates the generation of weather content. This is all possible.

[0093] After the target program obtains any first dialogue sample in the training sample set, in step S220, based on the first sample description, a first generated image is obtained through the image generation network. In this step, the target program can input the first sample description into the image generation network, and the image generation network processes the first sample description to obtain a first generated image corresponding to the first sample description, and the corresponding target program obtains the first generated image.

[0094] Then, in step S230, based on the second sample description and the sample instruction, a first generated description is obtained through the text generation network. In this step, the target program inputs the second sample description and the sample instruction into the text generation network, and the text generation network processes the second sample description and the sample instruction to process the second sample description based on the generation direction indicated by the sample instruction to obtain the first generated description.

[0095] In one implementation, based on the second sample description and the sample instruction, the output of the text generation network (for the sake of clarity, it can be referred to as the second output) is obtained through the text generation network. In one case, the second output may include a second word probability set corresponding to each of the several positions, wherein the second word probability set corresponding to each position includes: the second probability of each word (token) in the aforementioned preset word set appearing at the position, determined by the text generation network based on the second sample description and the sample instruction, and the preset word set is a set of words that can be predicted by the text generation network. Based on the second word probability set corresponding to each of the several positions included in the second output, the aforementioned first generated description can be determined. Specifically, the first generated description can be obtained by combining the second word probability set corresponding to each of the several positions included in the second output and the aforementioned first word determination rule. In another case, the second output may also include the first generated description.

[0096] Then in step S240, based on the first generated image and the first generated description, a second generated description is obtained through a multimodal generation network, and the second generated description is related to the output of the multimodal generation network.

[0097] In this step, the target program can input the first generated image and the first generated description into the multimodal generative network, so that the multimodal generative network processes the first generated image and the first generated description to obtain an image representation corresponding to the first generated image and a text representation corresponding to the first generated description, and then fuses the image representation and the text representation to obtain a fused representation that fuses image domain data features and text domain data features, and then based on the fused representation, the second generated description can be obtained.

[0098] In one implementation, based on the first generated image and the first generated description, the output of the multimodal generation network (for the sake of clarity, it can be referred to as the third output) can be obtained through the multimodal generation network. In one case, the third output may include a third word probability set corresponding to each of the multiple positions predicted by the multimodal generation network for the first generated image and the first generated description, and the third word probability set corresponding to each of the multiple positions may include a third probability that each word (token) in the specified word set appears in each of the multiple positions. Among them, the specified word set includes a set of words that can be predicted by the multimodal generation network, and the specified word set can be the same as the aforementioned preset word set. The specific number of the multiple positions can be determined based on the specified parameters for the multimodal generation network, which can characterize the longest length of the output of the multimodal generation network.

[0099] The target program can determine the aforementioned second generated description based on the third word probability set corresponding to each of the multiple positions included in the third output. Specifically, the second generated description is obtained by combining the third word probability set corresponding to each of the multiple positions included in the third output and the specified second word determination rule. The second generated description incorporates image domain data features (image representation corresponding to the first generated image) and text domain data features (text representation corresponding to the first generated description), which contains richer information for generating images. The second word determination rule can be any rule in the related art that can determine the corresponding word based on the third word probability set corresponding to each of the multiple positions mentioned above (the word at one / some positions can be empty). In one implementation, the parameters in the second word determination rule can include pre-set parameters or parameters adjusted following the training process of the multimodal generation network, which are all possible. The specified second word determination rule can be the same as or different from the aforementioned specified first word determination rule.

[0100] The multimodal generative network can be any fusion network in the related art that can fuse images and texts. In one embodiment, the multimodal generative network can include a text encoder, an image encoder, a fuser, and a decoder; accordingly, in step S240, the following steps 21-24 can be included: In step 21, using the first generated image, an image representation is obtained through an image encoder. In this step, the target program can input the first generated image into an image encoder, and the image encoder performs image encoding on the first generated image to obtain an image representation corresponding to the first generated image. The image encoder can be any encoder in the related art that can realize encoding of an image and extracting an image representation from the image.

[0101] Then, in step 22, the first generated description is used to obtain a text representation through a text encoder. In this step, the target program can input the first generated description into a text encoder, and the text encoder performs text encoding on the first generated description to obtain a text representation corresponding to the first generated description. The text encoder can be any encoder in the related art that can encode text and extract text representations from the text.

[0102] Then, in step 23, the image representation and the text representation are used to obtain a fused representation through a fuser. In this step, the image representation and the text representation can be input into the fuser, and the fuser performs a fusion operation on the image representation and the text representation to obtain a fused representation that is fused with the image representation and the text representation. In one implementation, the fusion operation may include but is not limited to a cross attention operation, that is, a fusion operation based on a cross attention mechanism.

[0103] Next, in step 24, the fused representation is used to obtain a second generated description through a decoder. In this step, the fused representation is input into the decoder, and the decoder can decode the fused representation to obtain a decoding result corresponding to the fused representation, and then based on the decoding result corresponding to the fused representation, a second generated description that fuses the image representation and the text representation can be obtained.

[0104] After the target program obtains the second generated description, in step S250, a second generated image is generated based on the second generated description. In one implementation, the target program may input the second generated description into the aforementioned image generation network to obtain a second generated image corresponding to the second generated description. In another implementation, the target program may input the second generated description into another trained image generation network to obtain a second generated image corresponding to the second generated description.

[0105] Then, in step S260, the target generation loss is determined based on the first sample description, the output of the multimodal generation network, and the second generated image. In this step, the target program can determine the target generation loss based on the difference between the generation result of the multimodal generation network represented by the output of the multimodal generation network and the first sample description (i.e., the second generated description) and the first sample description, and the matching degree, i.e., the correlation, between the second generated image and the first sample description.

[0106] Specifically, in one implementation, the target program can use a specified loss function (such as a cross entropy loss function, a mean square error loss function), determine the corresponding generation loss (i.e., the first generation loss) based on the output of the multimodal generation network and the first sample description, and determine the matching loss between the second generated image and the first sample description through the text image association model (called the first matching loss), and then combine the first generation loss and the first matching loss to determine the target generation loss. Among them, the target generation loss is positively correlated with the first generation loss and the first matching loss. Among them, minimizing the first generation loss can enable the text generation network to generate better quality image descriptions, and minimizing the first matching loss can enable the multimodal generation model to generate more accurate image descriptions.

[0107] In one implementation, the output of the multimodal generation network, i.e., the aforementioned third output, may include a third word probability set corresponding to each of the multiple positions predicted by the multimodal generation network for the first generated image and the first generated description, and the aforementioned process of determining the corresponding generation loss (i.e., the first generation loss) may be: using a cross-entropy loss function, i.e., according to the principle of maximizing log-likelihood, based on the third word probability set corresponding to each of the multiple positions in the third output, and the first sample description, constructing a cross-entropy loss between the two, and determining the cross-entropy loss as the first generation loss.

[0108] Specifically, it can be: from the third word probability set corresponding to each position in the multiple positions in the third output, determine the third probability of the word at each first position specified in the third output and the position corresponding to the first position in the first sample description; and then determine the cross entropy loss, i.e., the first generation loss, based on the third probability of the word at each first position specified and the position corresponding to the first position in the first sample description. Among them, the first generation loss is negatively correlated with the third probability of the word at each first position specified in the third output and the position corresponding to the first position in the first sample description, that is, the greater the third probability of the word at each first position specified in the third output and the position corresponding to the first position in the first sample description, it can be represented that the difference between the aforementioned second generation description and the first sample description is smaller, and the corresponding first generation loss is smaller.

[0109] In the above implementation, the second generated description can also be determined based on the third word probability set corresponding to each of the multiple positions included in the third output. Specifically, the second generated description is determined in combination with the third word probability set corresponding to each of the multiple positions included in the third output and the aforementioned second word determination rule, and then based on the difference between the second generated description and the first sample description, the aforementioned first generation loss is determined, wherein the first generation loss is positively correlated with the difference between the second generated description and the first sample description, and the smaller the difference between the second generated description and the first sample description, the smaller the first generation loss.

[0110] In yet another implementation, the output of the multimodal generation network includes the aforementioned second generation description, and the target program may determine the aforementioned first generation loss directly based on the difference between the second generation description and the first sample description.

[0111] The aforementioned process of determining the first matching loss may be: inputting the second generated image and the first sample description into the text image association model to obtain the matching value between the second generated image and the first sample description, and then determining the first matching loss between the two based on the matching value between the second generated image and the first sample description. It can be understood that the value range of the matching value between the second generated image and the first sample description may be [0, 1], and the larger the correlation value between the two, the stronger the matching degree, i.e., the correlation, between the second generated image and the first sample description. The first matching loss may be equal to the difference between 1 and the matching value between the second generated image and the first sample description, and the smaller the difference, the better the matching degree between the second generated image and the first sample description.

[0112] In one implementation, the text-image association model may be any model in the related art that can be used to determine the correlation between an image and text, for example, a CLIP model.

[0113] In one embodiment, in order to ensure that the second generated description is more accurate, the target generation loss can also be determined based on the relationship between the first generated image and the second generated description. Specifically, in step S260, it can specifically include: determining the target generation loss based on the first generated image, the first sample description, the output of the multimodal generation network and the second generated image.

[0114] Specifically, step S260 may include the following steps 31-33: In step 31, based on the output of the multimodal generation network and the first sample description, a first generation loss is determined. Specifically, reference may be made to the aforementioned process of determining the first generation loss, which will not be described in detail here.

[0115] Then, in step 32, based on the second generated image and the first sample description, a first matching loss is determined through a text-image association model; and based on the first generated image and the second generated description, a second matching loss is determined through a text-image association model. Specifically, the process of determining the second matching loss can refer to the process of determining the first matching loss described above, and is not limited here.

[0116] Then, in step 33, the target generation loss is determined by using the first generation loss, the first matching loss and the second matching loss. In this step, the target generation loss is determined by combining the first generation loss, the first matching loss and the second matching loss, wherein the target generation loss is positively correlated with the first generation loss, the first matching loss and the second matching loss, respectively. Determining the target generation loss by combining the first generation loss can prompt the text generation network to generate a higher quality image description, so that the multimodal generation network can learn the image generation content of the image generation network, and then the fusion of the image generated by the text generation network and the text (language) domain data (i.e., the first generation description) can be achieved; and then combining the first matching loss and the second matching loss to determine the target generation loss can better integrate the image generation capability of the image generation network in the multimodal generation model and the text generation capability of the text generation network, and then better integrate the image domain data and the text (language) domain data, and ensure the accuracy of the generated image description (i.e., the output of the multimodal generation network), and then improve the quality of the generated image (i.e., the image generated based on the second generation description) while ensuring the accuracy of the generated image.

[0117] In one embodiment, in order to ensure the stability of the image generation capability of the image generation network and to better improve the ability of the multimodal generation model to generate higher quality image descriptions, the first dialogue sample also includes a first sample image corresponding to the first sample description;

[0118] In step S260, the following steps are also specifically included: based on the difference between the first sample image and the first generated image, determine the second generation loss corresponding to the image generation network; and use the second generation loss, the first generation loss, the first matching loss, and the second matching loss to determine the target generation loss. The target program can use a preset loss function to determine the second generation loss corresponding to the image generation network based on the difference between the first sample image and the first generated image, wherein the second generation loss is positively correlated with the difference between the first sample image and the first generated image. In this embodiment, on the basis of the first generation loss, the first matching loss, and the second matching loss, the difference between the first sample image and the first generated image, i.e., the second generation loss, is combined to determine the target generation loss, so that the parameters of the image generation network can be fine-tuned in combination with the training sample set, i.e., the image generation ability of the image generation network can be fine-tuned, so that the image generation network can generate better images based on the image description, further improving the generation ability of the image description of the entire multimodal generation model.

[0119] Specifically, step S260 may include the following steps 41-44. In step 41, a first generation loss is determined based on the output of the multimodal generation network and the first sample description. Step 41 is the same as the aforementioned step 31 and will not be described in detail herein.

[0120] In step 42, based on the difference between the first sample image and the first generated image, a second generation loss corresponding to the image generation network is determined. In this step, a preset loss function is used to determine the second generation loss corresponding to the image generation network based on the difference between the first sample image and the first generated image. The preset loss function may include but is not limited to a cross entropy loss function, an MSE loss function, and a MAE loss function.

[0121] In step 43, based on the second generated image and the first sample description, a first matching loss is determined by a text-image association model; and based on the first generated image and the second generated description, a second matching loss is determined by a text-image association model. Here, step 43 is the same as step 32 above, and will not be described in detail here.

[0122] In step 44, the target generation loss is determined using the first generation loss, the second generation loss, the first matching loss and the second matching loss. In this step, the target generation loss is positively correlated with the first generation loss, the second generation loss, the first matching loss and the second matching loss, respectively.

[0123] In another embodiment, in step 44, it may also specifically include: determining the third generation loss corresponding to the text generation network based on the first sample description and the output of the text generation network; determining the target generation loss using the third generation loss, the first generation loss, the second generation loss, the first matching loss and the second matching loss. Among them, the target generation loss is also positively correlated with the third generation loss. In this embodiment, the aforementioned specified loss function (such as the cross entropy loss function and the mean square error loss function) may be used to determine the third generation loss corresponding to the text generation network based on the first sample description and the output of the text generation network. In this embodiment, on the basis of the first generation loss, the second generation loss, the first matching loss and the second matching loss, the third generation loss determined based on the output of the first sample description and the text generation network is combined to determine the target generation loss, so as to better improve the ability of the text generation network to generate image descriptions, and thus better improve the ability of the multimodal generation model to generate better quality image descriptions.

[0124] After obtaining the target generation loss in the aforementioned manner, in step S270, the parameters of the multimodal generation model are adjusted with the goal of minimizing the target generation loss. In this step, the parameters of the multimodal generation model are adjusted with the goal of minimizing the target generation loss, that is, with the goal of reducing the difference between the output of the multimodal generation network and the generation result of the multimodal generation network represented by the first sample description (i.e., the second generation description) and the first sample description, reducing the matching loss between the second generation description and the first generation image, reducing the matching loss between the second generation image and the first sample description (as well as reducing the difference between the first sample image and the first generation image, reducing the difference between the first sample description and the generation result of the text generation network represented by the output of the text generation network (i.e., the first generation description) and the first sample description), so as to enhance the ability of the multimodal generation model to generate higher quality image descriptions, so that the quality of the image descriptions generated by it is better, thereby ensuring that better quality images are generated using the better quality image descriptions.

[0125] Let's review the execution process of the aforementioned steps S210-S270. The above embodiment is described by taking a first dialogue sample as one sample. In another embodiment, the above steps S210-S260 can also be performed for a batch of samples, i.e., multiple first dialogue samples, to obtain the first generated image, the first sample description, the output of the multimodal generation network, and the second generated image (as well as the second generated description determined based on the output of the multimodal generation network, the first generated description determined based on the output of the text generation network, etc.) corresponding to each first dialogue sample, respectively, and then determine the target generation loss based on the above data corresponding to each first dialogue sample in step S260, and adjust the parameters of the multimodal generation model with the goal of minimizing the target generation loss in step S270. In this embodiment, the target generation loss is determined for a batch of samples, and then the parameters of the multimodal generation model are adjusted, which can reduce the number of parameter adjustments for the multimodal generation model and make it easier to implement the above process.

[0126] Adjust the parameters of the multimodal generation model until the multimodal generation model reaches a preset convergence condition, thereby obtaining a trained multimodal generation model. The preset convergence condition may include but is not limited to the number of times the multimodal generation model parameter is adjusted exceeding a preset number threshold, or the target generation loss is lower than a preset generation loss threshold.

[0127] In this embodiment, the idea of ​​multimodal learning is utilized to construct various fine-grained losses and adjust the multimodal generation model to enhance the ability of the multimodal generation model to generate higher quality image descriptions. The multimodal generation model can then be utilized to obtain higher quality image descriptions, thereby improving the quality of images generated based on the image descriptions.

[0128] Among them, the multimodal generation model can be a conversational multimodal generation model. For the trained multimodal generation model, the trained multimodal generation model can be used to generate a corresponding high-quality image description based on the data to be processed (the image to be processed and / or the image description to be processed) input by the user and the instructions (conversation content) input by the user indicating the generation direction of the data to be processed, and then generate a high-quality image based on the high-quality image description.

[0129] Corresponding to the above method embodiment, the embodiment of the present disclosure further provides an image generation method, which can be implemented by a first program. The first program can be the same physical program as the aforementioned target program, or can be a different physical program from the aforementioned target program, such as Figure 3 As shown, the image generation method may include the following steps S310-S340:

[0130] In step S310, data to be processed and a first instruction for indicating a generation direction of the data to be processed are obtained.

[0131] It is understandable that the image generation method provided in the embodiment of the present disclosure may be an image generation method based on a target multimodal generation model, wherein the target multimodal generation model may be a multimodal generation model trained by the aforementioned training method for the multimodal generation model.

[0132] In one implementation, the data to be processed and the first instruction are input by a user who has a need to generate an image. The data to be processed may be data in any form. In one embodiment, the data to be processed may be data in text form. Accordingly, the data to be processed includes a description of the image to be processed, i.e., a text prompt for generating an image. The user has a need to generate an image based on the description of the image to be processed and the first instruction. In another embodiment, the data to be processed may be data in image form. Accordingly, the data to be processed includes an image to be processed. The user has a need to generate a better quality image based on the image to be processed that is more in line with the intention (generation direction) expressed by the user through the first instruction.

[0133] In another implementation, the data to be processed may also be data in text form (i.e., image description) output by the target multimodal generation model, or may also be data in image form (i.e., generated image) generated by a target image generation network of the target multimodal generation model based on the data in text form output by the target multimodal generation model. The first instruction is data (in text form) input by the user indicating the generation direction of the data to be processed.

[0134] In one embodiment, the data to be processed includes an image to be processed (i.e., data in the form of an image). The image to be processed may be any image, and the first instruction may be an instruction input by a user to indicate a direction for generating the image to be processed, for example, indicating to generate more detailed content or to generate specified object content for the image to be processed, etc.

[0135] In another embodiment, the data to be processed includes a description of an image to be processed (i.e., data in text form). The first instruction may be an instruction input by a user indicating a direction for generating the description of the image to be processed, for example, indicating generating an image description containing more details or a specified object for the description of the image to be processed.

[0136] After the first program obtains the data to be processed and the first instruction, in step S320, based on the first instruction, a third generation description is obtained through a target text generation network of a target multimodal generation model.

[0137] In one implementation, when the data to be processed includes an image to be processed, the first program may input the first instruction into a target text generation network of a target multimodal generation model, and the target text generation network processes the first instruction to obtain a third generation description.

[0138] In another implementation, when the aforementioned data to be processed includes a description of an image to be processed, step S320 may specifically include: using the description of the image to be processed and the first instruction, through the target text generation network, to obtain a third generated description. In this step, the first program may input the description of the image to be processed and the first instruction into the target text generation network, and the target text generation network processes the description of the image to be processed and the first instruction, that is, processes the description of the image to be processed based on the generation direction indicated by the first instruction, to obtain the third generated description.

[0139] Next, in step S330, based on the data to be processed and the third generated description, a fourth generated description is obtained through a target multimodal generation network of a target multimodal generation model.

[0140] In one implementation, when the data to be processed includes an image to be processed, the first program can input the image to be processed and the third generation description into a target multimodal generation network of a target multimodal generation model, and the target multimodal generation network processes the image to be processed and the third generation description to obtain a fourth generation description, which incorporates features of the image to be processed and features of the third generation description.

[0141] In another implementation, when the data to be processed includes a description of an image to be processed, step S330 may specifically include: using the description of the image to be processed, through a target image generation network, to obtain a fourth generated image; using the third generated description and the fourth generated image, through a target multimodal generation network, to obtain a fourth generated description. In this implementation, the first program inputs the description of the image to be processed into the target image generation network of the target multimodal generation model to obtain a generated image corresponding to the description of the image to be processed, i.e., the fourth generated image; then, the third generated description and the fourth generated image are input into the target multimodal generation network to obtain a fourth generated description with better quality that incorporates features of the fourth generated image and features of the third generated description.

[0142] Afterwards, in step S340, based on the fourth generation description, a generated image corresponding to the data to be processed and the first instruction is obtained through the target image generation network of the target multimodal generation model. In this step, the first program inputs the fourth generation description into the target image generation network of the target multimodal generation model to obtain a generated image corresponding to the data to be processed and the first instruction.

[0143] In the above process, at least based on the first instruction representing the generation direction of the data to be processed indicated by the user, a third generation description is obtained, and then a fourth generation description is obtained by fusing the information of the data to be processed and the third generation description, and then a higher quality generated image can be obtained based on the fourth generation description.

[0144] In one implementation, the target multimodal generation model can generate corresponding image descriptions for generating better quality illustrations for data in text form such as novels and articles based on instructions indicating the generation direction, and then generate corresponding illustrations based on the image descriptions; it can also generate image descriptions for obtaining better quality images for original images based on instructions indicating the generation direction, and then generate better quality images based on the images. Accordingly, the multimodal generation model trained using the training method for the multimodal generation model provided in the embodiment of the present disclosure can optimize the input image description or input image through a conversational approach (i.e., the instructions indicating the generation direction input by the user) to obtain a good quality image description that meets the user's intention, and then obtain a better quality image.

[0145] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible, or may be advantageous.

[0146] Corresponding to the above method embodiment, the present disclosure embodiment provides a training device 400 for a multimodal generation model, wherein the multimodal generation model includes an image generation network, a text generation network, and a multimodal generation network, and its schematic block diagram is as follows: Figure 4 As shown, it is used to execute Figure 2 The method shown comprises:

[0147] A first acquisition module 410 is configured to acquire any first dialogue sample in a training sample set, wherein the first dialogue sample includes a first sample description and its corresponding second sample description and a sample instruction, wherein the sample instruction is used to indicate a generation direction of the second sample description;

[0148] A first obtaining module 420 is configured to obtain a first generated image through the image generation network based on the first sample description;

[0149] A second obtaining module 430 is configured to obtain a first generated description through the text generation network based on the second sample description and the sample instruction;

[0150] a third obtaining module 440, configured to obtain a second generated description through the multimodal generation network based on the first generated image and the first generated description, wherein the second generated description is related to an output of the multimodal generation network;

[0151] A generating module 450, configured to generate a second generated image based on the second generated description;

[0152] A first determination module 460, configured to determine a target generation loss based on the first sample description, the output of the multimodal generation network and the second generated image;

[0153] The first adjustment module 470 is configured to adjust the parameters of the multimodal generation model with the goal of minimizing the target generation loss.

[0154] In an optional implementation, it also includes:

[0155] A third acquisition module (not shown in the figure), configured to acquire the first sample description;

[0156] A first occlusion module (not shown in the figure), configured to determine the second sample description based on the first sample description, where the second sample description includes part of the content of the first sample description;

[0157] The fourth acquisition module (not shown in the figure) is configured to acquire, for the second sample description, a sample instruction for indicating a generation direction of the second sample description.

[0158] In an optional implementation, the first determination module 460 is specifically configured to determine the target generation loss based on the first generated image, the first sample description, the output of the multimodal generation network and the second generated image.

[0159] In an optional implementation manner, the first determining module 460 includes:

[0160] A first determining unit (not shown in the figure), configured to determine a first generation loss based on an output of the multimodal generation network and the first sample description;

[0161] a second determining unit (not shown in the figure), configured to determine a first matching loss based on the second generated image and the first sample description through a text-image association model; and to determine a second matching loss based on the first generated image and the second generated description through the text-image association model;

[0162] The third determining unit (not shown in the figure) is configured to determine the target generation loss by using the first generation loss, the first matching loss and the second matching loss.

[0163] In an optional implementation, the first dialogue sample further includes a first sample image corresponding to the first sample description;

[0164] The third determining unit (not shown in the figure) is specifically configured to determine a second generation loss corresponding to the image generation network based on a difference between the first sample image and the first generated image;

[0165] The target generation loss is determined using the second generation loss, the first generation loss, the first matching loss, and the second matching loss.

[0166] In an optional implementation, the third determination unit (not shown in the figure) is specifically configured to determine a third generation loss corresponding to the text generation network based on the first sample description and the output of the text generation network, wherein the output of the text generation network is related to the first generation description;

[0167] The target generation loss is determined using the third generation loss, the first generation loss, the second generation loss, the first matching loss, and the second matching loss.

[0168] In an optional implementation, the multimodal generation network includes a text encoder, an image encoder, a fusion unit, and a decoder;

[0169] The third obtaining module 440 is specifically configured to obtain an image representation by using the first generated image through the image encoder;

[0170] Using the first generated description, through the text encoder, to obtain a text representation;

[0171] Using the image representation and the text representation, through the fuser, to obtain a fused representation;

[0172] The second generated description is obtained by using the fused representation through the decoder.

[0173] In an optional embodiment, the device further comprises:

[0174] a fifth acquisition module (not shown in the figure), configured to acquire a third sample description and its corresponding second sample image before obtaining the first generated image;

[0175] a seventh obtaining module (not shown in the figure), configured to obtain a third generated image based on the third sample description through an image generation network to be trained;

[0176] A second determination module (not shown in the figure) is configured to determine a fourth generation loss corresponding to the image generation network to be trained by using a difference between the second sample image and the third generated image;

[0177] The second adjustment module (not shown in the figure) is configured to adjust the parameters of the image generation network to be trained with the goal of minimizing the fourth generation loss to obtain the image generation network that meets the first convergence condition.

[0178] In an optional embodiment, the device further comprises:

[0179] A sixth acquisition module (not shown in the figure), configured to acquire a fourth sample description before obtaining the first generated description;

[0180] A second occlusion module (not shown in the figure) is configured to use a preset occlusion algorithm to occlude part of the content described by the fourth sample to obtain a target occlusion description;

[0181] An eighth obtaining module (not shown in the figure) is configured to use the target occlusion description to obtain the output of the text generation network to be trained through the text generation network to be trained;

[0182] A third determination module (not shown in the figure) is configured to determine a fifth generation loss corresponding to the text generation network to be trained by using the output of the text generation network to be trained and the fourth sample description;

[0183] The third adjustment module (not shown in the figure) is configured to adjust the parameters of the text generation network to be trained with the goal of minimizing the fifth generation loss to obtain the text generation network that meets the second convergence condition.

[0184] Corresponding to the above method embodiment, the present disclosure provides an image generating device 500, whose schematic block diagram is as follows: Figure 5 As shown, it is used to execute Figure 3 The method shown, the device comprises:

[0185] A second acquisition module 510 is configured to acquire data to be processed and a first instruction for indicating a generation direction of the data to be processed;

[0186] A fourth obtaining module 520 is configured to obtain a third generated description based on the first instruction through a target text generation network of a target multimodal generation model;

[0187] A fifth obtaining module 530 is configured to obtain a fourth generation description through a target multimodal generation network of the target multimodal generation model based on the data to be processed and the third generation description;

[0188] The sixth obtaining module 540 is configured to obtain a generated image corresponding to the data to be processed and the first instruction through the image generation network of the target multimodal generation model based on the fourth generation description.

[0189] In an optional implementation, the data to be processed includes an image to be processed.

[0190] In an optional implementation, the data to be processed includes a description of an image to be processed;

[0191] The fourth obtaining module 520 is specifically configured to obtain the third generated description by using the to-be-processed image description and the first instruction through the target text generation network;

[0192] The fifth obtaining module 530 is specifically configured to use the description of the image to be processed to obtain a fourth generated image through the target image generation network; and use the third generated description and the fourth generated image to obtain the fourth generated description through the target multimodal generation network.

[0193] In an optional implementation, the target multimodal generation model is a model trained using the aforementioned multimodal generation model training device.

[0194] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0195] The present disclosure provides an electronic device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 2 or Figure 3 The method shown below. Figure 6 , which shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0196] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 403. In RAM 603, various programs and data required for the operation of the electronic device are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0197] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0198] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0199] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the following steps provided in the present disclosure: Figure 2 The training method of the multimodal generative model shown in Figure 3 The image generation method shown.

[0200] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0201] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains any first dialogue sample in the training sample set, the first dialogue sample includes a first sample description and its corresponding second sample description and a sample instruction, the sample instruction is used to indicate the generation direction of the second sample description; based on the first sample description, obtains a first generated image through the image generation network of the multimodal generation model; based on the second sample description and the sample instruction, obtains a first generated description through the text generation network of the multimodal generation model; based on the first generated image and the first generated description, obtains a second generated description through the multimodal generation network of the multimodal generation model, the second generated description is related to the output of the multimodal generation network; based on the second generated description, generates a second generated image; based on the first sample description, the output of the multimodal generation network and the second generated image, determines the target generation loss; and adjusts the parameters of the multimodal generation model with the goal of minimizing the target generation loss. Alternatively, the electronic device: obtains data to be processed and a first instruction for indicating a generation direction of the data to be processed; based on the first instruction, obtains a third generation description through a target text generation network of a target multimodal generation model; based on the data to be processed and the third generation description, obtains a fourth generation description through a target multimodal generation network of the target multimodal generation model; based on the fourth generation description, obtains a generated image corresponding to the data to be processed and the first instruction through a target image generation network of the target multimodal generation model.

[0202] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0203] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0204] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0205] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A training method for a multimodal generative model, the multimodal generative model comprising an image generation network, a text generation network and a multimodal generation network, the method comprising: Obtaining any first dialogue sample in a training sample set, where the first dialogue sample includes a first sample description and a corresponding second sample description and a sample instruction, where the sample instruction is used to indicate a generation direction of the second sample description; Based on the first sample description, obtaining a first generated image through the image generation network; Based on the second sample description and the sample instruction, obtaining a first generated description through the text generation network; Based on the first generated image and the first generated description, obtaining a second generated description through the multimodal generation network, wherein the second generated description is related to an output of the multimodal generation network; generating a second generated image based on the second generated description; Determining a target generation loss based on the first sample description, the output of the multimodal generation network, and the second generated image; The parameters of the multimodal generation model are adjusted with the goal of minimizing the target generation loss.

2. The method of claim 1, further comprising: Obtaining the first sample description; Based on the first sample description, determine the second sample description, where the second sample description includes part of the content of the first sample description; For the second sample description, a sample instruction for indicating a generation direction of the second sample description is obtained.

3. The method of claim 1, wherein: The determining of the target generation loss comprises: The target generation loss is determined based on the first generated image, the first sample description, an output of the multimodal generation network, and the second generated image.

4. The method of claim 3, wherein: The determining of the target generation loss comprises: Determining a first generation loss based on an output of the multimodal generation network and the first sample description; Determining a first matching loss based on the second generated image and the first sample description through a text-image association model; and determining a second matching loss based on the first generated image and the second generated description through the text-image association model; The target generation loss is determined using the first generation loss, the first matching loss, and the second matching loss.

5. The method of claim 4, wherein: The first dialogue sample also includes a first sample image corresponding to the first sample description; The determining the target generation loss by using the first generation loss, the first matching loss, and the second matching loss includes: Determining a second generation loss corresponding to the image generation network based on a difference between the first sample image and the first generated image; The target generation loss is determined using the second generation loss, the first generation loss, the first matching loss, and the second matching loss.

6. The method of claim 5, wherein: The determining the target generation loss by using the second generation loss, the first generation loss, the first matching loss and the second matching loss comprises: Determining a third generation loss corresponding to the text generation network based on the first sample description and the output of the text generation network, wherein the output of the text generation network is related to the first generation description; The target generation loss is determined using the third generation loss, the first generation loss, the second generation loss, the first matching loss, and the second matching loss.

7. The method of claim 1, wherein: The multimodal generation network includes a text encoder, an image encoder, a fusion unit and a decoder; The second generation description is obtained, including Using the first generated image, through the image encoder, to obtain an image representation; Using the first generated description, through the text encoder, to obtain a text representation; Using the image representation and the text representation, through the fuser, to obtain a fused representation; The second generated description is obtained by using the fused representation through the decoder.

8. The method according to claim 1, before obtaining the first generated image, further comprising: Obtaining a third sample description and its corresponding second sample image; Based on the third sample description, obtaining a third generated image through an image generation network to be trained; Determine a fourth generation loss corresponding to the image generation network to be trained by using a difference between the second sample image and the third generated image; With the goal of minimizing the fourth generation loss, the parameters of the image generation network to be trained are adjusted to obtain the image generation network that meets the first convergence condition.

9. The method according to claim 1, before obtaining the first generation description, further comprising: Get a fourth sample description; Using a preset occlusion algorithm, occlude part of the content described by the fourth sample to obtain a target occlusion description; Using the target occlusion description, through a text generation network to be trained, to obtain an output of the text generation network to be trained; Determine a fifth generation loss corresponding to the text generation network to be trained by using the output of the text generation network to be trained and the fourth sample description; With the goal of minimizing the fifth generation loss, the parameters of the text generation network to be trained are adjusted to obtain the text generation network that meets the second convergence condition.

10. A method for generating an image, comprising: Acquire data to be processed and a first instruction for indicating a generation direction of the data to be processed; Based on the first instruction, obtaining a third generated description through a target text generation network of a target multimodal generation model; Based on the data to be processed and the third generation description, a fourth generation description is obtained through a target multimodal generation network of the target multimodal generation model; Based on the fourth generation description, a generated image corresponding to the data to be processed and the first instruction is obtained through a target image generation network of the target multimodal generation model.

11. The method of claim 10, wherein: The data to be processed includes an image to be processed.

12. The method of claim 10, wherein: The data to be processed includes a description of an image to be processed; The obtaining of the third generation description includes: Using the to-be-processed image description and the first instruction, through the target text generation network, to obtain the third generated description; The obtaining of the fourth generation description includes: Using the description of the image to be processed, through the target image generation network, a fourth generated image is obtained; The fourth generated description is obtained by using the third generated description and the fourth generated image through the target multimodal generation network.

13. The method according to any one of claims 10 to 12, wherein: The target multimodal generation model is a model trained using the multimodal generation model training method described in any one of claims 1-9.

14. A training device for a multimodal generative model, the multimodal generative model comprising an image generation network, a text generation network and a multimodal generation network, the device comprising: A first acquisition module is configured to acquire any first dialogue sample in a training sample set, wherein the first dialogue sample includes a first sample description and a corresponding second sample description and a sample instruction, wherein the sample instruction is used to indicate a generation direction of the second sample description; A first obtaining module is configured to obtain a first generated image through the image generation network based on the first sample description; A second obtaining module is configured to obtain a first generated description through the text generation network based on the second sample description and the sample instruction; a third obtaining module, configured to obtain a second generated description through the multimodal generation network based on the first generated image and the first generated description, wherein the second generated description is related to an output of the multimodal generation network; A generating module, configured to generate a second generated image based on the second generated description; A first determination module configured to determine a target generation loss based on the first sample description, the output of the multimodal generation network, and the second generated image; The first adjustment module is configured to adjust the parameters of the multimodal generation model with the goal of minimizing the target generation loss.

15. An image generating device, comprising: A second acquisition module is configured to acquire data to be processed and a first instruction for indicating a generation direction of the data to be processed; a fourth obtaining module configured to obtain a third generated description based on the first instruction through a target text generation network of a target multimodal generation model; A fifth obtaining module, based on the data to be processed and the third generation description, obtains a fourth generation description through a target multimodal generation network of the target multimodal generation model; The sixth obtaining module obtains, based on the fourth generation description, a generated image corresponding to the data to be processed and the first instruction through a target image generation network of the target multimodal generation model.

16. An electronic device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 13 is implemented.

17. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 13.

Images