Model training method and apparatus, image generation method and apparatus, and device, medium and product

Abstract

Disclosed in the present application are a model training method and apparatus, an image generation method and apparatus, and a device, a medium and a product. The model training method comprises: firstly, acquiring sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, wherein the label image satisfies constraints described by the sample text and constraints described by the reference image; then, using a target model to perform first processing on the reference image, so as to obtain image features of the reference image, wherein the target model comprises a first network layer, which participates in the first processing; next, on the basis of the target model, the image features and the sample text, performing second processing, so as to obtain a generated image, wherein the first network layer does not participate in the second processing; and finally, on the basis of the difference between the generated image and the label image, updating the first network layer in the target model.

Description

Model training method, image generation method, device, equipment, medium and product

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Chinese Patent Application No. 202411390089.7, filed September 30, 2024. The entire teachings of the above application are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of data processing, and in particular to a model training method, an image generation method, a device, equipment, a medium and a product. BACKGROUND

[0004] For some image generation scenarios, there is a demand that a new image is generated according to user-specified text information and a user-provided image, so that the new image meets the constraints described by the text information and the constraints described by the user-provided image as much as possible. For ease of understanding, the following will be described in conjunction with an example.

[0005] As an example, after a user provides the text information "a person standing on the beach" and a clothing image, a new image can be generated according to the text information and the clothing image, so that the new image presents the following content as much as possible: a person standing on the beach, and the person is dressed in clothes or accessories described by the clothing image. SUMMARY

[0006] The present application provides a model training method, an image generation method, a device, equipment, a medium and a product.

[0007] To achieve the above purpose, the technical scheme provided by the present application is as follows:

[0008] The present application provides a model training method, which comprises: acquiring a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, the label image meeting the constraints described by the sample text and the constraints described by the reference image; performing first processing on the reference image by using a target model to obtain an image feature of the reference image, the target model comprising a first network layer, the first network layer participating in the first processing; performing second processing according to the target model, the image feature and the sample text to obtain a generated image, the first network layer not participating in the second processing; updating the first network layer in the target model according to the difference between the generated image and the label image.

[0009] In a possible implementation, the target model is obtained by adding the first network layer to a base model, and the base model has a text-to-image function.

[0010] In a possible implementation, the base model includes a plurality of target layers, and the plurality of target layers include linear layers in the base model and convolutional layers in the base model; the target model includes a plurality of merged layers corresponding to the plurality of target layers; for any target layer, the merged layer corresponding to the target layer is obtained by merging the first network layer into the target layer, and output data of the merged layer corresponding to the target layer is determined according to output data of the target layer, output data of the first network layer, and an enabling function of the first network layer, the enabling function being used to indicate whether the output data of the first network layer affects the output data of the merged layer corresponding to the target layer.

[0011] In a possible implementation, the first network layer includes two linear layers.

[0012] In a possible implementation, the target model further includes a second network layer, the second network layer does not participate in the first processing, the second network layer participates in the second processing, and input data of the second network layer is determined according to the image feature.

[0013] In a possible implementation, the updating the first network layer in the target model includes updating the first network layer and the second network layer in the target model.

[0014] In a possible implementation, the target model is obtained by adding the second network layer to a self-attention layer in a base model.

[0015] In a possible implementation, the target model includes an adjusted network layer corresponding to the self-attention layer, the adjusted network layer includes each module in the self-attention layer and the second network layer, and output data of the adjusted network layer is determined according to an enabling parameter of the second network layer, the enabling parameter being used to indicate whether the second network layer is used.

[0016] In a possible implementation, the target model comprises an adjusted network layer corresponding to the self-attention layer, the adjusted network layer comprising each module in the self-attention layer and the second network layer; input data of an attention module in the second network layer comprises output data of a key projection module in the second network layer, output data of a value projection module in the second network layer, and output data of a question projection module in the self-attention layer; input data of the key projection module in the second network layer and input data of the value projection module in the second network layer are determined according to the image feature; input data of the question projection module in the self-attention layer is determined according to output data of a previous network layer of the adjusted network layer, the previous network layer being adjacent to the adjusted network layer in the target model and being located in front of the adjusted network layer in the target model.

[0017] In a possible implementation, the key projection module in the second network layer is initialized by using the key projection module in the self-attention layer; and the value projection module in the second network layer is initialized by using the value projection module in the self-attention layer.

[0018] In a possible implementation, the clothing in the label image satisfies a constraint described by the reference image.

[0019] The present application provides an image generation method, comprising: obtaining a target text and a reference image corresponding to the target text; performing first processing on the reference image by using a target model to obtain an image feature of the reference image, the target model comprising a first network layer participating in the first processing, the target model being obtained by using a model training method provided by the present application; performing second processing according to the target model, the image feature and the target text to obtain a generated image, the first network layer not participating in the second processing.

[0020] In a possible implementation, the obtaining process of the target text comprises: obtaining an initial text provided by a user; performing rewriting processing on the initial text according to the reference image to obtain the target text, semantic information carried by the target text comprising semantic information carried by the initial text and semantic information carried by the reference image.

[0021] In a possible implementation, the second processing includes encoding processing and decoding processing; if the influence weight of the target text is higher than the influence weight of the reference image, the image feature does not participate in the encoding processing, the image feature participates in the decoding processing, the influence weight of the target text is used to describe the influence degree of the target text on the generated image, and the influence weight of the reference image is used to describe the influence degree of the reference image on the generated image; if the influence weight of the target text is not higher than the influence weight of the reference image, the image feature participates in the encoding processing, and the image feature participates in the decoding processing.

[0022] In a possible implementation, the method further includes: obtaining model indication information provided by a user; adding the first network layer to a model matched with the model description information according to a position of the first network layer in the target model, to obtain a new model; the first processing of the reference image by using the target model includes: first processing of the reference image by using the new model; and the second processing according to the target model, the image feature and the target text includes: second processing according to the new model, the image feature and the target text.

[0023] In a possible implementation, the target model and the model matched with the model description information are both constructed based on a same model.

[0024] The present application provides a model training apparatus, comprising: a first obtaining unit configured to obtain a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, the label image satisfying a constraint described by the sample text and a constraint described by the reference image; a first extracting unit configured to perform first processing on the reference image by using a target model to obtain an image feature of the reference image, the target model comprising a first network layer, the first network layer participating in the first processing; a first generating unit configured to perform second processing according to the target model, the image feature and the sample text to obtain a generated image, the first network layer not participating in the second processing; and a model updating unit configured to update the first network layer in the target model according to a difference between the generated image and the label image.

[0025] The application provides an image generation device, comprising: a second acquisition unit configured to acquire a target text and a reference image corresponding to the target text; a second extraction unit configured to perform first processing on the reference image by using a target model to obtain image features of the reference image, wherein the target model comprises a first network layer, the first network layer participates in the first processing, and the target model is obtained by using a model training method provided by the application; and a second generation unit configured to perform second processing according to the target model, the image features and the target text to obtain a generated image, wherein the first network layer does not participate in the second processing.

[0026] The application provides an electronic device, comprising: a processor and a memory; the memory is configured to store instructions or computer programs; and the processor is configured to execute the instructions or computer programs in the memory to enable the electronic device to perform the model training method or the image generation method provided by the application.

[0027] The application provides a computer readable medium, wherein the computer readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device is enabled to perform the model training method or the image generation method provided by the application.

[0028] The application provides a computer program product, comprising a computer program carried on a non-transitory computer readable medium, and the computer program comprises program codes for executing the model training method or the image generation method provided by the application.

[0029] Compared with the related art, the application has at least the following advantages:

[0030] In the technical solution provided by the application, a sample text, a reference image corresponding to the sample text and a label image corresponding to the sample text are first acquired, the label image meets a constraint described by the sample text and a constraint described by the reference image; then, first processing is performed on the reference image by using a target model to obtain image features of the reference image, the target model comprises a first network layer, and the first network layer participates in the first processing; then, second processing is performed according to the target model, the image features and the sample text to obtain a generated image, and the first network layer does not participate in the second processing; finally, the first network layer in the target model is updated according to a difference between the generated image and the label image, so that the updated target model has better performance. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments described in the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0032] FIG. 1 is a schematic diagram of an image generation scheme provided by an embodiment of the present application;

[0033] FIG. 2 is a flowchart of a model training method provided by an embodiment of the present application;

[0034] FIG. 3 is a schematic diagram of the working principle of a target model provided by an embodiment of the present application;

[0035] FIG. 4 is a schematic diagram of the working principle of an improved attention layer provided by an embodiment of the present application;

[0036] FIG. 5 is a flowchart of an image generation method provided by an embodiment of the present application;

[0037] FIG. 6 is a schematic diagram of an image generation process provided by an embodiment of the present application;

[0038] FIG. 7 is a structural schematic diagram of a model training device provided by an embodiment of the present application;

[0039] FIG. 8 is a structural schematic diagram of an image generation device provided by an embodiment of the present application;

[0040] FIG. 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] It is found through research that some image generation schemes are implemented by means of a double Unet network shown in FIG. 1, so that the schemes can be specific to: first, using a Unet network (such as network 1 shown in FIG. 1) to process an image provided by a user to obtain image features of the image; and then providing the image features by the Unet network to another Unet network (such as network 2 shown in FIG. 1) so that the latter can perform denoising processing based on the image features, noise data, and user-specified text information (such as text 1 shown in FIG. 1) to obtain a new image, such as generated image 1 shown in FIG. 1.

[0042] It is also found through research that the scheme shown in the above paragraph has the following defects: because the parameter quantity of the double Unet network is large, the image generation scheme implemented by means of the double Unet network needs to occupy more video memory, so that the resource consumption of the model training process involved in the scheme is relatively large, such as when the double Unet network is implemented by using a high-resolution diffusion model (Stable Diffusion, SD), such as an SD1.5 model or an SDXL model, it needs to be trained on a high-computing device (such as an A100 device), and the model training time involved in the scheme is relatively long, such as when the double Unet network is implemented by using a high-resolution diffusion model (such as an SDXL model with a resolution of 768x512), it takes 40 hours to train on 500,000 data.

[0043] Based on the above research, in order to better overcome the above defects, the present application provides a model training method, which comprises: first, obtaining a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, the label image satisfying the constraints described by the sample text and the constraints described by the reference image; second, performing first processing on the reference image by using a target model to obtain image features of the reference image, the target model comprising a first network layer, the first network layer participating in the first processing; third, performing second processing according to the target model, the image features and the sample text to obtain a generated image, the first network layer not participating in the second processing; and finally, updating the first network layer in the target model according to the difference between the generated image and the label image, so that the updated target model has better performance.

[0044] It can be seen that for the target model, when the first network layer in the target model is in a use state, the target model is used to implement image feature extraction processing, but when the first network layer in the target model is in a non-use state, the target model is used to implement image generation processing (such as denoising processing), so that the present application can complete different processing tasks by means of the same model through switching the working state of the first network layer, which can effectively overcome the defect of relatively large resource consumption when different models are used to complete different processing tasks, thereby facilitating the reduction of resource consumption.

[0045] In addition, because the training process provided by the present application only needs to update the first network layer, without updating the entire model, it can effectively overcome the defect of relatively long training time when different models are used to complete different processing tasks, thereby facilitating the reduction of training time.

[0046] In addition, the present application does not limit the execution subject of the model training method provided by the embodiments of the present application. For example, the model training method provided by the embodiments of the present application can be applied to a terminal device or a server. For another example, the model training method provided by the embodiments of the present application can also be implemented by means of a data interaction process between a terminal device and a server. The terminal device can be a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server can be a standalone server, a cluster server or a cloud server.

[0047] In order for those skilled in the art to better understand the technical scheme of the present application, the technical scheme of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] In order to better understand the technical scheme provided by the present application, the model training method provided by the present application will be described first in combination with some drawings. As shown in FIG. 2, the model training method provided by the embodiments of the present application includes the following S201-S204. Wherein, FIG. 2 is a flow chart of a model training method provided by the embodiments of the present application.

[0049] S201: obtaining a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, the label image satisfying the constraints described by the sample text and the constraints described by the reference image.

[0050] Wherein, the sample text refers to the text in the training data for describing the expected image, such as the text 2 shown in FIG. 3, so that the sample text can describe some constraints required by the finally generated image in a textual manner. For example, the sample text can be implemented by using the text “a person stands in a simple background, wearing a pink t-shirt with a large black pattern and a pair of black pants with color stitching details”.

[0051] The reference image corresponding to the sample text refers to an image existing in the training data and used to describe some details in the expected image in detail, such as the clothing image shown in FIG. 3, so that the reference image can describe some constraints, such as clothing, background, and posture, that the image generated based on the sample text needs to meet in an image manner. For example, in some scenarios, such as clothing refinement or clothing specification scenarios, the reference image can be implemented by using a clothing image, so that the reference image can describe what kind of clothing the object (such as a person, an animal, a virtual image, or the like) described by the sample text wears. For another example, in some scenarios, such as background refinement or background specification scenarios, the reference image can be implemented by using a background image, so that the reference image can describe what kind of environment, such as a beach, the object (such as a person, an animal, a virtual image, or the like) described by the sample text is in. For yet another example, in some scenarios, such as posture refinement or posture specification scenarios, the reference image can be implemented by using a posture image, so that the reference image can describe what kind of posture the object (such as a person, an animal, a virtual image, or the like) described by the sample text is in.

[0052] As can be seen, in a possible implementation, when the sample text is used to describe the state of the target object in the first dimension, the reference image corresponding to the sample text can be used to describe the state of the target object in the second dimension and / or the state details of the target object in the first dimension. The first dimension refers to a dimension that is constrained in a textual manner, such as a clothing dimension. The second dimension refers to one or more dimensions other than the first dimension, such as a posture dimension. The state details refer to information (such as a T-shirt style) described in an image manner and used to further refine the state of the object in the first dimension.

[0053] The label image corresponding to the sample text refers to an image existing in the training data and consistent with the sample text and the reference image corresponding to the sample text, such as the label image shown in FIG. 3, so that the label image meets the constraints described by the sample text and the constraints described by the reference image, thereby enabling the label image to serve as a true value to guide the image generation process based on the sample text and the reference image.

[0054] In addition, the present application does not limit the implementation of the label image corresponding to the sample text. For example, in some scenarios, such as clothing refinement or clothing specification scenarios, the label image can at least meet the following constraint: the clothing in the label image meets the constraint described by the reference image corresponding to the sample text.

[0055] It should be noted that the actual meaning of the above "the clothing in the tag image satisfies the constraint described by the reference image corresponding to the sample text" can be that the pixel points in the tag image for describing the clothing can cover as much information as possible carried by the reference image, such as color distribution information, etc. For example, when the reference image is used to describe a T-shirt, the tag image is used to describe a person wearing the T-shirt. For another example, when the reference image is used to describe an elephant, the pixel points in the tag image for describing the clothing can as comprehensively as possible describe the characteristics of the elephant, such as skin characteristics, facial feature distribution characteristics, etc.

[0056] In addition, the present application does not limit the implementation of S201 above, for example, in some scenarios such as clothing refinement or clothing designation scenarios, S201 can specifically include: first, obtaining an image pair by a certain manner (such as data query, etc.), so that the image pair includes a first image (such as a clothes image for showing characteristics of a certain clothes) and a second image (such as an image for describing a person wearing the clothes), and the first image is used to describe the characteristics of part or all of the clothing worn by the object in the second image, so that the first image can describe some constraints satisfied by the second image in an image manner, such as clothing constraints; second, performing semantic extraction processing on the second image to obtain a semantic extraction result, so that the semantic extraction result can describe the content described by the second image in a textual manner, so that the semantic extraction result can describe some constraints satisfied by the second image in a textual manner; and third, taking the semantic extraction result as a sample text, taking the first image as a reference image corresponding to the sample text, and taking the second image as a tag image corresponding to the sample text, so that subsequent model training can be performed using the training data {the sample text, the reference image corresponding to the sample text, the tag image corresponding to the sample text}.

[0057] It should be noted that the present application does not limit the implementation of the semantic extraction processing involved in the above paragraph, for example, it can adopt any method that can extract semantic information from an image, such as by means of a machine learning model with image semantic extraction function. In addition, the present application does not limit the implementation of the machine learning model, for example, it can be implemented by using a large multi-modal model (LMM).

[0058] Based on the related content of S201, for the training process of the current round, a triple of {sample text, reference image corresponding to the sample text, label image corresponding to the sample text} is obtained, so that the sample text can describe some constraints satisfied by the label image in a textual manner, and the reference image can describe some constraints satisfied by the label image in an image manner, so that the two kinds of data can more accurately and comprehensively describe the constraints satisfied by the label image, and thus the label image can better guide the image generation process based on the two kinds of data as the true value.

[0059] S202: performing first processing on the reference image by using a target model to obtain image features of the reference image, the target model comprising a first network layer participating in the first processing.

[0060] The target model is used to generate a new image according to the textual constraint and the image constraint, as shown in the generated image 2 in FIG. 3.

[0061] In addition, the present application does not limit the target model, for example, the target model belongs to a diffusion model, so that the target model can at least realize image generation processing by means of denoising, as shown in the image generation processing in FIG. 3.

[0062] In addition, in order to better save resource overhead, the target model can switch the function of the target model by controlling some network layers to be turned on or turned off, so as to achieve the effect of one model with multiple functions, so as to realize the image generation processing based on the textual constraint and the image constraint by means of only one model.

[0063] It can be seen that in a possible implementation, the target model can at least satisfy the following constraints: the target model comprises a first network layer; the first network layer participates in the first processing realized by the target model, but the first network layer does not participate in the second processing realized by the target model, so that the target model can switch the function of the target model by controlling the first network layer to be turned on or turned off. Wherein, the first processing is used to extract image features from an image, so that the first processing is used to realize the function of image feature extraction processing (such as the function of the image encoder shown in FIG. 3). The second processing is used to generate a new image based on some data, so that the second processing is used to realize the function of image generation processing (such as the function of the denoising network shown in FIG. 3).

[0064] For the first network layer described above, the first network layer refers to a network layer present in the target model and used to affect function switching, such as the newly added network layer shown in FIG. 3; and the present application does not limit the implementation of the first network layer, for example, the first network layer can include two linear layers, so that the first network layer can function as a lora layer in the target model.

[0065] In addition, the present application does not limit the working principle of the first network layer, for example, when the first network layer includes a first linear layer and a second linear layer arranged in sequence, the working principle of the first network layer is: first, the first linear layer processes the input data of the first network layer to obtain the output data of the first linear layer; and then the second linear layer processes the output data of the first linear layer to obtain the output data of the second linear layer, so that the output data of the second linear layer can be used as the output data of the first network layer for subsequent processing.

[0066] In addition, in order to better improve the effect, the target model described above can be obtained by adding a first network layer to a base model, so that the target model can use the inherent capabilities of the base model as a basis to better achieve image feature extraction and image generation functions. It can be seen that the target model is obtained by improving the base model.

[0067] Among them, the base model refers to a model that is pre-constructed and has good text-to-image generation capabilities, such as the denoising network, SD1.5 or SDXL diffusion model shown in FIG. 3, so that the model obtained by improving the base model can generate images based on text.

[0068] In addition, the present application does not limit the implementation of the base model, for example, in order to better improve the generalization, the base model can at least meet the following constraints: the base model is used for image generation processing based on text constraints; and the base model can be used as a bottom model to construct other models, such as various style corresponding diffusion models, Controlnet, IP-Adapter, Hpyper-SD, etc. Model, so that the newly added network layer in the target model based on the base model has better compatibility, so that these newly added network layers can be used as plug-and-play plugins and directly added to other models to achieve different types of image generation control.

[0069] It can be seen that in one possible implementation, the base model can be implemented using any diffusion model that can be used as a bottom model, such as SD1.5 or SDXL.

[0070] In addition, the application does not limit the improvement of the target model. For example, when the base model includes multiple target layers, the multiple target layers include linear layers in the base model and convolutional layers in the base model, and the target model can include a merged layer corresponding to each target layer. For any target layer, the merged layer corresponding to the target layer is obtained by merging a first network layer into the target layer, and the output data of the merged layer corresponding to the target layer is determined according to the output data of the target layer, the output data of the first network layer, and an enabling function of the first network layer. The enabling function is used to indicate whether the output data of the first network layer affects the output data of the merged layer corresponding to the target layer.

[0071] For the target layer, the target layer refers to a network layer in the base model that needs to add a lora layer during improvement, such as a convolutional layer or a linear layer. As can be seen, in a possible implementation, the multiple target layers can refer to all linear layers and all convolutional layers of the base model, so that the target model can be obtained by adding a first network layer to each linear layer and each convolutional layer of the base model.

[0072] For the ith target layer, the merged layer corresponding to the ith target layer refers to the improvement result of the ith target layer, so that the merged layer is obtained by merging a first network layer into the ith target layer, so that the merged layer includes the ith target layer and the first network layer, and the output data of the merged layer is determined according to the output data of the ith target layer, the output data of the first network layer, and an enabling function of the first network layer. The enabling function is used to indicate whether the output data of the first network layer affects the output data of the merged layer, so that the merged layer can present different data processing modes under different task processing requirements, thereby assisting the target model to realize function switching. i is a positive integer, i≤I, I is a positive integer, and I represents the number of target layers.

[0073] In addition, the application does not limit the implementation of the merged layer corresponding to the ith target layer. For example, in order to better reduce the improvement difficulty, the merged layer corresponding to the ith target layer can be implemented by using the following formula (1).

[0074] In the formula, h represents output data of a merging layer corresponding to the i-th target layer; x represents input data of the merging layer; f(x) represents output data of a target layer existing in the merging layer corresponding to the i-th target layer; AW(x) represents output data of a first network layer existing in the merging layer corresponding to the i-th target layer; and I(x) represents an enabling function of the first network layer, and can be specifically: if it is determined that the x is determined based on noise data (such as the noise addition result shown in FIG. 3), it can be determined that the current task processing requirement is de-noising processing, so I(x) = 0, so that the function value of I(x) can represent that AW(x) does not affect the output data of the merging layer, so that the function value of I(x) can represent that the first network layer is not used when de-noising processing is performed on the noise data; if it is determined that the x is determined based on a reference image (such as the clothing image shown in FIG. 3), it can be determined that the current task processing requirement is image feature extraction processing, so I(x) = 1, so that the function value of I(x) can represent that AW(x) affects the output data of the merging layer, so that the function value of I(x) can represent that the first network layer is used when image feature extraction processing is performed on the reference image.

[0075] It can be seen that for the merging layer corresponding to the i-th target layer, when the merging layer includes the i-th target layer and the first network layer added for the i-th target layer, the merging layer can use the enabling function of the first network layer as a control switch of the first network layer to ensure that the merging layer presents different data processing modes under different task processing requirements, which is conducive to automatically and flexibly switching functions.

[0076] Based on the above related content of the first network layer, in one possible implementation, the construction process of the target model can be: first, obtain a basic model, such as the SD1.5 model, so that the basic model can generate a high-quality new image based on text constraints, so that the basic model has good performance, such as image generation performance; then merge a first network layer, such as a lora layer, into each linear layer and each convolutional layer of the basic model to obtain a target model, so that the target model not only includes all network layers in the basic model, but also includes the first network layers, so that the working principle of the target model can include: when the first network layers are in an open state, the first network layers participate in the data processing process (such as the first processing above) implemented by the target model, so that the target model is used for image feature processing on an image; when the first network layers are in a closed state, the first network layers do not participate in the data processing process (such as the second processing below) implemented by the target model, so that the data processing process is implemented by all network layers in the basic model, so that the target model is used for image generation processing based on some information.

[0077] It can be seen that, for the target model including the first network layer, when the first network layer in the target model is in the open state, the target model can be regarded as a model having an image feature extraction function (such as the image encoder shown in FIG. 3) so that the target model is used for image feature extraction processing on an image; when the first network layer in the target model is in the closed state, the target model can be regarded as a model having an image generation function (such as the denoising network shown in FIG. 3) so that the target model is used for generating a new image according to some information.

[0078] Based on the related content of S202 above, for the training process of the current round, after obtaining the reference image (such as the clothing image shown in FIG. 3) corresponding to the sample text above, the first network layer (such as the newly added network layer shown in FIG. 3) in the target model can be opened first so that the first network layer participates in the data processing process realized by the target model; and the reference image is processed by the target model first (such as image encoding processing shown in FIG. 3) to obtain the image features (such as the data sent by the image encoder to the denoising network in FIG. 3) of the reference image, so that the image features can more accurately and comprehensively describe the information carried by the reference image.

[0079] S203: performing second processing according to the target model, the image features, and the sample text to obtain a generated image, and the first network layer does not participate in the second processing.

[0080] The generated image refers to an image generated according to the image features of the reference image above and the sample text, such as the generated image 2 shown in FIG. 3.

[0081] In addition, the present application does not limit the implementation of S203 above, for example, when the target model is obtained by improving a diffusion model, S203 can be specifically: when the first network in the target model is in a non-use state, performing denoising processing by the target model according to the noise-added result of the label image above, the image features of the reference image above, and the sample text above to obtain a generated image. The noise-added result of the label image refers to a result obtained by performing noise addition processing on the label image, so that the noise-added result carries a large amount of noise. It should be noted that the present application does not limit the implementation of the noise addition processing.

[0082] It can be seen that, in a possible implementation, when the target model is obtained by adding the first network layer to the base model, if the base model belongs to a diffusion model, the target model can realize the second processing through a denoising processing process, so that the second processing is used to represent denoising processing, such as the denoising processing realized by the denoising network in FIG. 3.

[0083] In addition, based on the related content of the above target model, in one possible implementation, when the target model is obtained by adding a first network layer to the base model, the S203 can specifically include: first closing the first network layer (such as the newly added network layer shown in FIG. 3) in the target model, so that the other network layers (such as all network layers in the base model) in the target model except the first network layer participate in the data processing process implemented by the target model; and then performing second processing (such as the denoising processing shown in FIG. 3) by the target model according to the image features of the above reference image and the sample text, to obtain a generated image (such as the generated image 2 shown in FIG. 3).

[0084] In addition, the present application does not limit the implementation of the above target model, for example, when the second processing implemented by the target model is denoising processing, in order to reduce the improvement difficulty as much as possible, the target model can at least meet the following constraints: the text features of the above sample text are introduced into the target model through cross-attention, and the image features of the above reference image are introduced into the target model through the self-attention layer in the target model.

[0085] For the text features of the above sample text (such as the output data of the text encoder shown in FIG. 3), the text features refer to the text features obtained by performing text feature extraction processing on the sample text, so that the text features can better describe the information carried by the sample text, so that the text features can be introduced into the target model through the cross-attention layer (such as the cross-attention layer shown in FIG. 3) in the target model, and then the text features can affect the data processing flow in the target model through cross-attention.

[0086] It should be noted that the present application does not limit the implementation of the text feature extraction processing involved in the above paragraph, for example, it can adopt any method capable of performing encoding processing on a text, such as by using a pre-constructed encoder with text encoding capability or the text encoder shown in FIG. 3.

[0087] For the image features of the above reference image, the image features can be introduced into the target model through the self-attention layer in the target model; and the implementation of this introduction method can be: determining the key (K) and value (V) involved in the self-attention layer by using the image features, and determining the query (Q) involved in the self-attention layer by using the output data of the previous network layer of the self-attention layer, so that the self-attention layer performs attention processing according to the three kinds of data, so as to introduce the image features by adjusting the input data of the self-attention layer while keeping the structure of the self-attention layer, which can effectively reduce the improvement difficulty and efficiency, thereby improving the model training efficiency.

[0088] In addition, the present application does not limit the implementation of the introduction process of the image features of the reference image shown in the above paragraph, such as if the base model belongs to the Unet network, the target model constructed based on the base model also belongs to the Unet network, so that when the base model includes N modules (such as multiple down-sampling modules and multiple up-sampling modules), the target model can be obtained by adding a first network layer to each module of the base model, so that the target model includes the improved modules corresponding to each module, and the improved module includes the first network layer, so that the process of sequentially implementing image feature extraction processing and denoising processing by means of the target model is: opening the first network layer in the improved module corresponding to the nth module, and processing the input data of the improved module corresponding to the nth module by the improved module corresponding to the nth module to obtain the image features corresponding to the nth module, so as to introduce the image features corresponding to the nth module by means of the self-attention layer in the improved module corresponding to the nth module after closing the first network layer. Wherein, the improved module corresponding to the nth module is obtained by adding a first network layer to the nth module. n is a positive integer, n≤N, and N is a positive integer.

[0089] It can be seen that when the target model includes multiple modules (such as the multiple modules shown in 3), and each module includes a first network layer, the image features of the reference image above can include the image features output by each module, so that the image features output by each module are introduced into the corresponding module of the target model for implementing image generation function in a certain way.

[0090] It should be noted that the present application does not limit the start time of the introduction of the image features in the target model, such as starting to introduce the image features when the target model performs decoding processing, starting to introduce the image features when the target model performs encoding processing, or randomly setting the introduction start time in different rounds of training process.

[0091] It is found through research that the self-attention layer in the target model is derived from the base model, so that the parameters in the self-attention layer in the target model are consistent with the parameters in the self-attention layer in the base model. Wherein, since the self-attention layer in the base model is suitable for implementing self-attention processing, so that the self-attention layer in the base model is suitable for the scene where QKV is determined according to the same data, so that when the image features of the reference image above are directly introduced by means of the self-attention layer in the target model, since this introduction scene is different from the scene suitable for the self-attention layer in the base model, so that this introduction method cannot effectively use the image features, thereby affecting the image generation effect.

[0092] Based on the above research, in order to better overcome the problems shown in the above paragraph, the self-attention layer in the target model is improved in the present application, so that the improved attention layer (such as the improved attention layer shown in FIG. 4) not only includes each module in the self-attention layer, but also includes other network layers, so that the improved attention layer can better introduce the image features of the above reference image while maintaining the original function of the self-attention layer.

[0093] It can be seen that, in a possible implementation, the target model can at least meet the following constraints: the target model further includes a second network layer, and the input data of the second network layer is determined according to the image features of the above reference image, so that the second network layer is used to introduce the image features, so that the second network layer does not participate in the above first processing, but participates in the above second processing, thereby enabling the target model to realize function switching by turning on or off the second network layer. The second network layer refers to a network layer in the target model that is used to introduce the image features of the above reference image, such as the network layer including the adapter in FIG. 4 and the attention processing module corresponding to the adapter.

[0094] In addition, the present application does not limit the implementation of the above second network layer, for example, the working principle of the second network layer can be: determining the Key and Value involved in the second network layer by using the image features of the above reference image, and determining the Query involved in the second network layer by using the output data (such as the forward propagation data shown in FIG. 4) of the previous network layer of the second network layer, so as to subsequently perform attention processing by the second network layer according to the three kinds of data.

[0095] In addition, the present application does not limit the construction method of the target model involved in the above paragraph, for example, the target model can be obtained by adding a second network layer to the self-attention layer in the base model, so that the target model can better introduce the image features of the above reference image while maintaining the original function of the self-attention layer.

[0096] It can be seen that, in a possible implementation, when the above base model includes a self-attention layer, the above target model can include an adjusted network layer corresponding to the self-attention layer, such as the improved attention layer shown in FIG. 4, so that the adjusted network layer is used to represent the result obtained by adding a second network layer to the self-attention layer, so that the adjusted network layer includes each module in the self-attention layer and the second network layer, thereby enabling the adjusted network layer to better introduce the image features of the above reference image while maintaining the original function of the self-attention layer.

[0097] Further, the application does not limit the implementation of the adjusted network layer above, for example, when the adjusted network layer includes the modules in the self-attention layer and the second network layer, the adjusted network layer can at least meet the following constraints: the output data of the adjusted network layer is determined according to the enabling parameter of the second network layer, and the enabling parameter is used to indicate whether the second network layer is used, so that the adjusted network layer can present different data processing modes under different task processing requirements, which is conducive to realizing automatic and flexible function switching.

[0098] It should be noted that the application does not limit the determination manner of the enabling parameter above, for example, the enabling parameter is determined according to the state of the first network layer in the target model (such as the function value of the enabling function of the first network layer), which can be specifically: if the first network layer in the target model is in an open state, it can be determined that the target model is used to realize image feature extraction processing, so that it can be determined that the target model does not have the demand of introducing image features, and then the enabling parameter is set to 0, so that the enabling parameter is used to indicate that the second network layer in the target model is in a closed state; if the first network layer in the target model is in a closed state, it can be determined that the target model is used to realize image generation processing, so that it can be determined that the target model has the demand of introducing image features, and then the enabling parameter is set to 1, so that the enabling parameter is used to indicate that the second network layer in the target model is in an open state.

[0099] It can be seen that for the target model, the target model can at least meet the following constraints: the state of the first network layer in the target model is opposite to the state of the second network layer in the target model, so as to ensure that the two network layers do not participate in the same data processing process at the same time.

[0100] Further, in order to better improve the effect, the application provides a possible implementation of the target model above, in which when the target model is obtained by adding the second network layer to the self-attention layer in the base model, the second network layer can at least meet the following constraints: the input data of the attention module in the second network layer includes the output data of the key projection module in the second network layer (such as K' shown in FIG. 4), the output data of the value projection module in the second network layer (such as V' shown in FIG. 4), and the output data of the problem projection module in the self-attention layer (such as Q shown in FIG. 4).

[0101] For the key projection module in the second network layer, the key projection module refers to a module existing in the second network layer and used for projecting the image feature of the image referred to above into a key space, so that the key projection module is used for determining Key according to the image feature. It can be seen that the input data of the key projection module is determined according to the image feature. In addition, the present application does not limit the implementation of the key projection module, for example, the key projection module can be implemented by using a linear projection layer.

[0102] For the value projection module in the second network layer, the value projection module refers to a module existing in the second network layer and used for projecting the image feature of the image referred to above into a value space, so that the value projection module is used for determining Value according to the image feature. It can be seen that the input data of the value projection module is determined according to the image feature. In addition, the present application does not limit the implementation of the value projection module, for example, the value projection module can be implemented by using a linear projection layer.

[0103] For the question projection module in the self-attention layer, the question projection module refers to a module existing in the self-attention layer and used for projecting the output data of the previous network layer (such as the previous network layer of the adjusted network layer corresponding to the self-attention layer) of the self-attention layer into a question space, so that the question projection module is used for determining Query according to the output data of the previous network layer. It can be seen that the input data of the question projection module in the self-attention layer is determined according to the output data of the previous network layer. The position of the previous network layer in the target model is adjacent to the position of the adjusted network layer in the target model, and the position of the previous network layer in the target model is earlier than the position of the adjusted network layer in the target model. In addition, the present application does not limit the implementation of the question projection module, for example, the question projection module can be implemented by using a linear projection layer.

[0104] It should be noted that the present application does not limit the implementation of the previous network layer referred to in the above paragraph, for example, in some scenarios, such as the scenario of performing cross-attention processing first and then performing self-attention processing in the base model, the previous network layer can refer to the cross-attention layer shown in FIG. 6. For another example, in some scenarios, such as the scenario of performing self-attention processing first and then performing cross-attention processing in the base model, the previous network layer can refer to a network layer existing in the base model, located in front of the self-attention layer used to construct the adjusted network layer, adjacent to the self-attention layer.

[0105] For the attention module in the second network layer as mentioned above (e.g., the attention processing module corresponding to the adapter shown in FIG. 4), the attention module is configured to perform attention processing on the output data of the key projection module in the second network layer, the output data of the value projection module in the second network layer, and the output data of the question projection module in the self-attention layer as mentioned above, i.e., the attention processing shown in FIG. 4. It should be noted that the present application does not limit the implementation of the attention module, which can be implemented by using any existing or future attention network layer.

[0106] Based on the related content of the second network layer as mentioned above, in one possible implementation, when the target model is obtained by adding the second network layer to the self-attention layer in the base model, the target model includes the adjusted network layer (e.g., the improved attention layer shown in FIG. 4) corresponding to the self-attention layer, and the adjusted network layer includes the modules in the self-attention layer and the second network layer, the adjusted network layer can at least satisfy the following constraints: if the enabling parameter of the second network layer indicates that the second network layer is in an open state, the output data of the adjusted network layer is determined based on the output data of the attention module in the self-attention layer and the output data of the attention module in the second network layer; if the enabling parameter of the second network layer indicates that the second network layer is in a closed state, the output data of the adjusted network layer is determined based on the output data of the attention module in the self-attention layer.

[0107] For the second network layer as shown in the above paragraph, the output data of the attention module in the second network layer is obtained by performing attention processing on the output data of the key projection module in the second network layer (e.g., K' shown in FIG. 4), the output data of the value projection module in the second network layer (e.g., V' shown in FIG. 4), and the output data of the question projection module in the self-attention layer as shown in the above paragraph (e.g., Q shown in FIG. 4).

[0108] For the self-attention layer as shown in the above two paragraphs, the output data of the attention module in the self-attention layer is obtained by performing attention processing on the output data of the question projection module in the self-attention layer (e.g., Q shown in FIG. 4), the output data of the key projection module in the self-attention layer (e.g., K shown in FIG. 4), and the output data of the value projection module in the self-attention layer (e.g., V shown in FIG. 4).

[0109] Based on the above three paragraphs, in one possible implementation, when the target model includes M modules, the adjusted network layer (e.g., the improved attention layer shown in FIG. 4) in the mth module can be implemented by using the following formulas (2)-(7), m is a positive integer, m≤M, and M is a positive integer. Q=z t W q(3) K = z t W k (4) V = z t W v (5) K' = c m W k ′ (6) V' = c m W' v (7)

[0110] In the formula, z represents the output data of the adjusted network layer in the mth module of the target model (the "update result of the forward propagation data" shown in FIG. 4); z t represents the output data of the previous network layer of the adjusted network layer (the forward propagation data shown in FIG. 4); c m represents the image features introduced by the adjusted network layer, such as the image features output by the mth module when the first network layer in the mth module of the target model is in an open state; P represents an enabling parameter of the second network layer in the adjusted network layer, P = 1 indicates that the second network layer in the adjusted network layer is in an open state, and P = 0 indicates that the second network layer in the adjusted network layer is in a closed state; W q represents a problem projection module in the self-attention layer used when the adjusted network layer is constructed; W k represents a key projection module in the self-attention layer; W v represents a value projection module in the self-attention layer; W' k represents a key projection module in the second network layer present in the adjusted network layer; W' v represents a value projection module in the second network layer present in the adjusted network layer; and d represents the feature dimension of Q, K, and K'.

[0111] It is found through research that, for the self-attention layer in the base model, since the self-attention layer itself has good Key and Value determination performance, in order to better improve the efficiency and performance, the application further provides a possible implementation manner of the above target model, in which manner, when the target model is obtained by adding a second network layer to the self-attention layer in the base model, the target model can at least meet the following constraints: the key projection module in the second network layer is initialized using the key projection module in the self-attention layer, and the value projection module in the second network layer is initialized using the value projection module in the self-attention layer, so that the initialized target model itself already has good image feature introduction performance, thereby facilitating the improvement of model training efficiency and model performance.

[0112] It can be seen that, for the above target model, when the target model is obtained by adding a second network layer to the self-attention layer in the base model, and the target model includes the adjusted network layer corresponding to the self-attention layer (such as the improved attention layer shown in FIG. 4), the adjusted network layer can at least meet the following constraints: the adjusted network layer includes all the modules in the self-attention layer (such as all the modules in the self-attention layer shown in FIG. 4 except the adapter and the attention processing module corresponding to the adapter) and the second network layer; all the modules in the adjusted network layer except the second network layer are modules present in the base model; the second network layer in the adjusted network layer is constructed in accordance with part of the “present modules”, so that the attention module in the second network layer is the same as the attention module in the “present modules”, the key projection module in the second network layer is initialized using the key projection module in the “present modules”, and the value projection module in the second network layer is initialized using the value projection module in the “present modules”, so as to facilitate the introduction effect of image features.

[0113] Based on the above related content of the target model, for the target model including the first network layer and the second network layer, when the first network layer in the target model is in an open state, and the second network layer in the target model is in a closed state, the target model can be regarded as a model having an image feature extraction function (such as the image encoder shown in FIG. 3), so that the target model is used for image feature extraction processing on an image; when the first network layer in the target model is in a closed state, and the second network layer in the target model is in an open state, the target model can be regarded as a model having an image generation function (such as the denoising network shown in FIG. 3), so that the target model is used to generate a new image according to some information.

[0114] Based on the related content of S202 to S203 above, for the training process of the current round, if the target model is obtained by adding the first network layer and the second network layer to the base model, after obtaining the reference image corresponding to the sample text above, the first network layer in the target model can be opened first, and the second network layer in the target model can be closed, so that the first network layer participates in the data processing process implemented by the target model, and the second network layer does not participate in the data processing process implemented by the target model, so that the target model presents better image feature extraction performance; then the target model is used to perform first processing on the reference image to obtain image features of the reference image, so that the image features can more accurately and more comprehensively describe the information carried by the reference image; then, the first network layer in the target model is closed, and the second network layer in the target model is opened, so that the first network layer does not participate in the data processing process implemented by the target model, and the second network layer participates in the data processing process implemented by the target model, so that the target model presents better image generation performance; then, the target model is used to perform second processing on the image features and the sample text to obtain a generated image, so that the performance of the target model can be measured by means of the generated image subsequently.

[0115] S204: updating the first network layer in the target model according to the difference between the generated image and the label image.

[0116] In this application, for the target model obtained by improving the base model, because the base model itself has better data processing performance, in order to better improve the efficiency, the network layer existing in the target model and coming from the base model can be frozen directly, so as to ensure that only the newly added network layer needs to be updated in the training process of the target model. In order to facilitate understanding, the following will be explained in combination with two cases.

[0117] Case 1, in some scenarios, such as scenarios where the improvement difficulty is as low as possible, when the target model is obtained by adding the first network layer to the base model, because the target model includes the first network layer and all network layers in the base model, in order to improve the efficiency, S204 above can be: updating the first network layer in the target model according to the difference between the generated image and the label image. Among them, because all the first network layers in the target model include more than 60 megabytes (M) of parameters, so that the training process of the target model only needs to update the more than 60 megabytes of parameters, which is conducive to better improving the training efficiency.

[0118] In case 2, in some scenarios, such as scenarios in which model performance is to be improved as much as possible, when the target model is obtained by adding the first network layer and the second network layer to the base model, because the target model includes the first network layer, the second network layer, and all network layers in the base model, in order to improve efficiency, S204 in the foregoing can be specifically: updating the first network layer and the second network layer in the target model according to the difference between the generated image and the label image.

[0119] It is found through research that, for the second network layer in the target model, because the data processing flow required to be implemented by the attention module in the second network layer is the same as the data processing flow required to be implemented by the attention module in the corresponding self-attention layer in the base model, in order to better improve efficiency, when the second network layer includes the attention module, the key projection module, and the value projection module, S204 in the foregoing can be specifically: updating the first network layer of the target model, the key projection module in the second network layer of the target model, and the value projection module in the second network layer of the target model according to the difference between the generated image and the label image. It can be seen that, because the amount of parameters that need to be updated for all second network layers in the target model is more than 17M, so that the training process for the target model only needs to update 83.4M parameters, this is conducive to better improving model performance under the premise of as little as possible increasing training time consumption.

[0120] It is also found through research that, in some scenarios, such as scenarios in which model performance requirements are relatively high, model training can be implemented by means of an iterative loop, so as to better improve model performance, S204 in the foregoing can be specifically: updating the first network layer in the target model (or updating the first network layer and the second network layer in the target model, or updating the first network layer of the target model, the key projection module in the second network layer of the target model, and the value projection module in the second network layer of the target model) according to the difference between the generated image and the label image, and returning to continue executing S201 and subsequent steps in the foregoing until, when a preset stopping condition is reached, ending the iterative training process for the target model.

[0121] The preset stopping condition refers to a condition that needs to be reached when the iterative training process for the target model is ended, and the preset stopping condition is not limited in implementation by the present application, for example, the preset stopping condition can include: the model loss of the target model being lower than a preset loss threshold. For another example, the preset stopping condition can include: a change rate of the model loss of the target model being lower than a preset change rate threshold. For yet another example, the preset stopping condition can include: the number of updates of the target model reaching a preset number threshold.

[0122] The model loss of the target model is used to describe the performance of the target model, and the model loss of the target model is determined according to the difference between the generated image and the label image. It should be noted that the application does not limit the calculation method of the loss.

[0123] Based on the above S201 to S204, for the model training process provided by the application, first, the sample text, the reference image corresponding to the sample text, and the label image corresponding to the sample text are obtained, the label image satisfies the constraint described by the sample text and the constraint described by the reference image; then the target model is used to perform first processing on the reference image to obtain the image feature of the reference image, the target model includes a first network layer, and the first network layer participates in the first processing; then, the second processing is performed according to the target model, the image feature and the sample text, to obtain a generated image, the first network layer does not participate in the second processing; finally, the first network layer in the target model is updated according to the difference between the generated image and the label image, so that the updated target model has better performance.

[0124] It can be seen that for the target model, when the first network layer in the target model is in a use state, the target model is used to realize image feature extraction processing, but when the first network layer in the target model is in a non-use state, the target model is used to realize image generation processing (such as denoising processing), so that the application can realize the completion of different processing tasks by means of the same model through switching the working state of the first network layer, which can effectively overcome the defect that the resource consumption is relatively large when different models are used to complete different processing tasks, thereby facilitating the reduction of resource consumption.

[0125] In addition, since the training process provided by the application only needs to update the first network layer, the entire model does not need to be updated, so that the defect that the training time is relatively long when different models are used to complete different processing tasks can be effectively overcome, thereby facilitating the reduction of training time.

[0126] In addition, based on the above related content of the target model, the application further provides an image generation method, as shown in FIG. 5, the image generation method includes the following S501-S503. Wherein, FIG. 5 is a flow chart of an image generation method provided by an embodiment of the application.

[0127] S501: obtaining a target text and a reference image corresponding to the target text.

[0128] Wherein, the target text refers to the text required to be used when performing an image generation task, so that the target text can describe some constraints specified by a user for the image generation task in a textual manner.

[0129] In addition, the present application does not limit the implementation of the target text, for example, the target text can be implemented by using the user-provided text.

[0130] The reference image corresponding to the target text refers to an image required to be used when performing the image generation task, for example, the clothing image shown in FIG. 6, so that the reference image can describe some constraints specified by the user for the image generation task in an image manner.

[0131] It should be noted that the implementation of the reference image corresponding to the target text is similar to the implementation of the reference image corresponding to the sample text described above, and is not described here for the sake of brevity. It can be seen that in some scenarios such as clothing refinement or clothing designation, the reference image corresponding to the target text can be implemented by using a clothing image, so that the reference image can describe what kind of clothing the object described by the target text wears.

[0132] It has been found through research that it is time-consuming and difficult for the user to provide text that can completely and accurately describe the expected image for any image generation task, but the image generation effect will be reduced when the user-provided text is inaccurate and incomplete.

[0133] Based on the above research, in order to better improve the image generation effect, the present application also provides a manner of obtaining the above target text, in which the obtaining process of the target text can include the following steps 11-12.

[0134] Step 11: Obtain the initial text provided by the user.

[0135] The initial text refers to the text provided by the user, for example, the user text shown in FIG. 6, so that the initial text can describe some constraints specified by the user for the current image generation task.

[0136] In addition, the present application does not limit the manner of obtaining the initial text, for example, the initial text can be provided by the user by means of any input device.

[0137] Step 12: After obtaining the reference image (for example, the clothing image shown in FIG. 6) provided by the user, the initial text is rewritten according to the reference image to obtain the target text (for example, the text 3 shown in FIG. 6), so that the semantic information carried by the target text includes the semantic information carried by the initial text and the semantic information carried by the reference image, so that the target text can describe some constraints required to be met by the current image generation task as completely and accurately as possible in a textual form.

[0138] It should be noted that the present application does not limit the implementation of the rewriting processing in the above paragraph, for example, it can adopt any one of the existing or future methods capable of realizing text rewriting processing according to multiple data, such as implementation by means of a machine learning model with text rewriting function. In addition, the present application does not limit the implementation of the machine learning model, for example, it can be implemented by LMM.

[0139] Based on the above steps 11 to 12, for some scenarios, after obtaining the text and image provided by the user for the current image generation task, the text is rewritten according to the image to obtain rewritten text, so that the rewritten text can better describe the constraints required to be met by the image generation task, so that the new image generated based on the rewritten text and the image is better.

[0140] In addition, in some scenarios, the user may not provide text constraints, therefore, in order to better improve the image generation effect, the present application further provides a method for obtaining the above target text, which can be specifically: after obtaining the reference image provided by the user, performing semantic extraction processing on the reference image to obtain the target text, so that the semantic information carried by the target text includes the semantic information carried by the reference image.

[0141] Based on the above two paragraphs, in a possible implementation, the above target text is determined according to the reference image provided by the user, so that the semantic information carried by the target text at least includes the semantic information carried by the reference image, so that the target text can at least represent the constraints described by the reference image.

[0142] S502: performing first processing on the reference image by using a target model to obtain image features of the reference image, the target model includes a first network layer participating in the first processing, and the target model is obtained by using any one of the embodiments of the model training method provided by the present application.

[0143] It should be noted that the related content of S502 is similar to the related content of S202, and for the sake of brevity, it will not be repeated here.

[0144] It can be seen that, in a possible implementation, when the target model above includes the first network layer and the second network layer, after the reference image corresponding to the target text above (such as the clothing image shown in FIG. 6) is acquired, the first network layer in the target model can be first opened, and the second network layer in the target model can be closed, so that the first network layer participates in the data processing process implemented by the target model, and the second network layer does not participate in the data processing process implemented by the target model, thereby making the target model switch to the image feature extraction function; the target model is used to perform first processing on the reference image, and the image feature of the reference image is obtained, so that the image feature can more accurately and more comprehensively describe the information carried by the reference image.

[0145] S503: performing second processing according to the target model, the image feature, and the target text to obtain a generated image, and the first network layer does not participate in the second processing.

[0146] It should be noted that the related content of S503 is similar to the related content of S203 above, and for the sake of brevity, it will not be repeated here.

[0147] It can be seen that, in a possible implementation, when the target model above includes the first network layer and the second network layer, after the reference image corresponding to the target text above (such as the clothing image shown in FIG. 6) is acquired, the first network layer in the target model can be first opened, and the second network layer in the target model can be closed, so that the first network layer participates in the data processing process implemented by the target model, and the second network layer does not participate in the data processing process implemented by the target model, thereby making the target model switch to the image feature extraction function; the target model is used to perform first processing on the reference image, and the image feature of the reference image is obtained, so that the image feature can more accurately and more comprehensively describe the information carried by the reference image.

[0148] In another possible implementation, when the target model implements the image generation processing in a denoising manner, S503 above can be specifically: when the first network in the target model is in a non-use state, the target model performs denoising processing on the random noise data (such as the noise data shown in FIG. 6), the image feature of the reference image corresponding to the target text above, and the target text to obtain a generated image.

[0149] Based on the related content of S501 to S503, for the image generation process provided by the present application, first, the target text and the reference image corresponding to the target text are obtained; then, the trained target model is used to perform first processing on the reference image to obtain image features of the reference image, and the first network layer in the target model participates in the first processing; then, the target model, the image features, and the target text are used to perform second processing to obtain a generated image, and the first network layer in the target model does not participate in the second processing, so that multiple processing tasks can be completed by means of one model, thereby effectively reducing resource consumption. Among them, because the trained target model has good performance, the generated image based on the target model is good, so as to improve the image generation quality.

[0150] In addition, the present application does not limit the execution subject of the image generation method provided by the embodiments of the present application. For example, the image generation method provided by the embodiments of the present application can be applied to a terminal device or a server. For another example, the image generation method provided by the embodiments of the present application can also be implemented by means of data interaction between a terminal device and a server.

[0151] It is found through research that for the target model, when the second processing is implemented by using the target model and the second processing includes encoding processing and decoding processing, if only the features of the image constraint are introduced in the decoding processing, the control force of the text constraint on the second processing is higher than the control force of the image constraint on the second processing; if the features of the image constraint are introduced in both the encoding processing and the decoding processing, the control force of the text constraint on the second processing is lower than the control force of the image constraint on the second processing.

[0152] Based on the above research, in order to better improve the image generation effect, the present application also provides a possible implementation manner of the above image generation method, in which manner the image generation method can at least include the following steps 21-22.

[0153] Step 21: obtaining an influence weight of the target text and an influence weight of the reference image corresponding to the target text, the influence weight of the target text being used to describe the influence degree of the target text on the above generated image, and the influence weight of the reference image being used to describe the influence degree of the reference image on the above generated image.

[0154] The influence weight of the target text is used to represent the influence degree of the constraint described by the target text on the image generation process, so that the influence weight can represent the influence degree of the target text on the finally obtained generated image, so that the influence weight can represent the constraint degree of the target text on the second processing, and further so that the influence weight can represent the control force of the text constraint on the second processing in the current image generation task.

[0155] The influence weight of the reference image corresponding to the target text is used to represent the influence degree of the constraint described by the reference image on the image generation process, so that the influence weight can represent the influence degree of the reference image on the finally obtained generated image, so that the influence weight can represent the constraint degree of the reference image on the second processing, and further, the influence weight can represent the control degree of the image constraint on the second processing in the current image generation task.

[0156] In addition, the present application does not limit the implementation of step 21 above, for example, it can specifically be: finding the influence weight matched with the current application scene from the pre-constructed mapping relationship to obtain the influence weight of the target text and the influence weight of the reference image corresponding to the target text. The mapping relationship is used to record the influence weight of the text constraint and the influence weight of the image constraint in different application scenes.

[0157] For example, in some scenarios, step 21 above can specifically be: receiving the weight specification information provided by the user, the weight specification information including the influence weight of the target text and the influence weight of the reference image corresponding to the target text.

[0158] Step 22: performing the second processing according to the influence weight of the target text, the influence weight of the reference image corresponding to the target text, the target model, the image feature of the reference image, and the target text to obtain a generated image.

[0159] In the present application, for the second processing realized by the target model, when the second processing includes encoding processing and decoding processing, if the influence weight of the target text is higher than the influence weight of the reference image above, the image feature of the reference image does not participate in the encoding processing, but the image feature participates in the decoding processing, so that the introduction start time (such as the introduction start time shown in FIG. 6) of the image feature can be effectively delayed, thereby effectively reducing the influence time of the image feature on the image generation process, and further ensuring that the control degree of the text constraint on the image generation process is higher than the control degree of the image constraint on the image generation process; if the influence weight of the target text is not higher than the influence weight of the reference image, the image feature participates in the encoding processing, and the image feature participates in the decoding processing, so that the introduction start time of the image feature can be as early as possible, thereby effectively increasing the influence time of the image feature on the image generation process, and further ensuring that the control degree of the text constraint on the image generation process is not higher than the control degree of the image constraint on the image generation process.

[0160] It can be seen that, in a possible implementation, when the target model includes an encoder and a decoder, the encoder includes a plurality of modules, and the decoder includes a plurality of modules, the target model for implementing the image generation process can at least satisfy the following constraints: the first network layer in the modules is in a closed state, but the second network layer in the modules is in an open state; if it is determined that the influence weight of the target text is higher than the influence weight of the reference image, the text features of the target text are introduced into the modules of the encoder by cross-attention, and for any module in the decoder, the module introduces the text features by cross-attention and introduces the image features corresponding to the module by the attention shown in FIG. 4; if it is determined that the influence weight of the target text is not higher than the influence weight of the reference image, for any module in the encoder, the module introduces the text features by cross-attention and introduces the image features corresponding to the module by the attention shown in FIG. 4, and for any module in the decoder, the module introduces the text features by cross-attention and introduces the image features corresponding to the module by the attention shown in FIG. 4. It should be noted that for any module in the target model, the image features corresponding to the module refer to the data output by the module when the target model is used for image feature extraction processing on the reference image.

[0161] Based on the above steps 21 to 22, in some scenarios, the introduction start time of the image features can be flexibly selected to ensure that the finally generated image is more in line with the image generation requirements, thereby facilitating the improvement of the image generation effect.

[0162] It has been found through research that for a trained target model, the network layer added by the target model relative to the base model can be regarded as a plug-and-play plug-in, so that the plug-in has a better compatibility effect with other models, which facilitates the improvement of flexibility.

[0163] Based on the above research, in order to better improve the flexibility, the present application also provides a possible implementation of the above image generation method, in which the image generation method can at least include the following steps 31-34.

[0164] Step 31: Obtain the model indication information provided by the user.

[0165] The model indication information is used to describe which model the user wants to use to perform the current image generation task.

[0166] In addition, the present application does not limit the implementation of the above model indication information, for example, the model indication information is used to describe at least one feature of the model specified by the user, such as the model is suitable for generating images of what style, the model is suitable for posture adjustment processing, the model is suitable for face adjustment processing, the model is suitable for improving the generation speed, etc.

[0167] In addition, the present application does not limit the way of obtaining the above model indication information, for example, the model indication information can be determined according to the model selection operation and / or information input operation triggered by the user. Wherein, the model selection operation is used to indicate that the user has directly selected which model. The information input operation is used to indicate which data has been input by the user, so that subsequent inference can be based on these data to determine which model is used for image generation processing.

[0168] Step 32: According to the position of the first network layer in the target model, add the first network layer to the model matched with the above model description information to obtain a new model.

[0169] Wherein, the model matched with the model description information refers to the model found from some candidate models, which is most suitable for the model description information, so that the "model matched with the model description information" can meet the user's model use requirements as much as possible.

[0170] In addition, the present application does not limit the implementation of the above candidate model, for example, it can include: various style corresponding diffusion model, Controlnet model with posture adjustment function, IP-Adapter model with face adjustment function, Hpyper-SD model with generation speed acceleration function.

[0171] In addition, in order to better improve the compatibility effect, the present application also provides a possible implementation of the above candidate model, in which when the above target model is obtained by improving the base model, each candidate model at least meets the following constraints: the candidate model is obtained by fine-tuning the base model, so as to ensure that these candidate models and the target model are constructed based on the same model, so that the similarity between these candidate models and the target model is relatively large, and the compatibility between the network layers added by these candidate models and the target model relative to the base model is relatively high, so as to facilitate improving the compatibility effect.

[0172] It can be seen that, in a possible implementation, when the target model is obtained by improving the base model, the "model described by the model description information" can at least satisfy the following constraint: the "model described by the model description information" is obtained by fine-tuning the base model, so that the compatibility between the "model described by the model description information" and the network layer added by the target model with respect to the base model is relatively high, which is conducive to improving the compatibility effect. Based on this, it can be known that the target model and the "model described by the model description information" are both constructed based on the same model.

[0173] In addition, the present application does not limit the implementation of step 32 above, and the following two examples are described for the convenience of understanding.

[0174] Example 1: when the target model is obtained by adding a first network layer to the base model, step 32 above can be specifically: according to the position of the first network layer in the target model, adding the first network layer to the "model described by the model description information" to obtain a new model, so that the new model includes the first network layer and all network layers in the "model described by the model description information", so that the new model can realize function switching by controlling the switch of the first network layer, and then the image feature extraction function is realized by the new model when the first network layer in the new model is in the open state, and the image generation function is realized by the new model when the first network layer in the new model is in the closed state, so that the function brought by the first network layer can be combined with the function possessed by the "model described by the model description information" itself, so that various image generation processes can be realized, such as style-controlled image generation process, pose-controlled image generation process, face-controlled image generation process, or speed-up image generation process.

[0175] In Example 2, when the target model is obtained by adding the first network layer and the second network layer to the base model, step 32 can be specifically: adding the first network layer and the second network layer to the model matched with the model description information according to the position of the first network layer in the target model and the position of the second network layer in the target model, to obtain a new model, so that the new model includes the first network layer, the second network layer, and all network layers in the model matched with the model description information, so that the new model can realize function switching by controlling the first network layer and the second network layer, so that the new model realizes image feature extraction function when the first network layer in the new model is in an open state and the second network layer in the new model is in a closed state, and so that the new model realizes image generation function when the first network layer in the new model is in a closed state and the second network layer in the new model is in an open state, so that the functions brought by the first network layer and the second network layer can be combined with the functions of the model matched with the model description information, so that various image generation processes can be realized.

[0176] Step 33: performing first processing on the reference image by using the new model to obtain image features of the reference image, the new model including the first network layer, the first network layer participating in the first processing.

[0177] It should be noted that the related content of step 33 is similar to the related content of S502, and for the sake of brevity, it will not be repeated here.

[0178] It can be seen that in a possible implementation, when the new model includes the first network layer and the second network layer, after obtaining the reference image corresponding to the target text, the first network layer in the new model can be opened first, and the second network layer in the new model can be closed, so that the first network layer participates in the data processing process realized by the new model, and the second network layer does not participate in the data processing process realized by the new model, so that the new model is switched to the image feature extraction function; then the first processing is performed on the reference image by using the new model to obtain the image features of the reference image, so that the image features can more accurately and more comprehensively describe the information carried by the reference image.

[0179] Step 34: performing second processing according to the new model, the image features, and the target text to obtain a generated image, the first network layer not participating in the second processing.

[0180] It should be noted that the related content of step 34 is similar to the related content of S503, and for the sake of brevity, it will not be repeated here.

[0181] It can be seen that, in a possible implementation, when the new model above includes the first network layer and the second network layer, after the image feature extraction processing is performed on the reference image corresponding to the target text by using the new model, the first network layer in the new model can be closed first, and the second network layer in the new model can be opened, so that the first network layer does not participate in the data processing process implemented by the new model, and the second network layer participates in the data processing process implemented by the new model, so that the new model is switched to the image generation function; and then the new model is used to perform second processing on the image feature and the target text, to obtain a generated image, so that the generated image can at least meet the constraints described by the target text and the constraints described by the reference image as much as possible, so that the generated image at least has high image quality and can be highly consistent with the target text and the reference image, which is beneficial to improve the image generation effect.

[0182] In another possible implementation, when the new model implements image generation processing by denoising, step 34 above can be specifically: when the first network in the new model is in a non-use state, performing denoising processing on the target text, the image feature of the reference image corresponding to the target text, and the target text according to randomly generated noise data by the target model, to obtain a generated image.

[0183] Based on the related content of steps 31 to 34 above, in some scenarios, the network layer added by the target model relative to the base model can be used as a plug-and-play plug-in, so that in the future, the plug-in can be combined with other models to obtain a model with specific image generation function, which is beneficial to better meet the image generation needs of different image generation tasks, thereby improving flexibility and image generation effect.

[0184] Based on the related content of the model training method and the image generation method above, the technical solutions provided by the present application have the advantages shown in ①-⑥ below.

[0185] ①The present application controls the function switching of the denoising network by means of the first network layer (such as the lora layer), so that the denoising network is used to implement different functions when the first network layer is in different states, which can realize multiple functions by using one model, thereby reducing the resource demand, and further expanding the application range, such as the model can be deployed on a device with less resources (such as A800) for training.

[0186] ②The present application introduces the features of the image constraint into part or all of the modules of the denoising network by means of the second network layer (such as the left half branch shown in FIG. 4), which is beneficial to better improve the influence effect of the image constraint on the image generation processing, thereby improving the image generation effect.

[0187] ③The application trains the model by only updating the newly added network layer, which can effectively improve the training speed. For example, when the target model is obtained by improving the SDXL model with a resolution of 768x512, the time for training the target model once on 500,000 data is 5.5 hours.

[0188] ④The application improves the text constraint provided by the user through text rewriting, which is beneficial to improve the image generation effect.

[0189] ⑤The technical solution provided by the application has higher image quality and constraint consistency, and when the technical solution is applied to a clothing-related scene, the solution has generalization for non-clothing image input.

[0190] ⑥The network layer added to the target model relative to the base model can be used as a plug-and-play plug-in, so that the plug-in can better be compatible with other models, which is beneficial to better realize various image generation controls, thereby improving the image generation effect.

[0191] Based on the model training method provided by the embodiments of the application, the embodiments of the application also provide a model training device. The following will be explained and described in combination with FIG. 7. FIG. 7 is a structural schematic diagram of a model training device provided by an embodiment of the application. It should be noted that the technical details of the model training device provided by the embodiments of the application are described in the above method for training a model.

[0192] As shown in FIG. 7, the model training device 700 provided by the embodiments of the application includes:

[0193] The first acquisition unit 701 is configured to acquire a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, wherein the label image satisfies a constraint described by the sample text and a constraint described by the reference image.

[0194] The first extraction unit 702 is configured to perform first processing on the reference image by using a target model to obtain an image feature of the reference image, wherein the target model includes a first network layer, and the first network layer participates in the first processing.

[0195] The first generation unit 703 is configured to perform second processing according to the target model, the image feature, and the sample text to obtain a generated image, wherein the first network layer does not participate in the second processing.

[0196] The model updating unit 704 is configured to update the first network layer in the target model according to a difference between the generated image and the label image.

[0197] In a possible implementation, the target model is obtained by adding the first network layer to a base model, and the base model has a text-to-image function.

[0198] In a possible implementation, the base model includes a plurality of target layers, and the plurality of target layers include linear layers in the base model and convolutional layers in the base model; the target model includes a plurality of merged layers corresponding to the plurality of target layers; for any target layer, the merged layer corresponding to the target layer is obtained by merging the first network layer into the target layer, and output data of the merged layer corresponding to the target layer is determined according to output data of the target layer, output data of the first network layer, and an enabling function of the first network layer, the enabling function being used to indicate whether the output data of the first network layer affects the output data of the merged layer corresponding to the target layer.

[0199] In a possible implementation, the first network layer includes two linear layers.

[0200] In a possible implementation, the target model further includes a second network layer, the second network layer does not participate in the first processing, the second network layer participates in the second processing, and input data of the second network layer is determined according to the image feature.

[0201] The model updating unit 704 is specifically configured to update the first network layer and the second network layer in the target model.

[0202] In a possible implementation, the target model is obtained by adding the second network layer to a self-attention layer in a base model.

[0203] In a possible implementation, the target model includes an adjusted network layer corresponding to the self-attention layer, the adjusted network layer includes each module in the self-attention layer and the second network layer, and output data of the adjusted network layer is determined according to an enabling parameter of the second network layer, the enabling parameter being used to indicate whether the second network layer is used.

[0204] In a possible implementation, the target model comprises an adjusted network layer corresponding to the self-attention layer, the adjusted network layer comprising the modules in the self-attention layer and the second network layer; input data of an attention module in the second network layer comprises output data of a key projection module in the second network layer, output data of a value projection module in the second network layer, and output data of a question projection module in the self-attention layer; input data of the key projection module in the second network layer and input data of the value projection module in the second network layer are determined according to the image feature; input data of the question projection module in the self-attention layer is determined according to output data of a previous network layer of the adjusted network layer, the previous network layer being adjacent to the adjusted network layer in the target model and being located in front of the adjusted network layer in the target model.

[0205] In a possible implementation, the key projection module in the second network layer is initialized using the key projection module in the self-attention layer; and the value projection module in the second network layer is initialized using the value projection module in the self-attention layer.

[0206] In a possible implementation, the clothing in the label image satisfies the constraint described by the reference image.

[0207] Based on the related content of the model training apparatus 700, the working principle of the model training apparatus 700 provided in this application includes: first, obtaining a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, the label image satisfying a constraint described by the sample text and a constraint described by the reference image; second, performing first processing on the reference image using a target model to obtain an image feature of the reference image, the target model comprising a first network layer, and the first network layer participating in the first processing; third, performing second processing according to the target model, the image feature, and the sample text to obtain a generated image, the first network layer not participating in the second processing; and fourth, updating the first network layer in the target model according to the difference between the generated image and the label image, so that the updated target model has better performance, which is conducive to reducing resource consumption, training time consumption, and image generation effect.

[0208] Based on the image generation method provided in the embodiments of this application, an image generation apparatus is further provided in the embodiments of this application, which is explained and described below with reference to FIG. 8. FIG. 8 is a structural schematic diagram of an image generation apparatus provided in the embodiments of this application. It should be noted that the technical details of the image generation apparatus provided in the embodiments of this application are described above with reference to the related content of the image generation method.

[0209] As shown in FIG. 8, the image generation apparatus 800 provided by the embodiment of the present application comprises:

[0210] The second acquisition unit 801 is configured to acquire a target text and a reference image corresponding to the target text.

[0211] The second extraction unit 802 is configured to perform first processing on the reference image by using a target model to obtain image features of the reference image, wherein the target model comprises a first network layer, the first network layer participates in the first processing, and the target model is obtained by using any embodiment of the model training method provided by the embodiment of the present application.

[0212] The second generation unit 803 is configured to perform second processing according to the target model, the image features and the target text to obtain a generated image, and the first network layer does not participate in the second processing.

[0213] In a possible implementation, the second acquisition unit 801 is specifically configured to: acquire an initial text provided by a user; and perform rewriting processing on the initial text according to the reference image to obtain the target text, wherein semantic information carried by the target text comprises semantic information carried by the initial text and semantic information carried by the reference image.

[0214] In a possible implementation, the second processing comprises encoding processing and decoding processing; if an influence weight of the target text is higher than an influence weight of the reference image, the image features do not participate in the encoding processing, and the image features participate in the decoding processing, the influence weight of the target text is used to describe an influence degree of the target text on the generated image, and the influence weight of the reference image is used to describe an influence degree of the reference image on the generated image; if the influence weight of the target text is not higher than the influence weight of the reference image, the image features participate in the encoding processing, and the image features participate in the decoding processing.

[0215] In a possible implementation, the second acquisition unit 801 is further configured to acquire model indication information provided by a user.

[0216] The image generation apparatus 800 further comprises:

[0217] The model construction unit is configured to add the first network layer into a model matched with the model description information to obtain a new model according to a position of the first network layer in the target model.

[0218] The second extraction unit 802 is specifically configured to perform first processing on the reference image by using the new model.

[0219] The second generation unit 803 is specifically configured to perform second processing according to the new model, the image feature, and the target text.

[0220] In a possible implementation, the target model and the model described in the model description information are both constructed based on a same model.

[0221] Based on the above-mentioned related content of the image generation apparatus 800, the working principle of the image generation apparatus 800 provided in the present application includes: first, obtaining a target text and a reference image corresponding to the target text; then, performing first processing on the reference image by using a trained target model, to obtain an image feature of the reference image, wherein the first network layer in the target model participates in the first processing; then, performing second processing according to the target model, the image feature, and the target text, to obtain a generated image, wherein the first network layer in the target model does not participate in the second processing, so that multiple processing tasks can be completed by using one model, thereby effectively reducing resource consumption. Wherein, the trained target model has relatively good performance, so that the generated image based on the target model is relatively good, so as to improve the image generation quality.

[0222] In addition, the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any embodiment of the model training method provided in the present application, or executes any embodiment of the image generation method provided in the present application.

[0223] Referring to FIG. 9, a structural schematic diagram of an electronic device 900 suitable for implementing the embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 9 is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0224] As shown in FIG. 9, the electronic device 900 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or loaded into a random access memory (RAM) 903 from a storage device 908. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0225] In general, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication devices 909 can allow the electronic device 900 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 9 shows the electronic device 900 with various devices, it should be understood that all of the shown devices are not required to be implemented or possessed. More or less devices can be alternatively implemented or possessed.

[0226] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 909, or installed from the storage devices 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0227] The electronic device provided by the embodiments of the present disclosure and the method provided by the above-mentioned embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiments can be referred to the above-mentioned embodiments, and the present embodiments have the same beneficial effects as the above-mentioned embodiments.

[0228] The embodiments of the present application also provide a computer readable medium, wherein instructions or computer programs are stored in the computer readable medium, and when the instructions or computer programs are run on a device, the device is caused to perform any of the embodiments of the model training method provided by the embodiments of the present application, or perform any of the embodiments of the image generation method provided by the embodiments of the present application.

[0229] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, which bears computer-readable program code. Such a propagated data signal can take many forms, including but not limited to electro-magnetic, optical or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transfer a program for use by or in connection with an instruction execution system, apparatus or device. Program code contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, optical fiber, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0230] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0231] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and is not assembled into the electronic device.

[0232] The computer-readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device can execute the method described above.

[0233] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0234] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0235] The units involved in the embodiments of the present disclosure can be implemented by software, or can be implemented by hardware. In some cases, the name of the unit / module does not constitute a limitation on the unit itself.

[0236] The functions described in the above embodiments of the present disclosure can be performed at least in part by one or more hardware logic components. For example, non-limiting examples of exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0237] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0238] It should be noted that the various embodiments described in the specification are progressive and each embodiment focuses on the differences from other embodiments. The same or similar parts between embodiments can be mutually referred to. For the system or device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0239] It should be understood that in this application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, "A and / or B" can represent: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0240] It is also to be noted that, as used in the specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless otherwise indicated. Furthermore, to the extent that the terms "including," "includes," "having," "has," "with," or "contains" are used in either the detailed description and the claims, such terms are intended to be inclusive in a manner similar to the term "comprising" as an open transition term without precluding any additional or other elements.

[0241] The embodiments disclosed herein can each be implemented as a method, apparatus, or article of manufacture using programming instructions. The embodiments disclosed herein can be implemented using software, firmware, hardware, or a combination thereof. The various elements of the disclosed embodiments, as well as the embodiments themselves, can be constructed from any combination of hardware, software, and / or firmware. The software implementation can be implemented by one or more software modules using object-oriented design methodology, among other techniques. The software modules can be stored on any computer-readable medium, including RAM, ROM, EEPROM, flash memory, or a hard disk, to name a few. The software modules can include one or more routines.

[0242] The above description of disclosed embodiments is intended to be illustrative and not restrictive. Many embodiments will be apparent to those of skill in the art upon reading and understanding the above description, and the applications should not be limited to the embodiments disclosed in this application. Numerous modifications and variations are possible in light of the above teachings without departing from the scope and spirit of the application, which include those modifications and variations that result from the exercise of the options set forth herein. The scope of the application should be determined, not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

Claims

1. A model training method, comprising: obtaining a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, the label image satisfying a constraint described by the sample text and a constraint described by the reference image; performing first processing on the reference image by using a target model to obtain an image feature of the reference image, the target model comprising a first network layer participating in the first processing; performing second processing according to the target model, the image feature, and the sample text to obtain a generated image, the first network layer not participating in the second processing; updating the first network layer in the target model according to a difference between the generated image and the label image.

2. The method of claim 1, wherein the target model is obtained by adding the first network layer to a base model, the base model having a text-to-image function.

3. The method of claim 2, wherein the base model comprises a plurality of target layers, the plurality of target layers comprising a linear layer in the base model and a convolutional layer in the base model; the target model comprises a merged layer corresponding to each of the target layers; for any target layer, the merged layer corresponding to the target layer is obtained by merging the first network layer into the target layer, and output data of the merged layer corresponding to the target layer is determined according to output data of the target layer, output data of the first network layer, and an enable function of the first network layer, the enable function being used to indicate whether the output data of the first network layer affects the output data of the merged layer corresponding to the target layer.

4. The method of claim 1, wherein the first network layer comprises two linear layers.

5. The method of claim 1, wherein the target model further comprises a second network layer, the second network layer not participating in the first processing, the second network layer participating in the second processing, and input data of the second network layer being determined according to the image feature; the updating the first network layer in the target model comprises: updating the first network layer and the second network layer in the target model.

6. The method of claim 5, wherein the target model is obtained by adding the second network layer to a self-attention layer in a base model.

7. The method of claim 6, wherein the target model comprises an adjusted network layer corresponding to the self-attention layer, the adjusted network layer comprising each module in the self-attention layer and the second network layer; output data of the adjusted network layer is determined according to an enable parameter of the second network layer, the enable parameter being used to indicate whether the second network layer is used.

8. The method of claim 6, wherein the target model comprises an adjusted network layer corresponding to the self-attention layer, the adjusted network layer comprising each module in the self-attention layer and the second network layer; The input data of the attention module in the second network layer comprises the output data of the key projection module in the second network layer, the output data of the value projection module in the second network layer, and the output data of the question projection module in the self-attention layer; The input data of the key projection module in the second network layer and the input data of the value projection module in the second network layer are determined according to the image feature; The input data of the question projection module in the self-attention layer is determined according to the output data of a previous network layer of the adjusted network layer, the previous network layer is adjacent to the position of the adjusted network layer in the target model, and the position of the previous network layer in the target model is earlier than the position of the adjusted network layer in the target model.

9. The method of claim 6, wherein the key projection module in the second network layer is initialized using the key projection module in the self-attention layer; The value projection module in the second network layer is initialized using the value projection module in the self-attention layer.

10. The method of any one of claims 1-9, wherein the clothing in the label image satisfies the constraint described by the reference image.

11. An image generation method, comprising: obtaining a target text and a reference image corresponding to the target text; performing first processing on the reference image using a target model to obtain an image feature of the reference image, the target model comprising a first network layer participating in the first processing, the target model being obtained using the model training method of any one of claims 1-10; performing second processing according to the target model, the image feature, and the target text to obtain a generated image, the first network layer not participating in the second processing.

12. The method of claim 11, wherein the obtaining process of the target text comprises: obtaining an initial text provided by a user; rewriting the initial text according to the reference image to obtain the target text, the semantic information carried by the target text comprising the semantic information carried by the initial text and the semantic information carried by the reference image.

13. The method of claim 11, wherein the second processing comprises encoding processing and decoding processing; if the influence weight of the target text is higher than the influence weight of the reference image, the image feature does not participate in the encoding processing, the image feature participates in the decoding processing, the influence weight of the target text being used to describe the influence degree of the target text on the generated image, and the influence weight of the reference image being used to describe the influence degree of the reference image on the generated image; if the influence weight of the target text is not higher than the influence weight of the reference image, the image feature participates in the encoding processing, and the image feature participates in the decoding processing.

14. The method of claim 11, further comprising: obtaining model indication information provided by a user; adding the first network layer to a model matched with the model description information according to a position of the first network layer in the target model, to obtain a new model; the first processing of the reference image by using the target model comprises: the first processing of the reference image by using the new model; the second processing according to the target model, the image feature and the target text comprises: the second processing according to the new model, the image feature and the target text.

15. The method of claim 14, wherein the target model and the model matched with the model description information are both constructed based on a same model.

16. A model training apparatus, comprising: a first obtaining unit configured to obtain a sample text, a reference image corresponding to the sample text, and a label image corresponding to the sample text, the label image satisfying a constraint described by the sample text and a constraint described by the reference image; a first extracting unit configured to perform first processing of the reference image by using a target model to obtain an image feature of the reference image, the target model comprising a first network layer, the first network layer participating in the first processing; a first generating unit configured to perform second processing according to the target model, the image feature and the sample text to obtain a generated image, the first network layer not participating in the second processing; a model updating unit configured to update the first network layer in the target model according to a difference between the generated image and the label image.

17. An image generation apparatus, comprising: a second obtaining unit configured to obtain a target text and a reference image corresponding to the target text; a second extracting unit configured to perform first processing of the reference image by using a target model to obtain an image feature of the reference image, the target model comprising a first network layer, the first network layer participating in the first processing, the target model being obtained by using the model training method of any one of claims 1-10; a second generating unit configured to perform second processing according to the target model, the image feature and the target text to obtain a generated image, the first network layer not participating in the second processing.

18. An electronic device comprising: a processor and a memory; the memory is configured to store instructions or a computer program; the processor is configured to execute the instructions or the computer program in the memory, so that the electronic device performs the method of any one of claims 1-15.

19. A computer readable medium, wherein the computer readable medium stores instructions or a computer program, when the instructions or the computer program are executed on a device, the device is caused to perform the method of any one of claims 1-15.

20. A computer program product, comprising a computer program carried on a non-transitory computer readable medium, the computer program comprising program code for performing the method of any one of claims 1-15.

Images