Model training method, device and equipment in text map scene and storage medium

Abstract

The invention provides a model training method and device in a text graph scene, equipment and a storage medium, and belongs to the technical field of artificial intelligence. For a text map scene, a training framework provided by the present disclosure includes a first branch and a second branch. Wherein each of the first branch and the second branch comprises a text graph model. In the first training stage, the model parameters of the text graph model in the first branch are frozen, and the model parameters in the second branch are updated. According to the scheme, a branch capable of freezing model parameters is additionally introduced on the basis of an original branch. The text graph model on the branch does not participate in training in the first training stage and only performs parameter updating in the second training stage, but only structural information guidance is applied to the text graph model of the first branch in the training process, and the original generation capability of the text graph model cannot be damaged. Therefore, the target of aligning the original generation capability of the text graph model is provided for the second branch.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, and storage medium in a cultural image scenario. Background Art

[0002] Controllable text-based images, as an important branch in the field of image generation, means that when generating images through text descriptions, users can impose control conditions on the generation process to make the generated images more in line with user expectations.

[0003] To achieve controllable text graphs, it's often necessary to train a conditional injection module on top of the existing text graph model. However, while this conditional injection module helps the text graph model achieve conditional control, it can also undermine the model's inherent generative capabilities. For example, training the conditional injection module often requires fine-tuning some or all model parameters. This can overwrite the general generative capabilities learned by the text graph model from massive amounts of data.

[0004] Therefore, how to maintain the original generation capability of the cultural graph model while introducing control conditions has become an urgent problem to be solved in this field. Summary of the Invention

[0005] The present disclosure provides a model training method, apparatus, device, and storage medium for a cultural graph scenario. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a model training method in a culture graph scenario is provided. The training framework in the culture graph scenario includes a first branch and a second branch. The first branch includes a culture graph model and a structural information extraction unit; the second branch includes the culture graph model and a condition injection module. In a first training phase, model parameters of the culture graph model in the first branch are frozen; the condition injection module is used to encode input control conditions and inject the encoded results into the culture graph model. In a second training phase, the method includes:

[0007] Extracting structural information from the control condition by the structural information extraction unit to obtain external features containing structural information;

[0008] Based on the external features and the internal features containing structural information output by the culture graph model in the first branch, an original noise prediction result output at each time step is optimized to obtain an optimized noise prediction result;

[0009] For the first branch, a first loss is obtained based on the noise added at each time step and the optimized noise prediction result output at the corresponding time step; and based on the first loss, the model parameters of the Wensheng graph model in the first branch are updated.

[0010] In some embodiments, optimizing the noise prediction result output at each time step based on the external features and the internal features containing structural information output by the culture graph model in the first branch to obtain the optimized noise prediction result includes:

[0011] For any time step, constructing a structural alignment constraint term based on the difference between the external features and the internal features in the latent space;

[0012] The original noise prediction result output at the time step is subtracted from the structural alignment constraint term to obtain an optimized noise prediction result output at the time step.

[0013] In some other embodiments, constructing a structural alignment constraint based on the difference between the external features and the internal features in the latent space includes:

[0014] Obtaining a gradient of the difference, where the gradient of the difference is used to indicate adjustment of a sampled output of the time step; wherein the sampled output is obtained by sampling a data distribution of the time step;

[0015] The structural alignment constraint term is constructed based on the weight coefficient to be updated, the dynamic attenuation factor that decays with time steps, and the gradient of the difference.

[0016] In some other embodiments, the method further comprises:

[0017] The mean of the difference between the external features and the internal features at each network level is taken as the difference between the external features and the internal features in the latent space.

[0018] In some other embodiments, in the second training phase, the method further includes:

[0019] For the second branch, obtaining a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step;

[0020] Obtaining a third loss based on the text interaction features corresponding to the first branch and the second branch; wherein the text interaction features are used to reflect the response of the features output by the intermediate layer of the text graph model to the text prompt; and the third loss is used to measure the consistency of the text interaction behavior between the first branch and the second branch;

[0021] Obtaining a fourth loss based on features output by the intermediate layer of the Wensheng graph model in the first branch and the second branch; the fourth loss is used to measure the consistency of the first branch and the second branch in image details;

[0022] Based on the second loss, the third loss, and the fourth loss, the model parameters of the Wensheng graph model in the second branch are updated.

[0023] In some other embodiments, the method further comprises:

[0024] Encode the input text prompt to obtain text features;

[0025] Using the text features as a key matrix and the features output by the intermediate layer of the text graph model in the first branch as a first query matrix, and obtaining the text interaction features corresponding to the first branch based on the key matrix and the first query matrix;

[0026] The text features are used as a key matrix, and the features output by the intermediate layer of the text graph model in the second branch are used as a second query matrix. Based on the key matrix and the second query matrix, the text interaction features corresponding to the second branch are obtained.

[0027] In some other embodiments, obtaining the text interaction feature corresponding to the first branch based on the key matrix and the first query matrix includes:

[0028] Obtaining an interaction weight between the key matrix and the first query matrix;

[0029] A weighted sum is performed on the first query matrix based on the interaction weight to obtain the text interaction feature corresponding to the first branch.

[0030] In some other embodiments, the text graph model includes an encoding block, an intermediate block, and a decoding block, and the number of the encoding blocks is consistent with the number of the decoding blocks; the conditional injection module is a trainable copy of the encoding block and the intermediate block;

[0031] In the first training phase, the method further includes:

[0032] The control condition is encoded by the condition injection module to obtain a conditional feature; wherein the condition injection module is a trainable copy of the encoding block and the intermediate block;

[0033] inputting the conditional features into the decoding block;

[0034] For the second branch, a second loss is obtained based on the noise added at each time step and the noise prediction result output at the corresponding time step; based on the second loss, the model parameters of the Wensheng graph model in the second branch are updated.

[0035] In some other embodiments, inputting the conditional features into the decoding block includes:

[0036] Inputting the conditional features processed by the zero convolution layer into the decoding block in a skip connection manner;

[0037] The weights and biases of the zero convolution layer are initialized to zero.

[0038] According to a second aspect of an embodiment of the present disclosure, a model training device for a culture graph scenario is provided. The training framework for the culture graph scenario includes a first branch and a second branch. The first branch includes a culture graph model and a structural information extraction unit; the second branch includes the culture graph model and a conditional injection module. In a first training phase, model parameters of the culture graph model of the first branch are frozen; the conditional injection module is used to encode input control conditions and inject the encoded results into the culture graph model. In a second training phase, the device includes:

[0039] An extraction unit is configured to extract structural information from the control condition through the structural information extraction unit in a second training phase to obtain an external feature containing structural information;

[0040] an optimization unit configured to optimize, in the second training phase, an original noise prediction result output at each time step based on the external features and the internal features containing structural information output by the culture graph model in the first branch, to obtain an optimized noise prediction result;

[0041] an acquisition unit configured to, in the second training phase, acquire, for the first branch, a first loss based on the noise added at each time step and the optimized noise prediction result output at the corresponding time step;

[0042] A first updating unit updates model parameters of the culture graph model in the first branch based on the first loss.

[0043] In some embodiments, the optimization unit is configured to:

[0044] For any time step, constructing a structural alignment constraint term based on the difference between the external features and the internal features in the latent space;

[0045] The original noise prediction result output at the time step is subtracted from the structural alignment constraint term to obtain an optimized noise prediction result output at the time step.

[0046] In some other embodiments, the optimization unit is configured to:

[0047] Obtaining a gradient of the difference, where the gradient of the difference is used to indicate adjustment of a sampled output of the time step; wherein the sampled output is obtained by sampling a data distribution of the time step;

[0048] The structural alignment constraint term is constructed based on the weight coefficient to be updated, the dynamic attenuation factor that decays with time steps, and the gradient of the difference.

[0049] In some other embodiments, the optimization unit is configured to:

[0050] The mean of the difference between the external features and the internal features at each network level is taken as the difference between the external features and the internal features in the latent space.

[0051] In some other embodiments, the apparatus further comprises:

[0052] The acquisition unit is further configured to acquire, for the second branch, a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step;

[0053] The acquisition unit is further configured to acquire a third loss based on text interaction features corresponding to the first branch and the second branch; wherein the text interaction features are used to reflect the response of the features output by the intermediate layer of the text graph model to the input text prompt; and the third loss is used to measure the consistency of the text interaction behavior between the first branch and the second branch;

[0054] The acquisition unit is further configured to acquire a fourth loss based on features output by the intermediate layer of the Wensheng graph model in the first branch and the second branch; the fourth loss is used to measure the consistency of the first branch and the second branch in image details;

[0055] The second updating unit is configured to update the model parameters of the Wensheng graph model in the second branch based on the second loss, the third loss and the fourth loss.

[0056] In some other embodiments, the acquiring unit is further configured to:

[0057] Encoding the text prompt to obtain text features;

[0058] Using the text features as a key matrix and the features output by the intermediate layer of the text graph model in the first branch as a first query matrix, and obtaining the text interaction features corresponding to the first branch based on the key matrix and the first query matrix;

[0059] The text features are used as a key matrix, and the features output by the intermediate layer of the text graph model in the second branch are used as a second query matrix. Based on the key matrix and the second query matrix, the text interaction features corresponding to the second branch are obtained.

[0060] In some other embodiments, the acquiring unit is further configured to:

[0061] Obtaining an interaction weight between the key matrix and the first query matrix;

[0062] A weighted sum is performed on the first query matrix based on the interaction weight to obtain the text interaction feature corresponding to the first branch.

[0063] In some other embodiments, the culture graph model includes an encoding block, an intermediate block, and a decoding block, and the number of the encoding blocks is consistent with the number of the decoding blocks; the conditional injection module is a trainable copy of the encoding block and the intermediate block; and the apparatus further includes:

[0064] an encoding unit configured to, during the first training phase, encode the control condition through the condition injection module to obtain a conditional feature; and input the conditional feature into the decoding block;

[0065] The acquisition unit is further configured to acquire, for the second branch, a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step;

[0066] The third updating unit is configured to update the model parameters of the culture graph model in the second branch based on the second loss.

[0067] In some other embodiments, the encoding unit is configured to:

[0068] Inputting the conditional features processed by the zero convolution layer into the decoding block in a skip connection manner;

[0069] The weights and biases of the zero convolution layer are initialized to zero.

[0070] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device including:

[0071] one or more processors;

[0072] a memory for storing program codes executable by the processor;

[0073] The processor is configured to execute the program code to implement the model training method in the above-mentioned cultural image scenario.

[0074] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, and when the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to execute the model training method in the above-mentioned Wenshengtu scenario.

[0075] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor of an electronic device, implements the model training method in the above-mentioned cultural graph scenario.

[0076] For the Wensheng graph scenario, the training framework provided by the disclosed embodiments includes a first branch and a second branch. The first branch includes a Wensheng graph model and a structural information extraction unit, while the second branch includes a Wensheng graph model and a conditional injection module. During the first training phase, the model parameters of the Wensheng graph model in the first branch are frozen, while the conditional injection module is used to encode the input control conditions and inject the resulting encoding results into the Wensheng graph model. Since the Wensheng graph model in the first branch does not participate in training in the first training phase, its parameters are only updated in the second training phase. However, during training, only the structural information guidance is applied to the Wensheng graph model in the first branch. Compared to the method of implementing conditional control by training an additional conditional injection module, since only structural guidance is applied in this branch to ensure that the generated image meets the control conditions, without the need for training an additional module, the original generation capability of the Wensheng graph model is not compromised. Therefore, the second branch is aligned with the original generation capability of the Wensheng graph model. In other words, this solution enables the Wensheng graph model to achieve controllable generation capability while maintaining the original generation capability, thus ensuring image generation quality.

[0077] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0079] Figure 1 The figure is a schematic diagram of an implementation environment of a model training method in a cultural graph scenario according to an exemplary embodiment.

[0080] Figure 23 is a schematic diagram of a training framework in a text graph scenario according to an exemplary embodiment.

[0081] Figure 3 The figure is a flowchart of a model training method in a text graph scenario according to an exemplary embodiment.

[0082] Figure 4 The figure is a flowchart of another model training method in a text-based graph scenario according to an exemplary embodiment.

[0083] Figure 5 The figure is a block diagram of a model training device in a cultural graph scenario according to an exemplary embodiment.

[0084] Figure 6 The figure is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0085] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0086] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0087] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0088] Figure 1 FIG2 is a schematic diagram showing an implementation environment of a model training method in a cultural graph scenario according to an exemplary embodiment. Figure 1 , the implementation environment includes: a terminal 101 and a server 102.

[0089] Terminal 101 is an electronic device used by a user. In the embodiment of the present disclosure, a target application is installed on terminal 101. The target application can be an instant messaging application, such as a short video application with text-to-image functionality, or an application dedicated to image processing, which is not limited in the present disclosure.

[0090] In some embodiments, the terminal 101 is a device such as a smart phone, a desktop computer, or a laptop computer. Figure 1 The terminal 101 is merely an example of a smartphone. Furthermore, those skilled in the art will appreciate that the number of terminals may be greater or lesser. For example, there may be only a few terminals, or dozens, hundreds, or even more. The embodiments of the present disclosure do not limit the number and type of terminals.

[0091] In other embodiments, the server 102 provides background services for the target application. In addition, the server 102 is connected to the terminal 101 via a wireless network or a wired network.

[0092] In addition, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server, which is not limited by this disclosure. In addition, the server involved in the embodiment of the present disclosure can also include other functional servers to provide more comprehensive and diversified services. In addition, server 102 is responsible for executing the model training method in the context of the text map provided in the embodiment of the present disclosure.

[0093] The following examples will introduce the model training scheme in the context of the text graph provided by the embodiments of the present disclosure.

[0094] Figure 2 3 is a schematic diagram of a training framework in a text graph scenario according to an exemplary embodiment.

[0095] See also Figure 2 The training framework for this text-graph scenario revolves around a dual branch. The dual branch consists of a first branch and a second branch. The first branch includes the text-graph model and a structural information extraction unit, while the second branch includes the text-graph model and a conditional injection module. In other words, these two branches share the model structure but have independent parameters.

[0096] In the embodiment of the present disclosure, the first branch is also called the prototype branch, and the second branch is also called the training branch. In addition, the input of the model includes: a natural image x0, a control condition c corresponding to the image, and a text prompt E txt .

[0097] As an example, the text graph model is an SD model, which mainly includes the following parts: an image compression model based on the encoder-decoder architecture, a text encoder, and a noise prediction network (such as U-Net). Furthermore, U-Net includes: encoding blocks, intermediate blocks, and decoding blocks. The number of encoding blocks is the same as the number of decoding blocks. The conditional injection module is also called a copy encoder. It is called a copy encoder because the conditional injection module is a trainable copy of the encoding block and intermediate block included in U-Net.

[0098] As another example, the structural information extraction unit utilizes DDIM (Denoising Diffusion Implicit Models) to inversely extract structural information.

[0099] In the embodiment of the present disclosure, model training is divided into two stages, referred to as the first training stage and the second training stage.

[0100] In the first training stage, the Wensheng graph model on the prototype branch does not participate in the training. In other words, the purpose of setting the prototype branch is to provide a reference target for the training branch to maintain the original generation ability of the Wensheng graph model. Therefore, the Wensheng graph model on the prototype branch does not participate in the training in the first training stage.

[0101] The Wensheng graph model on the prototype branch undergoes parameter updates only in the second training phase, but only structural information is applied to it during training. In other words, during the generation process, structural information needs to be applied during sampling to ensure that the generated image meets the control condition c of the image form.

[0102] For the training branch, in the first training phase, the conditional injection module encodes the control condition c and injects the resulting encoding into the linguistic graph model to achieve controllable generation. After the first phase of training, the conditional injection module acquires the ability to recognize the control condition c.

[0103] In the second training phase, the text-based graph model on the training branch is aligned with the text-based graph model on the prototype branch in terms of two aspects: on the one hand, the generated content controlled by the text, and on the other hand, the fine details such as layout and ID, which are also called image details.

[0104] Figure 3 is a flowchart of a model training method in a cultural graph scenario according to an exemplary embodiment. Figure 3 As shown in Figure 2, the model training method in the text-based image scenario is applied to electronic devices, such as Figure 1 The server 102 shown. The model training method in the text-graph scenario includes the following steps:

[0105] In 301 , in the second training phase, the electronic device extracts structural information from the control condition through a structural information extraction unit to obtain external features containing structural information.

[0106] In the embodiment of the present disclosure, the input image is a natural image including a target object (such as a person or an animal), and the control condition refers to an additional description other than the text prompt, such as a skeleton map, posture key points, graffiti sketch, edge map, segmentation map or depth map, etc., which is not limited by the present disclosure. txt Working together, they can guide image generation more accurately.

[0107] As an example, the structural information extraction unit extracts structural information using DDIM inversion. In other words, in the controllable generation task, the control condition c is converted from the pixel space to the latent space (latent) feature through DDIM inversion. Inversion refers to the process of converting the noise state X into the latent space. T The process of gradually restoring (close to pure noise) to a clean image X0, where T refers to the Tth time step.

[0108] In detail, DDIM inversion maps the control condition c into the intermediate features of the latent space through the reverse diffusion process. The specific steps are as follows: First, the control condition c is encoded as the noise state X T ; Then, through deterministic branching (rather than random sampling), gradually from X T Recover the underlying intermediate state; that is, at each time step t of the reverse diffusion process, extract the intermediate features including structural information (Feature map form). Where t∈[0, T], C refers to the number of channels of the feature map, H refers to the height of the feature map, and W refers to the width of the feature map. s It is a feature obtained by encoding and inverting the control conditions of the external input, so it is also called an external feature that includes structural information.

[0109] In 302, in the second training phase, the electronic device optimizes the original noise prediction result output at each time step based on the extracted external features and the internal features containing structural information output by the text graph model in the first branch to obtain an optimized noise prediction result.

[0110] In the embodiment of the present disclosure, internal features are relative to external features. The state variables within the internal feature model including structural information refer to the potential intermediate states in the generation process, which are used to gradually denoise and generate the final image. This internal feature is marked as

[0111] In some embodiments, based on the extracted external features and the internal features containing structural information output by the text graph model, the noise prediction results output at each time step are optimized to obtain optimized noise prediction results, including but not limited to the following methods:

[0112] For time step t, based on the extracted external features F s The internal features output by the Wensheng graph model in the first branch The difference in latent space between Construct structural alignment constraints; output the original noise prediction results at time step t Subtract the structural alignment constraint term to obtain the optimized noise prediction result output at time step t Here, θ refers to the model parameters.

[0113] In other embodiments, based on the extracted external features F s The internal features output by the Wensheng graph model in the first branch The difference in latent space between The structural alignment constraint is constructed as follows:

[0114] Get the gradient of the difference between the two in the latent space; the gradient of the difference is used to indicate the adjustment of the sampled output at time step t; the sampled output at time step t is obtained by sampling the data distribution (probability distribution of the image) at time step t; next, based on the weight coefficient λ to be updated and the dynamic attenuation factor ρ that decays with time step t And the gradient of this difference is used to construct the structural alignment constraint term.

[0115] The above construction method can be expressed by the following formula (1):

[0116]

[0117] in, refers to the gradient of that difference; refers to the structural alignment constraint; x t It refers to the potential intermediate state (noisy image) output at time step t during the generation process; the weight coefficient λ to be updated is updated during the model training process.

[0118] It should be noted that the external feature F s With internal features are all multi-level features. Therefore, this solution will transform the external features F s With internal features The mean of the differences at each network level is used as the external feature F s With internal features The above process can be expressed by the following formula (2):

[0119]

[0120] Here, the value of i is a positive integer starting from 1; l refers to the network depth, that is, the number of intermediate layers included in the Wensheng graph model; ||.||2 refers to the L2 norm.

[0121] In the embodiment of the present disclosure, the external feature F s As a guiding condition, the generation process of the prototype branch text graph model will be regulated so that the generated content meets the control condition c. That is, the constraint external feature F s With internal features Be consistent at every level.

[0122] In 303, in the second training phase, for the first branch, the electronic device obtains a first loss based on the noise added at each time step and the optimized noise prediction result output at the corresponding time step; and updates the model parameters of the Wensheng graph model in the first branch based on the first loss.

[0123] In the embodiment of the present disclosure, the first loss can be calculated using the loss function shown in the following formula (3):

[0124]

[0125] Among them, ∈ refers to the noise added at time step t during the forward noise addition process; ∈~N(0,1) means that the noise added at time step t follows a normal distribution; L diff-1 refers to the first loss; E refers to the expected value.

[0126] As an example, this solution updates the model parameters of the first branch of the Wensheng graph model by minimizing the first loss. Alternatively, the training process may be terminated after the first loss is less than a preset loss value or the number of iterations reaches a preset number, which is not limited in this disclosure.

[0127] For the Wensheng graph scenario, the training framework provided by the disclosed embodiments includes a first branch and a second branch. The first branch includes a Wensheng graph model and a structural information extraction unit, while the second branch includes a Wensheng graph model and a conditional injection module. During the first training phase, the model parameters of the Wensheng graph model in the first branch are frozen, while the conditional injection module is used to encode the input control conditions and inject the resulting encoding results into the Wensheng graph model. Since the Wensheng graph model in the first branch does not participate in training in the first training phase, its parameters are only updated in the second training phase. However, during training, only the structural information guidance is applied to the Wensheng graph model in the first branch. Compared to the method of implementing conditional control by training an additional conditional injection module, since only structural guidance is applied in this branch to ensure that the generated image meets the control conditions, without the need for training an additional module, the original generation capability of the Wensheng graph model is not compromised. Therefore, the second branch is aligned with the original generation capability of the Wensheng graph model. In other words, this solution enables the Wensheng graph model to achieve controllable generation capability while maintaining the original generation capability, thus ensuring image generation quality.

[0128] Furthermore, in the second training phase, external features including structural information guide the sampling process. Specifically, these external features serve as guiding conditions and participate in the image generation process. At each time step, this structural information influences the model's decisions, ensuring that the generated image meets the control conditions. In other words, this scheme uses structural alignment constraints to correct the predicted noise, and then uses this corrected noise to guide denoising, ensuring that the model generates images that meet the control conditions. Furthermore, gradient calculations enforce alignment of generated edges with the control conditions, improving the accuracy of the generated structure. Furthermore, by introducing a dynamic decay factor that decays over time, the dynamic variation of constraint strength can be controlled, achieving strong constraints early on while gradually releasing generation freedom later, balancing controllability and diversity. Furthermore, the introduction of weight coefficients balances contextual and structural constraints. Furthermore, calculating and averaging the differences between external and internal features at multiple network levels enables more precise control of the generation direction.

[0129] above Figure 3 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 4 is a flowchart of another model training method in a cultural graph scenario according to an exemplary embodiment. Figure 4 As shown in Figure 2, the model training method in the text-based image scenario is applied to electronic devices, such as Figure 1 The server 102 shown. The model training method in the text-graph scenario includes the following steps:

[0130] In 401 , the electronic device obtains an input image, a text prompt corresponding to the input image, and a control condition.

[0131] In the embodiment of the present disclosure, the input image x0, the text prompt E describing the input image x0 txt and the control condition c form a triplet. By collecting such triplets, we can construct a sample set for model training.

[0132] In 402 , in a first training phase, the electronic device encodes the control condition through a condition injection module to obtain a condition feature, and inputs the condition feature into a decoding block of a Wensheng graph model.

[0133] Among them, the conditional injection module is a trainable copy of the encoding block and intermediate block contained in the U-Net in the Wensheng graph model.

[0134] As an example, the conditional injection module is connected to the decoding block of the Wensheng graph model whose model parameters are frozen through a zero convolution layer (such as a 1*1 convolution layer). It is called a zero convolution layer because the weights and biases of this or these convolution layers are initialized to zero. Accordingly, the conditional features obtained after encoding will be injected into the decoding block of the Wensheng graph model in the form of a skip connection after passing through the zero convolution layer. In other words, this scheme will input the conditional features processed by the zero convolution layer into the decoding block of the Wensheng graph model in the form of a skip connection. Exemplarily, the zero convolution layer performs channel adjustment or spatial alignment on the input conditional features, which is not limited in this disclosure.

[0135] In summary, during training, the zero-convolution layer does not add noise to the network, making it possible for the Wensheng graph model to consistently generate high-quality images. Furthermore, passing conditional information to the Wensheng graph model's decoding block via skip connections not only accelerates training but also preserves detailed information about the control conditions, thereby improving the quality of generated images.

[0136] In 403, in the first training phase, for the second branch, the electronic device obtains a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step; and updates the model parameters of the Wensheng graph model in the second branch based on the second loss.

[0137] In the embodiment of the present disclosure, the second loss can be calculated using the loss function shown in the following formula (4):

[0138]

[0139] Where,∈ refers to the noise added at time step t during the forward noisy process; It refers to the noise prediction result output at time step in the reverse denoising process; ∈~N(0,1) means that the noise added at time step t obeys the normal distribution; L diff-2 Refers to the second loss.

[0140] Among them, the Wensheng graph model obtained by training in the first stage is the backbone network of the training branch, and the original Wensheng graph model with frozen model parameters is the backbone network of the prototype branch.

[0141] In summary, by extracting conditional features through the copy encoder, we achieve independent encoding of the control condition. Furthermore, during the first training phase, the second branch of the text graph model is trained by predicting noise. During training, the difference between the predicted noise and the added noise in the latent space is calculated to constrain the generated direction to conform to the conditional features. This ensures that the generated image meets the input control condition.

[0142] In 404 , in the second training phase, the electronic device extracts structural information from the control condition through a structural information extraction unit to obtain external features containing structural information.

[0143] The second training phase is later than the first training phase in time.

[0144] In 405 , in the second training phase, the electronic device optimizes the original noise prediction result output at each time step based on the extracted external features and the internal features containing structural information output by the text graph model to obtain an optimized noise prediction result.

[0145] This step is similar to the above step 303 and will not be described again here.

[0146] In 406, in the second training phase, for the first branch, the electronic device obtains a first loss based on the noise added at each time step and the optimized noise prediction result output at the corresponding time step; and updates the model parameters of the Wensheng graph model in the first branch based on the first loss.

[0147] This step is similar to the above step 304 and will not be described again here.

[0148] In 407, in the second training phase, for the second branch, the electronic device obtains a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step; and obtains a third loss based on the text interaction features corresponding to the first branch and the second branch; and obtains a fourth loss based on the features output by the intermediate layer of the text graph model in the first branch and the second branch.

[0149] Among them, the text interaction feature is used to reflect the response of the features output by the intermediate layer of the text-based graph model to the text prompt; the third loss is used to measure the consistency of the text interaction behavior between the first branch and the second branch; the fourth loss is used to measure the consistency of the first branch and the second branch in image details (such as layout or ID and other fine content).

[0150] In the embodiment of the present disclosure, in order to achieve the two aforementioned alignment goals, after the first training phase, the text-based graph model of the training branch needs to be constrained in the following two aspects: on the one hand, aligning the generated content of the text control, and on the other hand, aligning the layout, ID and other fine content.

[0151] In some embodiments, the solution achieves aligned text control of generated content by constraining the consistency of text interaction behavior between the training branch and the prototype branch. That is, the solution constructs a text alignment loss function to calculate the text alignment loss (also known as the third loss), and constrains the training branch and the prototype branch to be consistent in text interaction behavior by minimizing the text alignment loss.

[0152] In some other embodiments, the text interaction features corresponding to the first branch and the second branch are obtained in the following manner:

[0153] First, the text prompt E is trained by the text encoder of CLIP (Contrastive Language-Image Pre-Training, a pre-training method or model based on contrastive text-image pairs). txt Encode and obtain text features.

[0154] Next, the extracted text features are used as the key matrix The features output by the middle layer of the Wensheng graph model in the first branch are used as the first query matrix Q ori , based on the bond matrix and the first query matrix Q ori , obtain the text interaction features corresponding to the first branch. For example, the scheme first obtains the key matrix With the first query matrix Q ori Next, the first query matrix Q is calculated based on the obtained interaction weights. ori Perform weighted summation to obtain the text interaction features corresponding to the first branch Among them, in the attention mechanism, the interaction weight usually refers to the weight obtained by calculating the correlation between the query matrix and the key matrix.

[0155] Next, the extracted text features are used as the key matrix The features output by the middle layer of the Wensheng graph model in the second branch are used as the second query matrix Q train, based on the bond matrix and the second query matrix Get the text interaction features corresponding to the second branch. For example, the scheme first obtains the key matrix With the first query matrix Q train Next, the second query matrix Q is calculated based on the obtained interaction weights. train Perform weighted summation to obtain the text interaction features corresponding to the second branch

[0156] Among them, the bond matrix With the key matrix are the same. In addition, the text alignment loss function is shown in the following formula (5):

[0157]

[0158] In other embodiments, the scheme achieves alignment of layout, ID and other fine contents by constraining the consistency of the first branch and the second branch in image details. That is, the scheme calculates the fine alignment loss (also known as the fourth loss) by constructing a fine alignment loss function, and constrains the training branch and the prototype branch to be consistent in image details by minimizing the fine alignment loss. In addition, the text alignment loss function is shown in the following formula (6):

[0159] L fine-grained =||Q ori -Q train ||2 (6)

[0160] That is, the query matrices of the two branches’ respective networks are constrained by the two norms, so that the training branch can generate the same image as the prototype branch in terms of finer content.

[0161] In 408 , in the second training phase, the electronic device updates the model parameters of the Vincent graph model in the second branch based on the second loss, the third loss, and the fourth loss.

[0162] In the second training phase, for the training branch, the scheme calculates the total loss based on the following formula (7) and updates the model parameters of the Wensheng graph model in the training branch by minimizing the total loss.

[0163] L total =L text-consistent +λ1L fine-grained +λ2L diff-2 (7)

[0164] Among them, λ1 and λ2 are weight coefficients to be updated.

[0165] For the Wensheng graph scenario, the training framework provided by the disclosed embodiments includes a first branch and a second branch. The first branch includes a Wensheng graph model and a structural information extraction unit, while the second branch includes a Wensheng graph model and a conditional injection module. During the first training phase, the model parameters of the Wensheng graph model in the first branch are frozen, while the conditional injection module is used to encode the input control conditions and inject the resulting encoding results into the Wensheng graph model. Since the Wensheng graph model in the first branch does not participate in training in the first training phase, its parameters are only updated in the second training phase. However, during training, only the structural information guidance is applied to the Wensheng graph model in the first branch. Compared to the method of implementing conditional control by training an additional conditional injection module, since only structural guidance is applied in this branch to ensure that the generated image meets the control conditions, without the need for training an additional module, the original generation capability of the Wensheng graph model is not compromised. Therefore, the second branch is aligned with the original generation capability of the Wensheng graph model. In other words, this solution enables the Wensheng graph model to achieve controllable generation capability while maintaining the original generation capability, thus ensuring image generation quality.

[0166] In addition, for the training branch, in addition to the original diffusion loss of the text-to-graph model, by calculating the text alignment loss and fine alignment loss, it is possible to control the content generated by the training branch in terms of text control and more detailed content to be consistent with the images generated by the prototype branch, thereby achieving better text control capabilities and generating better image quality.

[0167] Furthermore, by semantically aligning the image features of these two branches, the text-based graph model learns how to embed the control condition without affecting the behavior of the original model. That is, if the presence of the control condition does not affect the original model behavior, then the response of the image features to the text prompt should be consistent in both branches. This approach avoids the problem of the presence of the control condition affecting image quality.

[0168] In addition, through the above-mentioned text interaction feature calculation process, the results obtained reflect the "attention" of image features (Q) to text features (K), that is, which image features are extracted and integrated because they are related to the text, thereby establishing a relationship between text features and image features. This provides a basic guarantee for subsequent generation tasks and realizes the alignment of text to control the generated content.

[0169] In summary, in terms of generation effect, the controllable generation model trained by this scheme has a consistent improvement in fidelity indicators, achieving higher-quality image generation and improving the quality and diversity of image generation.

[0170] Figure 5This is a block diagram of a model training device in a culture graph scenario according to an exemplary embodiment. The training framework in the culture graph scenario includes a first branch and a second branch. The first branch includes a culture graph model and a structural information extraction unit; the second branch includes the culture graph model and a condition injection module; in the first training phase, the model parameters of the culture graph model in the first branch are frozen; the condition injection module is used to encode the input control conditions and inject the obtained encoding results into the culture graph model. Figure 5 , the device comprises:

[0171] The extraction unit 501 is configured to extract structural information from the control condition through the structural information extraction unit in the second training phase to obtain external features containing structural information;

[0172] an optimization unit 502 configured to optimize, in the second training phase, an original noise prediction result output at each time step based on the external features and the internal features containing structural information output by the culture graph model in the first branch, to obtain an optimized noise prediction result;

[0173] an acquiring unit 503 configured to, in the second training phase, acquire, for the first branch, a first loss based on the noise added at each time step and the optimized noise prediction result output at the corresponding time step;

[0174] The first updating unit 504 is configured to update the model parameters of the culture graph model in the first branch based on the first loss in the second training phase.

[0175] For the Wensheng graph scenario, the training framework provided by the disclosed embodiments includes a first branch and a second branch. The first branch includes a Wensheng graph model and a structural information extraction unit, while the second branch includes a Wensheng graph model and a conditional injection module. During the first training phase, the model parameters of the Wensheng graph model in the first branch are frozen, while the conditional injection module is used to encode the input control conditions and inject the resulting encoding results into the Wensheng graph model. Since the Wensheng graph model in the first branch does not participate in training in the first training phase, its parameters are only updated in the second training phase. However, during training, only the structural information guidance is applied to the Wensheng graph model in the first branch. Compared to the method of implementing conditional control by training an additional conditional injection module, since only structural guidance is applied in this branch to ensure that the generated image meets the control conditions, without the need for training an additional module, the original generation capability of the Wensheng graph model is not compromised. Therefore, the second branch is aligned with the original generation capability of the Wensheng graph model. In other words, this solution enables the Wensheng graph model to achieve controllable generation capability while maintaining the original generation capability, thus ensuring image generation quality.

[0176] In some embodiments, the optimization unit is configured to:

[0177] For any time step, constructing a structural alignment constraint term based on the difference between the external features and the internal features in the latent space;

[0178] The original noise prediction result output at the time step is subtracted from the structural alignment constraint term to obtain an optimized noise prediction result output at the time step.

[0179] In some other embodiments, the optimization unit is configured to:

[0180] Obtaining a gradient of the difference, where the gradient of the difference is used to indicate adjustment of a sampled output of the time step; wherein the sampled output is obtained by sampling a data distribution of the time step;

[0181] The structural alignment constraint term is constructed based on the weight coefficient to be updated, the dynamic attenuation factor that decays with time steps, and the gradient of the difference.

[0182] In some other embodiments, the optimization unit is configured to:

[0183] The mean of the difference between the external features and the internal features at each network level is taken as the difference between the external features and the internal features in the latent space.

[0184] In some other embodiments, the apparatus further comprises:

[0185] The acquisition unit is further configured to acquire, for the second branch, a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step;

[0186] The acquisition unit is further configured to acquire a third loss based on text interaction features corresponding to the first branch and the second branch; wherein the text interaction features are used to reflect the response of the features output by the intermediate layer of the text graph model to the input text prompt; and the third loss is used to measure the consistency of the text interaction behavior between the first branch and the second branch;

[0187] The acquisition unit is further configured to acquire a fourth loss based on features output by the intermediate layer of the Wensheng graph model in the first branch and the second branch; the fourth loss is used to measure the consistency of the first branch and the second branch in image details;

[0188] The second updating unit is configured to update the model parameters of the Wensheng graph model in the second branch based on the second loss, the third loss and the fourth loss.

[0189] In some other embodiments, the acquiring unit is further configured to:

[0190] Encoding the text prompt to obtain text features;

[0191] Using the text features as a key matrix and the features output by the intermediate layer of the text graph model in the first branch as a first query matrix, and obtaining the text interaction features corresponding to the first branch based on the key matrix and the first query matrix;

[0192] The text features are used as a key matrix, and the features output by the intermediate layer of the text graph model in the second branch are used as a second query matrix. Based on the key matrix and the second query matrix, the text interaction features corresponding to the second branch are obtained.

[0193] In some other embodiments, the acquiring unit is further configured to:

[0194] Obtaining an interaction weight between the key matrix and the first query matrix;

[0195] A weighted sum is performed on the first query matrix based on the interaction weight to obtain the text interaction feature corresponding to the first branch.

[0196] In some other embodiments, the culture graph model includes an encoding block, an intermediate block, and a decoding block, and the number of the encoding blocks is consistent with the number of the decoding blocks; the conditional injection module is a trainable copy of the encoding block and the intermediate block; and the apparatus further includes:

[0197] an encoding unit configured to, during the first training phase, encode the control condition through the condition injection module to obtain a conditional feature; and input the conditional feature into the decoding block;

[0198] The acquisition unit is further configured to acquire, for the second branch, a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step;

[0199] The third updating unit is configured to update the model parameters of the culture graph model in the second branch based on the second loss.

[0200] In some other embodiments, the encoding unit is configured to:

[0201] Inputting the conditional features processed by the zero convolution layer into the decoding block in a skip connection manner;

[0202] The weights and biases of the zero convolution layer are initialized to zero.

[0203] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0204] It should be noted that the model training device for the Vincent graph scenario provided in the above embodiment only uses the division of the above-mentioned functional units as an example when training the Vincent graph model. In actual applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the model training device for the Vincent graph scenario provided in the above embodiment and the model training method embodiment for the Vincent graph scenario are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0205] Regarding the apparatus in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0206] In some embodiments, when the electronic device is provided as a server, Figure 6 6 is a block diagram of a server 600 according to an exemplary embodiment. The server 600 may vary significantly due to different configurations or performance, and includes one or more processors (Central Processing Units, CPUs) 601 and one or more memories 602, wherein the memories 602 store at least one program code, which is loaded and executed by the processor 601 to implement the image processing methods provided by the various method embodiments described above. Of course, the server also has components such as a wired or wireless network interface, a keyboard, and input and output interfaces for input and output. The server 600 also includes other components for implementing device functions, which will not be described in detail here.

[0207] In some embodiments, a computer-readable storage medium including instructions is further provided, such as a memory including instructions, wherein the instructions are executed by a processor of an electronic device to implement the model training method in the above-mentioned Vincent graph scenario. In other embodiments, the computer-readable storage medium is a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0208] In some embodiments, a computer program product is also provided, including a computer program, which, when executed by a processor of an electronic device, implements the model training method in the above-mentioned cultural graph scenario.

[0209] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0210] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A model training method in a cultural image scenario, characterized in that: The training framework in the context of the culture graph includes a first branch and a second branch, wherein the first branch includes a culture graph model and a structural information extraction unit, and the second branch includes the culture graph model and a conditional injection module; In the first training phase, the model parameters of the Wensheng graph model in the first branch are frozen; The condition injection module is used to encode the input control conditions and inject the obtained encoding results into the Wensheng graph model; In the second training phase, the method includes: Extracting structural information from the control condition by the structural information extraction unit to obtain external features containing structural information; Based on the external features and the internal features containing structural information output by the culture graph model in the first branch, an original noise prediction result output at each time step is optimized to obtain an optimized noise prediction result; For the first branch, a first loss is obtained based on the noise added at each time step and the optimized noise prediction result output at the corresponding time step; and based on the first loss, the model parameters of the Wensheng graph model in the first branch are updated.

2. The model training method in the context of cultural images according to claim 1, characterized in that: The step of optimizing the noise prediction result output at each time step based on the external features and the internal features containing structural information output by the culture graph model in the first branch to obtain an optimized noise prediction result includes: For any time step, constructing a structural alignment constraint term based on the difference between the external features and the internal features in the latent space; The original noise prediction result output at the time step is subtracted from the structural alignment constraint term to obtain an optimized noise prediction result output at the time step.

3. The model training method in the context of a cultural image according to claim 2, characterized in that: The constructing of a structural alignment constraint based on the difference between the external features and the internal features in the latent space includes: Obtaining a gradient of the difference, where the gradient of the difference is used to indicate adjustment of a sampled output of the time step; wherein the sampled output is obtained by sampling a data distribution of the time step; The structural alignment constraint term is constructed based on the weight coefficient to be updated, the dynamic attenuation factor that decays with time steps, and the gradient of the difference.

4. The model training method in the context of a Vincent graph according to any one of claims 1 to 3, characterized in that: The method further comprises: The mean of the difference between the external features and the internal features at each network level is taken as the difference between the external features and the internal features in the latent space.

5. The model training method in the context of a cultural image according to claim 1, characterized in that: In the second training phase, the method further includes: For the second branch, obtaining a second loss based on the noise added at each time step and the noise prediction result output at the corresponding time step; Obtaining a third loss based on the text interaction features corresponding to the first branch and the second branch; wherein the text interaction features are used to reflect the response of the features output by the intermediate layer of the text graph model to the input text prompt; and the third loss is used to measure the consistency of the text interaction behavior between the first branch and the second branch; Obtaining a fourth loss based on features output by the intermediate layer of the Wensheng graph model in the first branch and the second branch; the fourth loss is used to measure the consistency of the first branch and the second branch in image details; Based on the second loss, the third loss, and the fourth loss, the model parameters of the Wensheng graph model in the second branch are updated.

6. The model training method in the context of a cultural image according to claim 5, characterized in that: The method further comprises: Encoding the text prompt to obtain text features; Using the text features as a key matrix and the features output by the intermediate layer of the text graph model in the first branch as a first query matrix, and obtaining the text interaction features corresponding to the first branch based on the key matrix and the first query matrix; The text features are used as a key matrix, and the features output by the intermediate layer of the text graph model in the second branch are used as a second query matrix. Based on the key matrix and the second query matrix, the text interaction features corresponding to the second branch are obtained.

7. The model training method in the context of a cultural image according to claim 6, characterized in that: The acquiring, based on the key matrix and the first query matrix, the text interaction feature corresponding to the first branch includes: Obtaining an interaction weight between the key matrix and the first query matrix; A weighted sum is performed on the first query matrix based on the interaction weight to obtain the text interaction feature corresponding to the first branch.

8. The model training method in the context of cultural images according to claim 1, characterized in that: The Wensheng graph model includes a coding block, an intermediate block and a decoding block, and the number of the coding blocks is consistent with the number of the decoding blocks; The conditional injection module is a trainable copy of the encoding block and the intermediate block; In the first training phase, the method further includes: Encoding the control condition through the condition injection module to obtain a condition feature; inputting the conditional features into the decoding block; For the second branch, a second loss is obtained based on the noise added at each time step and the noise prediction result output at the corresponding time step; based on the second loss, the model parameters of the Wensheng graph model in the second branch are updated.

9. The model training method in the context of a cultural image according to claim 8, characterized in that: The step of inputting the conditional features into the decoding block comprises: Inputting the conditional features processed by the zero convolution layer into the decoding block in a skip connection manner; The weights and biases of the zero convolution layer are initialized to zero.

10. A model training device in a cultural image scenario, characterized in that: The training framework in the context of the culture graph includes a first branch and a second branch, wherein the first branch includes a culture graph model and a structural information extraction unit; the second branch includes the culture graph model and a conditional injection module; In a first training phase, the model parameters of the culture graph model of the first branch are frozen; The condition injection module is used to encode the input control conditions and inject the obtained encoding results into the Wensheng graph model; The device comprises: An extraction unit is configured to extract structural information from the control condition through the structural information extraction unit in a second training phase to obtain an external feature containing structural information; an optimization unit configured to optimize, in the second training phase, an original noise prediction result output at each time step based on the external features and the internal features containing structural information output by the culture graph model in the first branch, to obtain an optimized noise prediction result; The first update unit is configured to, in the second training phase, obtain a first loss for the first branch based on the noise added at each time step and the optimized noise prediction result output at the corresponding time step; and update the model parameters of the Wensheng graph model in the first branch based on the first loss.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the model training method in the Wenshengtu scenario as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the model training method in the Wenshengtu scenario as described in any one of claims 1 to 9.

13. A computer program product, characterized in that It includes a computer program, which, when executed by a processor of an electronic device, implements the model training method in the Wenshengtu scenario as described in any one of claims 1 to 9.

Images