Model training method and device, electronic equipment, storage medium and product

Abstract

This application discloses a model training method, apparatus, electronic device, storage medium, and product, belonging to the field of computer technology. The method includes obtaining user instructions including first text and a first layout bounding box, where the first text specifies the category of the object to be generated, and the first layout bounding box constrains the spatial position and size of the object; based on the user instructions, using a feature decoupling network of a first diffusion model, performing feature decoupling processing layer by layer on the intermediate features generated during model propagation to obtain a first image; training the first diffusion model according to sampling weight information and reward information of the first image to obtain a second diffusion model; the reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box, and the sampling weight information is determined by inputting the user instructions and the second image into the first diffusion model and the third diffusion model respectively, and based on the degree of difference between the outputs of the first diffusion model and the third diffusion model.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to a model training method, apparatus, electronic device, storage medium and product. Background Technology

[0002] Diffusion models have become one of the mainstream models in the field of image generation. Users only need to input Gaussian noise into the model and combine it with control factors such as text. After multiple iterations of denoising, high-quality images can be generated. In order to further improve the control over the generated images, a diffusion model variant based on layout control has recently emerged. In addition to text control factors, this type of model allows users to input multiple object names and their corresponding layout bounding boxes to constrain the spatial position and size of the objects, thereby achieving accurate rendering of the objects to the positions of the corresponding bounding boxes.

[0003] However, training a diffusion model based on layout control relies on a large number of images and training data paired with user commands. This paired training data requires providing a corresponding text description and layout bounding box for each object in the image. Because this paired training data requires a high degree of detail, existing publicly available data cannot be directly reused. Object annotation, layout bounding box annotation, and command organization must be done manually. This annotation process is cumbersome and time-consuming, increasing the human and time costs of training data preparation and resulting in high training costs for the diffusion model. Summary of the Invention

[0004] The purpose of this application is to provide a model training method, apparatus, electronic device, storage medium, chip, and computer program product that can reduce the training cost of diffusion models.

[0005] In a first aspect, embodiments of this application provide a model training method, including: Obtain user instructions, which include first text and a first layout bounding box. The first text is used to specify the category of the object to be generated, and the first layout bounding box is used to constrain the spatial position and size of the object. Based on user instructions, the intermediate features generated during the model propagation process are decoupled layer by layer through the feature decoupling network of the first diffusion model to obtain the first image. Based on the sampling weight information and the reward information of the first image, a first diffusion model is trained to obtain a second diffusion model. The reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting the user command and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image.

[0006] Secondly, embodiments of this application provide a model training apparatus, including: The acquisition module is used to acquire user instructions. The user instructions include first text and a first layout bounding box. The first text is used to specify the category of the object to be generated, and the first layout bounding box is used to constrain the spatial position and size of the object. The processing module is used to perform feature decoupling processing on the intermediate features generated during the model propagation process layer by layer through the feature decoupling network of the first diffusion model, based on user instructions, to obtain the first image. The training module is used to train a first diffusion model based on sampling weight information and reward information of the first image to obtain a second diffusion model. The reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting user instructions and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image.

[0007] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the model training method as shown in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the model training method as shown in the first aspect are implemented.

[0009] Fifthly, embodiments of this application provide a chip, which includes a processor and a display interface, the display interface and the processor being coupled together, the processor being used to run programs or instructions to implement the steps of the model training method as shown in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the model training method as described in the first aspect.

[0011] In this embodiment, a user instruction is obtained, which includes a first text and a first layout bounding box. The first text specifies the category of the object to be generated, and the first layout bounding box constrains the spatial position and size of the object. Based on the user instruction, the intermediate features generated during the model propagation process are decoupled layer by layer through the feature decoupling network of the first diffusion model to obtain a first image. The first diffusion model is trained according to the sampling weight information and the reward information of the first image to obtain a second diffusion model. The reward information is used to characterize the matching degree between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting the user instruction and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image. In this way, user commands including the first text and the first layout bounding box, combined with the intermediate feature decoupling network of the first diffusion model, can decouple intermediate features generated during model propagation layer by layer. This enables constraint learning of object spatial position and size within the model, eliminating the need for manual annotation, layout bounding box annotation, and command organization for each object. Reward information is automatically calculated based on the generated first image, allowing for automated evaluation of the matching degree between the generated result and layout constraints. Supervision signals are no longer dependent on manually labeled data. Model training combines sampling weight information and reward information from noisy images, effectively guiding the diffusion model to learn correct object generation and spatial layout rules even without finely labeled data. Therefore, there is no need to reuse or modify existing public datasets, saving the tedious and time-consuming manual annotation process, significantly reducing the human and time costs of training data preparation, and overall reducing the training cost of layout-controlled diffusion models. Attached Figure Description

[0012] Figure 1 Schematic diagrams of the structure of a model training system provided for some embodiments of this application; Figure 2 Flowcharts of model training methods provided for some embodiments of this application; Figure 3 A schematic diagram of the structure of a model training device provided for some embodiments of this application; Figure 4A schematic diagram of the structure of an electronic device is provided for some embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided for some embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. Embodiments of this application provide a model training method, apparatus, electronic device, and storage medium.

[0015] Diffusion models have become one of the most important models in the field of image generation today. Users only need to input Gaussian noise data into the model, combine it with control factors such as text, and iterate for a total of T time steps to obtain high-quality generated images. This capability is due to the model's simulation of data diffusion states during the training phase using a large number of image and text data pairs. Generally speaking, from Gaussian noise... The evolution of image x0 follows Langevin dynamics, and the noisy image x at time t can be obtained through sampling equations such as Denoising Diffusion Implicit Models (DDIM) or Flow Matching. t For ease of description, the model training method provided in this application is based on Flow Matching to illustrate the subsequent training process.

[0016] In this context, the noisy image corresponding to a certain time t is: During training, the model Given time t, noisy image xt, and control factors such as text c as input, predict the velocity field of the noisy image added to the image at that time, thereby constructing the training objective function as shown in formula (1): in, That is, the velocity field of the noisy image, which represents the true dynamic velocity field from noise to image. The model needs to input x based on a series of conditions at the current time t. t , ...to predict the velocity field at the next time step and compare it with the actual dynamic velocity field. The L2 norm is calculated to learn how to predict. Time t is constrained to between 0 and 1, and samples are randomly and uniformly distributed in [0,1) during the training phase. After training, during the test inference phase, for a given noise level... And control text c, the model starts from time t=1 and initial noise Start by following the time step Predict the next time step t- The velocity field of t is determined, and the Gaussian noise data is iteratively updated according to the following formula (2): This continues until x0 is obtained as the final generated image, i.e., the first image in this embodiment. Generally, the inference phase is divided into approximately 1000 time steps ( This is used to gradually complete the reasoning process from noise to generated image.

[0017] To further enhance control over generated images, a recent variant of the diffusion model based on layout control has emerged. This variant allows users to specify multiple objects and their corresponding bounding boxes, thereby rendering the objects of interest to their corresponding locations, significantly improving the model's usability and scalability. Specifically, for a resolution of W... For the image H, in addition to providing a text c describing the overall information of the generated image, the user can also specify multiple objects of interest in c using the noun pi, and specify their corresponding layout bounding boxes bi=[x0 / W, y0 / H, x1 / W , y1 / H]. That is, by drawing a rectangle on the canvas, the user can specify the coordinates of the top-left corner (x0, y0) and the bottom-right corner (x1, y1) of the bounding box, thus rendering the corresponding objects at their respective positions according to the image coordinate system. In existing technologies, solving this task only requires replacing the text c with a comprehensive instruction feature y composed of {c, {(pi,bi) | 1≤ i≤ n}} compared to the traditional training process. This feature y is extracted by an independent network and is related to x... tFeature concatenation is performed as input to the diffusion model for training. For example, the model input instruction y includes the overall image text c, the object pi, and their respective layout bounding boxes. It should be noted that the object in the embodiments of this application may also be referred to as an instance or object.

[0018] While the layout-controlled diffusion model, a variant of the diffusion model, demonstrates greater ease of use and controllable generation capabilities, training it faces numerous challenges. These challenges include the susceptibility of instances to mutual influence; when generating two spatially close instances, the attributes of one instance, such as color or number, can be affected by the other, leading to incorrect rendering—a problem known as attribute leakage. Furthermore, as discussed earlier, training the layout-controlled diffusion model requires similar training data as the standard text-based image diffusion model, relying on a large number of images and training data paired with user commands. This paired training data requires providing corresponding text descriptions and layout bounding boxes for each object in the image. Due to the high level of detail required for this paired training data, existing publicly available data cannot be directly reused. Object annotation, layout bounding box annotation, and command organization must be done manually, a cumbersome and time-consuming process that increases the human and time costs of training data preparation, resulting in high training costs for the diffusion model.

[0019] To address the aforementioned problems, embodiments of this application provide a model training method, apparatus, device, storage medium, and program product, aiming to overcome the limitations of existing diffusion model training methods, alleviate the problem of attribute leakage during image generation and improve generation quality, while eliminating the dependence on image data during training and completing model training solely through text commands, thereby improving training efficiency and reducing training costs.

[0020] The following is in conjunction with the appendix Figures 1 to 5 The model training method provided in this application will be described in detail through specific embodiments and application scenarios.

[0021] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0022] The terminology used in the embodiments of this invention will be explained below.

[0023] A diffusion model is a deep learning model. It simulates the diffusion process in physics using large amounts of data and learns how images and noise can be converted into each other. In this embodiment, the diffusion model can be either a Transformer diffusion model or a U-Net diffusion model.

[0024] Feature disentanglement is an operation that selects intermediate features from the output of intermediate layers of a diffusion model. It improves the quality of the final image generation by removing the parts of the features that are harmful to the final generated result.

[0025] The Instance Disentanglement Module (IDM) is used to infer the action to be performed from the state representations of each successive state. Its role is to model the mapping relationship between state transitions and actions.

[0026] Reinforcement learning is a model training method that encourages the model to generate outputs that are more in line with human preferences by assigning different rewards to the model's output.

[0027] The bounding box is the smallest rectangle that encloses an object on a two-dimensional plane.

[0028] A trajectory is a set of forward and inverse relationships between noise and image constructed by the diffusion model over a total of T time steps. The set of output features of the diffusion model at each time step t (1≤t≤T) is called the trajectory.

[0029] A Visual Language Model (VLM) is a type of model that performs visual tasks through text instructions. The GDINO involved in this application embodiment is a type of VLM, which can specify the second layout bounding box of the object to be generated in the input image (i.e., the first image) and the confidence value of the detected object based on the first text.

[0030] The intermediate layer refers to all feature transformation layers in the diffusion model from the input layer to the output layer. In other words, it refers to all network layers that process intermediate features Z in the process of gradually denoising and generating an image from Gaussian noise-based data.

[0031] The following is combined with Figure 1 The model training system provided in the embodiments of this application will be described in detail.

[0032] like Figure 1 As shown, in some embodiments of this application, the model training system 10 provided in this application embodiment can be an improved model of the diffusion model. The model training system may include an input layer, an intermediate layer, an instance disentanglement module (IDM), a diffusion generation unit, an output layer, and a reinforcement learning reward module.

[0033] The input layer receives user commands, Gaussian noise data, and time steps, and encodes this data before transmitting it to the intermediate layer. The intermediate layer comprises at least two intermediate layers, each followed by a feature decoupling network. The first intermediate layer can mark the spatial region of the generated object based on user commands, obtaining a layout mask M. The first intermediate layer can then send the layout mask M and the original intermediate features Z generated during model propagation to the feature decoupling network. This feature decoupling network can determine feature weighting information based on M and Z. And filter the effective features output by the first intermediate layer. The effective features are those in the original intermediate features that match the spatial location and size of the object constrained by the user instruction and are used in the subsequent image generation process. These effective features are then transmitted to the next intermediate layer after the first intermediate layer. Based on this, the next intermediate layer can output its intermediate features based on the effective features, and decouple these intermediate features from the layout mask input to the feature decoupling network after the intermediate layer. This process is repeated layer by layer until the effective features output by the last intermediate layer are obtained. The effective features output by the last intermediate layer are then used as the input to the next diffusion generation unit to iteratively predict the velocity field of the noisy image, gradually reducing noise and generating the first image. The first image is then output through the output layer.

[0034] Based on this, the generated first image is input into the reinforcement learning reward module. This module can send the first image to the GDINO reward module to calculate the reward information of the first image. Furthermore, the reinforcement learning reward module can synthesize a noisy image at each of the T time steps using Gaussian noise data and the first image. Based on the noisy image at each time step and the user instruction, the sampling weight information is determined. Based on this, the reinforcement learning reward module can train a diffusion model based on the reward information and sampling weight information of the first image, where the noisy image is obtained by adding Gaussian noise data to the first image.

[0035] Furthermore, in other embodiments of this application, in order to enable the IDM to output highly accurate feature weighting information, the model training system 10 may further include a feature decoupling network reinforcement learning module, which can calculate the instance decoupling constraint loss function. This first bias information is used during the training phase of the IDM. During inference, the IDM only outputs valid features, discards invalid features, and passes the clean, valid features to the next intermediate layer, completely blocking feature interference between different instances and mitigating the attribute leakage problem.

[0036] Based on this, the model training system provided in this application embodiment can achieve a heuristic feature decoupling method by explicitly considering and designing an automatic feature selection module and applying feature decoupling constraints, thereby reducing the possibility of attribute leakage from a mechanistic perspective. Moreover, the reinforcement learning training framework reduces the requirement for training data; only text and layout information are needed to complete training, which not only accelerates model iteration but also lowers the training threshold. Furthermore, it has good compatibility with different diffusion model architectures. Although there are various variations of diffusion models, the model training system provided in this application embodiment can be easily applied to the intermediate outputs of various diffusion models, exhibiting strong versatility. Specifically, after deploying the IDM to each intermediate layer in the diffusion model, the IDM can receive intermediate features and M, fuse them to obtain the final effective features, and use them as input to subsequent intermediate layers. Instance decoupling constraints can be applied to the obtained effective and ineffective features, and these constraints can be used as part of the optimization loss function to optimize the model. Simultaneously, starting from the M input, a series of trajectories and the final generated graph are sampled and fed into GDINO to calculate reward information. A reward is assigned to each trajectory based on the reward information, and then the reward and KL divergence value are used as the loss function value to complete the diffusion model optimization. In this way, there is no need to reuse or modify existing public datasets, saving the tedious and time-consuming manual annotation process, significantly reducing the human and time costs of training data preparation, and reducing the overall training cost of diffusion models based on layout control.

[0037] It should be noted that the model training system provided in this application embodiment can be set on electronic devices such as mobile phones, tablets, laptops, PDAs, and wearable devices, or it can be set on a server.

[0038] The following is combined with Figure 2 This application provides a detailed description of a model training method based on an embodiment.

[0039] Figure 2 A flowchart of a model training method provided for some embodiments of this application.

[0040] like Figure 2 As shown, the model training method provided in this application embodiment can be applied to electronic devices or servers. Based on this, the model training method may include steps 210 to 230, as detailed below.

[0041] Step 210: Obtain user instructions. User instructions include first text and a first layout bounding box. The first text is used to specify the category of the object to be generated, and the first layout bounding box is used to constrain the spatial position and size of the object.

[0042] In this step, the first text explicitly specifies the objects to be generated. This text can be descriptive information about the objects; for example, if the first text is "Generate a cat and a book," it specifies that the generated objects are two instances: "cat" and "book." The first layout bounding box is used to constrain the spatial position and size of the two objects. A corresponding first layout bounding box is set for "cat" and "book," respectively. Specifically, the first layout bounding box for the cat can be represented by coordinates (50, 80) to (200, 250), constraining the cat to be generated in the left area of ​​the image with a size of 150×170. The first layout bounding box for the book can be represented by coordinates (300, 80) to (450, 180), constraining the book to be generated in the right area of ​​the image with a size of 150×100. Therefore, this step eliminates the need for manual annotation of the existing image, directly specifying the objects and layout constraints through user instructions, thus avoiding the tedious process of manual annotation.

[0043] Step 220: Based on user instructions, the intermediate features generated during the model propagation process are decoupled layer by layer through the feature decoupling network of the first diffusion model to obtain the first image.

[0044] In this step, the user instruction obtained in step 210, namely the first text and two first layout bounding boxes, are input into the first diffusion model. The first diffusion model starts the image generation process. The at least two intermediate layers inside it can be exemplarily displayed as three intermediate layers, namely intermediate layer 1, intermediate layer 2, and intermediate layer 3. During the feature propagation process, corresponding intermediate features will be generated.

[0045] Each intermediate layer is followed by a feature decoupling network. The layout mask generated based on user instructions corresponds to the valid and invalid regions of the two first layout bounding boxes. Each feature decoupling network decouples the intermediate features generated by its corresponding intermediate layer layer by layer.

[0046] Specifically, after intermediate layer 1 generates the original intermediate features, its corresponding feature decoupling network receives these original intermediate features and the layout mask, outputs feature weighting information, and filters out valid features that match the first layout bounding boxes of "cat" and "book" through Hadamard product processing, while eliminating invalid features outside the layout. Next, the valid features output by intermediate layer 1 are fed into intermediate layer 2, and the feature decoupling network corresponding to intermediate layer 2 repeats the above decoupling process to further optimize the valid features and suppress invalid interference. Then, intermediate layer 3 is the last intermediate layer. After its corresponding feature decoupling network completes the final feature decoupling, it outputs the final valid features. Based on these valid features, a first image including a cat and a book is generated, in which the two objects are located within their respective first layout bounding boxes.

[0047] Step 230: Train the first diffusion model based on the sampling weight information and the reward information of the first image to obtain the second diffusion model; the reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box; the sampling weight information is determined by inputting the user command and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model; the third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round; the second image is a noisy image obtained by adding Gaussian noise data to the first image.

[0048] In this step, the first image obtained in step 220 is input into a preset detection model to obtain a second bounding box corresponding to the object in the first image. Then, the intersection-union ratio (IUR) of the second bounding box and the first bounding box obtained in step 210 is calculated. The IUR is then combined with the object confidence value to obtain the reward information for the first image, which characterizes the matching degree between the second and first bounding boxes. Additionally, Gaussian noise is added to the first image obtained in step 220 to obtain a second image. The second image and user commands are then input into a first diffusion model and a third diffusion model, respectively, to obtain a first forward inference result output by the first diffusion model and a second forward inference result output by the third diffusion model. Based on this, the ratio of the first forward inference result to the second forward inference result is determined as sampling weight information, which characterizes the degree of change in the output of the first diffusion model relative to the third diffusion model. In this way, based on the reward information and sampling weight information obtained above, the first reward weight value can be determined, the sampling weight information can be pruned to obtain the second reward weight value within the preset interval, and the maximum value between the first reward weight value and the second reward weight value can be taken as the policy gradient reward value. The first diffusion model is then trained, and the second diffusion model is obtained after training.

[0049] In this way, by using user instructions including the first text and the first layout bounding box, and combining the intermediate layer feature decoupling network to decouple the propagation features layer by layer, the constraint learning of object spatial position and size can be automatically achieved within the model, without the need for additional manual annotation, layout bounding box annotation, and instruction organization for each object. Reward information is automatically calculated based on the generated first image, enabling automated evaluation of the degree of matching between the generated result and the layout constraints, eliminating reliance on manually labeled data for supervision. Furthermore, by combining the sampling weight information and reward information of noisy images for model training, the diffusion model can be effectively guided to learn the correct object generation and spatial layout rules even without finely labeled data. Therefore, there is no need to reuse or modify existing public datasets, saving the tedious and time-consuming manual annotation process, significantly reducing the human and time costs of training data preparation, and overall reducing the training cost of the layout-controlled diffusion model.

[0050] The steps described above are explained in detail below.

[0051] Regarding step 210, the first text in this embodiment is a user-inputted text instruction that explicitly specifies the objects to be generated in the image. It describes one or more instances or objects included in the image to be generated, providing semantic constraints on the generated content for the diffusion model. For example, the first text could be "Generate a cat and a book," which explicitly specifies that the model needs to generate two objects: a cat and a book.

[0052] In this embodiment, the first layout bounding box refers to the spatial constraint information corresponding to each object in the first text. It defines the position, size, and range of the corresponding instance in the generated image using coordinates, thereby controlling the layout of the instance. For example, the first layout bounding box for the instance "cat" can be represented by coordinates (50, 80)-(200, 250) to constrain the cat to be generated in the left area of ​​the image, with a size of 150×170. The first layout bounding box for the instance "book" can be represented by coordinates (300, 80)-(450, 180) to constrain the book to be generated in the right area of ​​the image, with a size of 150×100.

[0053] Regarding step 220, in some embodiments of this application, an IDM is provided for heuristically optimizing the output of intermediate layers of a diffusion model, improving the final generation effect and mitigating the attribute leakage problem by discarding useless intermediate features. For example... Figure 1 As shown, the IDM is deployed after each intermediate layer. During the inference testing phase, for intermediate features... Ideal feature decoupling network Construct a layout mask based on the first text pi input by the user and the first layout bounding box bi, i.e., {(pi,bi) |1≤i≤n}. To predict feature-weighted information ,Right now This feature weighting information will be combined with the intermediate features. Calculate the Hadamard product Used to determine valid features And suppress ineffective features This is to construct the first image based on the effective features of each intermediate layer input.

[0054] Based on this, the number of intermediate layers of the feature decoupling network is at least two, and each of the at least two intermediate layers is followed by a feature decoupling network. Based on this, step 220 may specifically include steps 2201 to 2204 to decouple features through the feature decoupling network to obtain the first image.

[0055] Step 2201: Based on user instructions, mark the spatial region generated by the object to obtain a layout mask; the layout mask includes a first mask region and a second mask region, the first mask region is used to mark the valid image region surrounded by the first layout bounding box, and the second mask region is used to mark the invalid image region outside the first layout bounding box.

[0056] In this step, the user instruction includes a first text and a first layout bounding box, which can be represented as {(pi, bi) | 1≤ i ≤ n}, where the first text can describe n objects, pi is the information of the object, and bi is the first layout bounding box corresponding to each object. Based on this, the first mask region enclosed by the first layout bounding box bi can be marked as 1, and the second mask region outside the first layout bounding box bi can be marked as 0. Thus, it can be used to distinguish between spatial regions that match the user instruction and irrelevant regions in the intermediate features.

[0057] Step 2202: Based on the original intermediate features and layout mask generated by the first intermediate layer during model propagation in at least two intermediate layers, feature weighting information is obtained by setting a feature decoupling network after the first intermediate layer. The feature weighting information is used to characterize the probability that each pixel feature of the generated image is retained.

[0058] In this step, the feature decoupling network ε(z, M) is used to predict the feature weighting information α, i.e., α=ε(z, M), by combining the intermediate feature z and the layout mask M. This α is used to quantify the effectiveness of each part of the intermediate feature z and together with the layout mask M, it achieves feature selection.

[0059] Step 2203: Perform Hadamard product processing on the feature weighted information and the original intermediate features of the first intermediate layer to obtain the effective features and invalid features output by the first intermediate layer. The effective features are those in the original intermediate features that match the spatial position and size of the object constrained by the user instruction and are used in the subsequent image generation process. The invalid features are those in the original intermediate features that do not match the spatial position and size of the object constrained by the user instruction and are not involved in the subsequent image generation process.

[0060] In this step, the effective features output by the first intermediate layer can be obtained using the following formula (3). That is, by retaining the valid part of the intermediate feature z that matches the layout mask M, and the invalid features output by the first intermediate layer, α is used. That is, by suppressing and separating the invalid parts of the intermediate feature z that do not match the layout mask M through α.

[0061] Step 2204: If the first intermediate layer is the last of at least two intermediate layers, generate the first image based on the valid features output by the first intermediate layer.

[0062] Therefore, by setting a feature decoupling network after multiple intermediate layers and combining it with a layout mask to weight and filter the original intermediate features, it is possible to accurately retain effective features within the layout and eliminate invalid interference features outside the layout, so that the generated image strictly follows the spatial position and size constraints specified by the user, thereby improving the layout accuracy of the generated image.

[0063] It should be noted that during the process of generating the first image using the diffusion model, only effective features are considered. Only those features will be applied in subsequent generation processes, while invalid features... These features will be discarded directly, thus completing the feature decoupling of the current intermediate layer. If the first intermediate layer is the last of at least two intermediate layers, the first image can be generated based on the effective features output by the first intermediate layer. Specifically, the effective features output by the first intermediate layer can be fed into the diffusion generation unit of the inlet diffusion model, iteratively predicting the velocity field of the noisy image layer by layer, gradually reducing noise along the reverse path, and finally generating the first image completely from the effective features.

[0064] In some other embodiments of this application, the first intermediate layer may not be the last intermediate layer of at least two intermediate layers. In that case, step 220 may include steps 2205 to 2206.

[0065] Step 2205: If the first intermediate layer is not the last intermediate layer of at least two intermediate layers, the invalid features output by the first intermediate layer are removed, and the valid features output by the first intermediate layer are propagated to the second intermediate layer, so that the second intermediate layer and the feature decoupling network set after the second intermediate layer can determine the valid features output by the second intermediate layer. The second intermediate layer is the next intermediate layer after the first intermediate layer among at least two intermediate layers.

[0066] In this step, it is assumed that the first intermediate layer is not the last intermediate layer, and a feature decoupling network is set after the first intermediate layer. Based on this, the first intermediate layer 1 generates original intermediate features during the model feature propagation process. Its corresponding feature decoupling network, combined with the layout mask generated by the user instruction, decouples the original intermediate features and outputs valid and invalid features. Among them, the valid features are the features that match the first layout bounding box of "cat" and "book" in the user instruction, and the invalid features are the interference features outside the layout bounding box, such as meaningless pixel features of image edges. Since the first intermediate layer 1 is not the last intermediate layer of at least two intermediate layers, all invalid features output by the first intermediate layer 1 can be removed, and only the valid features are retained. The remaining valid features after removing invalid features are propagated to the second intermediate layer, that is, the next intermediate layer after the first intermediate layer 1, namely the second intermediate layer 2. After receiving the propagated valid features, the second intermediate layer 2, combined with the supplementary features it generated during the feature propagation process, is decoupled again by the feature decoupling network set after the second intermediate layer 2, thereby determining the valid features output by the second intermediate layer 2, completing the layer-by-layer optimization and propagation of features.

[0067] Step 2206: In the case that the second intermediate layer is the last intermediate layer of at least two intermediate layers, a first image is generated based on the effective features output by the second intermediate layer.

[0068] In this step, the second intermediate layer 2 completes feature decoupling through its corresponding feature decoupling network and outputs its own valid features. It is then determined whether the second intermediate layer 2 is the last intermediate layer. If there are three intermediate layers, and intermediate layer 3 is the last, then the second intermediate layer 2 is not the last. Its output valid features continue to propagate to the subsequent intermediate layer 3, where intermediate layer 3 and its corresponding feature decoupling network perform further decoupling. As the last intermediate layer of at least two intermediate layers, intermediate layer 3's corresponding feature decoupling network completes the final feature decoupling and outputs the final valid features. Thus, based on the valid features output by intermediate layer 3, the first image is generated.

[0069] It should be noted that, based on the effective features output by intermediate layer 3, the first image can still be generated in accordance with the above-mentioned method of sending the effective features output by the first intermediate layer into the diffusion generation unit of the inlet diffusion model, which will not be repeated here.

[0070] Therefore, by propagating effective features layer by layer and eliminating invalid features, the multi-layer features are gradually optimized, ensuring that only features that meet the layout constraints participate in the final image generation, thereby further improving the stability and spatial accuracy of image generation.

[0071] It should be noted that the embodiments of this application also provide a way to update the parameters of the feature decoupling network, that is, a way to train the network. Based on this, the data method may further include steps 3101 to 3104, as shown below.

[0072] Step 3101: Based on the valid features output by each intermediate layer, determine the first deviation information of the valid features output by each intermediate layer in the second mask region of the corresponding layout mask; the first deviation information is used to characterize the feature anomaly distribution of the invalid image region of the intermediate layer.

[0073] In this step, in order for the feature decoupling network to output correctly... This application proposes an instance decoupling constraint loss function. Used for the training phase of IDM. Based on this, a CAS metric is first relied upon to evaluate the extent to which the instance generation results exceed the bounding box of a given layout, i.e., the abnormal distribution of features in the invalid image regions of the intermediate layers. In this application, the first deviation information can be calculated by the following formula (4): in, The layout mask is calculated for the bounding boxes {bi | 1 ≤ i ≤ n} of all given instances. In the layout mask, all instances have 1s inside their bounding boxes and 0s outside. i is one of the n instances given by the user. The response value of the effective features output by the instance in the intermediate layer. For example, taking a diffusion model with two intermediate layers, namely intermediate layer 1 and intermediate layer 2, as an example, the effective features output by intermediate layer 1... The response value can be expressed as Effective features output by intermediate layer 2 The response value can be expressed as The larger the value, the greater the response of the diffusion model to the instance, and the better the generation. The CAS metric evaluates the generation quality by assessing whether the diffusion model incorrectly generates instances outside the bounding box of a given layout in terms of spatial location. A higher value indicates a greater likelihood of errors in the spatial location of the generated instance. Based on this, effective features... Should trigger invalid features A smaller CAS value is needed to decouple the features.

[0074] In this way, the characteristic anomalies of the second mask region of each intermediate layer can be calculated by formula (4), and the degree to which instance errors are generated outside the layout can be quantified.

[0075] Step 3102: Based on the feature fidelity constraint term within the layout, determine the second deviation information of the effective features output by each intermediate layer in the first mask region of the corresponding layout mask. The second deviation information is used to characterize the degree of feature deviation in the effective image region of the intermediate layer.

[0076] In this step, the feature fidelity constraint term within the layout can be λ*sum( +( zi Mi ), where λ is the preset balance coefficient, zi is the original intermediate layer feature, Mi is the layout mask, and the first mask region has a value of 1. Based on this, taking intermediate layer 1 as an example, only the features within the first mask region (i.e., the layout bounding box) of the original feature z1 are retained, and the absolute deviation between the effective features and the original features within the layout is calculated pixel by pixel. The second deviation information is calculated according to the aforementioned constraint terms. Repeat the above operation for each intermediate layer to obtain the second deviation information for each intermediate layer.

[0077] In this way, feature deviation within the bounding box, i.e., the first mask region, can be calculated through the in-layout fidelity constraint term, thus ensuring feature fidelity within the layout.

[0078] Step 3103: Determine the penalty layout loss value based on the first deviation information of each intermediate layer; and determine the feature deviation value within the layout based on the second deviation information of each intermediate layer.

[0079] In this step, the first deviation information of each intermediate layer can be summarized to obtain the summarized first deviation information, and the summarized first deviation information is determined as the penalty layout loss value. Also, the second deviation information of each intermediate layer can be summarized to obtain the summarized second deviation information, and the summarized second deviation information is determined as the layout feature deviation value.

[0080] Step 3104: Based on the penalized layout loss value and the feature bias value within the layout, update the parameters of the feature decoupling network set after each intermediate layer.

[0081] In this step, the total loss of IDM includes the penalty layout loss value and the in-layout feature deviation value. Based on this, the loss function value can be calculated using the instance decoupling constraint loss function shown in the following formula (5). : Wherein, SoftPlus = ln(1 + exp(·)) is a monoincreasing function used for the smooth training process. The instance decoupling constraint loss function can be deployed after all IDMs and combined with formula (1) to assist the training of the entire IDM. Heuristically, the model learns how each instance should be correctly rendered to the corresponding position. This improves the quality of the generated image in the subsequent inference stage. Based on this, the loss function value can be used to determine the quality of the generated image. The parameters of the feature decoupling network can be updated along the negative gradient direction using gradient descent. This process is repeated iteratively until the preset number of iterations or the output of the IDM is met. The accuracy of filtering effective features is greater than or equal to a preset threshold, thereby completing the parameter update of IDM.

[0082] Therefore, by using the invalid region bias penalty and the valid region feature fidelity constraint, the feature decoupling network is accurately trained, enabling it to learn to automatically distinguish between valid and invalid features, thereby further reducing the generation of out-of-layout errors and improving instance rendering accuracy and image quality.

[0083] Based on steps 210 and 220 above, the feature decoupling network can improve the generation quality and accuracy of the diffusion model. However, the entire model training still relies on a large number of image and user instruction pairs. The collection of such data is time-consuming and labor-intensive. Therefore, this application further proposes a reinforcement learning-based training method to reduce the model training's dependence on paired data and achieve the goal of training solely based on user instruction pairs. Specifically, for a given user instruction y, the diffusion model first infers the generation of the first image x0 according to formula (2), and uses VLM to give a reward information A to the generated first image, thereby using (y, A, x0) to fine-tune the current diffusion model. Based on this, step 230 can be executed.

[0084] In some embodiments of this application, before performing step 230, the model training method provided in this application may further include steps 3201 to 3203 to determine the reward information of the first image.

[0085] Step 3201: Input the first image and user instructions into the preset detection model to obtain the second layout bounding box of the object specified by the first text output by the preset detection model and the confidence value of the object. The confidence value is used to characterize the degree of distortion of the object.

[0086] In this step, the first image generated in step 120, such as an image containing "red cup" generated based on user instructions, and the user instructions, such as the first text "red cup," can be input into a preset detection model, namely the GDINO detector. The first layout bounding box Bgt: (50,60) to (150,200) is input. The GDINO detector performs instance detection on the first image and outputs two results: first, the second layout bounding box Bpred, for example, the example output is (55,65) to (145,195), which is the bounding box of the actual generation position of "red cup" detected by the model. Confidence value G(i, The example output is 0.92. This value represents the degree of distortion of the "red cup" instance. The higher the value, the less distortion the instance has. For example, a confidence value of 0.5 indicates that the cup outline is blurred and the shape is distorted.

[0087] Step 2302: Generate a comprehensive reward score for the object based on the intersection-union ratio (IUR) information of the first and second layout bounding boxes and the confidence value of the object. The IUR information is used to characterize the accuracy of the object's spatial location.

[0088] In this step, the crossover-union ratio information can be calculated using the following formula (6): Based on this, in this example, Bgt=(50,60)-(150,200) and Bpred=(55,65)-(145,195), the calculated IoU value is 0.88. The closer the value is to 1, the more accurate the spatial location.

[0089] Based on this, the comprehensive reward score of the object can be calculated using the following formula (7): For example, the overall reward score = 0.88 (IoU) + 0.92 (confidence value) = 1.8. The higher the value, the more accurate and distortion-free the spatial location of the object is, and the higher the generation quality.

[0090] Step 2303: Determine the reward information for the first image based on the comprehensive reward score of the object.

[0091] Therefore, by simultaneously acquiring predicted bounding boxes and confidence scores through the detection model, a comprehensive reward score is constructed that integrates cross-union information and instance fidelity (i.e., confidence score). This enables automated and accurate evaluation of the quality of generated images, providing reliable supervision signals for model training. Based on this, the feature decoupling network can heuristically search for and eliminate useless intermediate features. Through reinforcement learning, it utilizes an externally pre-trained detection model as the reward function, enabling model training solely through text instructions. This significantly reduces the dependence of the diffusion model training on paired data and improves the training efficiency of the diffusion model in the task of generating images based on layout.

[0092] It should be noted that in some embodiments, the number of objects is at least two. Based on this, step 2303 above may specifically include steps 2304 and 2305.

[0093] Step 2304: Summarize the comprehensive reward scores for each of the at least two objects to obtain the summarized comprehensive reward score.

[0094] In this step, the user command generates a scene with two objects, such as a red cup and a black laptop. Steps 3201 and 3202 have already calculated the comprehensive reward scores for the two objects. For example, the red cup has an IoU of 0.88 and a confidence score of 0.92. The comprehensive reward score A1 is 0.88 + 0.92 = 1.8. The black laptop, detected by GDINO, has a second layout bounding box Bpred ranging from (200, 60) to (350, 200), meaning the user provides a first layout bounding box Bgt ranging from (195, 55) to (355, 205). The calculated IoU is 0.90, the confidence score is 0.85, and the comprehensive reward score A2 is 0.90 + 0.85 = 1.75.

[0095] Based on this, in this embodiment, the comprehensive reward scores of the two objects can be summarized by summation, or by averaging, weighted summation, etc., depending on actual needs. This example uses summation as an illustration, and the total comprehensive reward score = A1 + A2 = 1.8 + 1.75 = 3.55. Alternatively, the average can be used as an example, and the total comprehensive reward score = (1.8 + 1.75) / 2 = 1.775. The choice can be made according to the model training requirements.

[0096] Step 2305: The overall reward score is summarized and determined as the reward information for the first image.

[0097] Therefore, by deploying a separate GDINO algorithm to score the final generated image of the diffusion model, the model is encouraged to optimize in a more correct direction based on the score, thereby improving the performance of the diffusion model. This process eliminates the dependence on image training data, requiring only text instructions and the bounding boxes of corresponding instances, thus greatly reducing the reliance on training data. Moreover, summarizing the reward scores for multiple objects allows for the overall quantification of the multi-object generation quality, enabling the model to learn the layout and generation effects of multiple instances simultaneously, making it suitable for layout-controlled generation in complex scenes.

[0098] In some embodiments of this application, the second image includes a noisy image at each of T time steps, where T is an integer greater than or equal to 2; before performing step 230, the model training method provided in the embodiments of this application may further include steps 3301 to 3303 to determine sampling weight information.

[0099] Step 3301: Using Gaussian noise data and the first image, synthesize a noisy image for each of the T time steps in reverse.

[0100] In this step, for each time step T, such as T=10, T=25, T=50, the Gaussian noise is fused with the first image x0 through the reverse process of the diffusion model, for example, by referring to formula (2), to generate a noisy image for each time step.

[0101] Step 3302: Input the noisy image and user command at each time step into the first diffusion model and the third diffusion model respectively to obtain the first model forward inference result output by the first diffusion model and the second model forward inference result output by the third diffusion model.

[0102] In this step, the noisy image and user commands at each time step are input into the first diffusion model, and the forward inference result of the first model output by the first diffusion model can be obtained. Furthermore, by inputting the noisy image and user commands at each time step into the third diffusion model, the forward inference result of the third model output can be obtained. .

[0103] Step 3303: The ratio of the forward inference result of the first model to the forward inference result of the second model is determined as the sampling weight information.

[0104] In this step, the sampling weight information can be determined using the following formula (8). : in, This refers to the sampling weight information at time step t, representing the degree of change in the output of the current version of the first diffusion model relative to the output of the historical version of the third diffusion model at this time point. If the current version of the first diffusion model can obtain a better reward than the historical version of the third diffusion model in generating the first image, then the corresponding loss... It will decrease, thus affirming and encouraging the model's prediction at this moment, thereby improving the model's capabilities; conversely, it will punish and suppress the output of the current version of the first model.

[0105] Therefore, sampling weights are obtained by comparing the inference results of noisy images on the old and new models, which can stably measure the magnitude of model updates, provide a reliable weight basis for reinforcement learning training, and avoid oscillations in the training process.

[0106] Based on this, step 230 is involved. In some embodiments, step 230 may specifically include steps 2301 to 2304.

[0107] Step 2301: Determine the first reward weight value based on the sampling weight information and the reward information of the first image. The first reward weight value is used to characterize the degree of matching between the reward information and the sampling weight information.

[0108] In this step, the first image and user command obtained in step 220 are input into a preset detection model. The detection model outputs the second bounding boxes of "cat" and "book," which are the detected actual bounding boxes and their respective confidence values. The intersection-union ratio (IUR) of the second bounding box of each object with the corresponding first bounding box is calculated. Combined with the confidence value, a comprehensive reward score for each object is obtained. The summation yields the reward information for the first image, for example, 8.6 points. The higher the score, the higher the matching degree of the layout constraints between the first image and the user command. Furthermore, the first image is synthesized by reverse engineering with Gaussian noise data for T time steps, for example, 50 time steps, resulting in a noisy image, i.e., the second image. There can be multiple second images, forming a noisy image sequence. The degree of noise varies at each time step. Based on this, the noisy image and user command at each time step are input into the first diffusion model and the third diffusion model, respectively, to obtain the forward inference results of the two models. The ratio of the two results is determined as the sampling weight information, used to characterize the degree of change in the output of the first diffusion model relative to the historical model. Then, the product of the reward information value of the first image and the sampling weight information value of the second image is determined as the first reward weight value. .

[0109] Step 2302: Crop the sampling weight information of the second image to obtain the second reward weight value. The second reward weight value is within a preset stable range. The preset stable range is […]. ], ( ).

[0110] In this step, the sampling weight information is pruned to a preset stable range, such as [1-1×10]. , 1+1×10 ], thus obtaining the second reward weight value.

[0111] Step 2303: Take the maximum value between the first reward weight value and the second reward weight value as the policy gradient reward value.

[0112] Step 2304: Use the policy gradient reward value and KL divergence value as the loss function value to train the first diffusion model and obtain the second diffusion model; the KL divergence value is the difference in the output probability distribution between the first diffusion model and the third diffusion model under the same input conditions.

[0113] In this step, the reward value can be based on the policy gradient, i.e., max[ Combining the KL divergence value, i.e., KL( || The loss function value is determined by the following formula (9). : Based on this, the parameters of the first diffusion model can be updated and trained using the calculated loss function value. After training, a second diffusion model is obtained, which can generate images more accurately according to the layout constraints of the user's instructions.

[0114] In this way, the process eliminates the need for manual labeling of any paired training data; model training can be completed solely through user commands, effectively reducing training costs while improving the layout accuracy of generated images.

[0115] It should be noted that since this training process relies entirely on the images inferred by the model itself and the scores given by the reward function, there may be training instability. Therefore, in order to maintain training stability, formula (9) adds an additional KL divergence term and a clip term. The former measures the degree of change of the current model compared to the historical model, preventing training collapse due to large gradient updates during optimization, while the latter controls... The absolute change is limited to a preset stable range. In order to further improve training stability, when training the model, in order to calculate the reinforcement learning loss function, it is necessary to first store the historical version of the model, namely the third diffusion model, and then use the user input to perform an inference on the current model, namely the first diffusion model, to supplement the missing image parts. Then, the reward function is used to evaluate the generated image, and finally the loss function mentioned in formula (8) and formula (9) is calculated to optimize the current model.

[0116] Therefore, by pruning weights and combining policy gradients with KL divergence constraints, the model is guided towards higher quality and better fit of layout constraints while ensuring training stability, thus improving the performance of the final generated model. It is evident that the reinforcement learning part relies solely on user input and the reward function to fine-tune the model's noise and image mapping trajectory online. By changing the generation trajectory, it generates images that better match user instructions, pleasing the reward model and awarding higher reward scores, thereby enhancing its generation capabilities. This process does not require external pre-collection of images and user instruction pairs, thus significantly reducing the training difficulty of the diffusion model.

[0117] The model training method provided in this application can be executed by a model training device. This application uses the example of a model training device performing a display to illustrate the apparatus of the model training method provided in this application.

[0118] This application also provides a model training device. (Specifically combined with...) Figure 3 Please provide a detailed explanation.

[0119] Figure 3 This is a schematic diagram of the structure of a model training device provided for some embodiments of this application.

[0120] like Figure 3 As shown, the model training device 30 can be applied to electronic devices, and the model training device 30 may specifically include: The acquisition module 301 is used to acquire user instructions, which include first text and a first layout bounding box. The first text is used to specify the category of the object to be generated, and the first layout bounding box is used to constrain the spatial position and size of the object. Processing module 302 is used to perform feature decoupling processing on intermediate features generated during model propagation layer by layer through the feature decoupling network of the first diffusion model based on user instructions, so as to obtain the first image; The training module 303 is used to train a first diffusion model based on the sampling weight information and the reward information of the first image to obtain a second diffusion model. The reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting the user command and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image.

[0121] The model training device 30 in the embodiments of this application will be described in detail below.

[0122] In some embodiments of this application, the model training device 30 may further include a labeling module, used to label the spatial region generated by the object based on user instructions to obtain a layout mask when the number of intermediate layers of the feature decoupling network is at least two, and a feature decoupling network is set after each of the at least two intermediate layers; the layout mask includes a first mask region and a second mask region, the first mask region is used to label the valid image region surrounded by the first layout bounding box, and the second mask region is used to label the invalid image region outside the first layout bounding box; The processing module 302 is used to obtain feature weighting information based on the original intermediate features and layout mask generated by the first intermediate layer in the model propagation process of at least two intermediate layers, through a feature decoupling network set after the first intermediate layer. The feature weighting information is used to characterize the probability that each pixel feature of the generated image is retained. Processing module 302 is used to perform Hadamard product processing on the feature weighted information and the original intermediate features of the first intermediate layer to obtain the effective features and invalid features output by the first intermediate layer. The effective features are those in the original intermediate features that match the spatial position and size of the object constrained by the user instruction and are used in the subsequent image generation process. The invalid features are those in the original intermediate features that do not match the spatial position and size of the object constrained by the user instruction and are not used in the subsequent image generation process. The model training device 30 may further include a generation module for generating a first image based on the valid features output by the first intermediate layer, provided that the first intermediate layer is the last of at least two intermediate layers.

[0123] In some embodiments of this application, the processing module 302 is configured to, when the first intermediate layer is not the last intermediate layer of at least two intermediate layers, remove invalid features output by the first intermediate layer and propagate valid features output by the first intermediate layer to the second intermediate layer, so that the second intermediate layer and the feature decoupling network set after the second intermediate layer determine the valid features output by the second intermediate layer, wherein the second intermediate layer is the next intermediate layer after the first intermediate layer among at least two intermediate layers. The model training device 30 may further include a generation module for generating a first image based on the effective features output by the second intermediate layer, provided that the second intermediate layer is the last of at least two intermediate layers.

[0124] In some embodiments of this application, the processing module 302 is used to input the first image and user instructions into a preset detection model to obtain the second layout bounding box of the object and the confidence value of the object specified by the first text output by the preset detection model. The confidence value is used to characterize the degree of distortion of the object. The model training device 30 may also include a generation module for generating a comprehensive reward score for the object based on the intersection-union information of the first and second layout bounding boxes and the confidence value of the object. The intersection-union information is used to characterize the accuracy of the object's spatial location. The model training device 30 may also include a determination module for determining reward information of the first image based on the object-based comprehensive reward score.

[0125] In some embodiments of this application, the model training device 30 may further include a synthesis module for inversely synthesizing a noisy image at each of T time steps using Gaussian noise data and a first image, where T is an integer greater than or equal to 2. Processing module 302 is used to input the noisy image and user command at each time step into the first diffusion model and the third diffusion model respectively, and obtain the first model forward inference result output by the first diffusion model and the second model forward inference result output by the third diffusion model. The model training device 30 may also include a determination module for determining the ratio of the forward inference result of the first model to the forward inference result of the second model as sampling weight information.

[0126] In some embodiments of this application, the model training device 30 may further include a determination module, which is used to determine a first reward weight value based on the sampling weight information and the reward information of the first image. The first reward weight value is used to characterize the degree of matching between the reward information and the sampling weight information. The processing module 302 is used to perform cropping processing on the sampling weight information of the second image to obtain a second reward weight value, wherein the second reward weight value is within a preset stable range; The determination module is also used to take the maximum value between the first reward weight value and the second reward weight value as the policy gradient reward value; The processing module 302 is used to train the first diffusion model and obtain the second diffusion model by using the policy gradient reward value and KL divergence value as loss function values; the KL divergence value is the difference in the output probability distribution of the first diffusion model and the third diffusion model under the same input conditions.

[0127] The model training device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0128] The model training device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0129] The model training device provided in this application embodiment can achieve... Figure 2 The various processes implemented in the model training method examples shown achieve the same technical effect, and will not be described again here to avoid repetition.

[0130] Based on this, the model training apparatus provided in this application obtains user instructions, which include first text and a first layout bounding box. The first text is used to specify the category of the object to be generated, and the first layout bounding box is used to constrain the spatial position and size of the object. Based on the user instructions, the intermediate features generated during the model propagation process are decoupled layer by layer through the feature decoupling network of the first diffusion model to obtain a first image. The first diffusion model is trained according to the sampling weight information and the reward information of the first image to obtain a second diffusion model. The reward information is used to characterize the matching degree between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting the user instructions and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image. In this way, user commands including the first text and the first layout bounding box, combined with the intermediate feature decoupling network of the first diffusion model, can decouple intermediate features generated during model propagation layer by layer. This enables constraint learning of object spatial position and size within the model, eliminating the need for manual annotation, layout bounding box annotation, and command organization for each object. Reward information is automatically calculated based on the generated first image, allowing for automated evaluation of the matching degree between the generated result and layout constraints. Supervision signals are no longer dependent on manually labeled data. Model training combines sampling weight information and reward information from noisy images, effectively guiding the diffusion model to learn correct object generation and spatial layout rules even without finely labeled data. Therefore, there is no need to reuse or modify existing public datasets, saving the tedious and time-consuming manual annotation process, significantly reducing the human and time costs of training data preparation, and overall reducing the training cost of layout-controlled diffusion models.

[0131] Optional, such as Figure 4 As shown, this application embodiment also provides an electronic device 40, including a processor 401 and a memory 402. The memory 402 stores a program or instructions that can run on the processor 401. When the program or instructions are executed by the processor 401, they implement the various steps of the above-described model training method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0132] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0133] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided for some embodiments of this application.

[0134] The electronic device 500 includes, but is not limited to, components such as: radio frequency unit 501, network module 502, audio output unit 503, input unit 504, sensor 505, display unit 506, user input unit 507, interface unit 508, memory 509, and processor 510.

[0135] Those skilled in the art will understand that the electronic device 500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0136] In this embodiment, the processor 510 is used to acquire user instructions, which include first text and a first layout bounding box. The first text specifies the category of the object to be generated, and the first layout bounding box constrains the spatial position and size of the object. The processor 510 can also be used to, based on the user instructions, perform feature decoupling processing on the intermediate features generated during model propagation layer by layer through the feature decoupling network of the first diffusion model to obtain a first image. The processor 510 can also be used to train the first diffusion model according to sampling weight information and reward information of the first image to obtain a second diffusion model. The reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting the user instructions and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before training the first diffusion model or the diffusion model obtained in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image.

[0137] Therefore, by using user commands including the first text and the first layout bounding box, combined with the intermediate feature decoupling network of the first diffusion model, the intermediate features generated during model propagation can be decoupled layer by layer. This enables constraint learning of object spatial position and size within the model, eliminating the need for manual annotation, layout bounding box annotation, and command organization for each object. Reward information is automatically calculated based on the generated first image, enabling automated evaluation of the matching degree between the generated result and the layout constraints. Supervision signals are no longer dependent on manually labeled data. Model training combines sampling weight information and reward information from noisy images, effectively guiding the diffusion model to learn correct object generation and spatial layout rules even without finely labeled data. Thus, there is no need to reuse or modify existing public datasets, saving the tedious and time-consuming manual annotation process, significantly reducing the human and time costs of training data preparation, and overall reducing the training cost of layout-controlled diffusion models.

[0138] The electronic device 500 is described in detail below.

[0139] In some embodiments of this application, the processor 510 is configured to mark the spatial region of an object generated based on user instructions to obtain a layout mask when the number of intermediate layers of the feature decoupling network is at least two, and a feature decoupling network is provided after each of the at least two intermediate layers; the layout mask includes a first mask region and a second mask region, the first mask region is used to mark the valid image region surrounded by the first layout bounding box, and the second mask region is used to mark the invalid image region outside the first layout bounding box; Based on the original intermediate features and layout mask generated by the first intermediate layer during model propagation in at least two intermediate layers, feature weighting information is obtained by setting a feature decoupling network after the first intermediate layer. The feature weighting information is used to characterize the probability that each pixel feature of the generated image is retained. The weighted feature information and the original intermediate features of the first intermediate layer are processed by Hadamard product to obtain the effective features and invalid features output by the first intermediate layer. The effective features are those in the original intermediate features that match the spatial position and size of the object constrained by the user instruction and are used in the subsequent image generation process. The invalid features are those in the original intermediate features that do not match the spatial position and size of the object constrained by the user instruction and are not involved in the subsequent image generation process. In the case where the first intermediate layer is the last of at least two intermediate layers, a first image is generated based on the valid features output by the first intermediate layer.

[0140] In some embodiments of this application, processor 510 is configured to, when the first intermediate layer is not the last intermediate layer of at least two intermediate layers, remove invalid features output by the first intermediate layer and propagate valid features output by the first intermediate layer to the second intermediate layer, so that the second intermediate layer and the feature decoupling network set after the second intermediate layer determine the valid features output by the second intermediate layer, wherein the second intermediate layer is the next intermediate layer after the first intermediate layer among at least two intermediate layers. In the case where the second intermediate layer is the last of at least two intermediate layers, a first image is generated based on the effective features output by the second intermediate layer.

[0141] In some embodiments of this application, the processor 510 is configured to input a first image and a user instruction into a preset detection model to obtain a second layout bounding box of an object and a confidence value of the object specified by the first text output by the preset detection model. The confidence value is used to characterize the degree of distortion of the object. Based on the intersection-union ratio (IUR) information of the first and second layout bounding boxes and the confidence value of the object, a comprehensive reward score for the object is generated. The IUR information is used to characterize the accuracy of the object's spatial location. Based on the comprehensive reward score of the object, the reward information of the first image is determined.

[0142] In some embodiments of this application, processor 510 is configured to inversely synthesize a noisy image at each of T time steps using Gaussian noise data and a first image, where T is an integer greater than or equal to 2. The noisy image and user command at each time step are input into the first diffusion model and the third diffusion model respectively, to obtain the first model forward inference result output by the first diffusion model and the second model forward inference result output by the third diffusion model. The ratio of the forward inference result of the first model to the forward inference result of the second model is determined as the sampling weight information.

[0143] In some embodiments of this application, the processor 510 is configured to determine a first reward weight value based on the sampling weight information and the reward information of the first image, wherein the first reward weight value is used to characterize the degree of matching between the reward information and the sampling weight information; The sampling weight information of the second image is cropped to obtain the second reward weight value, which is within a preset stable range; The maximum value between the first reward weight value and the second reward weight value is used as the policy gradient reward value; The policy gradient reward value and KL divergence value are used as loss function values ​​to train the first diffusion model and obtain the second diffusion model; the KL divergence value is the difference in the output probability distribution between the first diffusion model and the third diffusion model under the same input conditions.

[0144] It should be understood that the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image information of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 506 may include a display panel, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include a touch detection device and a touch display. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as volume display buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0145] The memory 509 can be used to store software programs and various information. The memory 509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing information. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 509 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0146] Processor 510 may include one or more processing units; optionally, processor 510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless display signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 510.

[0147] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model training method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0148] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0149] In addition, this application embodiment provides another chip, which includes a processor and a display interface. The display interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described model training method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0150] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0151] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the model training method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0153] Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0155] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model training method, characterized in that, include: Obtain user instructions, the user instructions including first text and first layout bounding box, the first text being used to specify the category of the object to be generated, and the first layout bounding box being used to constrain the spatial position and size of the object; Based on the user instructions, the intermediate features generated during the model propagation process are decoupled layer by layer through the feature decoupling network of the first diffusion model to obtain the first image. The first diffusion model is trained based on the sampling weight information and the reward information of the first image to obtain the second diffusion model. The reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting the user command and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image.

2. The method according to claim 1, characterized in that, The feature decoupling network has at least two intermediate layers, and each of the at least two intermediate layers is followed by a feature decoupling network; the first image is obtained by performing feature decoupling processing layer by layer on the intermediate features generated during model propagation through the feature decoupling network of the first diffusion model based on the user instruction, including: Based on the user instruction, the spatial region generated by the object is marked to obtain a layout mask; the layout mask includes a first mask region and a second mask region, the first mask region is used to mark the valid image region surrounded by the first layout bounding box, and the second mask region is used to mark the invalid image region outside the first layout bounding box. Based on the original intermediate features generated by the first intermediate layer during model propagation and the layout mask, feature weighting information is obtained through a feature decoupling network set after the first intermediate layer. The feature weighting information is used to characterize the probability that each pixel feature of the generated image is retained. The weighted feature information is subjected to Hadamard product processing with the original intermediate features of the first intermediate layer to obtain the effective features and invalid features output by the first intermediate layer. The effective features are those in the original intermediate features that match the spatial position and size of the object constrained by the user instruction and are used in the subsequent image generation process. The invalid features are those in the original intermediate features that do not match the spatial position and size of the object constrained by the user instruction and are not involved in the subsequent image generation process. When the first intermediate layer is the last of the at least two intermediate layers, the first image is generated based on the valid features output by the first intermediate layer.

3. The method according to claim 2, characterized in that, The step of performing feature decoupling processing on intermediate features generated during model propagation layer by layer through the feature decoupling network of the first diffusion model based on the user instruction includes: If the first intermediate layer is not the last intermediate layer of the at least two intermediate layers, invalid features output by the first intermediate layer are removed, and valid features output by the first intermediate layer are propagated to the second intermediate layer, so that the second intermediate layer and the feature decoupling network set after the second intermediate layer determine the valid features output by the second intermediate layer, wherein the second intermediate layer is the next intermediate layer after the first intermediate layer among the at least two intermediate layers. When the second intermediate layer is the last of the at least two intermediate layers, the first image is generated based on the valid features output by the second intermediate layer.

4. The method according to claim 1, characterized in that, The method further includes: The first image and the user instruction are input into a preset detection model to obtain the second layout bounding box of the object specified by the first text and the confidence value of the object, which are output by the preset detection model. The confidence value is used to characterize the degree of distortion of the object. Based on the intersection-union ratio (IUR) information of the first and second layout bounding boxes and the confidence value of the object, a comprehensive reward score for the object is generated, wherein the IUR information is used to characterize the accuracy of the object's spatial location. The reward information for the first image is determined based on the comprehensive reward score of the object.

5. The method according to claim 1, characterized in that, The second image includes a noisy image at each of T time steps, where T is an integer greater than or equal to 2; the method further includes: By combining the Gaussian noise data with the first image, a noisy image is synthesized in reverse at each of the T time steps; The noisy image at each time step and the user command are respectively input into the first diffusion model and the third diffusion model to obtain the first model forward inference result output by the first diffusion model and the second model forward inference result output by the third diffusion model. The ratio of the forward inference result of the first model to the forward inference result of the second model is determined as the sampling weight information.

6. The method according to claim 1, characterized in that, The step of training the first diffusion model based on the sampling weight information and the reward information of the first image to obtain the second diffusion model includes: Based on the sampling weight information and the reward information of the first image, a first reward weight value is determined, which is used to characterize the degree of matching between the reward information and the sampling weight information. The sampling weight information of the second image is cropped to obtain a second reward weight value, which is within a preset stable range. The maximum value between the first reward weight value and the second reward weight value is taken as the policy gradient reward value; The policy gradient reward value and KL divergence value are used as loss function values ​​to train the first diffusion model and obtain the second diffusion model; the KL divergence value is the difference in the output probability distribution of the first diffusion model and the third diffusion model under the same input conditions.

7. A model training device, characterized in that, include: The acquisition module is used to acquire user instructions, which include first text and a first layout bounding box. The first text is used to specify the category of the object to be generated, and the first layout bounding box is used to constrain the spatial position and size of the object. The processing module is used to perform feature decoupling processing on the intermediate features generated during the model propagation process layer by layer through the feature decoupling network of the first diffusion model based on the user instruction, so as to obtain the first image; The training module is used to train the first diffusion model and obtain the second diffusion model based on the sampling weight information, the reward information of the first image, and the second image. The reward information is used to characterize the degree of matching between the second layout bounding box of the object in the first image and the first layout bounding box. The sampling weight information is determined by inputting the user command and the second image into the first diffusion model and the third diffusion model respectively, based on the degree of difference between the outputs of the first diffusion model and the third diffusion model. The third diffusion model is the initial diffusion model before the first diffusion model is trained or the diffusion model obtained by the first diffusion model in any training round before the current training round. The second image is a noisy image obtained by adding Gaussian noise data to the first image.

8. An electronic device, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the model training method as described in any one of claims 1-6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the model training method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the model training method as described in any one of claims 1-6.

Images