Training method and device of controllable generation network based on modularized self-encoding

Abstract

This invention discloses a training method and apparatus for a controllable generative network based on a modular autoencoder, belonging to the field of artificial intelligence technology. The training method includes: acquiring training data; training a convolutional network autoencoder using equivariant constraints based on the training data to obtain a trained autoencoder; constructing k adapters for k different modules based on a pre-trained latent diffusion model; and inputting an image I from the training data into the trained autoencoder to obtain the feature map f of the i-th module. (i) ; using the obtained feature map f (i) The i-th adapter of the latent diffusion model is trained with the objective optimization function; the trained autoencoder and the adapter together constitute a controllable generative network based on modular autoencoders. The self-supervised training method provided by this invention does not require manual design of control conditions or additional manual annotation, simplifying the training process of the controllable generative network while improving the richness of control conditions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method and apparatus for a controllable generative network based on modular autoencoders. Background Technology

[0002] Currently, controllable image generation technologies can be categorized into three types based on the control conditions they are based on: text-based image generation, such as DALL-E2, Stable Diffusion, and Midjoumey; image-based image generation, such as StyleGAN, CycleGAN, DeepFill, and ControlNet; and image generation based on other modal information, such as generating images from audio. Image-based controllable generation often provides better control information and generates images that meet requirements, hence its increased research in recent years. Based on the type of generative model, they are mainly divided into: Generative Adversarial Networks (GANs): generating realistic images through adversarial training between the generator and discriminator; Diffusion Models: generating high-quality images by learning the noise process of data distribution; and Variational Autoencoders (VAEs): generating images with specific features by learning the latent representation of data. Existing generative models, especially those based on diffusion models, can generate highly realistic images, even comparable to real photographs. Furthermore, they offer strong controllability, allowing the style, content, and details of the generated image to be controlled based on user-input text descriptions, reference images, or other information.

[0003] Therefore, controllable generation based on diffusion models, using images as control conditions, has received widespread attention in the past two years and has achieved good results, such as ControlNet. However, current methods often train models through supervised training, such as using semantic segmentation maps, edge maps, pose maps, and depth maps as control conditions. This requires both manually defining control conditions and a large number of labeled images as training samples. This increases the difficulty of training image-based controllable generation models and limits the availability of richer control conditions. Summary of the Invention

[0004] In order to solve the problems existing in the prior art, the present invention provides the following technical solution.

[0005] The first aspect of this invention provides a training method for a controllable generative network based on a modular autoencoder, comprising:

[0006] Acquire training data, which includes image I and image I′ obtained by transforming image I;

[0007] Based on the training data, the convolutional network autoencoder is trained using equivariant constraints to obtain a trained autoencoder; wherein, the equivariant constraint L... EC The following formula is used for calculation:

[0008]

[0009] L equ =∑ i ||f′ (i) -M (i) (δ)f (i) || 2 , i∈{0,1,...,k-1},

[0010]

[0011] L EC =L recon +λ1L equ +λ2L sym ,

[0012] f = I * W,

[0013] f′=I′*W,

[0014] Where, I′=L δ (I), where Lδ is a certain transformation with transformation parameter δ; W is the convolution kernel of the autoencoder of the convolutional network, and W = [W (0) W (1) , ..., W (k-1) ], that is, the convolution kernel is divided into k modules; f is the feature image obtained after convolving image I, and f = [f (0) f (1) , ..., f (k-1) f′ is the feature image obtained after convolving image I′, f′=[f′ (0) f′ (1) ,...,f′ (k-1) ]; * indicates a convolution operation. Indicates the deconvolution operation; f (i) f′ is the feature map obtained after convolving image I through the i-th module of the convolution kernel; (i) M is the feature map obtained after convolving image I′ through the i-th module of the convolution kernel; (i) (δ) is the prediction matrix of the i-th module with parameter δ, which is obtained by random initialization and optimization through model training; L recon To rebuild the losses, L equ For equal loss, L sym For symmetrical loss, L ECFor equivariant constraints; λ1 and λ2 are the weighting coefficients of equivariant loss and symmetric loss, respectively; l represents the number of dimensions of each module; This represents the feature map of the i-th module where only the m-th dimension is 1 and the other dimensions are 0. This represents the feature map of the i-th module where only the n-th dimension is 1 and the other dimensions are 0. From arrive The optimal transformation parameters; Let the parameter be the prediction matrix M. (i) The value of (δ) in the m-th row and n-th column; The prediction matrix M with parameter δ′ (i) The value of (δ′) in the m-th row and n-th column; T is a hyperparameter greater than 0; For the transformation parameters are The prediction matrix at time; δ′ is the integration variable, used to distinguish the transformation parameter δ under different integration terms;

[0015] Construct k adapters for k different modules based on a pre-trained latent diffusion model;

[0016] The image I from the training data is input into the trained autoencoder to obtain the feature map f of the i-th module. (i) ,i=0,1,2,...,k-1;

[0017] Using the obtained feature map f (i) Train the i-th adapter of the latent diffusion model using the objective optimization function shown below:

[0018]

[0019] Where z0 is the latent representation of the latent diffusion model for the input image I, z t The noisy latent characterization at time step t, c txt This is a text prompt, where ∈ represents the actual added random noise with a distribution of N(0,1), where N(0,1) is a normal distribution with a mean of 0 and a variance of 1. The noise is predicted by optimizing the parameter θ. Represents the noise in the prediction The expected value of the square of the modulus of the deviation from the actual added random noise ∈, L mc (i) For the feature map f of the i-th module (i) The loss function of the adapter;

[0020] The trained autoencoder and adapter together form a controllable generative network based on modular autoencoders.

[0021] Preferably, the transformation includes a translation transformation or a translation-rotation transformation.

[0022] The second aspect of the present invention provides a controllable generative network based on modular autoencoders, which is trained using the method described in the first aspect.

[0023] A third aspect of the present invention provides a training apparatus for a controllable generative network based on a modular autoencoder, comprising:

[0024] The training data acquisition module is used to acquire training data, which includes image I and image I′ obtained by transforming image I;

[0025] The autoencoder training module is used to train the convolutional network autoencoder based on the training data using equivariant constraints, thereby obtaining a trained autoencoder; wherein, the equivariant constraint L... EC The following formula is used for calculation:

[0026]

[0027] L equ =∑ i ||f′ (i) -M (i) (δ)f (i) || 2 , i∈{0,1,...,k-1},

[0028]

[0029]

[0030] L EC =L recon +λ1L equ +λ2L sym ,

[0031] f = I * W,

[0032] f′=I′*W,

[0033] Where, I′=L δ (I), where Lδ is a certain transformation with transformation parameter δ; W is the convolution kernel of the autoencoder of the convolutional network, and W = [W (0) W (1) , ..., W (k-1) ], that is, the convolution kernel is divided into k modules; f is the feature image obtained after convolving image I, and f = [f (0) f (1) , ..., f (k-1) f′ is the feature image obtained after convolving image I′, f′=[f′ (0)f′ (1) ,...,f′ (k-1) ]; * indicates a convolution operation. Indicates the deconvolution operation; f (i) f′ is the feature map obtained after convolving image I through the i-th module of the convolution kernel; (i) M is the feature map obtained after convolving image I′ through the i-th module of the convolution kernel; (i) (δ) is the prediction matrix of the i-th module with parameter δ, which is obtained by random initialization and optimization through model training; L recon To rebuild the losses, L equ For equal loss, L sym For symmetrical loss, L EC For equivariant constraints; λ1 and λ2 are the weighting coefficients of equivariant loss and symmetric loss, respectively; l represents the number of dimensions of each module; This represents the feature map of the i-th module where only the m-th dimension is 1 and the other dimensions are 0. This represents the feature map of the i-th module where only the n-th dimension is 1 and the other dimensions are 0. From arrive The optimal transformation parameters; Let the parameter be the prediction matrix M. (i) The value of (δ) in the m-th row and n-th column; The prediction matrix M with parameter δ′ (i) The value of (δ′) in the m-th row and n-th column; T is a hyperparameter greater than 0; For the transformation parameters are The prediction matrix at time; δ′ is the integration variable, used to distinguish the transformation parameter δ under different integration terms;

[0034] The adapter building module is used to build k adapters for k different modules based on a pre-trained latent diffusion model;

[0035] The adapter training module is used to input the image I from the training data into the trained autoencoder to obtain the feature map f of the i-th module. (i) , i = 0, 1, 2, ..., k-1; using the obtained feature map f (i) Train the i-th adapter of the latent diffusion model using the objective optimization function shown below:

[0036]

[0037] Where z0 is the latent representation of the latent diffusion model for the input image I, z t The noisy latent characterization at time step t, c txtThis is a text prompt, where ∈ represents the actual added random noise with a distribution of N(0,1), where N(0,1) is a normal distribution with a mean of 0 and a variance of 1. The noise is predicted by optimizing the parameter θ. Represents the noise in the prediction The expected value of the square of the modulus of the deviation from the actual added random noise ∈, L mc (i) For the feature map f of the i-th module (i) The loss function of the adapter;

[0038] The controllable generative network building module is used to combine the trained autoencoder and adapter to form a controllable generative network based on modular autoencoders.

[0039] Preferably, the transformation includes a translation transformation or a translation-rotation transformation.

[0040] A fourth aspect of the present invention provides a memory that stores a plurality of instructions for implementing the training method of the controllable generative network based on modular autoencoders as described in the first aspect.

[0041] The fifth aspect of the present invention provides an electronic device, including a processor and a memory connected to the processor, the memory storing a plurality of instructions which can be loaded and executed by the processor to enable the processor to perform a training method for a controllable generative network based on modular autoencoders as described in the first aspect.

[0042] The beneficial effects of this invention are as follows: The training method and apparatus for a controllable generative network based on a modular autoencoder provided by this invention first constructs equivariant constraints and uses these constraints to train the modular network autoencoder on training data of homologous image pairs, obtaining a trained autoencoder. This enhances the correlation within the autoencoder modules and the independence between modules, promoting the specialization of network module functions. Then, the automatically differentiated modular features are used as control conditions for controllable generation to train the controllable generative network. This self-supervised training method does not require manual design of control conditions or additional manual annotation, simplifying the training process of the controllable generative network while improving the richness of control conditions. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the training method for the controllable generative network based on modular autoencoder described in this invention.

[0044] Figure 2 This is a schematic diagram of the training process of a modular autoencoder.

[0045] Figure 3 This is a schematic diagram of the training process of a controllable generative network based on modular autoencoders.

[0046] Figure 4 This is a schematic diagram illustrating the functional differentiation effect under modular and equivariant constraints.

[0047] Figure 5 The images are generated by the sketch-based supervised method (ControlNet) and the self-supervised controllable generation (SCG) method described in this invention.

[0048] Figure 6 The images are generated by the supervised method and self-supervised controllable generation method based on oil painting described in this invention.

[0049] Figure 7 , Figure 8 All images are generated by the self-supervised and controllable generation method based on ink painting described in this invention;

[0050] Figure 9 The images are generated by the supervised method and self-supervised controllable generation method based on murals as described in this invention;

[0051] Figure 10 This is a schematic diagram of the functional structure of the training device for the controllable generative network based on modular autoencoder described in this invention. Detailed Implementation

[0052] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0053] The method provided by this invention can be implemented in a terminal environment that may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0054] A processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the terminal, and performs various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in memory, and by calling data stored in memory.

[0055] Memory can include random access memory (RAM) or read-only memory (ROM). Memory can be used to store instructions, programs, code, code sets, or instructions.

[0056] The display screen is used to show the user interface of each application.

[0057] In addition, those skilled in the art will understand that the structure of the terminal described above does not constitute a limitation on the terminal. The terminal may include more or fewer components, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, input units, sensors, audio circuits, power supplies, and other components, which will not be described in detail here.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment of the invention provides a training method for a controllable generative network based on modular autoencoders, comprising:

[0060] S101, Obtain training data, the training data including image I and image I′ obtained by transforming image I;

[0061] S102, Based on the training data, the convolutional network autoencoder is trained using equivariant constraints to obtain a trained autoencoder; wherein, the equivariant constraint L EC The following formula is used for calculation:

[0062]

[0063] L equ =∑ i ||f′ (i) -M (i) (δ)f (i) || 2 , i∈{0,1,...,k-1},

[0064]

[0065] L EC =L recon +λ1L equ +λ2L sym ,

[0066] f = I * W,

[0067] f′=I′*W,

[0068] Where, I′=L δ (I), where Lδ is a certain transformation with transformation parameter δ; W is the convolution kernel of the autoencoder of the convolutional network, and W = [W (0) W (1) , ..., W (k-1) ], that is, the convolution kernel is divided into k modules; f is the feature image obtained after convolving image I, and f = [f (0) f (1) , ..., f (k-1)f′ is the feature image obtained after convolving image I′, f′=[f′ (0) f′ (1) , ..., f, (k-1) ]; * indicates a convolution operation. Indicates the deconvolution operation; f (i) f′ is the feature map obtained after convolving image I through the i-th module of the convolution kernel; (i) M is the feature map obtained after convolving image I′ through the i-th module of the convolution kernel; (i) (δ) is the prediction matrix of the i-th module with parameter δ, which is obtained by random initialization and optimization through model training; L recon To rebuild the losses, L equ For equal loss, L sym For symmetrical loss, L EC For equivariant constraints; λ1 and λ2 are the weighting coefficients of equivariant loss and symmetric loss, respectively; l represents the number of dimensions of each module; This represents the feature map of the i-th module where only the m-th dimension is 1 and the other dimensions are 0. This represents the feature map of the i-th module where only the n-th dimension is 1 and the other dimensions are 0. From arrive The optimal transformation parameters; Let the parameter be the prediction matrix M. (i) The value of (δ) in the m-th row and n-th column; The prediction matrix M with parameter δ′ (i) The value of (δ′) in the m-th row and n-th column; T is a hyperparameter greater than 0; For the transformation parameters are The prediction matrix at time; δ′ is the integration variable, used to distinguish the transformation parameter δ under different integration terms;

[0069] S103, construct k adapters for k different modules based on the pre-trained latent diffusion model;

[0070] S104, Input the image I from the training data into the trained autoencoder to obtain the feature map f of the i-th module. (i) i = 0, 1, 2, ..., k-1;

[0071] S105, using the obtained feature map f (i) Train the i-th adapter of the latent diffusion model using the objective optimization function shown below:

[0072]

[0073] Where z0 is the latent representation of the latent diffusion model for the input image I, zt The noisy latent characterization at time step t, c txt This is a text prompt, where ∈ represents the actual added random noise with a distribution of N(0,1), where N(0,1) is a normal distribution with a mean of 0 and a variance of 1. The noise is predicted by optimizing the parameter θ. Represents the noise in the prediction The expected value of the square of the modulus of the deviation from the actual added random noise ∈, L mc (i) For the feature map f of the i-th module (i) The loss function of the adapter;

[0074] S106 combines the trained autoencoder and adapter to form a controllable generative network based on modular autoencoders.

[0075] It should be noted that, The subscript "m" in the middle The subscript "m" in the matrix, although representing the dimension of the feature map and the row number of the matrix respectively, is used because the feature map and the matrix correspond. The subscript "m" in the middle The subscript "m" in the original text corresponds to the same subscript. In this invention, to illustrate the correspondence between the two, the same letter "m" is used. Similarly, The subscript "n" in the text is related to... The subscript "n" in the matrix, although representing the dimension of the feature map and the column number of the matrix respectively, is used because the feature map and the matrix correspond. The subscript "n" in the text is related to... The subscript "n" in the text is corresponding. In order to illustrate the correspondence between the two, the same letter "n" is used to represent them in this invention.

[0076] In a preferred embodiment of the present invention, step S101 may include a translation transformation or a translation-rotation transformation, thereby obtaining the source data pair image I and image I′.

[0077] In the above training method, the training process of the modular autoencoder can be as follows: Figure 2 As shown, the training process of a controllable generative network based on modular autoencoders can be as follows: Figure 3 As shown.

[0078] In practical applications, this invention can be implemented according to the following steps:

[0079] Step 1: Construct an autoencoder consisting of convolutions. Divide the convolution kernels into k modules. During training, input both the untransformed image I and the transformed image I′ into the constructed autoencoder, and perform convolution operations on each of the k modules to obtain the feature image for each module. Additionally, construct equivariant constraints and use these constraints to train and optimize the convolution kernel W and prediction matrix M of the constructed convolutional network autoencoder. After training, all parameters are fixed, resulting in a functionally differentiated modular autoencoder.

[0080] Step 2: Obtain the modular autoencoder with k modules trained in Step 1, with convolutional kernels W = [W (0) W (1) , ..., W (k-1) ].

[0081] Step 3: Construct k adapters for k different modules based on the pre-trained latent diffusion model.

[0082] Step four: Input image I from the training data into the modular autoencoder from step two to obtain the modular feature image f = I * W, where W = [W (0) W (1) , ..., W (k-1) ], f = [f (0) f (1) , ..., f (k-1) ].

[0083] Step 5: Utilize the feature map f of the i-th module (i) The i-th adapter of the latent diffusion model is trained using the objective optimization function constructed as shown below:

[0084]

[0085] Where z0 is the latent representation of the latent diffusion model for the input image I, z t The noisy latent characterization at time step t, c txt This is a text prompt, where ∈ represents the actual added random noise with a distribution of N(0,1), where N(0,1) is a normal distribution with a mean of 0 and a variance of 1. The noise is predicted by optimizing the parameter θ. Represents the noise in the prediction The expected value of the square of the modulus of the deviation from the actual added random noise ∈, L mc (i) For the feature map f of the i-th module (i) The loss function of the adapter;

[0086] Step six: Perform self-supervised training, such as stochastic gradient descent, as in steps two through five to obtain the final trained k adapters. The trained modular autoencoder and the k adapters together constitute a controllable generative network based on the modular autoencoder.

[0087] In this invention, equivariant constraints are used to enhance the correlation within modules and the independence between modules, thereby enabling modules to spontaneously differentiate their functions. For example... Figure 4 This demonstrates the effect of functional differentiation under modular and equivariant constraints. Figure 4 In the diagram, Figure a shows the convolutional kernels trained on the MNIST black-and-white image dataset; Figures b and c show the convolutional kernels trained on the ImageNet color image dataset. Each row in Figures a, b, and c represents a module, and each square visualizes a learned convolutional kernel. The figures show clear functional differentiation between modules, such as differentiation in different spatial frequencies, brightness, and color. Figure d shows the visualization of the reconstruction of features extracted from each module in Figure c, providing a more intuitive view of the functional differentiation between modules in terms of color, brightness, and edges at different spatial frequencies. This modular feature differentiation is beneficial for providing a better representation space for downstream tasks, such as classification, detection, and associative generation tasks. It also helps improve the robustness and generalization of downstream tasks. For example, when the style of the input image for a recognition task changes significantly, conventional classification networks may fail to recognize the image, resulting in poor generalization. However, with the modular representation in this invention, even if the image style changes, it only affects the representation of a few modules, while having little impact on others.

[0088] This invention provides a controllable generative network model obtained through self-supervised training, which can be well transferred and generalized to more tasks, such as sketch association generation, picture association generation, and mural association generation. Notably, retraining on these new tasks is unnecessary; the model trained on standard datasets, such as the COCO dataset, can automatically exhibit generative capabilities for these new tasks. Compared to supervised methods, the self-supervised approach of this invention does not require a specific feature extractor or labeled data for a particular feature. A modular autoencoder is used to process the input sketch (e.g., ...) Figure 5 ), oil paintings (such as Figure 6 ), ink painting (such as Figure 7 , 8 ), murals (such as Figure 9 By extracting modular features and using some of these features as control conditions for generation, images can be generated that not only conform to the structure and content of the original image but are also more natural, vivid, and richer in detail. Figure 5In the diagram, the first line contains the input sketch image and text prompt; the second line contains the image generated using supervised ControlNet (for comparison); and the third line contains the image generated using the self-supervised controllable generation (SCG) method of this invention, which uses feature maps extracted by unsupervised automatic function specialization or differentiation as control conditions. Figure 6 In the diagram, the first column is the input image (original), the second column is the edge information extracted using the manually designed Canny operator as the control condition for ControlNet, the third and fourth columns are the generation results of ControlNet (Generations), the fifth column is the visualization result of the automatically differentiated feature deconvolution of the present invention, and the sixth and seventh columns are the results generated by SCG using the method of the present invention (Generations). Compared with the generation results of ControlNet, the present invention does not require manually designed feature extraction operators, and the generated images have better aesthetics while remaining faithful to the content and structure of the original image. Figure 7 In the image, the first row represents the input ink painting; the second row shows the automatically differentiated brightness features visualized through deconvolution; and the third and fourth rows represent the generated image. The generated image faithfully reproduces the input image in terms of content, overall structure, and brightness, while automatically supplementing or regenerating reasonable and aesthetically pleasing color information. Figure 8 In the output, the first row shows the input ink painting; the second row shows the automatically differentiated edge features visualized through deconvolution; and the third and fourth rows show the generated images. The generated images are faithful to the input images in terms of content and overall structure, while automatically supplementing or regenerating reasonable and aesthetically pleasing colors, brightness, and some detailed information. Figure 9 In the image, the left side shows an ancient mural image from over 10,000 years ago. The right side shows images generated using ControlNet and the SCG method proposed in this invention, respectively. Compared to ControlNet, the image generated by the method of this invention is clearer and more aesthetically pleasing while remaining faithful to the content and structure of the input image.

[0089] Example 2

[0090] like Figure 10 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow. That is, the embodiments of the present invention also provide a training device based on a modular autoencoder controllable generative network, including:

[0091] The training data acquisition module 1001 is used to acquire training data, which includes image I and image I′ obtained by transforming image I.

[0092] The autoencoder training module 1002 is used to train the convolutional network autoencoder based on the training data using equivariant constraints, thereby obtaining a trained autoencoder; wherein, the equivariant constraint L... EC The following formula is used for calculation:

[0093]

[0094] L equ =∑ i ||f′ (i) -M (i) (δ)f (i) || 2 , i∈{0,1,...,k-1},

[0095]

[0096] L EC =L recon +λ1L equ +λ2L sym ,

[0097] f = I * W,

[0098] f′=I′*W,

[0099] Where, I′=L δ (I), where Lδ is a certain transformation with transformation parameter δ; W is the convolution kernel of the autoencoder of the convolutional network, and W = [W (0) W (1) , ..., W (k-1) ], that is, the convolution kernel is divided into k modules; f is the feature image obtained after convolving image I, and f = [f (0) f (1) , ..., f (k-1) f′ is the feature image obtained after convolving image I′, f′=[f′ (0) f′ (1) ,...,f′ (k-1) ]; * indicates a convolution operation. Indicates the deconvolution operation; f (i) f′ is the feature map obtained after convolving image I through the i-th module of the convolution kernel; (i) M is the feature map obtained after convolving image I′ through the i-th module of the convolution kernel; (i) (δ) is the prediction matrix of the i-th module with parameter δ, which is obtained by random initialization and optimization through model training; L recon To rebuild the losses, L equ For equal loss, L sym For symmetrical loss, L ECThe constraint is equivariant; λ1 and λ2 are the weight coefficients of the equivariant loss and symmetric loss, respectively; l represents the number of dimensions of each module. This represents the feature map of the i-th module where only the m-th dimension is 1 and the other dimensions are 0. This represents the feature map of the i-th module where only the n-th dimension is 1 and the other dimensions are 0. From arrive The optimal transformation parameters; Let the parameter be the prediction matrix M. (i) The value of (δ) in the m-th row and n-th column; The prediction matrix M with parameter δ′ (i) The value of (δ′) in the m-th row and n-th column; T is a hyperparameter greater than 0; For the transformation parameters are The prediction matrix at time; δ′ is the integration variable, used to distinguish the transformation parameter δ under different integration terms;

[0100] Adapter building module 1003 is used to build k adapters for k different modules based on a pre-trained latent diffusion model;

[0101] The adapter training module 1004 is used to input the image I from the training data into the trained autoencoder to obtain the feature map f of the i-th module. (i) , i = 0, 1, 2, ..., k-1; using the obtained feature map f (i) Train the i-th adapter of the latent diffusion model using the objective optimization function shown below:

[0102]

[0103] Where z0 is the latent representation of the latent diffusion model for the input image I, z t The noisy latent characterization at time step t, c txt This is a text prompt, where ∈ represents the actual added random noise with a distribution of N(0,1), where N(0,1) is a normal distribution with a mean of 0 and a variance of 1. The noise is predicted by optimizing the parameter θ. Represents the noise in the prediction The expected value of the square of the modulus of the deviation from the actual added random noise ∈, L mc (i) For the feature map f of the i-th module (i) The loss function of the adapter;

[0104] The controllable generative network building module 1005 is used to combine the trained autoencoder and adapter to form a controllable generative network based on modular autoencoder.

[0105] The device can be implemented using the training method of the controllable generative network based on modular autoencoder provided in Embodiment 1 above. For the specific implementation method, please refer to the description in Embodiment 1, which will not be repeated here.

[0106] The present invention also provides a memory that stores multiple instructions for implementing the training method of the controllable generative network based on modular autoencoder as described in Embodiment 1.

[0107] The present invention also provides an electronic device, including a processor and a memory connected to the processor, the memory storing a plurality of instructions which can be loaded and executed by the processor to enable the processor to perform a training method for a controllable generative network based on modular autoencoders as described in Embodiment 1.

[0108] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A training method for a controllable generative network based on modular autoencoders, characterized in that, include: Acquire training data, which includes image I and image I′ obtained by transforming image I; Based on the training data, the convolutional network autoencoder is trained using equivariant constraints to obtain a trained autoencoder; wherein, the equivariant constraint L... EC The following formula is used for calculation: L equ =∑ i ||f′ (i) -M (i) (δ)f (i) || 2 ,i∈{0,1,...,k-1}, L EC =L recon +λ1L equ +λ2L sym , f = I * W, f′=I′*W, Where, I′=L δ (I), L δ Let W be a certain transformation with transformation parameter δ; W is the convolution kernel of the autoencoder of the convolutional network, and W = [W (0) W (1) , ..., W (k-1) ], that is, the convolution kernel is divided into k modules; f is the feature image obtained after convolving image I, and f = [f (0) f (1) , ..., f (k-1) f′ is the feature image obtained after convolving image I′, f′=[f′ (0) f′ (1) ,...,f′ (k-1) ]; * indicates a convolution operation. Indicates the deconvolution operation; f (i) f′ is the feature map obtained after convolving image I through the i-th module of the convolution kernel; (i) M is the feature map obtained after convolving image I′ through the i-th module of the convolution kernel; (i) (δ) is the prediction matrix of the i-th module with parameter δ, which is obtained by random initialization and optimization through model training; L recon To rebuild the losses, L equ For equal loss, L sym For symmetrical loss, L EC For equivariant constraints; λ1 and λ2 are the weighting coefficients of equivariant loss and symmetric loss, respectively; l represents the number of dimensions of each module; This represents the feature map of the i-th module where only the m-th dimension is 1 and the other dimensions are 0. This represents the feature map of the i-th module where only the n-th dimension is 1 and the other dimensions are 0. From arrive The optimal transformation parameters; Let the parameter be the prediction matrix M. (i) The value of (δ) in the m-th row and n-th column; The prediction matrix M with parameter δ′ (i) The value of (δ′) in the m-th row and n-th column; T is a hyperparameter greater than 0; For the transformation parameters are The prediction matrix at time; δ′ is the integration variable, used to distinguish the transformation parameter δ under different integration terms; Construct k adapters for k different modules based on a pre-trained latent diffusion model; The image I from the training data is input into the trained autoencoder to obtain the feature map f of the i-th module. (i) ,i=0,1,2,...,k-1; Using the obtained feature map f (i) Train the i-th adapter of the latent diffusion model using the objective optimization function shown below: Where z0 is the latent representation of the latent diffusion model for the input image I, z t The noisy latent characterization at time step t, c txt This is a text prompt, where ∈ represents the actual added random noise with a distribution of N(0,1), where N(0,1) is a normal distribution with a mean of 0 and a variance of 1. The noise is predicted by optimizing the parameter θ. Represents the noise in the prediction The expected value of the square of the modulus of the deviation from the actual added random noise ∈, L mc (i) For the feature map f of the i-th module (i) The loss function of the adapter; The trained autoencoder and the trained adapter together form a controllable generative network based on modular autoencoders.

2. The training method for a controllable generative network based on modular autoencoders as described in claim 1, characterized in that, The transformation includes translation transformation or translation-rotation transformation.

3. A controllable generative network based on modular autoencoders, characterized in that, It is trained using the method described in any one of claims 1-2.

4. A training device based on a modular autoencoder controllable generative network, characterized in that, include: The training data acquisition module is used to acquire training data, which includes image I and image I′ obtained by transforming image I; The autoencoder training module is used to train the convolutional network autoencoder based on the training data using equivariant constraints, thereby obtaining a trained autoencoder; wherein, the equivariant constraint L... EC The following formula is used for calculation: L equ =∑ i ||f′ (i) -M (i) (δ)f (i) || 2 ,i∈{0,1,...,k-1}, L EC =L recon +λ1L equ +λ2L sym , f = I * W, f′=I′*W, Where, I′=L δ (I), L δ Let W be a certain transformation with transformation parameter δ; W is the convolution kernel of the autoencoder of the convolutional network, and W = [W (0) W (1) , ..., W (k-1) ], that is, the convolution kernel is divided into k modules; f is the feature image obtained after convolving image I, and f = [f (0) f (1) , ..., f (k-1) f′ is the feature image obtained after convolving image I′, f′=[f′ (0) f′ (1) ,...,f′ (k-1) ]; * indicates a convolution operation. Indicates the deconvolution operation; f (i) f′ is the feature map obtained after convolving image I through the i-th module of the convolution kernel; (i) M is the feature map obtained after convolving image I′ through the i-th module of the convolution kernel; (i) (δ) is the prediction matrix of the i-th module with parameter δ, which is obtained by random initialization and optimization through model training; L recon To rebuild the losses, L equ For equal loss, L sym For symmetrical loss, L EC For equivariant constraints; λ1 and λ2 are the weighting coefficients of equivariant loss and symmetric loss, respectively; l represents the number of dimensions of each module; This represents the feature map of the i-th module where only the m-th dimension is 1 and the other dimensions are 0. This represents the feature map of the i-th module where only the n-th dimension is 1 and the other dimensions are 0. From arrive The optimal transformation parameters; Let the parameter be the prediction matrix M. (i) The value of (δ) in the m-th row and n-th column; The prediction matrix M with parameter δ′ (i) The value of (δ′) in the m-th row and n-th column; T is a hyperparameter greater than 0; For the transformation parameters are The prediction matrix at time; δ′ is the integration variable, used to distinguish the transformation parameter δ under different integration terms; The adapter building module is used to build k adapters for k different modules based on a pre-trained latent diffusion model; The adapter training module is used to input the image I from the training data into the trained autoencoder to obtain the feature map f of the i-th module. (i) , i = 0, 1, 2, ..., k-1; using the obtained feature map f (i) Train the i-th adapter of the latent diffusion model using the objective optimization function shown below: Where z0 is the latent representation of the latent diffusion model for the input image I, z t The noisy latent characterization at time step t, c txt This is a text prompt, where ∈ represents the actual added random noise with a distribution of N(0,1), where N(0,1) is a normal distribution with a mean of 0 and a variance of 1. The noise is predicted by optimizing the parameter θ. Represents the noise in the prediction The expected value of the square of the modulus of the deviation from the actual added random noise ∈, L mc (i) For the feature map f of the i-th module (i) The loss function of the adapter; The controllable generative network building module is used to combine the trained autoencoder and the trained adapter to form a controllable generative network based on modular autoencoders.

5. The training device based on a modular autoencoder-based controllable generative network as described in claim 4, characterized in that, The transformation includes translation transformation or translation-rotation transformation.

6. A memory, characterized in that, The system stores multiple instructions for implementing the training method of the controllable generative network based on modular autoencoders as described in any one of claims 1-2.

7. An electronic device, characterized in that, The system includes a processor and a memory connected to the processor, the memory storing multiple instructions that can be loaded and executed by the processor to enable the processor to perform the training method for a controllable generative network based on modular autoencoders as described in any one of claims 1-2.

Images