A lightweight latent diffusion model design method and system based on evolutionary neural architecture search

Abstract

The present invention provides a lightweight potential diffusion model design method and system based on evolutionary neural architecture search. The method uses supernet technology to simultaneously train multiple network architectures, combines evolutionary genetic algorithms in the architecture search process, and trains a multi-layer perceptron as a score predictor to assist the evolutionary genetic algorithm in efficiently screening out the optimal architecture in a large-scale search space. This method significantly reduces the resource consumption and computing costs required in the traditional architecture design process. The present invention introduces a pre-trained teacher model to supervise the training of the student model by combining knowledge distillation technology, which can make the potential diffusion model lightweight while ensuring the stability of the performance of the potential diffusion model. Through this method, not only can an efficient neural network architecture be automatically designed, but also the efficient deployment of the potential diffusion model can be achieved, providing a new and low-cost solution for the optimization of the diffusion model.

Description

Technical Field

[0001] The present invention relates to the field of automated machine learning, and in particular to a lightweight potential diffusion model design method and system based on evolutionary neural architecture search. Background Art

[0002] With the rapid development of deep learning technology, the diffusion model, as a generative model, has achieved remarkable results in many fields such as image generation, speech synthesis, and video processing. The core idea of ​​the diffusion model is to simulate the forward diffusion process of an image from a clear state to a pure noise state, and then gradually restore the original clear image through the reverse denoising process, thereby achieving high-quality image generation. The diffusion model uses the U-shaped network architecture UNet to predict the noise in the image during the reverse denoising process. However, the U-shaped network architecture UNet used in the existing diffusion model usually relies on manually designed network architectures. Although these manually designed network architectures can complete some generation tasks to a certain extent, they have poor generalization capabilities in different data sets and task backgrounds, and consume huge computing resources. When faced with new data sets or task backgrounds, it is almost impossible to quickly design a good network architecture by relying solely on manual design and trial and error.

[0003] The huge consumption of computing resources has always been a major problem faced by the diffusion model. To address this problem, researchers have proposed a potential diffusion model based on the diffusion model. , the latent diffusion model is constructed by introducing a variational autoencoder , the input image is converted from the original pixel space to the latent space, and the denoising and denoising operations of the diffusion model are performed in the latent space. The latent diffusion model not only significantly reduces the dependence on computing resources, but also ensures a high consistency in the generated quality, thereby achieving further improvements in efficiency and adaptability.

[0004] In order to improve the performance and adaptability of potential diffusion models, researchers have begun to explore solutions for automated design of neural network architectures in recent years. Neural network architecture search technology has become an important research direction. By automatically searching for the optimal architecture, neural network architecture search technology can reduce the complexity of manual design and quickly design efficient network architectures for specific tasks. However, most neural network architecture search methods use traditional optimization strategies, such as reinforcement learning and gradient optimization, which usually require a lot of computing resources and time in large-scale search spaces, making the search process time-consuming and costly.

[0005] In addition, despite the introduction of the automated search strategy of neural network architecture search, how to efficiently combine the search strategy with specific generation tasks in the network architecture design of the latent diffusion model remains an important challenge. Especially in a resource-constrained environment, how to significantly reduce the computational overhead while ensuring the performance of the latent diffusion model is still one of the difficulties in the practical application of the latent diffusion model. At the same time, some studies have proposed training the network architecture through supernet inheritance. Although this method can greatly shorten the training time, the subnets obtained by weight inheritance from the supernet often have a large deviation from the actual performance when trained from scratch. In view of the above problems, there is an urgent need for an efficient and low-cost solution that can not only optimize the architecture design of the latent diffusion model through the automated neural network architecture search method, but also combine the distillation technology to improve the adaptability and performance of the latent diffusion model in different tasks and data sets, while reducing the consumption of computing resources. Therefore, a lightweight latent diffusion model design method and system based on evolutionary neural architecture search is proposed, which can achieve the lightweight of the latent diffusion model while performing efficient architecture search, and improve the performance and adaptability of the latent diffusion model in different application scenarios. Summary of the invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a lightweight potential diffusion model design method and system based on evolutionary neural architecture search in view of the deficiencies of the prior art. The method comprises the following steps:

[0007] Step 1: Select a pre-trained variational autoencoder , using a variational autoencoder Encoder in The input image data is converted from the original pixel space to the latent space, where the input image data is represented by a latent data distribution. Next, a noise addition process is performed in the latent space, that is, Gaussian noise is gradually added to the latent data distribution until the latent data distribution reaches a pure noise state. Subsequently, the operations of steps 2 to 8 are sequentially completed in the latent space, and after waiting for step 8 to be completed, the network architecture obtained in step 8 is used to perform a denoising operation on the pure noise data. The subsequent series of steps (except step 10) are all completed in the latent space.

[0008] Step 2: Select a pre-trained model related to image generation as the teacher model, and design the network architecture of the student model based on the network architecture of the teacher model. Then, divide the network architectures of the teacher model and the student model into two or more blocks based on the different resolution levels.

[0009] Step 3, define the architecture search space of the student model, build a student supernet according to the student model architecture designed in step 2, and train the student supernet;

[0010] Step 4: randomly initialize a population of architectures in each student model block, and evaluate the performance of individuals (architectures) in the initial population in each block;

[0011] Step 5: Perform crossover and mutation operations on the parent architecture through the existing technology based on integer-coded uniform crossover operator and random uniform mutation operator to generate a child architecture, gradually expand the search space, and explore potential excellent architectures;

[0012] Step 6: For the newly generated offspring population after the evolution operation, the fitness value of the population individual (the architecture of the student model) is calculated using the same evaluation method as in step 4; after the fitness value of the newly generated offspring architecture is evaluated, the offspring architecture population generated by evolution is merged with the parent architecture population to form a population called the extended population. Then, an elite selection strategy is used to sort all architectures according to their fitness values, and only the expanded population is retained after the sorting. Medium fitness value (usually the first ) architecture as a new generation population , as the basic population for the next round of evolution and evaluation; this method can not only ensure the inheritance of excellent architectures and avoid the loss of excellent individuals during crossover and mutation, but also gradually improve the overall performance of the population by selecting high-quality architectures;

[0013] Step 7: Repeat steps 5 to 6 for each block of the student model until the preset stop condition is met (i.e., when the number of population evolution iterations reaches the upper limit of the defined total number of search iterations), the search is stopped; then, the architecture with the highest fitness value is selected from each block of the student model as the optimal architecture finally searched in each block of the student model;

[0014] Step 8, connecting the optimal architecture finally searched in each block of the student model in order of block size from small to large to form a complete network architecture, which is called a subnet;

[0015] Step 9: Retrain the subnet obtained in step 8. During the subnet retraining process, a dynamic joint loss function is introduced to further improve the retraining effect of the subnet and the diffusion model. The main function of the neural network is to predict the noise in the noisy data distribution; the dynamic joint loss function is composed of the distillation loss between the teacher model and the student model and the prediction loss of the subnet on the noisy data. By using this dynamic joint loss function, it can ensure that the student model can efficiently learn the knowledge of the teacher model and enhance the noise prediction ability of the subnet; the retraining refers to using the subnet searched in step 8 to train the training data set The above operation continues in step 1 to perform the noise removal operation on the pure noise data. The dataset used in the subnet retraining process ( ) is a public dataset in the field of image generation, which contains The resolution size is of face images, where images as training dataset ;

[0016] Step 10: Use the subnet retrained in step 9 to sample and generate the final image. During the sampling process, the variational autoencoder is used. Decoder in The latent data distribution obtained after the subnet removes noise is mapped back to the pixel space, and the generated image samples are finally output.

[0017] In step 2, the network architecture of the teacher model and the student model is partitioned and blocked because the use of this partitioning method can significantly reduce the size of the search space. In the traditional global architecture search method, it is necessary to find the optimal solution in the search space of the entire network architecture, which will cause the search space to grow exponentially and the computational complexity is extremely high. For example, in the present invention, if the traditional search method is used, the search space is approximately If the network architecture is divided into multiple smaller blocks according to different resolution levels, and the evolutionary architecture search is performed independently in each block, the search space will be reduced to , the search efficiency will be greatly improved.

[0018] In step 2, a pre-trained model related to image generation is selected as a teacher model. The pre-trained model must use the same face image dataset as the present invention. , and performs well in the image generation task, that is, the selected pre-trained model has good performance in the evaluation index It is superior to all existing pre-training models in terms of evaluation indicators. It is a commonly used indicator for evaluating the quality of generated images, which is used to measure the difference between generated images and real images. The smaller the value, the smaller the difference between the generated image and the real image, that is, the higher the quality of image generation, and the evaluation index It is particularly suitable for the evaluation of generative models such as diffusion models. Then, a lightweight student model is designed based on the network architecture of the teacher model. The backbone networks of the teacher model and the student model are both Network architecture , Network architecture It includes an encoder (downsampling) stage, an intermediate stage, and a decoder (upsampling) stage. In each block of the encoder (downsampling) stage and the decoder (upsampling) stage, the number of residual blocks in the student model network architecture is only half of that in the teacher model network architecture, and the other parts of the architecture remain unchanged; finally, the network architectures of the teacher model and the student model are divided into two or more blocks according to the different resolution levels, among which the blocks divided in the encoder (downsampling) stage are called encoder (downsampling) blocks, and the blocks divided in the decoder (upsampling) stage are called decoder (upsampling) blocks;

[0019] The encoder (downsampling) block consists of multiple attention downsampling blocks. The bottom layer of each attention downsampling block contains a downsampling operation module, which is used to reduce the resolution of the input features to the original The middle block consists of two residual network blocks and an attention mechanism block; the decoder (up-sampling) block consists of multiple attention up-sampling blocks, and the last layer of each attention up-sampling block contains an up-sampling operation module, which is used to increase the resolution of the input features to twice the original. Therefore, the present invention uses different levels of resolution as the basis for partitioning the teacher model and the student model. Network architecture Decompose the network architecture of the teacher model and the student model into two or more blocks respectively.

[0020] Step 3 includes:

[0021] Step 3.1, define the teacher model and student model in blocks:

[0022] ,

[0023] ,

[0024] in, =0, 1, ..., -1 indicates the resolution level, there are level, The smaller it is, the higher the resolution level is and the larger the feature map resolution is; represents the teacher model, and They represent the resolution level in the teacher model. The encoder (downsampling) block and the decoder (upsampling) block of represents the intermediate stage of the teacher model between the encoder (downsampling) block and the decoder block (upsampling); represents the student model, and They represent the resolution level in the student model. The encoder (downsampling) block and the decoder (upsampling) block of Represents the intermediate stage of the student model between the encoder (downsampling) block and the decoder (upsampling) block;

[0025] Because the intermediate stages in the teacher model and the student model do not involve the operation of changing the resolution, the network architecture of the intermediate stage between the teacher model and the student model will not be considered in the present invention, that is, the network architecture of the intermediate stage is kept unchanged and is not searched;

[0026] Step 3.2, define the architecture search space of the student model, and build a student supernet based on the student model network architecture designed in step 2; the search space refers to the set of design parameters available in the student model architecture, which is used to define potential architecture variants; the search space of the student model only includes the design options of the residual block in the student model network architecture: the convolution kernel size of the convolution layer in the residual block ranges from , the normalization method in the residual block selects group normalization or layer normalization; the student supernet is a network structure that includes all possible student model architectures, and the student supernet can include and support one or more different student model architectures in one training; the present invention adopts a single path sampling method when training the student supernet, and randomly selects a path each time for forward propagation and parameter update, wherein each path corresponds to a specific student model architecture; in the process of training the student supernet, by extracting the real feature map of the teacher model, aligning the feature map generated by the student model with the real feature map of the corresponding block of the teacher model, each block of the student model is supervised and optimized, and the mean square error between the intermediate feature map of the teacher model and the feature map generated by the corresponding block of the student model is calculated loss:

[0027] Loss function during student supernet training for:

[0028] ,

[0029] , =0,1,...,2 -1,

[0030] in, =0,1,...,2 -1 indicates 2 in the student model network architecture blocks, The resolution level defined in step 3.1 The numbers are equal. Here, the encoder (downsampling) block and the decoder (upsampling) block are regarded as separate blocks. In the teacher model and student model network architecture, each resolution level corresponds to an encoder (downsampling) block and a decoder (upsampling) block, so the total number of blocks is 2. , which is also equal to 2 in value ; Among them, the serial number index is Before The blocks are encoder (downsampling) blocks, which are used to gradually extract and compress features. The serial number index is After The first block is the decoder (upsampling) block, which is used to gradually restore the resolution and generate output features;

[0031] Indicates the current The subnetwork architecture being trained in the block, Indicates the student model The input feature map of each block comes from the teacher model. Indicates when The encoder (downsampling) block belonging to the student model hour, The teacher model Output of blocks ; Indicates when Decoder (upsampling) block belonging to the student model hour, Is a teacher model Network architecture Middle Skip connection features of blocks With the teacher model Output feature map of each block The splicing result ; Indicates that in the student model blocks, using the currently selected subnetwork architecture For the input feature map The feature map generated after forward propagation; Indicates the first The real feature map of each block is used to supervise and guide the training of the student supernet;

[0032] Is a teacher model Network architecture Middle The skip connection feature of the encoder block, in the present invention, Network architecture The skip connection Plays a vital role, jump connection It is connected to the teacher model Network architecture A path of feature maps from different blocks in the teacher model is usually used to establish a direct connection between the encoder (downsampling) stage and the decoder (upsampling) stage; specifically, Network architecture In the skip connection The feature information extracted in the encoder (downsampling) stage is directly passed to the corresponding block in the decoder (upsampling) stage, which helps to preserve the detailed features of the image during the decoding process and improve the accuracy and quality of image generation. Network architecture This skip connection method is also adopted, inheriting the advantages of the teacher model in image detail restoration and feature transfer;

[0033] Step 3.3, by minimizing the loss function To optimize each block in the student model Trainable network parameters , so that the student model performs better during the training process, the training data set used in the student supernet training process and the subnet retraining process of the present invention Consistent, both are public datasets in the field of image generation , the dataset contains The resolution size is of face images, where images as training set , images as validation set , the optimization process is expressed as:

[0034] ,

[0035] , =0,1,...,2 -1,

[0036] in, It is the first The trainable network parameters of each block, including the weight matrix used for convolution operations, such as convolution kernel weights, and scaling parameters and offset parameters in normalization layers; In the student model The ultimate goal that we hope to achieve during the optimization process of each block is: Indicates The architecture search space of blocks;

[0037] The optimization process is to adjust the Trainable network parameters for blocks , minimize the loss function , thereby obtaining the current block The optimal network parameters ,Right now ,in Indicates that in the current The architecture search space of blocks In the example, use the current network parameters , calculate the student model Output of blocks With the teacher model The real feature map of the block The error between .

[0038] In step 4, each individual of the initial population or the offspring population individual evolved from the parent generation will be evaluated to obtain their corresponding fitness values, but the present invention only uses a part of the individuals in the population for real evaluation. Therefore, the present invention sets a threshold , as the end signal of the population evaluation process, when this threshold is reached, the present invention uses the historical record dictionary obtained in the real evaluation process To train a multilayer perceptron As a score predictor ; In the subsequent prediction and evaluation stage, the trained score predictor can be used directly To predict the scores of population individuals (architectures). Step 4 specifically includes:

[0039] Step 4.1: Initialize an architecture population in each block of the student model as the initial population , defines the number of architectures to be evaluated in each generation , Total number of search iterations , the number of true evaluations in each iteration ;

[0040] Step 4.2, define the current number of iterations in the evolutionary architecture search and subpopulations , for the initial population Conduct an assessment.

[0041] Step 4.2 includes:

[0042] Step 4.2.1, for the initial population The individuals in the population are encoded, and the encoding of each individual in the population is standardized. In the evolutionary algorithm, the encoding operation is to represent the individual in a form that is convenient for algorithm operation. The present invention uses the integer encoding method in the existing encoding technology to represent the population individuals in each block. After the encoding is completed, the duplicate individuals and invalid individuals in the population are removed, and the remaining individuals are called valid individuals; according to the current number of iterations and the number of actual evaluations required in each iteration The size relationship between them divides the evaluation phase into two stages: This is the actual evaluation stage. This is the prediction and evaluation stage;

[0043] Step 4.2.2, if , then the sequence index in the current evaluation population is The individuals are truly evaluated and the obtained fitness values ​​are Recorded in the fitness value list In the above example, the encoding of the individual being evaluated and the fitness value corresponding to the individual being evaluated are both Recorded in a history dictionary In the fitness value list, the fitness value of the individual that has not been truly evaluated is set to 0 and recorded in the fitness value list middle;

[0044] In the real evaluation phase, the subnetwork architecture in the student model with the same architecture as the teacher model is called the baseline network , for the student model The architecture search space in blocks , select the architecture search space Loss assessment Outperforms the baseline network The subnetwork architecture for evaluating the loss , and then in the filtered sub-network architecture Further search for the sub-network architecture with the smallest fitness value , where the fitness value The calculation formula is:

[0045] , =0, 1, ..., 2 -1,

[0046] ,

[0047] ,

[0048] in, Restricted to, is a relaxation factor used to control the The sub-network architecture selected from With the benchmark network The degree of relaxation of the loss value between Indicates that in the validation data set On the network, use the currently trained sub-network architecture The obtained student model Output of blocks With the teacher model The real feature map of the block The error between Cannot be larger than the baseline network The obtained student model Output of blocks With the teacher model The real feature map of the block The error between , validation dataset From a public dataset separated from A face image;

[0049] in, = , used to calculate from the current block The sub-network architecture selected from The real feature map of the corresponding block of the teacher model In the validation dataset Loss on ; = , used to calculate the benchmark network The real feature map of the corresponding block of the teacher model In the validation dataset Loss on ;

[0050] Indicates the current block The sub-network architecture selected from The relevant cost calculation function is mainly calculated by the number of multiplication and addition operations ( ) indicators to measure The size of the cost, Indicates that from The architecture search space of blocks Select cost Minimal architecture ; Fitness function For the selected architecture The computational cost A negative value of indicates that the lower the cost, the greater the fitness value of the current architecture, and the better the corresponding architecture;

[0051] Step 4.2.3, in the first generation of the prediction evaluation phase, i.e. When using the history dictionary obtained in step 4.2.2 To train a multilayer perceptron As a score predictor , using the score predictor To predict the scores of individuals in the population can indirectly reflect the quality of individuals in the population, but at this time the score predictor The prediction range and the actual evaluation result range are not in the same numerical range, so the parent population needs to use a score predictor Re-evaluate once; When , directly use the trained score predictor Scoring all valid individuals in the population without obtaining their actual performance values ​​through real evaluation can greatly reduce the consumption of computing resources.

[0052] In step 5, each population individual (network architecture) is represented by a set of integer codes, and each integer code corresponds to an operation in the architecture. The present invention uses a uniform crossover operator based on integer codes and a random uniform mutation operator to retain the excellent characteristics of the parent generation and integrate diversity, avoid falling into a local optimal solution and expand the search space. The generated new generation of child architecture can gradually guide the population to converge to the optimal solution and explore potential excellent architectures. For example, two parent architectures are randomly selected from the current population. and , for each position in the architecture encoding, a binary code is randomly generated , =0 means the child inherits the parent at this position The value of =1 means the child inherits the parent at this position The value of

[0053] In the present invention Network architecture The encoder (downsampling) block is taken as an example, and the residual block The convolution kernel size and normalization method in are searched. There are three candidate operations for the convolution kernel size: {1, 3, 5}. The normalization methods include layer normalization and group normalization. The search range includes 4 convolution layers and 4 normalization layers. Therefore, the encoding of the parent A is set to {0, 1, 0, 2, 1, 1, 1, 0}. The encoding is set to {1,2,0,0,1,0,1,1}. When the random mask is designed as =[0,1,1,0,0,1,0,1], then the offspring obtained by the crossover operation The encoding is represented as {0,2,0,2,1,0,1,1};

[0054] In order to further enhance the diversity of the population, the present invention uses a randomly uniformly selected mutation operator to randomly modify the architecture code; similarly, in the mutation operation, a binary code is randomly generated for each position of the architecture code. {0,1}, if =1, it means that the code of the position will change. For the position that needs to change, the present invention will randomly select a candidate operation code from the current block to replace the original operation;

[0055] In the present invention Network architecture For example, the decoder (up-sampling) block of The convolution kernel size and normalization method in the convolution kernel size have three candidate operations of {1,3,5}. The normalization method includes layer normalization and group normalization. Unlike the encoder (downsampling) block architecture, the search range of the decoder (upsampling) block includes 6 convolutional layers and 6 normalization layers. Therefore, the encoding of the parent A is set to {0,1,0,2,1,1,1,0,1,2,1,1}. =[0,0,1,0,0,1,0,1,0,1,0,0], the offspring after the mutation operation The architecture encoding is represented as {0,1,1,2,1,0,1,1,1,0,1,1}.

[0056] In step 6, for the newly generated offspring architecture after several generations of evolution, the fitness value of the population individuals is calculated using the same evaluation method as in step 4.2. After the fitness value of the newly generated offspring architecture is evaluated, the offspring architecture population generated by evolution is merged with the parent architecture population to form a population called the extended population. Then, an elite selection strategy is used to sort all architectures according to their fitness values, and only the expanded population is retained after the sorting. Medium fitness value (usually the first ) architecture as a new generation population , used as the basic population for the next round of evolution and evaluation. This method can not only ensure the inheritance of excellent architectures and avoid the loss of excellent individuals during crossover and mutation, but also gradually improve the overall performance of the population by selecting high-quality architectures.

[0057] Step 9 includes:

[0058] Step 9.1, retrain the subnet obtained in step 8 in the latent space using a dynamic joint loss function;

[0059] Step 9.2: Design the weight coefficients in the dynamic joint loss function An adjustment function to smoothly control the loss function weights.

[0060] Step 9.1 includes: retraining the subnetwork obtained in step 8 in the latent space using a dynamic joint loss function to ensure that the student model better inherits the characteristics of the teacher model in terms of performance and generalization ability. The dynamic joint loss function for:

[0061] (1 ) ,

[0062] ,

[0063] , =0,1,...,2 -1,

[0064] in, Is used to balance the distillation losses and noise prediction loss The weight coefficient of the present invention is In the initial stage of subnet retraining, the weight coefficient is set to 0. As the retraining progresses, the weight coefficient The value of eventually grows to 1; is used to balance the noise prediction loss and distillation losses A parameter for the learning rate between Indicates the actual noise level added during the noise addition process in step 1, is a trainable network parameter in the subnet used to predict the noise in the data with noise distribution, represents the current time step in the denoising process, represents the noise data distribution, where represents the mean of the noise data distribution, Represents the variance of the noise data distribution, noise prediction loss It is a mean square error based The loss function is used to measure the accuracy of the subnetwork in predicting the noise size in the noisy distribution data, specifically by calculating the noise prediction loss The actual noise added to the original data distribution This is achieved by the error between

[0065] =0,1,...,2 -1, indicating that there are 2 blocks, It means to accumulate the distillation loss between the teacher model and the student model in all blocks, where , and the loss function set in step 3 Same, distillation loss It is also a method based on mean square error The loss function is used to calculate the generated feature map of the student model in all blocks of the subnet The true feature map with the teacher model The error between and.

[0066] The dynamic joint loss function is composed of the distillation loss and noise prediction loss It consists of two parts, among which the distillation loss Used to measure the distance between the student model and the teacher model on the intermediate feature map, supervise and encourage the student model to imitate the excellent representation ability of the teacher model; Noise prediction loss It is used to measure the error between the noise size predicted by the student model and the actual noise size added, ensuring that the student model can generate high-quality images; specifically, the distillation loss is used in the initial stage of subnet retraining. Based on the prediction of noise loss This method of dynamically adjusting the weights of the two losses enables the student model to further optimize its own task performance while imitating the teacher model, thus achieving a good balance between performance and generalization ability.

[0067] Step 9.2 includes:

[0068] Choose a tuning function to dynamically adjust the distillation loss and noise prediction loss The weight coefficient between , to optimize the retraining process of the subnet, ensure that it efficiently inherits the characteristics of the teacher model, and focus on the generation ability of the subnet in the later retraining stage;

[0069] Weight coefficient There are two options for the adjustment function: linear function and step function; the linear function represents the coefficient During the subnet retraining process, it gradually increases from 0 to 1 in a smooth linear manner. In contrast, the step function is adjusted in a more abrupt manner during the training process. In the early stage of subnet retraining, the coefficient The value of remains at 0 until after a certain number of training steps, the coefficient The value of will suddenly jump to 1. The present invention chooses to use a step function to Adjustment can save the time of subnet retraining and make the coefficient When a certain threshold point is reached, it directly jumps to a state with a value of 1, avoiding excessive calculations and parameter adjustments. On the other hand, the use of a step function can also ensure the effect of subnet retraining.

[0070] In step 10, the subnet retrained in step 9 is used to predict the noise component in the current data and remove the noise component to gradually obtain a representation closer to the actual data distribution, and then the variational autoencoder is used Decoder in The latent representation obtained after noise removal is mapped back to the pixel space, and finally the generated image samples are output to complete the image generation task of the lightweight latent diffusion model.

[0071] A system implemented by the method comprises:

[0072] Student supernet training module based on knowledge distillation: used to train a student model so that the student model is close to or exceeds the teacher model in performance, thereby achieving lightweight potential diffusion model;

[0073] Evolutionary Neural Architecture Search Module Based on Score Predictor: Used to Train a Multilayer Perceptron , as a score predictor for evaluating architecture performance, enabling the evolutionary genetic algorithm to find the optimal architecture in a shorter time;

[0074] Retraining module based on dynamic joint loss function: used to retrain the subnets obtained during the architecture search process; a dynamic joint loss function that combines the distillation loss and the noise prediction loss is used to smoothly adjust the weights of different loss functions, thereby improving the stability and efficiency of the training process.

[0075] This paper proposes a lightweight latent diffusion model design method based on evolutionary neural architecture search, combining knowledge distillation technology and automated neural network architecture search method, aiming to improve the performance and adaptability of the latent diffusion model while reducing the consumption of computing resources. Specifically, this paper first uses a pre-trained variational autoencoder Encoder in The input image is mapped from the original pixel space to the latent space. This operation provides efficient feature representation for the subsequent operations of the diffusion model and greatly reduces the computational complexity. On this basis, the guidance of the teacher model is introduced in combination with the knowledge distillation technology to optimize the performance of the student model during the training process of the student supernet, so that the student model can better inherit the characteristics of the teacher model. Subsequently, the network architecture is automatically searched in combination with the evolutionary architecture search method, avoiding the inefficiency and high cost of the traditional manually designed architecture. Finally, the retrained complete network (subnet) is used to perform sampling of the lightweight latent diffusion model to generate high-quality images. By combining the evolutionary neural architecture search method with the knowledge distillation strategy, the present invention can reduce the consumption of computing resources while ensuring the performance of the diffusion model generation task, solving the bottleneck of the diffusion model in terms of large-scale computing overhead and generalization ability.

[0076] The beneficial effects of the present invention are as follows: the method of the present invention proposes a lightweight latent diffusion model design method based on evolutionary neural architecture search for the latent diffusion model. Different from the traditional method that only uses weight inheritance, the present invention introduces the guidance of the teacher model in the student supernet training process, and uses distillation technology to optimize the student model, thereby improving the performance and generalization ability of the student model. By performing automated neural network architecture search through the evolutionary architecture search method, a suitable model architecture can be found more quickly in different data sets and task contexts. In addition, the lightweight latent diffusion model designed by the present invention not only improves the adaptability of the model, but also greatly reduces the consumption of computing resources, solving the problems of traditional latent diffusion models in generalization ability and large-scale computing overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0078] Figure 1 For the classic latent diffusion model on the face dataset The sampling result graph on .

[0079] Figure 2 This is a framework diagram of the student supernet training phase of the present invention.

[0080] Figure 3 This is a framework diagram of the evolutionary architecture search phase of the present invention.

[0081] Figure 4 This is a framework diagram of the subnet retraining phase of the present invention.

[0082] Figure 5 For the present invention, the face dataset The sampling result graph on . DETAILED DESCRIPTION

[0083] The embodiment of the present invention proposes a lightweight potential diffusion model design method based on evolutionary neural architecture search. The present invention mainly consists of four parts, namely, variational autoencoder The encoding and decoding phases, the student supernet training phase, the evolutionary neural architecture search phase, and the subnet retraining phase; from using variational autoencoders Encoder in The input image is mapped from the original pixel space to the latent space, and then the knowledge distillation technology is used to assist the student supernet in training. After the student supernet training is completed, the evolutionary architecture search strategy is executed in each block. Then, the optimal architecture searched in each block of the student model is integrated into a complete network (subnet). Then, the retrained subnet is used for noise prediction, and the noise in the noise data is gradually removed to restore the original potential data distribution. Finally, the variational autoencoder is used Decoder in The restored potential data distribution is mapped back to the pixel space to complete the image generation task; the entire process introduces in detail the specific implementation steps and details of the lightweight potential diffusion model design method based on evolutionary neural architecture search proposed in the present invention. The method of the present invention specifically includes the following steps:

[0084] Step 1: Use a pre-trained variational autoencoder Encoder in , mapping the input image from the original pixel space to the latent space. This process extracts the key features of the image and reduces the computational complexity of the data. The input image data is represented by the latent data distribution in the latent space. Next, the noise addition process is performed in the latent space, that is, Gaussian noise is gradually added to the latent data distribution until the latent data distribution reaches a pure noise state; then, the operations of steps 2 to 8 are completed in the latent space in sequence, and after waiting for step 8 to complete, the network architecture obtained in step 8 is used to perform the denoising operation of the pure noise data; the subsequent series of steps (except step 10) are all completed in the latent space.

[0085] Step 2: If Figure 2 As shown, the input of the lightweight latent diffusion model is the variational autoencoder in step 1. Encoder Potential data distribution after processing ,in, represents data with a noisy distribution, represents the current time step; first, select a time step that is relevant to image generation and has a high evaluation index The best performing pre-trained model among the existing pre-trained models is used as the teacher model, and a lightweight student model is designed based on the network architecture of the teacher model. The backbone networks of the teacher model and the student model are both Network architecture ,Should Network architecture It consists of an encoder (downsampling) stage, an intermediate stage, and a decoder (upsampling) stage. In each block of the encoder (downsampling) and decoder (upsampling) stages, the student model uses half the number of residual blocks of the teacher model, and the other parts of the architecture remain unchanged. Network architecture The resolution conversion operations in the training data (for example, the downsampling operation reduces the image resolution by half, and the upsampling operation increases the image resolution by twice the original) are used to divide the network architectures of the teacher model and the student model into two or more blocks based on the different levels of resolution, and each block corresponds to a specific resolution level.

[0086] Step 3: First, define the search space of the present invention, and build a student supernet according to the student model architecture designed in step 2; then, define the teacher model and the student model in a block-based manner, and then, as shown in Figure 2 As shown in the figure, in the process of training the student supernet, the intermediate real feature map of the teacher model is extracted as the supervision signal in the student supernet training process, and compared with the feature map generated by the corresponding block of the student model. By minimizing the loss function, the network parameters of the student model are adjusted to gradually approach the feature expression ability of the teacher model.

[0087] Step 3.1: Define the teacher model and student model in blocks:

[0088] ,

[0089] ,

[0090] in, =0, 1, ..., -1 indicates the resolution level, there are level, The smaller it is, the higher the resolution level is and the larger the feature map resolution is; represents the teacher model, and They represent the resolution level in the teacher model. The encoder (downsampling) block and the decoder (upsampling) block of represents the intermediate stage of the teacher model between the encoder (downsampling) block and the decoder (upsampling) block; represents the student model, and They represent the resolution level in the student model. The encoder (downsampling) block and the decoder (upsampling) block of Represents the intermediate stage of the student model between the encoder (downsampling) block and the decoder (upsampling) block;

[0091] Step 3.2: Define the architecture search space of the student model and build a student supernet based on the student model network architecture designed in step 2; the search space refers to the set of design parameters available in the student model architecture, which is used to define potential architecture variants; the search space of the student model only includes the design options of the residual block in the student model network architecture: the convolution kernel size of the convolution layer in the residual block ranges from , the normalization method in the residual block selects group normalization or layer normalization; the student supernet is a network structure that contains all possible student model architectures. The student supernet can contain and support multiple different student model architectures in one training; the single path sampling method is used when training the student supernet, and a path is randomly selected each time for forward propagation and parameter update. Each path corresponds to a specific student model architecture; while training the student supernet, the real feature map of the teacher model is extracted, and the feature map generated by the student model is aligned with the real feature map of the corresponding block of the teacher model, so as to supervise and optimize each block of the student model, and calculate the mean square error between the intermediate feature map of the teacher model and the feature map generated by the corresponding block of the student model loss:

[0092] Loss function of the student supernet training process Designed to:

[0093] ,

[0094] , =0,1,...,2 -1,

[0095] in, =0,1,...,2 -1 indicates 2 in the student model network architecture blocks, The resolution level defined in step 3.1 The encoder (downsampling) block and decoder (upsampling) block of the teacher model and the student model are considered as separate blocks. In the network architecture of the teacher model and the student model, each resolution level corresponds to an encoder (downsampling) block and a decoder (upsampling) block, so the total number of blocks is 2. , which is also equal to 2 in value ; Among them, the serial number index is Before The blocks are encoder (downsampling) blocks, which are used to gradually extract and compress features. The serial number index is After The first block is the decoder (upsampling) block, which is used to gradually restore the resolution and generate output features;

[0096] Indicates the current The subnetwork architecture being trained in the block, Indicates the student model The input feature map of each block comes from the teacher model. Indicates when The encoder (downsampling) block belonging to the student model hour, The teacher model Output of blocks ; Indicates when Decoder (upsampling) block belonging to the student model hour, Is a teacher model Network architecture Middle Skip connection features of blocks With the teacher model Output feature map of each block The splicing result ; In the student model blocks, using the currently selected subnetwork architecture For the input feature map The feature map generated after forward propagation; Indicates the first The real feature map of each block is used to supervise and guide the training of the student supernet;

[0097] Step 3.3: By minimizing the loss function To optimize each block in the student model Trainable network parameters , so that the student model performs better during the training process, the training data set used in the training process of the present invention Public datasets from the field of image generation ; For network architecture parameters The optimization process is updated by back propagation. This optimization process is usually performed in an iterative manner. Through multiple rounds of optimization, the student model gradually approaches the feature expression ability of the teacher model. The optimization process is expressed as:

[0098] ,

[0099] , =0, 1, ..., 2 -1,

[0100] in, It is the first The trainable network parameters of each block, including the weight matrix used for convolution operations, such as convolution kernel weights, and scaling parameters and offset parameters in normalization layers; In the student model The ultimate goal that we hope to achieve during the optimization process of each block is: Indicates The architecture search space of blocks;

[0101] The optimization process is to adjust the Trainable network parameters for blocks , minimize the loss function , thereby obtaining the current block The optimal network parameters ,Right now ,in Indicates that in the current The architecture search space of blocks In the example, use the current network parameters , calculate the student model Output of blocks With the teacher model The real feature map of the block The error between .

[0102] Step 4: Figure 3 As shown, in each block of the student model, a population of architectures is randomly initialized, and then the architectures with the serial index in the front are randomly initialized. The individuals are truly evaluated and the corresponding fitness values ​​are obtained. ; Then, in the first generation of the prediction evaluation phase, the historical record dictionary obtained through the real evaluation is used To train a simple Network as score predictor , when the number of iterations reaches a certain number of generations, it does not directly rely on the real evaluation, but through the trained score predictor To directly predict the scores of all valid individuals in the population. It includes the following steps:

[0103] Step 4.1: Initialize an architecture population as the initial population , defines the number of architectures to be evaluated in each generation , Total number of search iterations , the number of true evaluations in each iteration ;

[0104] Step 4.2: For the initial population ( / Subpopulation( The encoding operation in the evolutionary algorithm is to represent the individual in a form that is convenient for algorithm operation. The present invention uses the integer encoding method in the existing encoding technology to represent the individuals in each block of the population, and removes the individuals with repeated and invalid encoding in the population. Because the present invention only truly evaluates some individuals in the population, it will first be based on the current number of iterations. and the number of true evaluations required in each generation The size relationship between them is divided into two stages, namely the real evaluation stage ( ) and the prediction evaluation phase ( );like , then the sequence index in the current evaluation population will be The individuals are evaluated truly, and the evaluation values ​​are List of fitness values ​​recorded in the current population In the equation, the encoding of the individual being evaluated and the fitness value corresponding to the individual being evaluated are both Recorded in a history dictionary In the above example, the fitness values ​​of individuals that have not been truly evaluated are set to 0 and recorded in The fitness value of the present invention in the real evaluation stage ( ) is calculated as:

[0105] , =0,1,...,2 -1,

[0106] ,

[0107] ,

[0108] Among them, the benchmark network Represents the subnetwork architecture in the student model with the same architecture as the teacher model. The architecture search space in blocks , the present invention selects the architecture search space Loss assessment Outperforms the baseline network The subnetwork architecture for evaluating the loss , and then in the filtered sub-network architecture Further search for the sub-network architecture with the smallest fitness value ;

[0109] Restricted to, is a relaxation factor used to control the The sub-network architecture selected from With the benchmark network The degree of relaxation of the loss value between Indicates that in the validation data set On the network, use the currently trained sub-network architecture The obtained student model Output of blocks With the teacher model The real feature map of the block The error between Cannot be larger than the baseline network The obtained student model Output of blocks With the teacher model The real feature map of the block The error between , validation dataset From a public dataset separated from face images; among them, = , used to calculate from the current block The sub-network architecture selected from The real feature map of the corresponding block of the teacher model In the validation dataset losses; = , used to calculate the benchmark network The real feature map of the corresponding block of the teacher model In the validation dataset losses;

[0110] Indicates the current block The sub-network architecture selected from The relevant cost calculation function is mainly calculated by the number of multiplication and addition operations ( ) indicators to measure The size of the cost, Indicates that from The architecture search space of blocks Select cost Minimal architecture ; Fitness function For the selected architecture The computational cost A negative value of indicates that the lower the cost, the greater the fitness value of the current architecture, and the better the corresponding architecture;

[0111] Current iteration number achieve When , the system enters the first generation of the prediction evaluation phase, using the historical record dictionary obtained in the real evaluation To train a multilayer perceptron As a score predictor , at this point, the fitness value no longer depends entirely on the true evaluation, but is instead obtained through the score predictor To predict the fitness value of individuals; however, in the first generation of the prediction evaluation phase When the real evaluation range and the predicted evaluation range are inconsistent, the first generation of individuals need to use the trained score predictor Re-evaluate once; When , we directly use the score predictor To predict the scores of all valid individuals in the population.

[0112] Step 5: In the process of evolving offspring, the existing technology of uniform crossover operator based on integer coding and mutation operator based on random uniform selection is used to perform crossover and mutation operations on the parent architecture to generate offspring, so as to explore potential excellent architectures; in the crossover process, the architectural features of the parent are randomly combined to generate new offspring architectures, and in the mutation process, some coding bits are randomly changed, which increases the diversity of the architecture.

[0113] Step 6: After the evolution of the offspring architecture, the fitness value of the offspring population individuals is calculated using the same evaluation method as in step 4.2 After the fitness value evaluation operation of the newly generated offspring architecture is completed, the offspring architecture population generated by evolution is merged with the parent architecture population to form a population called the extended population. ,at this time, The size is 2 ,in, is the fixed value size of the population in each generation; then the elite selection strategy is used to sort all architectures according to the fitness value, and only the expanded population is retained after the sorting is completed Medium fitness value (usually the first ) architecture as a new generation population , as the basic population for the next round of evolution and evaluation; this method can not only ensure the inheritance of excellent architectures and avoid the loss of excellent individuals during crossover and mutation, but also gradually improve the overall performance of the population by selecting high-quality architectures.

[0114] Step 7: Repeat steps 5 to 6 for each block of the student model until the stopping condition is reached (that is, when the number of population evolution iterations reaches the upper limit of the defined total number of search iterations), and then stop the search; then, select the architecture with the highest fitness value from each block of the student model as the optimal architecture finally searched in each block of the student model.

[0115] Step 8: Figure 3 As shown, the optimal architecture finally selected from each block of the student model is connected in order from small to large in block order to form a complete network architecture, which is called a subnet. In the process of splicing the network architecture, it is necessary to ensure that the input and output dimensions of each block match to maintain the coherence and accuracy of the network architecture.

[0116] Step 9: Figure 4 As shown, a dynamic joint loss function is used to retrain the subnet obtained in step 8 in the latent space. The following steps are included:

[0117] Step 9.1: Design a method based on distillation losses and noise prediction loss Dynamic joint loss function composed of , of which, distillation loss Used to measure the distance between the student model and the teacher model on the intermediate feature map, supervise and encourage the student model to imitate the excellent representation ability of the teacher model; Noise prediction loss It is used to measure the error between the noise size predicted by the student model and the actual noise size added, ensuring that the student model can generate high-quality images; by dynamically adjusting the weights of these two parts of the loss, the distillation loss is used in the initial training. Based on the noise, the loss is gradually transitioned to The dynamic joint loss function is mainly used to make the student model further optimize its own task performance while imitating the teacher model, thus achieving a good balance between performance and generalization ability. for:

[0118] (1 ) ,

[0119] ,

[0120] , =0,1,...,2 -1,

[0121] in, Is used to balance the distillation losses and noise prediction loss The weight coefficient of the present invention is In the initial stage of subnet retraining, the weight coefficient is set to 0. As the retraining progresses, the weight coefficient The value of eventually grows to 1; is used to balance the noise prediction loss and distillation losses A parameter for the learning rate between Indicates the actual noise level added during the noise addition process in step 1, is a trainable network parameter in the subnet used to predict the noise in the data with noise distribution, represents the time step of the lightweight latent diffusion model during the denoising process, represents the noise data distribution, where represents the mean of the noise data distribution, represents the variance of the noise data distribution;

[0122] =0,1,...,2 -1, indicating that there are 2 blocks, It means to accumulate the distillation loss between the teacher model and the student model in all blocks, where , and the loss function set in step 3 Same; distillation loss It is a mean error based The loss function is used to calculate the generated feature map of the student model in all blocks of the subnet The true feature map with the teacher model The error between

[0123] Step 9.2: The present invention uses a step function to dynamically adjust the weight coefficient , in the early stage of subnet retraining, The value of will remain at 0 until a certain number of training steps have passed. back, The value of will suddenly jump to 1; on the one hand, the use of this step function adjustment method can save the time of subnet retraining and avoid excessive calculations and parameter adjustments; on the other hand, the use of step functions can also ensure the effect of subnet retraining.

[0124] Step 10: Use the retrained subnet in step 9 in the latent space to predict the noise component in the data with noise distribution and remove it, gradually obtain a representation closer to the real data distribution, and then use the variational autoencoder Decoder in The latent data representation is mapped back to the original pixel space, and the generated image samples are finally output, completing the image generation task of the lightweight latent diffusion model.

[0125] like Figure 5 As shown, the face image generated by the method of the present invention is obviously better than Figure 1 The classic latent diffusion model in the same face dataset The results of generating face images under the dataset. Compared with the classic latent diffusion model method, the sampling results in this invention are not only more outstanding in visual intuitive effect, but also have better image evaluation indicators. In addition, compared with the classic latent diffusion model as the teacher model, the network parameters of the lightweight latent diffusion model in the present invention are relatively reduced. , which effectively reduces computing and storage costs and significantly improves operational efficiency.

[0126] Table 1 shows the classical latent diffusion model In different datasets ( Datasets and Dataset) and comparative experimental results under different network architecture settings. ( ) dataset is a high-quality public dataset of facial images, which is widely used in the research of image generation, image editing and face-related tasks in the field of computer vision. Selected high-quality face images, each with a resolution of ; The dataset was proposed by the University of Toronto in Canada. It is a classic natural image dataset and is widely used in the research of image classification and generative models. The dataset contains The resolution size is A color image of Zhang is used for training, Zhang is used for testing, The images in the dataset are divided into categories, namely airplane, car, bird, cat, deer, dog, frog, horse, boat and truck, each category contains images.

[0127] Table 1

[0128] Latent Diffusion Model Dataset Evaluation index (FID) GPU days LDM(f=8,ddim-250) CelebA-HQ 9.67 10.2 LDM(f=8,ddim-250) CIFAR-10 N / A N / A LDM(f=8,ddim-250,parameters changed) CIFAR-10 26.94 0.66 LDM(f=8,ddim-250,updownsample modified) CIFAR-10 12.16 0.5 LDM(f=8,ddim-250,updownsample modified) CelebA-HQ 326.74 14.0

[0129] This experiment is based on the classic latent diffusion model Expanding, the classic latent diffusion model uses a downsampling factor The setting is through the variational autoencoder Encoder After mapping the input image from the original pixel space to the latent space, the resolution of the image is reduced to the original image , which significantly reduces the amount of calculation and improves the operating efficiency of the model. The sampling process uses method, namely, using the denoising diffusion implicit model , and the sampling steps Set to This method can generate images in fewer time steps than the traditional diffusion model. The specific experimental results are summarized as follows:

[0130] The first row of experimental results: Based on the classic potential diffusion model, the data set is , the face image actually generated by sampling is as follows Figure 1 As shown in the figure, it can be seen that the classic latent diffusion model has shown excellent performance in the field of face generation, but there are still some flaws in details such as eyes and teeth. The second row of experimental results: Based on the first row of experiments, only The dataset is changed to -10 data sets, other experimental parameters and architecture settings remain unchanged. and -10 The resolution levels of the datasets vary greatly (face image datasets The resolution of the image is , while the classic natural image dataset The resolution of the image is ), which causes an error in the potential diffusion model during runtime. The error message is " ",Right now -10 The dataset is downsampled by a factor of -10 in the latent diffusion model After the effect of , the resolution in the latent space is too small to generate valid output results. Therefore, the results of other related experiments cannot be obtained. The third row of experimental results: To solve the problem of the classical potential diffusion model in the second row of experiments -10 dataset, the number of channels and attention mechanism of the potential diffusion model were adjusted (the experiment was marked as " ”), the experiment can run normally, but the generation effect is not good, and the sampled image is blurry. The score is as high as 26.94; among them, The score is a common indicator for evaluating the quality and diversity of generated images. The lower the value, the higher the quality of the generated images. The fourth row of experimental results: Based on the third row of experiments, the upsampling and downsampling modules in the network architecture are further adjusted (the experiments are marked as " Specifically, the downsampling pooling operation in the classic potential diffusion model is replaced by a convolution layer with a convolution kernel size of 1, which can reduce the information loss of the data and better control the change of the feature map; at the same time, the interpolation module of the upsampling operation in the classic potential diffusion model is cancelled. The interpolation module is used to estimate the missing pixel values ​​through interpolation methods (such as nearest neighbor interpolation or bilinear interpolation) during the upsampling process, and fill the estimated values ​​into the target position, thereby generating a higher resolution image. The experimental results show that after the above adjustments to the upsampling and downsampling modules in the network architecture, the quality of the generated images is significantly improved, and the image evaluation indicators are The fifth row of experimental results: Based on the first row of experiments, the same network architecture adjustment method as the fourth row of experiments was applied. However, the sampling effect collapsed, and the image evaluation index It rises sharply to 326.74, indicating that this architecture adjustment method has great limitations in applicability in different data sets or task contexts. This result also further emphasizes the necessity of introducing automated neural network architecture search technology in the potential diffusion model so that it can better adapt to different task requirements, optimize the network architecture and improve the stability and performance of the potential diffusion model.

[0131] The above experimental results show that the potential diffusion model has poor generalization ability between different task backgrounds and datasets, and takes a long time to train. The longest training time is as long as The sampling effect of the last row of experiments collapsed, and the image evaluation index It rose sharply to 326.74, which further shows that it is difficult to achieve ideal sampling effects in different task contexts by manual design and trial and error methods alone. It also emphasizes the necessity of introducing automated neural network architecture search technology in the potential diffusion model; automated neural network architecture search technology can better adapt to different task requirements, optimize network architecture, and improve the stability and performance of the model to meet the challenges of the potential diffusion model in practical applications.

[0132] The present invention provides a lightweight potential diffusion model design method and system based on evolutionary neural architecture search. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A lightweight latent diffusion model design method based on evolutionary neural architecture search, wherein the lightweight latent diffusion model is used for image generation tasks, characterized in that: The following steps are involved: Step 1: Select a pre-trained variational autoencoder (VAE) and use the encoder E in the variational autoencoder (VAE) to transform the input image data from the original pixel space to the latent space. The input image data is represented by the latent data distribution in the latent space. Then, perform the noise addition process in the latent space until the latent data distribution reaches a pure noise state. Then, the operations from step 2 to step 8 are completed in the latent space in sequence, and after step 8 is completed, the denoising operation of pure noise data is performed using the network architecture obtained in step 8; Step 2: Select a pre-trained model related to image generation as the teacher model, and design the network architecture of the student model based on the network architecture of the teacher model. Then, divide the network architectures of the teacher model and the student model into two or more blocks based on the different resolution levels. Step 3, define the architecture search space of the student model, build a student supernet according to the student model architecture designed in step 2, and train the student supernet; Step 4: randomly initialize a population of architectures in each student model block, and evaluate the performance of individuals in the initial population in each block; Step 5: Perform crossover and mutation operations on the parent architecture to generate a child architecture, gradually expand the search space, and explore potential excellent architectures; Step 6: For the newly generated offspring population after the evolution operation, the fitness value of the population individuals is calculated using the same evaluation method as in step 4; after the fitness value of the newly generated offspring architecture is evaluated, the offspring architecture population generated by evolution is merged with the parent architecture population to form a population called the extended population P extended Then, the elite selection strategy is used to sort all architectures according to their fitness values, and only the expanded population P is retained after the sorting. extended The architecture with high medium fitness value is used as the new generation population P next , as the basic population for the next round of evolution and evaluation; Step 7: Repeat steps 5 to 6 for each block of the student model until the preset stop condition is met and the search stops; then, select the architecture with the highest fitness value from each block of the student model as the optimal architecture finally searched in each block of the student model; Step 8, connecting the optimal architecture finally searched in each block of the student model in order of block size from small to large to form a complete network architecture, which is called a subnet; Step 9, retraining the subnet obtained in step 8, and introducing a dynamic joint loss function during the subnet retraining process; the retraining refers to using the subnet searched in step 8 to perform a noise removal operation on the pure noise data on the training data set train following the operation in step 1; Step 10: Use the subnet retrained in step 9 to perform sampling to generate the final image. During the sampling process, the decoder D in the variational autoencoder VAE is used to map the potential data distribution obtained after the subnet removes noise back to the pixel space, and finally output the generated image sample. In step 2, a pre-trained model related to image generation is selected as the teacher model, and a lightweight student model is designed according to the network architecture of the teacher model. The backbone networks of the teacher model and the student model both adopt the U-type network architecture UNet, which includes an encoder stage, an intermediate stage, and a decoder stage. In each block of the encoder stage and the decoder stage, the number of residual blocks in the student model network architecture is only half of that in the teacher model network architecture; finally, the network architectures of the teacher model and the student model are divided into two or more blocks according to different resolution levels, wherein the blocks divided in the encoder stage are called encoder blocks, and the blocks divided in the decoder stage are called decoder blocks; Step 3 includes: Step 3.1, define the teacher model and student model in blocks: Where i=0, 1, ..., n-1 represents the level of resolution, and there are n levels in total; T represents the teacher model, and denote the encoder block and decoder block of resolution level i in the teacher model, respectively, and M t represents the teacher model in the intermediate stage between the encoder block and the decoder block; S represents the student model, and denote the encoder block and decoder block of resolution level i in the student model, respectively, and M s Represents the intermediate stage of the student model between the encoder block and the decoder block; Step 3.2, define the architecture search space of the student model, and build a student supernet according to the student model network architecture designed in step 2; the search space refers to the set of design parameters available in the student model architecture, which is used to define potential architecture variants; the search space of the student model only includes the design options of the residual block in the student model network architecture: the convolution kernel size of the convolution layer in the residual block ranges from {1,3,5}, and the normalization method in the residual block selects group normalization or layer normalization; the student supernet is a network structure that contains all possible student model architectures, and the student supernet can be trained in one time Contains and supports one or more different student model architectures; a single path sampling method is used when training the student supernet, and a path is randomly selected each time for forward propagation and parameter update, where each path corresponds to a specific student model architecture; in the process of training the student supernet, by extracting the real feature map of the teacher model, aligning the feature map generated by the student model with the real feature map of the corresponding block of the teacher model, each block of the student model is supervised and optimized, and the mean square error L2 loss between the intermediate feature map of the teacher model and the feature map generated by the corresponding block of the student model is calculated: The loss function L during the student supernet training process train for: Wherein, j = 0, 1, ..., 2m-1 represents 2m blocks in the student model network architecture, m is numerically equal to the resolution level n defined in step 3.1, and the encoder block and the decoder block are regarded as separate blocks. In the teacher model and the student model network architecture, each resolution level corresponds to an encoder block and a decoder block, so the total number of blocks is 2n, that is, the numerical size is also equal to 2m; wherein, the first m blocks with serial number index 0, 1, ..., m-1 are encoder blocks, which are used to gradually extract and compress features, and the last m blocks with serial number index m, m+1, ..., 2m-1 are decoder blocks, which are used to gradually restore the resolution and generate output features; α j represents the sub-network architecture being trained in the current j-th block, X j represents the input feature map of the jth block in the student model, which comes from the teacher model; Indicates when X j The encoder block belonging to the student model When X j is the output Y of the j-1th block of the teacher model j-1 ; Indicates when X j Decoder block belonging to the student model When X j is the skip connection feature Skip of the jth block in the U-type network architecture UNet of the teacher model j The output feature map Y of the j-1th block of the teacher model j-1 The splicing result Cat(Y j-1 ,Skip j ); Indicates that the currently selected sub-network architecture α is used on the jth block of the student model j For the input feature map X j The feature map generated after forward propagation; Y j represents the true feature map of the jth block in the teacher model, which is used to supervise and guide the training of the student supernet; Skip j It is the skip connection feature of the jth encoder block in the U-shaped network architecture UNet of the teacher model. In the U-shaped network architecture UNet of the teacher model, the skip connection Skip directly transfers the feature information extracted in the encoder stage to the corresponding block in the decoder stage. The U-shaped network architecture UNet of the student model also adopts the skip connection method. Step 3.3, by minimizing the loss function L′ train To optimize each block X in the student model j The trainable network parameters W j , the optimization process is expressed as: Among them, W j are the trainable network parameters of the jth block in the student model, including the weight matrix used for convolution operations, and the scaling parameters and offset parameters in the normalization layer; A represents the final goal to be achieved in the optimization process of the jth block of the student model. j represents the architecture search space of the jth block; The optimization process is to adjust the trainable network parameters W of the jth block j , minimize the loss function L′ train , thereby obtaining the current block X j The optimal network parameters Right now where L′ train (W j ,A j ; L train (X j ,Y j )) represents the architecture search space A in the current j-th block j In the example, the current network parameters W are used. j , calculate the output Block of the jth block of the student model α (X j ) and the true feature map Y of the jth block of the teacher model j The error between Step 9 includes: Step 9.1, retrain the subnet obtained in step 8 in the latent space using a dynamic joint loss function; Step 9.2, design an adjustment function for the weight coefficient β in the dynamic joint loss function to smoothly control the weight of the loss function; Step 9.1 includes: the dynamic joint loss function L retrain for: L retrain =l·(1-β)L dis +βL ori , Where β is used to balance the distillation loss L dis and noise prediction loss L ori The weight coefficient β is set to 0 at the beginning of subnet retraining, and the value of the weight coefficient β eventually increases to 1 as the retraining progresses; l is used to balance the noise prediction loss L ori and distillation loss L dis A parameter of the learning rate between steps; ∈ represents the actual noise size added in the noise addition process in step 1, ∈ θ is a trainable network parameter in the subnet used to predict the noise in the data with noise distribution, t represents the current time step in the denoising process, α t x+σ t ∈ represents the noise data distribution, where α t represents the mean of the noise data distribution, σ t represents the variance of the noise data distribution, and the noise prediction loss L ori It is a loss function based on the mean square error L2, which is used to measure the accuracy of the subnetwork in predicting the noise level in data with noisy distribution; ∑ j L train It means to accumulate the distillation loss between the teacher model and the student model in all blocks, where Distillation loss L dis It is a loss function based on the mean square error L2, which is used to calculate the generated feature map of the student model in all blocks of the subnet The true feature map Y of the teacher model j The error between Step 9.2 includes: adjusting the coefficient β using a step function.

2. The method according to claim 1, characterized in that Step 4 includes: Step 4.1, initialize an architecture population within each block of the student model as the initial population P, define the number S of architectures for true evaluation in each generation of the population, the total number T of search iterations, and the number Ng of true evaluations in each iteration; Step 4.2, define the current iteration number G and the sub-population Q in the evolutionary architecture search, and evaluate the initial population P.

3. The method according to claim 2, characterized in that Step 4.2 includes: Step 4.2.1, encode the individuals in the initial population P, and standardize the encoding of each individual in the population. Use integer encoding to represent the population individuals in each block. After encoding, remove the duplicate and invalid individuals in the population. The remaining individuals are called valid individuals; according to the size relationship between the current iteration number G and the number Ng of true evaluations required in each iteration, the evaluation stage is divided into two stages: the true evaluation stage when G < Ng, and the prediction evaluation stage when G ≥ Ng; Step 4.2.2, if G < Ng, perform true evaluations on the individuals with serial indices in the first S in the current evaluation population, record the obtained fitness value Fitness in the fitness value list Q.Fitness, and at the same time record the encoding of the individuals being truly evaluated and the fitness value Fitness corresponding to the individuals being truly evaluated in a historical record dictionary History. Set the fitness values of the individuals not being truly evaluated to 0 and record them in the fitness value list Q.fitness; In the real evaluation phase, the subnetwork architecture in the student model with the same architecture as the teacher model is called the benchmark network α base , for the architecture search space A in the jth block of the student model j , select the architecture search space A j Evaluation loss L′ val Outperforms the baseline network α base The subnetwork architecture for evaluating the loss Then in the selected sub-network architecture Further search for the sub-network architecture with the smallest fitness value Among them, the calculation formula of fitness value Fitness is: Among them, st means restricted by, γ is a relaxation coefficient used to control the j The sub-network architecture selected from Compared with the baseline network α base The degree of relaxation of the loss value between Indicates that the currently trained sub-network architecture is used on the validation dataset val The output of the jth block of the student model is obtained The true feature map Y of the jth block of the teacher model j The error between Cannot be greater than the baseline network α base The output of the jth block of the student model is obtained The true feature map Y of the jth block of the teacher model j The error between in, Used to calculate from the current block X j The sub-network architecture selected from The real feature map Y of the corresponding block of the teacher model j The loss between the validation dataset val Used to calculate the baseline network α base The real feature map Y of the corresponding block of the teacher model j The loss between the validation dataset val Indicates the current block X j The sub-network architecture selected from Related cost calculation functions, Represents the architecture search space A from the jth block j Select cost Minimal architecture Fitness function For the selected architecture The computational cost Negative value of Step 4.2.3, in the first generation of the prediction evaluation stage, i.e., when G = Ng, use the historical record dictionary History obtained in Step 4.2.2 to train a multi-layer perceptron MLP as a score predictor M, and the parent population needs to be re-evaluated using the score predictor M; when G > Ng, directly use the trained score predictor M to score all valid individuals in the population.

4. The method according to claim 3, characterized in that In Step 10, use the subnet retrained in Step 9 to predict the noise component in the current data, remove the noise component, gradually obtain a representation closer to the true data distribution, and then use the decoder D in the variational autoencoder VAE to map the latent representation obtained after removing the noise back to the pixel space, and finally output the generated image sample to complete the image generation task of the lightweight latent diffusion model.

5. A system implemented by the method according to any one of claims 1 to 4, characterized in that: Includes: Student supernet training module based on knowledge distillation: used to train a student model; Evolutionary neural architecture search module based on a score predictor: used to train a multi-layer perceptron MLP as a score predictor for evaluating the architecture performance; Retraining module based on a dynamic joint loss function: used to retrain the subnet obtained during the architecture search process; adopt a dynamic joint loss function that combines the distillation loss and the noise prediction loss, and adjust the weights of different loss functions.

Images