Model determination method and apparatus, image generation method and apparatus, and device and medium

Abstract

Disclosed in the present disclosure are a model determination method and apparatus, an image generation method and apparatus, and a device and a medium. The model determination method comprises: according to a pre-constructed diffusion model, constructing a search space for providing at least one candidate time step sequence; then searching the search space according to a preset number-of-steps constraint and a preset performance condition, so as to obtain a search result, such that performance representation data of a model constructed on the basis of the search result meets the preset performance condition, and a target time step sequence in the search result meets the preset number-of-steps constraint; and then determining an image generation model by using the model constructed on the basis of the search result.

Description

Model determination method, image generation method, device, equipment, medium

[0001] This application claims priority to Chinese Patent Application No. 202310786950.0 filed on June 29, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to a model determination method, an image generation method, an apparatus, a device, and a medium. Background Art

[0003] The diffusion model is an image generation model, and the diffusion model can be applied in many fields (for example, image generation, image super-resolution reconstruction, text image generation, etc.).

[0004] However, because the diffusion model divides the image generation process into multiple time steps, and each time step requires calling a denoising neural network, the image generation speed of the diffusion model is relatively slow, resulting in a relatively long image generation time consumption.

[0005] Summary of the Invention

[0006] The present disclosure provides a model determination method, an image generation method, an apparatus, a device, and a medium.

[0007] The technical solutions provided by this disclosure are as follows:

[0008] The present disclosure provides a model determination method, the method comprising:

[0009] Constructing a search space based on a pre-constructed diffusion model, wherein the search space is used to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model;

[0010] Searching the search space according to a preset step number constraint and a preset performance condition to obtain search results, the search results including a target time step sequence, the target time step sequence satisfying the preset step number constraint, performance characterization data of a model constructed based on the search results satisfying the preset performance condition, and the number of steps defined by the preset step number constraint being less than the number of time steps in the reference time step sequence;

[0011] An image generation model is determined using the model constructed based on the search results.

[0012] In one possible implementation, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence, the candidate network structure including at least one feature channel of at least one network layer in the denoising neural network in the diffusion model;

[0013] The searching of the search space according to the preset step number constraint and the preset performance condition to obtain search results includes:

[0014] The search space is searched according to a preset step number constraint, a preset structural constraint, and a preset performance condition to obtain search results, wherein the search results also include a target network structure corresponding to each time step in the target time step sequence, and the target network structure satisfies the preset structural constraint.

[0015] In one possible implementation, the process of determining the search results includes:

[0016] Randomly sampling a number of individuals from the search space, wherein the individuals include a sampling time step sequence, and the sampling time step sequence satisfies the preset step number constraint;

[0017] Initializing a population based on the individuals;

[0018] Determining a ranking result of individuals corresponding to the population based on the fitness of each individual in the population; the fitness of the individual is determined based on performance characterization data of a model constructed based on the individual;

[0019] Based on the individual ranking results, the population is evolved, and the step of determining the individual ranking results corresponding to the population based on the fitness of each individual in the population is continued, until the search stop condition is reached, and the search result is determined from the population based on the preset performance condition.

[0020] In one possible implementation, the evolution process includes:

[0021] Deleting individuals that meet a preset deletion condition from the population according to the individual sorting result;

[0022] The time steps within at least one pair of individuals in the population are randomly exchanged, and / or the time steps within the individuals in the population are modified according to a preset probability.

[0023] In one possible implementation, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence;

[0024] The individual also includes a sampling network structure corresponding to each time step in the sampling time step sequence, and the sampling network structure satisfies the preset structural constraints;

[0025] The randomly exchanging time steps of at least one pair of individuals in the population comprises:

[0026] Randomly exchanging information combinations within at least one pair of individuals in the population, the information combinations including corresponding time steps and network structures existing within the individuals;

[0027] The modifying of the time steps within the individuals in the population according to the preset probability includes:

[0028] The time steps and / or network structures within the individuals in the population are modified according to a preset probability.

[0029] In one possible implementation, the performance characterization data is determined based on a reference image and a first generated image generated by a model constructed based on the search results.

[0030] In a possible implementation manner, the diffusion model is constructed based on a training image; the reference image includes the training image;

[0031] and / or,

[0032] The reference image includes a second generated image generated by the diffusion model.

[0033] The present disclosure provides an image generation method, the method comprising:

[0034] Get noise data;

[0035] An image generation model is used to perform image generation processing on the noise data to obtain a third generated image; the image generation model is determined using the model determination method provided by the present disclosure.

[0036] The present disclosure provides a model determination device, comprising:

[0037] A construction unit is configured to construct a search space based on a pre-constructed diffusion model, wherein the search space is configured to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model;

[0038] A search unit is configured to search the search space according to a preset step number constraint and a preset performance condition to obtain a search result, wherein the search result includes a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search result satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps in the reference time step sequence;

[0039] A determination unit is used to determine an image generation model using the model constructed based on the search results.

[0040] The present disclosure provides an image generating device, comprising:

[0041] an acquisition unit, used for acquiring noise data;

[0042] A generating unit is configured to perform image generation processing on the noise data using an image generation model to obtain a third generated image; the image generation model is determined using the model determination method provided in the present disclosure.

[0043] The present disclosure provides an electronic device, the device comprising: a processor and a memory;

[0044] The memory is used to store instructions or computer programs;

[0045] The processor is configured to execute the instructions or computer programs in the memory, so that the electronic device executes the model determination method or image generation method provided by the present disclosure.

[0046] The present disclosure provides a computer-readable medium having instructions or a computer program stored therein. When the instructions or the computer program are executed on a device, the device executes the model determination method or the image generation method provided by the present disclosure.

[0047] The present disclosure provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, wherein the computer program contains program code for executing the model determination method or image generation method provided by the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings required for the description are briefly introduced. Obviously, the accompanying drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0049] FIG1 is a flow chart of a model determination method provided by an embodiment of the present disclosure;

[0050] FIG2 is a schematic diagram of a model determination process provided by an embodiment of the present disclosure;

[0051] FIG3 is a flow chart of an image generation method provided by an embodiment of the present disclosure;

[0052] FIG4 is a schematic structural diagram of a model determination device provided by an embodiment of the present disclosure;

[0053] FIG5 is a schematic structural diagram of an image generating device provided by an embodiment of the present disclosure; and

[0054] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] Research has found that because the diffusion model divides the image generation process into multiple time steps, and each time step requires a call to the denoising neural network, the diffusion model's image generation speed is relatively slow, resulting in a relatively long image generation time consumption. Therefore, for the diffusion model, the image generation speed of the diffusion model is related to the number of time steps involved in the generation process of the diffusion model. Specifically, the larger the number of time steps involved in the generation process of the diffusion model, the slower the image generation speed of the diffusion model; the smaller the number of time steps involved in the generation process of the diffusion model, the faster the image generation speed of the diffusion model. Based on this, it can be seen that the present disclosure can increase the image generation speed by reducing the time step, thereby reducing the image generation time consumption.

[0056] Based on the above research, it can be known that in order to better reduce the time consumption of image generation, the present disclosure provides a model determination method (that is, a model compression scheme), which may specifically include: for any pre-constructed diffusion model, first construct a search space based on the diffusion model, so that the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; then search the search space based on a preset step number constraint (for example, the constraint of 10 time steps) and a preset performance condition (for example, the image generation performance reaches a preset threshold, or the image generation performance reaches the best, etc.), and obtain a search result, so that the performance characterization data of the model constructed based on the search result meets the preset performance condition, and the target time step sequence in the search result meets the preset Step number constraint. At this time, the number of steps limited by the preset step number constraint is smaller than the number of time steps of the reference time step sequence, so that the number of steps of the target time step sequence in the search result is smaller than the number of time steps of the reference time step sequence, thereby making the number of time steps in the model constructed based on the search result smaller than the number of time steps in the diffusion model; then, using the model constructed based on the search result, an image generation model is determined, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model, so that compression processing can be performed on the diffusion model while ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption.

[0057] In addition, the present disclosure does not limit the execution subject of the model determination method. For example, the model determination method provided in the embodiment of the present disclosure can be applied to a device with data processing function such as a terminal device or a server. For another example, the model determination method provided in the embodiment of the present disclosure can also be implemented with the help of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a stand-alone server, a cluster server or a cloud server.

[0058] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0059] To better understand the technical solutions provided by the present disclosure, the model determination method provided by the present disclosure is described below with reference to some figures. As shown in Figure 1, the model determination method provided by an embodiment of the present disclosure includes the following steps S101-S103. Figure 1 is a flow chart of a model determination method provided by an embodiment of the present disclosure.

[0060] S101: constructing a search space based on a pre-constructed diffusion model, where the search space is used to provide at least one candidate time step sequence, where the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model.

[0061] The diffusion model is used to perform image generation processing on the input data of the diffusion model (for example, noise data); and the present disclosure is not limited to the diffusion model. For example, it can be implemented using any existing or future trained diffusion model with image generation function (for example, a pre-trained diffusion model, etc.).

[0062] In addition, for the diffusion model above, the diffusion model may include the time step sequence required for the image generation process (e.g., [t T ,t T-1 ,…,t1] time step sequence) and denoising neural network, so that the diffusion model can divide the image generation process into data processing stages corresponding to each time step according to the time step sequence, and the data processing stage corresponding to each time step in the time step sequence is implemented by calling the denoising neural network. It can be seen that when the diffusion model includes [t T ,t T-1 ,…,t1], the diffusion model can implement the image generation process by calling the denoising neural network T times. The denoising neural network is used to denoise the input data of the denoising neural network. The present disclosure does not limit the implementation of the denoising neural network. For example, it can be implemented using a neural network with denoising capabilities within any existing or future diffusion model. T is a positive integer representing the length of the reference time step sequence (i.e., the number of time steps).

[0063] The reference time step sequence is used to represent the time step sequence required when the diffusion model above performs the image generation process (for example, [t T ,t T-1 ,…,t1] this time step sequence).

[0064] The search space refers to the search space required for compression processing on the above diffusion model (for example, the search space shown in FIG2 ).

[0065] In addition, for the above search space, the search space can be determined based on the diffusion model that needs to be compressed, so that the search space is used to provide some compression results of the diffusion model (for example, compression results presented in terms of time steps and / or network structures), so that a compression model with better image generation performance can be screened out from these compression results, thereby realizing compression processing for the diffusion model.

[0066] In addition, the present disclosure does not limit the implementation method of the above search space. For example, in some application scenarios (for example, when compression processing is achieved by reducing the time step), the search space can be used to provide at least one candidate time step sequence. The candidate time step sequence is used to represent a time step sequence that can be sampled from the search space; and the present disclosure does not limit the candidate time step sequence. For example, the candidate time step sequence can refer to a time step sequence sampled from the above reference time step sequence, so that the candidate time step sequence can include at least one time step in the reference time step sequence. For another example, the candidate time step sequence can refer to a time step sequence sampled from the above reference time step sequence according to a preset step number threshold, so that the number of time steps in the candidate time step sequence does not exceed the preset step number threshold. The preset step number threshold can be pre-set, for example, the preset step number threshold can be any value that is less than the number of time steps in the reference time step sequence. It can be seen that when the reference time step sequence is [t T ,t T-1 ,…,t1], the “at least one candidate time step sequence” may include at least the candidate time step sequence shown in the following formula (1).

[0067]

[0068] Where, represents a candidate time step sequence of length K; t′ K Indicates the time step that is ranked most forward among the K time steps sampled from the reference time step sequence above, t′ K-1 represents the second-most advanced time step among the K time steps sampled from the reference time step sequence, ..., t′1 represents the most advanced time step among the K time steps sampled from the reference time step sequence, t′ i Used to represent any one of the K time steps sampled from the reference time step sequence; time step t′ i+1 With time step t′ i adjacent, and time step t′ i+1 The corresponding arrangement position is greater than the time step t′ iThe corresponding arrangement position is at the front; [tT, tT -1 ,…,t1] represents the reference time step sequence, K is a positive integer, K<T.

[0069] Based on the above content, it can be seen that in one possible implementation, the above search space can be based on the above reference time step sequence (for example, [tT, tT -1 ,…,t1] time step sequence) is determined (for example, the search space includes the reference time step sequence), so that the search space can provide the time step sequence and the sampling results for the time step sequence (for example, [t′ K ,t′ K-1 ,…,t′1], so that the final time step sequence required can be determined by sampling and searching the search space.

[0070] It should be noted that the present disclosure does not limit the sampling method for the candidate time step sequence above. For example, in order to better improve the sampling effect, the time step in the candidate time step sequence can refer to at least one time step arbitrarily sampled from the reference time step sequence above. For example, when the reference time step sequence is [t 100 ,t 99 ,…,t1], we can 100 ,t 99 ,…,t1] and randomly sample some time steps (e.g., t 74 , t 53 , t 21 , t6, etc.) to construct a candidate time step sequence (e.g., [t 74 ,t 53 ,t 21 ,t6] this time step sequence).

[0071] S102: Search the search space according to a preset step constraint and a preset performance condition to obtain search results, where the search results include a target time step sequence, the target time step sequence satisfies the preset step constraint, performance characterization data of a model constructed based on the search results satisfies the preset performance condition, and the number of steps limited by the preset step constraint is less than the number of time steps in the reference time step sequence.

[0072] The preset step number constraint refers to the constraint required when performing time step compression processing on the diffusion model described above; and the number of steps limited by the preset step number constraint is less than the number of time steps in the reference time step sequence described above. In addition, the preset step number constraint can be set in advance based on the application scenario. For example, when the number of time steps in the reference time step sequence is T, the preset step number constraint can be: the number of time steps in the final compressed model time step sequence is K (that is, the length of the final compressed model time step sequence is K), where K is a positive integer and K<T.

[0073] The preset performance condition refers to the constraint required in terms of image generation performance when searching the above search space; and the preset performance condition can be set in advance according to the application scenario. For example, the preset performance condition can specifically be: the image generation performance reaches the maximum, or the image generation performance reaches a preset threshold.

[0074] The search results are used to represent objects (eg, time step sequences) obtained from the search space above and satisfying both the preset step number constraint and the preset performance condition above.

[0075] In addition, the present disclosure is not limited to the above search results. For example, in some application scenarios (e.g., when compression processing is achieved by reducing the time step), the search results may include a target time step sequence. The target time step sequence refers to a candidate time step sequence obtained from the search space above and meeting the two constraints of the preset step number constraint and the preset performance condition. Since each candidate time step sequence provided by the search space is obtained from any sampled time step in the reference time step sequence above, the search space can represent the time step sampling results for the reference time step sequence as comprehensively as possible, thereby enabling the search space to represent as comprehensively as possible the time step compression schemes that can be selected when performing time step compression on the diffusion model above, and further enabling the target time step sequence finally searched from the search space to better represent the search for the optimal sequence under the premise of meeting the preset step number constraint, which is conducive to improving the compression effect on the diffusion model.

[0076] Based on the above content, it can be seen that in some application scenarios (for example, when compression processing is achieved by reducing the time step), the target time step sequence mentioned above not only satisfies the preset step number constraint, but also the performance characterization data of the model constructed based on the target time step sequence meets the preset performance conditions. Among them, the "model constructed based on the target time step sequence" can be used to perform image generation processing on the input data of the model (for example, a noise data). It should be noted that the determination process of the "performance characterization data of the model constructed based on the target time step sequence" is similar to the determination process of the "performance characterization data of the model constructed based on the search results" below. For the sake of brevity, it will not be repeated here.

[0077] In addition, the present disclosure does not limit the construction method of the "model constructed based on the target time step sequence" in the above paragraph. For example, it can be specifically: based on the target time step sequence and the denoising neural network in the above diffusion model, construct the "model constructed based on the target time step sequence", so that the constructed "model constructed based on the target time step sequence" includes the target time step sequence and the denoising neural network, so that the working principle of the "model constructed based on the target time step sequence" is specifically: the "model constructed based on the target time step sequence" divides the image generation process into data processing stages corresponding to each time step according to the target time step sequence, and the data processing stage corresponding to each time step in the target time step sequence is realized by calling the denoising neural network.

[0078] Based on the above two paragraphs, it can be seen that in one possible implementation, the "model constructed based on the search results" in S102 above can be implemented using the "model constructed based on the target time step sequence" shown in the above two paragraphs.

[0079] The "performance characterization data of the model constructed based on the search results" described above is used to characterize the image generation performance of the "model constructed based on the search results." Furthermore, this disclosure does not limit the method for determining this performance characterization data. For example, it can be implemented using any existing or future method capable of evaluating the image generation performance of a diffusion model, such as the Fréchet Inception Distance (FID) scoring metric. For ease of understanding, the following examples are provided for illustration.

[0080] As an example, the above process of determining “performance characterization data of a model constructed based on search results” may specifically include the following steps 11 to 13.

[0081] Step 11: Get a reference image.

[0082] Here, the reference image refers to image data required to be referenced when performing image generation performance determination processing on a model (for example, the reference image shown in FIG. 2 ).

[0083] In addition, the present disclosure does not limit the implementation of the above reference image. For ease of understanding, some situations are described below.

[0084] Case 1: In some application scenarios, a model's image generation performance can be evaluated using real images. Based on this, this disclosure provides a possible implementation of the reference image described above. In this implementation, when the diffusion model described above is constructed based on training images, the reference image can include the training image.

[0085] Case 2: In some application scenarios, to ensure that the image generation performance of the compressed model is as close as possible to that of the diffusion model, the image generation performance of a compressed model can also be evaluated using image data generated by the diffusion model. Based on this, the present disclosure provides a possible implementation of the reference image described above, in which the reference image can include a second generated image generated by the diffusion model. The second generated image refers to image data generated by the diffusion model.

[0086] Case 3: In some application scenarios, the image generation performance of a compressed model can be evaluated using both real images and image data generated by the diffusion model. Based on this, the present disclosure provides a possible implementation of the reference image described above. In this implementation, when the diffusion model is constructed based on a training image, the reference image can include the training image and a second generated image generated by the diffusion model.

[0087] Furthermore, the present disclosure does not limit the number of reference images mentioned above, which can be determined based on actual application scenarios. For example, when the present disclosure uses the FID score metric to characterize the image generation performance of a model, the number of reference images can be 50,000.

[0088] Research has found that, because the search process only requires determining the relative size of the image generation performance of two models, rather than accurately determining the image generation performance of each model, a rough estimate of a model's image generation performance can be used to improve search efficiency. Based on this, the present disclosure can speed up searches by reducing the number of reference images. Thus, in one possible implementation, when the present disclosure uses the FID score metric to characterize a model's image generation performance, the number of reference images can be 5,000.

[0089] Step 12: Obtain a first generated image generated by the model constructed above based on the search results.

[0090] The first generated image refers to image data obtained by performing image generation processing on the input data of the model (for example, noise data) by the model constructed based on the search results above.

[0091] In addition, the present disclosure does not limit the number of the first generated images mentioned above. For example, the relevant content of the "number of first generated images" is similar to the relevant content of the "number of reference images" mentioned above. For the sake of brevity, it will not be repeated here.

[0092] In addition, the present disclosure does not limit the association relationship between the first generated image and the second generated image. For example, the two images may be generated based on different noise data.

[0093] For another example, to better improve the performance evaluation effect, the first generated image and the second generated image can be generated based on the same noise data. It can be seen that in one possible implementation, when the reference image includes Q second generated images, and the number of the first generated images is Q, the qth second generated image is obtained by performing image generation processing on the qth noise data using the diffusion model, and the qth first generated image is obtained by performing image generation processing on the qth noise data using the model constructed based on the search results; q is a positive integer, q≤Q, and Q is a positive integer. It should be noted that any two of the Q noise data are different.

[0094] Step 13: According to the above reference image and the above first generated image, determine the performance characterization data of the model constructed based on the search results.

[0095] It should be noted that the present disclosure does not limit the implementation method of step 13. For example, it can be implemented by using any existing or future method that can determine the performance characterization data of a model by comparing two image data (for example, the FID evaluation indicator calculation formula shown in formula (2) below).

[0096] In the formula, Score represents the performance characterization data of the model constructed based on the search results above; m r represents the mean value determined based on the image features of all reference images; m g represents the mean value determined based on the image features of all first generated images; C r represents the variance determined based on the image features of all reference images; C grepresents the variance determined based on the image features of all first generated images; Tr() represents the trace of a matrix. The image features of an image data are used to represent the image information carried by the image data; and the present disclosure does not limit the method for determining the image features. For example, it can be implemented using any existing or future method capable of extracting image features, such as the VGG (Visual Geometry Group) model.

[0097] Based on the relevant content of steps 11 to 13 above, it can be seen that in one possible implementation, the performance characterization data of the model constructed based on the search results above is determined based on the reference image and the first generated image generated by the model constructed based on the search results, so that the performance characterization data can represent the image generation performance of the model constructed based on the search results.

[0098] In addition, the present disclosure does not limit the process of determining the search results above. For example, it can be implemented using any existing or future method (such as an evolutionary algorithm) that can search for objects that meet certain constraints in a search space.

[0099] In fact, in order to better improve the search effect, the present disclosure also provides a possible implementation method of the above search result determination process. In this implementation method, when the above search space is used to provide at least one candidate time step sequence, the search result determination process can specifically include the following steps 21 to 24.

[0100] Step 21: Randomly sample a number of individuals from the search space, where the individuals include a sampling time step sequence that satisfies a preset step number constraint.

[0101] The sampling time step sequence refers to a candidate time step sequence randomly sampled from the search space above and meeting the preset step number constraint above.

[0102] Based on the relevant content of step 21 above, it can be seen that if the above search space is determined based on the above reference time step sequence, then after obtaining the search space, some individuals can be randomly sampled from the search space based on the above preset step number constraint, so that the time step sequence within each individual satisfies the preset step number constraint, so that the initial population can be constructed based on these individuals later.

[0103] It should be noted that the present disclosure does not limit the specific meaning of “several”; for example, it may mean at least two.

[0104] Step 22: Initialize the population based on the individuals above.

[0105] In the present disclosure, after randomly sampling some individuals that meet the preset step number constraint from the above search space, these individuals can be used to form a population, so that the search process for the search space can be realized through the subsequent evolution process for the population.

[0106] Step 23: Determine the individual ranking result corresponding to the population based on the fitness of each individual in the population; the fitness of the individual is determined based on the performance characterization data of the model constructed based on the individual.

[0107] The fitness of an individual is used to characterize the likelihood of the individual being retained (or deleted); and the process for determining the fitness of the individual can be: determining the fitness of the individual based on performance characterization data of a model constructed based on the individual. The "performance characterization data of the model constructed based on the individual" is used to characterize the image generation performance of the model constructed based on the individual, and the process for determining the "performance characterization data of the model constructed based on the individual" is similar to the process for determining the "performance characterization data of the model constructed based on the search results" described above.

[0108] Furthermore, the present disclosure does not limit the implementation of the step of "determining the fitness of the individual based on the performance characterization data of the model constructed based on the individual" in the preceding paragraph. For example, the step may specifically include directly determining the fitness of the individual using the performance characterization data of the model constructed based on the individual. Alternatively, the step may include performing some processing (e.g., multiplying the performance characterization data of the model constructed based on the individual by a coefficient) to obtain the fitness of the individual.

[0109] The above “individual ranking result corresponding to the population” is used to indicate the order in which the individuals in the population are deleted (or the order in which the individuals are retained); and the present disclosure does not limit the “individual ranking result corresponding to the population”.

[0110] Step 24: Based on the individual ranking results corresponding to the population above, the population is evolved, and the process returns to step 23 and subsequent steps until the search stop condition is reached, and the search results are determined from the population based on the preset performance conditions.

[0111] The evolution process is used to update a population; and the present disclosure does not limit the implementation of the evolution process. For example, it may include one or more of individual deletion processing, mutation processing, and crossover processing.

[0112] In fact, in order to better improve the search effect, the present disclosure also provides a possible implementation of the above evolutionary processing, which may specifically include one or more steps of the following steps 241 to 243.

[0113] Step 241: Based on the individual sorting result corresponding to the population, individuals that meet the preset deletion conditions are deleted from the population to implement individual deletion processing for the population.

[0114] Among them, the preset deletion condition refers to the condition that the individual in the population needs to be deleted meets; and the present disclosure does not limit the preset deletion condition. For example, the higher the position of an individual in the individual ranking result, the lower the possibility of the individual being deleted. The preset deletion condition can be specifically: the position of the individual in the individual ranking result is later than the preset position.

[0115] In fact, in order to improve the flexibility of individual deletion processing, the present disclosure also provides a possible implementation method of the above-mentioned preset deletion conditions. Under this implementation method, the preset deletion conditions can be determined based on the above-mentioned preset performance conditions, so that the population individual screening process achieved based on the preset deletion conditions is similar to the screening process achieved based on the preset performance conditions.

[0116] Step 242: randomly exchange the time steps of at least one pair of individuals in the population to achieve crossover processing for the population.

[0117] The random exchange process may refer to exchanging some time steps involved in one individual with other time steps involved in another individual to obtain a new individual. For example, when an individual includes [t 74 ,t 53 ,t 21 ,t6] this time step sequence, and another individual includes [t 100 ,t 87 ,t 39 ,t 10 ] this time step sequence, the [t 74 ,t 53 ,t 21 ,t6] in the time step sequence 74 With the [t 100 ,t 87 ,t 39 ,t 10 ]Time step t in this time step sequence 87 Exchange and get [t 87 ,t 53 ,t 21 ,t6] the new individual of this time step sequence and / or including [t 100 ,t 74 ,t 39 ,t 10 ]A new individual for this sequence of time steps.

[0118] In addition, the present disclosure does not limit the implementation of the above random exchange process. For example, when the population includes the first individual (for example, [t 74 ,t 53 ,t 21 ,t6] this time step sequence) and the second individual (for example, [t 100 ,t 87 ,t 39 ,t 10 ] this time step sequence), all the time steps involved in the first individual can be combined with all the time steps involved in the second individual (for example, {t 100 ,t 87 ,t 74 ,t 53 ,t 39 ,t 21 ,t 10 ,t6} this set), and then extract some time steps from the set (for example, t 87 , t 53 , t 21 , t6) to construct a new time step sequence (for example, [t 87 ,t 53 ,t 21 ,t6] this time step sequence) so that some time steps in the new time step sequence (for example, t 53 , t 21 , t6) belong to the first body, and the other time steps (for example, t 87 This time step) belongs to the second individual; finally, a new individual is determined based on the new time step sequence.

[0119] Based on the relevant content of step 242 above, it can be known that for a population (for example, a population after some individuals have been deleted), the time steps within at least one pair of individuals in the population can be randomly exchanged to obtain a new individual, and the new individual can be added to the population to enrich the individuals in the population, so that cross-processing of the population can be achieved.

[0120] It should be noted that the present disclosure does not limit the execution time of the above step 242. For example, the execution time of the above step 242 is later than the execution time of the above step 241.

[0121] Step 243: Modify the time steps within the individuals in the population according to the preset probability to achieve mutation processing for the population.

[0122] The preset probability refers to the probability of occurrence of a mutation required for performing individual mutations; and the preset probability can be set in advance according to the application scenario.

[0123] It should be noted that the present disclosure does not limit the implementation of the above step 243. For example, when the above population includes a third individual, and the third individual includes [t 88 ,t 80 ,t 60 ,t 53 ] this time step sequence, the [t 88 ,t 80 ,t 60 ,t 53 ]Time step t in this time step sequence 80 Modified to t 70 , to obtain [t 88 ,t 70 ,t 60 ,t 53 ]A new individual in this time step sequence can be created, thus completing a mutation process for the third individual.

[0124] It should also be noted that the present disclosure does not limit the execution time of the above step 243. For example, the execution time of the above step 243 is later than the execution time of the above step 241.

[0125] Based on the relevant contents of steps 241 to 243 above, it can be known that, in one possible implementation, after obtaining a population, individuals corresponding to some models with poor image generation performance can be deleted from the population based on the individual sorting results corresponding to the population, so that the retained individuals in the population can be used as parents; then crossover processing and / or mutation processing can be performed on the individuals in the parent generation to obtain an updated population, so that the updated population not only includes the parent generation, but also includes new individuals obtained by crossover processing and / or mutation processing on the parent generation, so that step 23 and its subsequent steps can be continued to be executed based on the updated population to achieve the next round of processing for the population.

[0126] The "search stopping condition" mentioned above refers to the condition that must be met to terminate the search process in the search space. This disclosure does not limit the search stopping condition. For example, the search stopping condition may specifically be: traversing all time step sequences in the search space that meet a preset step number constraint. In another example, the search stopping condition may specifically be: the number of iterations reaches a preset threshold.

[0127] Furthermore, the present disclosure does not limit the timing of determining the "search stop condition" described above. For example, in some application scenarios, the step of "determining whether the search stop condition has been met" may be performed after executing step 23. For another example, in other application scenarios, the step of "determining whether the search stop condition has been met" may be performed after executing the step of "performing an evolutionary process on the population based on the individual ranking results described above."

[0128] In addition, the present disclosure does not limit the implementation method of "determining search results from the population based on preset performance conditions" in step 24 above. For example, it can specifically be: based on the individual sorting results corresponding to the population, determine the individuals that meet the preset performance conditions from the population as search results.

[0129] Based on the relevant content of steps 21 to 24 above, it can be seen that in some application scenarios, for a search space, after obtaining the search space, an evolutionary algorithm can be used to search the search space to obtain search results that meet certain constraints (for example, preset step constraints, preset performance conditions, etc.).

[0130] Based on the relevant content of S102 above, it can be known that after constructing a search space based on the pre-constructed diffusion model, the search space can be searched based on a preset step number constraint and a preset performance condition to obtain search results, so that the performance characterization data of the model constructed based on the search results meets the preset performance condition, and the target time step sequence in the search results meets the preset step number constraint, so that the compression result corresponding to the diffusion model can be determined based on the search results later.

[0131] S103: Determine an image generation model using the model constructed based on the search results.

[0132] The image generation model refers to the compression result of the above diffusion model; and the image generation model is used to perform image generation processing on the input data of the image generation model (for example, noise data).

[0133] In addition, because the number of time steps in the time step sequence within the image generation model is smaller than the number of time steps in the above reference time step sequence, the image generation speed of the image generation model is faster than that of the diffusion model.

[0134] In addition, the present disclosure does not limit the implementation of S103. For example, it may specifically be: directly determining the model constructed based on the search results as the image generation model.

[0135] Based on the relevant contents of S101 to S103 above, it can be known that for any pre-constructed diffusion model, a search space is first constructed based on the diffusion model, so that the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; then, based on a preset step number constraint (for example, the constraint of 10 time steps) and a preset performance condition (for example, the image generation performance reaches a preset threshold, or the image generation performance reaches the best, etc.), the search space is searched to obtain a search result, so that the performance characterization data of the model constructed based on the search result meets the preset performance condition, and the target time step sequence in the search result meets the preset step number constraint. At this time, the preset step number constraint limits the The number of steps is smaller than the number of time steps of the reference time step sequence, so that the number of steps of the target time step sequence in the search result is smaller than the number of time steps of the reference time step sequence, thereby making the number of time steps in the model constructed based on the search result smaller than the number of time steps in the diffusion model; then, using the model constructed based on the search result, an image generation model is determined, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model, so that compression processing can be performed on the diffusion model while ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption.

[0136] Research has found that in order to better improve the image generation speed, the diffusion model can be further compressed in terms of network structure, so that the network structure corresponding to one or more time steps in the compressed diffusion model is different from the network structure corresponding to the corresponding time step in the diffusion model before compression, and the size of the network structure corresponding to the one or more time steps in the compressed diffusion model (for example, the size in terms of depth and / or width) is smaller than the size of the network structure corresponding to the corresponding time step in the diffusion model before compression, so that the total size of the network structure of the compressed diffusion model is smaller than the total size of the network structure of the diffusion model before compression.

[0137] The study also found that for a time step sequence, the optimal network structures corresponding to different time steps in the time step sequence may be the same or different (for example, there are differences in depth and / or width, etc.).

[0138] Based on the above two paragraphs, it can be seen that in order to better improve the image generation speed, the present disclosure also provides a possible implementation of the model determination method, which may specifically include the following steps 31 to 33.

[0139] Step 31: Based on the pre-constructed diffusion model, a search space is constructed, where the search space is used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step in the time step sequence in the diffusion model, where the candidate time step sequence includes at least one time step in the reference time step sequence, and the candidate network structure includes at least one feature channel of at least one network layer in the denoising neural network in the diffusion model.

[0140] For the relevant content of the diffusion model, please refer to the relevant content of the diffusion model in S101 above.

[0141] In addition, please refer to the above for the definition of search space.

[0142] In addition, the present disclosure does not limit the implementation of the search space in step 31 above. For example, the search space can be used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step in the reference time step sequence corresponding to the diffusion model above. The at least one candidate network structure corresponding to a time step refers to a network structure that can be selected when executing the data processing stage corresponding to the time step; and the present disclosure does not limit the implementation of the "at least one candidate network structure". For example, it can include the denoising neural network in the diffusion model above and the sampling results for the denoising neural network (for example, sampling results based on depth, sampling results based on width, or sampling results based on depth + width).

[0143] It should be noted that the "sampling results based on depth" in the above paragraph refers to sampling by selecting one or more network layers from the above denoising neural network, so that the "sampling results based on depth" include at least one network layer in the denoising neural network, so that model compression processing based on depth can be achieved; the "sampling results based on width" in the above paragraph refers to sampling by selecting one or more feature channels from the denoising neural network, so that some network layers in the "sampling results based on depth" include at least one feature channel of the corresponding network layer in the denoising neural network, so that model compression processing based on width can be achieved; the "sampling results based on depth + width" in the above paragraph refers to sampling by selecting one or more network layers from the denoising neural network and selecting only one or more feature channels from these selected network layers, so that the number of network layers in the "sampling results based on depth + width" does not exceed the number of network layers in the denoising neural network, and each network layer in the "sampling results based on depth + width" includes at least one feature channel of the corresponding network layer in the denoising neural network, so that model compression processing based on depth + width can be achieved.

[0144] Based on the content of the above paragraph, it can be known that in a possible implementation mode, the above search space can be determined based on the above reference time step sequence and the denoising neural network (for example, the search space can include the reference time step sequence and the denoising neural network), so that the search space can not only provide the time step sequence and the sampling results for the time step sequence, but also provide the denoising neural network and the sampling results for the denoising neural network for each time step, so that the time step sequence ultimately required for use and the network structure ultimately selected for each time step in the time step sequence can be determined by performing a sampling search on the search space. It can be seen that in some application scenarios, the search space can be implemented using the search space shown in Figure 2.

[0145] Step 32: Search the search space based on the preset step number constraint, the preset structure constraint, and the preset performance condition to obtain search results, which include a target time step sequence and a target network structure corresponding to each time step in the target time step sequence. The target time step sequence satisfies the preset step number constraint, the target network structure satisfies the preset structure constraint, and the performance characterization data of the model constructed based on the search results satisfies the preset performance condition.

[0146] For details about the preset step number constraints and preset performance conditions, please refer to the above text.

[0147] The preset structural constraint refers to the constraint required when performing compression processing on the network structure of the diffusion model mentioned above; and the preset structural constraint can be determined in advance based on the application scenario. For example, in some application scenarios (such as depth-based model compression processing), the preset structural constraint can specifically be: the sum of the number of network layers of the target network structure corresponding to all time steps in a time step sequence does not exceed the preset layer number threshold (such as the constraint shown in formula (3) below).

[0148] Where K represents the time series [t′ K ,t′ K-1 ,…,t′1]; n i Used to represent the time step t′ in the time series i The number of network layers of the corresponding target network structure; t′ i Used to represent the time step sequence [t′ K ,t′ K-1 ,…,t′1]; N max Indicates the preset layer threshold.

[0149] For example, in some application scenarios (e.g., width-based model compression processing), the preset structural constraint can specifically be: the sum of the number of feature channels of each network layer in the target network structure corresponding to all time steps in a time step sequence does not exceed the preset channel number threshold (e.g., the constraint shown in formula (4) below).

[0150] Where K represents the time series [t′ K ,t′ K-1 ,…,t′1]; n i Used to represent the time step t′ in the time series i The number of network layers of the corresponding target network structure; t′ i Used to represent the time step sequence [t′ K ,t′ K-1 ,…,t′1]; Represents the time step t′ i The number of feature channels of the jth network layer in the corresponding target network structure; M max Indicates the preset channel number threshold.

[0151] For example, in some application scenarios (such as model compression processing based on depth + width), the preset structural constraint can be specifically: the sum of the number of network layers of the target network structure corresponding to all time steps in a time step sequence does not exceed the preset layer threshold, and the sum of the number of feature channels of each network layer in the target network structure corresponding to all time steps in the time step sequence does not exceed the preset channel threshold.

[0152] In fact, in some application scenarios (for example, scenarios that focus on model runtime), network structure compression can be achieved by constraining runtime. Based on this, it can be seen that in one possible implementation, the preset structural constraint above can specifically be: the sum of the runtimes of the target network structure corresponding to all time steps in a time step sequence does not exceed a preset runtime threshold.

[0153] The "search results" in step 32 above are used to represent objects obtained from the search in the above search space that meet the three constraints of the above preset step number constraint, the above preset structure constraint, and the above preset performance condition (for example, the time step sequence + the network structure corresponding to each time step in the time sequence).

[0154] In addition, the present disclosure does not limit the implementation of the search results in the above paragraph. For example, in one possible implementation, the search results may include a target time step sequence and a target network structure corresponding to each time step in the target time step sequence. The target network structure refers to the candidate network structure that is ultimately selected for a time step in the target time step sequence above; and the target network structures corresponding to all time steps in the target time step sequence satisfy the preset structural constraints above (for example, the sum of the number of network layers of the target network structures corresponding to all time steps in the target time step sequence does not exceed a preset layer threshold).

[0155] Based on the content in the previous paragraph, it can be seen that for the search results in the previous paragraph, in one possible implementation, the target time step sequence in the search results satisfies the preset step number constraint, the target network structure corresponding to all time steps in the target time step sequence satisfies the above preset structural constraint, and the performance characterization data of the model constructed based on the search results meets the preset performance conditions. Among them, the "model constructed based on search results" is used to perform image generation processing on the input data of the model (for example, noise data); and the present disclosure does not limit the construction method of the "model constructed based on search results", for example, it can be specifically: according to the target time step sequence and the target network structure corresponding to each time step in the target time step sequence, construct the "model constructed based on search results", so that the constructed "model constructed based on search results" includes the target time step sequence and the target network structure corresponding to each time step in the target time step sequence, so that the working principle of the "model constructed based on search results" is specifically: the "model constructed based on search results" divides the image generation process into data processing stages corresponding to each time step according to the target time step sequence, and the data processing stage corresponding to each time step in the target time step sequence is realized by calling the target network structure corresponding to the corresponding time step.

[0156] In addition, the present disclosure does not limit the process of determining the search results in step 32 above. For example, it can be implemented using any existing or future method (e.g., evolutionary search algorithm, etc.) that can search for objects that meet certain constraints in a search space.

[0157] In fact, in order to better improve the search effect, the present disclosure also provides a possible implementation method of the above search result determination process. Under this implementation method, when the above search space is used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step in the above reference time step sequence, the search result determination process can specifically include the following steps 321-324.

[0158] Step 321: Randomly sample a number of individuals from the search space, where the individuals include a sampling time step sequence and a sampling network structure corresponding to each time step in the sampling time step sequence, where the sampling time step sequence satisfies a preset step number constraint, and where the sampling network structure satisfies a preset structure constraint.

[0159] For details about the sampling time step sequence, please refer to the above text.

[0160] The sampling network structure refers to the candidate network structure obtained by sampling a time step in the above sampling time step sequence; and the sampling network structures corresponding to all time steps in the sampling time step sequence meet the above preset structural constraints (for example, the sum of the number of network layers of the sampling network structures corresponding to all time steps in the sampling time step sequence does not exceed the preset layer number threshold).

[0161] In addition, the present disclosure does not limit the individual implementation methods in the above step 321. For example, it can be expressed using the following formulas (5)-(7). 0<t′ i+1 -t′ i <t T -t1;t′ i ∈[t T ,t T-1 ,…,t1] (6)

[0162] In the formula, cand represents an individual sampled from the search space; represents the sampling time step sequence within the individual cand; t′ i It is used to represent any time step in the sampling time step sequence, time step t′ i+1 With time step t′ i adjacent, and time step t′ i+1 The corresponding arrangement position is greater than the time step t′ i The corresponding arrangement position is at the front; Represents the set of sampling network structures corresponding to each time step in the sampling time step sequence within the individual cand; L k Represents the time step t′ in the sampling time step sequence K The corresponding sampling network structure; L k-1 Represents the time step t′ in the sampling time step sequence K-1 The corresponding sampling network structure; ...; L1 represents the sampling network structure corresponding to time step t′1 in the sampling time step sequence; L i Represents the time step t′ in the sampling time step sequence within the individual cand i The corresponding sampling network structure; Represents the sampling network structure Li The network layer located at the first arrangement position; Represents the sampling network structure L i The network layer located in the second arrangement position; Represents the sampling network structure L i The nth i Network layers in the arrangement position; n i Represents the sampling network structure L i The number of network layers in n i ≤The number of network layers in the denoising neural network above.

[0163] Based on the relevant content of step 321 above, it can be known that after constructing the search space based on the pre-constructed diffusion model, some individuals can be sampled from the search space according to the preset step number constraint and the preset structure constraint, so that the sampling time step sequence within the individual satisfies the preset step number constraint, and the sampling network structure corresponding to all time steps in the sampling time step sequence within the individual satisfies the preset structure constraint, so that the image generation speed of the model constructed based on the individual is faster than the image generation speed of the diffusion model.

[0164] Step 322: Initialize the population based on the above individuals.

[0165] It should be noted that the relevant content of step 322 can be found in step 22 above, and for the sake of brevity, it will not be repeated here.

[0166] Step 323: Determine the individual ranking result corresponding to the population based on the fitness of each individual in the population; the fitness of the individual is determined based on the performance characterization data of the model constructed based on the individual.

[0167] It should be noted that the relevant content of step 323 can be found in step 23 above, and for the sake of brevity, it will not be repeated here.

[0168] Step 324: Based on the individual ranking results corresponding to the above population, the population is evolved, and the process returns to step 323 and subsequent steps until the search stop condition is reached, and the search results are determined from the population based on the preset performance conditions.

[0169] It should be noted that the implementation of the "evolution processing" in step 324 is similar to the implementation of the "evolution processing" in step 24 above. For ease of understanding, it is described below with reference to examples.

[0170] As an example, the “evolution process” in step 324 above may include one or more steps of steps 3241 to 3243 below.

[0171] Step 3241: Based on the individual sorting result corresponding to the above population, individuals that meet the preset deletion conditions are deleted from the population to implement individual deletion processing for the population.

[0172] It should be noted that for the relevant content of step 3241, please refer to step 241 above. For the sake of brevity, it will not be repeated here.

[0173] Step 3242: Randomly exchange information combinations within at least one pair of individuals in the population to achieve crossover processing for the population.

[0174] Here, an information combination refers to an object that can be exchanged within an individual (for example, a time step and the network structure corresponding to that time step); and the information combination can include the time steps and network structures that have a corresponding relationship within the individual. In other words, an information combination can include a time step and the network structure corresponding to that time step.

[0175] In addition, the "random exchange processing" in step 3242 above refers to exchanging some information combinations involved in one individual (for example, a time step and its corresponding network structure) with other information combinations involved in another individual to obtain a new individual (for example, the result after cross processing shown in Figure 2).

[0176] In addition, the implementation of the "random exchange process" in step 3242 is similar to the implementation of the "random exchange process" in step 242 above.

[0177] Based on the relevant content of step 3242 above, it can be seen that for a population (for example, a population after some individuals have been deleted), the information combination within at least one pair of individuals in the population can be randomly exchanged to obtain a new individual, and the new individual can be added to the population to enrich the individuals in the population, so that cross-processing of the population can be achieved.

[0178] It should be noted that the present disclosure does not limit the execution time of the above step 3242. For example, the execution time of the above step 3242 is later than the execution time of the above step 3241.

[0179] Step 3243: Modify the time steps and / or network structures within the individuals in the population according to the preset probability to achieve variation processing for the population.

[0180] For details on preset probabilities, please refer to the above text.

[0181] It should be noted that the present disclosure does not limit the implementation method of the modification in step 3243 above. For example, a modification may only modify the time step without modifying the network structure. For another example, a modification may only modify the network structure without modifying the time step. For another example, a modification may modify both the time step and the network structure.

[0182] It should also be noted that the present disclosure does not limit the execution time of the above step 3243. For example, the execution time of the above step 3243 is later than the execution time of the above step 3241.

[0183] Based on the relevant contents of steps 3241 to 3243 above, it can be known that, in one possible implementation, after obtaining a population, individuals corresponding to some models with poor image generation performance can be deleted from the population based on the individual sorting results corresponding to the population, so that the retained individuals in the population can be used as parents; then crossover processing and / or mutation processing can be performed on the individuals in the parent generation to obtain an updated population, so that the updated population not only includes the parent generation, but also includes new individuals obtained by crossover processing and / or mutation processing on the parent generation, so that step 323 and its subsequent steps can be continued to be executed based on the updated population to achieve the next round of processing for the population.

[0184] For the relevant content of the “search stop condition” in step 324 above, please refer to the relevant content of the “search stop condition” in step 24 above.

[0185] Based on the relevant content of steps 321 to 324 above, it can be seen that in some application scenarios, for a search space, after obtaining the search space, an evolutionary algorithm can be used to search the search space to obtain search results that meet certain constraints (for example, preset step constraints, preset structural constraints, and preset performance conditions, etc.).

[0186] Based on the relevant content of step 32 above, it can be known that after constructing a search space based on the pre-constructed diffusion model, the search space can be searched based on the preset step number constraint, the preset structural constraint and the preset performance condition to obtain search results, so that the performance characterization data of the model constructed based on the search result meets the preset performance condition, and the target time step sequence in the search result meets the preset step number constraint, and the target network structure corresponding to each time step in the target time step sequence meets the preset structural constraint, so that the compression result corresponding to the diffusion model can be determined based on the search result later.

[0187] It can be seen that in one possible implementation, the above search space can be used to perform compression processing on the diffusion model from the two dimensions of time and space. Specifically, the compression processing on the time dimension is achieved by searching the time step sequence from the search space, and the compression processing on the space dimension is achieved by searching the network structure corresponding to each time step from the search space. In this way, the search processing for the time step sequence and the network structure corresponding to each time step in the time step sequence can be completed with the help of the same search process, thereby avoiding the problems caused by separately executing the search processing on these two dimensions, and thus helping to improve the model compression effect.

[0188] Step 33: Determine an image generation model using the model built based on the search results.

[0189] It should be noted that the relevant content of step 33 can be found in S103 above, and for the sake of brevity, it will not be repeated here.

[0190] Based on the relevant contents of steps 31 to 33 above, it can be known that for any pre-constructed diffusion model, a search space is first constructed based on the diffusion model, so that the search space includes the reference time step sequence and the denoising neural network, so that the search space can be used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step involved in these candidate time step sequences; then, based on the preset step number constraint, the preset structure constraint and the preset performance condition, the search space is searched to obtain the search result, so that the performance characterization data of the model constructed based on the search result meets the preset performance condition, and the target time step sequence in the search result meets the preset step number constraint, and the target time step sequence meets the preset step number constraint. The target network structure corresponding to each time step in the sequence satisfies the preset structural constraint; then, the image generation model is determined using the model constructed based on the search results, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the scale of the network structure required to be called for each time step in the image generation model is smaller than the scale of the denoising neural network, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model. In this way, multi-faceted compression processing can be performed on the diffusion model while ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the time consumption of image generation.

[0191] Based on the above description of the model determination method, the present disclosure further provides an image generation method. For ease of understanding, the following description is provided with reference to the accompanying drawings. As shown in FIG3 , the image generation method provided in the present embodiment includes steps S301-S302 below. FIG3 is a flow chart of the image generation method provided in the present embodiment.

[0192] S301: Acquire noise data.

[0193] The noise data is used to generate image data; and the present disclosure does not limit the noise data.

[0194] S302: Perform image generation processing on the noise data using an image generation model to obtain a third generated image; the image generation model is determined using any implementation of the model determination method provided in the embodiments of the present disclosure.

[0195] The image generation model is used to perform image data generation processing on the input data of the image generation model (for example, noise data); and the relevant content of the image generation model can be found above.

[0196] The third generated image refers to image data obtained by processing the above noise data by the image generation model.

[0197] Based on the relevant contents of S301 to S302 above, it can be seen that for the image generation method provided in the embodiment of the present disclosure, a predetermined image generation model can be used to perform image generation processing on any noise data to obtain a generated image corresponding to the noise data. In particular, because the image generation model has a relatively low image generation time consumption and relatively good image generation performance, the image generation model can generate relatively high-quality image data at a relatively fast speed, which is conducive to reducing image generation time consumption while ensuring image generation quality.

[0198] Furthermore, the present disclosure does not limit the execution subject of the image generation method. For example, the image generation method provided in the embodiments of the present disclosure can be applied to a device with data processing capabilities, such as a terminal device or a server. Furthermore, the image generation method provided in the embodiments of the present disclosure can also be implemented via a data communication process between different devices (e.g., a terminal device and a server, two terminal devices, or two servers).

[0199] Based on the model determination method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide a model determination device, which will be explained and illustrated below in conjunction with Figure 4. Figure 4 is a schematic diagram of the structure of a model determination device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the model determination device provided in the embodiments of the present disclosure, please refer to the relevant content of the model determination method above.

[0200] As shown in FIG4 , the model determination device 400 provided in an embodiment of the present disclosure includes:

[0201] A construction unit 401 is configured to construct a search space based on a pre-constructed diffusion model, wherein the search space is configured to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model;

[0202] A search unit 402 is configured to search the search space according to a preset step number constraint and a preset performance condition to obtain search results, wherein the search results include a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search results satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps in the reference time step sequence;

[0203] The determining unit 403 is configured to determine an image generation model using the model constructed based on the search results.

[0204] In one possible implementation, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence, the candidate network structure including at least one feature channel of at least one network layer in the denoising neural network in the diffusion model;

[0205] The search unit 402 is specifically used to search the search space according to a preset step number constraint, a preset structural constraint, and a preset performance condition to obtain search results, wherein the search results also include a target network structure corresponding to each time step in the target time step sequence, and the target network structure satisfies the preset structural constraint.

[0206] In a possible implementation manner, the search unit 402 includes:

[0207] An individual sampling subunit, configured to randomly sample a number of individuals from the search space, wherein the individuals include a sampling time step sequence, and the sampling time step sequence satisfies the preset step number constraint;

[0208] A population initialization subunit, configured to initialize the population based on the plurality of individuals;

[0209] a ranking determination subunit, configured to determine a ranking result of individuals corresponding to the population based on the fitness of each individual in the population; the fitness of the individual is determined based on performance characterization data of a model constructed based on the individual;

[0210] The population evolution subunit is used to evolve the population according to the individual sorting result, and return to the sorting determination subunit to continue to execute the step of determining the individual sorting result corresponding to the population according to the fitness of each individual in the population.

[0211] The result determination subunit is configured to determine the search result from the population according to the preset performance condition until a search stop condition is reached.

[0212] In one possible implementation, the population evolution subunit includes:

[0213] An individual deletion subunit, configured to delete individuals that meet a preset deletion condition from the population based on the individual sorting result;

[0214] The crossover mutation subunit is used to randomly exchange the time steps of at least one pair of individuals in the population, and / or modify the time steps of individuals in the population according to a preset probability.

[0215] In one possible implementation, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence;

[0216] The individual also includes a sampling network structure corresponding to each time step in the sampling time step sequence, and the sampling network structure satisfies the preset structural constraints;

[0217] The crossover mutation subunit is specifically used to: randomly exchange information combinations within at least one pair of individuals in the population, where the information combinations include corresponding time steps and network structures existing in the individuals; and / or modify the time steps and / or network structures within the individuals in the population according to a preset probability.

[0218] In one possible implementation, the performance characterization data is determined based on a reference image and a first generated image generated by a model constructed based on the search results.

[0219] In a possible implementation manner, the diffusion model is constructed based on a training image; the reference image includes the training image; and / or the reference image includes a second generated image generated by the diffusion model.

[0220] Based on the relevant contents of the above-mentioned model determination device 400, it can be known that for the model determination device 400 provided in the embodiment of the present disclosure, a search space is first constructed based on a pre-constructed diffusion model, so that the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; then, based on a preset step number constraint (for example, the constraint of 10 time steps) and a preset performance condition (for example, the image generation performance reaches a preset threshold, or the image generation performance reaches the best, etc.), the search space is searched to obtain a search result, so that the performance characterization data of the model constructed based on the search result meets the preset performance condition, and the target time step sequence in the search result meets the preset step number constraint. At this time, due to the preset step number constraint The limited number of steps is smaller than the number of time steps of the reference time step sequence, so that the number of steps of the target time step sequence in the search result is smaller than the number of time steps of the reference time step sequence, thereby making the number of time steps in the model constructed based on the search result smaller than the number of time steps in the diffusion model; then, using the model constructed based on the search result, an image generation model is determined, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model, so that compression processing can be performed on the diffusion model while ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption.

[0221] Based on the image generation method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide an image generation device, which will be explained and illustrated below in conjunction with Figure 5. Figure 5 is a schematic structural diagram of the image generation device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the image generation device provided in the embodiments of the present disclosure, please refer to the relevant content of the image generation method above.

[0222] As shown in FIG5 , an image generating apparatus 500 provided by an embodiment of the present disclosure includes:

[0223] An acquisition unit 501 is used to acquire noise data;

[0224] The generating unit 502 is configured to perform image generation processing on the noise data using an image generation model to obtain a third generated image; the image generation model is determined using the model determination method according to any one of claims 1 to 7.

[0225] Based on the above-described information regarding the image generation device 500, it can be seen that the image generation device 500 provided in the embodiments of the present disclosure can utilize a predetermined image generation model to perform image generation processing on any noise data to obtain a generated image corresponding to the noise data. In particular, because the image generation model consumes relatively little time and has relatively good image generation performance, the image generation model can generate relatively high-quality image data at a relatively fast speed, thereby facilitating the reduction of image generation time while ensuring image generation quality.

[0226] In addition, an embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the model determination method provided by the embodiment of the present disclosure or executes any implementation of the image generation method provided by the embodiment of the present disclosure.

[0227] Referring to FIG6 , a schematic diagram of the structure of an electronic device 600 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG6 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0228] As shown in Figure 6, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of electronic device 600 are also stored in RAM 603. Processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0229] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 shows the electronic device 600 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0230] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0231] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0232] The embodiments of the present disclosure also provide a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the model determination method provided by the embodiments of the present disclosure or executes any implementation of the image generation method provided by the embodiments of the present disclosure.

[0233] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0234] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0235] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0236] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0237] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0238] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0239] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.

[0240] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0241] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0242] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0243] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0244] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0245] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0246] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A model determination method, comprising: According to the pre-constructed diffusion model, a search space is constructed, wherein the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; According to a preset step number constraint and a preset performance condition, the search space is searched to obtain search results, wherein the search results include a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search results satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps of the reference time step sequence; and An image generation model is determined using the model constructed based on the search results.

2. The method according to claim 1, wherein: The search space is also used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence, wherein the candidate network structure includes at least one feature channel of at least one network layer in the denoising neural network in the diffusion model; The searching of the search space according to the preset step number constraint and the preset performance condition to obtain search results includes: The search space is searched according to the preset step number constraint, the preset structure constraint and the preset performance condition to obtain the search result, wherein the search result also includes a target network structure corresponding to each time step in the target time step sequence, and the target network structure satisfies the preset structure constraint.

3. The method according to claim 1, wherein: The process of determining the search results includes: Randomly sampling a number of individuals from the search space, wherein the individuals include a sampling time step sequence, and the sampling time step sequence satisfies the preset step number constraint; Initializing a population according to the individuals; Determining the individual ranking result corresponding to the population according to the fitness of each individual in the population; the fitness of the individual is determined according to the performance characterization data of the model constructed based on the individual; According to the individual sorting result, the population is evolved, and the The step of determining the individual ranking result corresponding to the population according to the fitness of each individual in the population, until the search stop condition is reached, and then determining the search result from the population according to the preset performance condition.

4. The method according to claim 3, wherein: The evolution process comprises: According to the individual sorting result, deleting individuals that meet a preset deletion condition from the population; The time steps within at least one pair of individuals in the population are randomly exchanged, and / or the time steps within the individuals in the population are modified according to a preset probability.

5. The method according to claim 4, wherein: The search space is also used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence; The individual also includes a sampling network structure corresponding to each time step in the sampling time step sequence, and the sampling network structure satisfies the preset structural constraint; The randomly exchanging time steps of at least one pair of individuals in the population comprises: Performing random exchange processing on the information combination within the at least one pair of individuals in the population, wherein the information combination includes the time steps and network structures with corresponding relationships existing within the individuals; The modifying of the time steps within individuals in the population according to a preset probability comprises: The time steps and / or network structures within the individuals in the population are modified according to the preset probability.

6. The method according to claim 1, wherein: The performance characterization data is determined based on a reference image and a first generated image generated by the model constructed based on the search results.

7. The method according to claim 6, wherein: The diffusion model is constructed based on a training image; the reference image includes the training image; and / or, The reference image includes a second generated image generated by the diffusion model.

8. A method for generating an image, wherein: The method comprises: obtaining noise data; and The noise data is processed for image generation using an image generation model to obtain a third generated image; the image generation model is determined using the model determination method described in any one of claims 1 to 7.

9. A model determination device, comprising: A construction unit is configured to construct a search space according to a pre-constructed diffusion model, wherein the search The space is used to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; A search unit is configured to search the search space according to a preset step number constraint and a preset performance condition to obtain a search result, wherein the search result includes a target time step sequence, the target time step sequence satisfies the preset step number constraint, the performance characterization data of the model constructed based on the search result satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps of the reference time step sequence; as well as A determination unit is configured to determine an image generation model using the model constructed based on the search results.

10. An image generating device, comprising: an acquisition unit configured to acquire noise data; as well as A generation unit is configured to perform image generation processing on the noise data using an image generation model to obtain a third generated image; the image generation model is determined using the model determination method described in any one of claims 1-7.

11. An electronic device, comprising: Processor and memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or the computer program in the memory so that the electronic device executes the method according to any one of claims 1 to 8.

12. A computer-readable medium, wherein: The computer-readable medium stores instructions or a computer program, and when the instructions or the computer program are executed on a device, the device is enabled to execute the method according to any one of claims 1 to 8.

Images