Model determination method and device, image generation method and device, equipment and medium

Abstract

The invention discloses a model determination method and device, an image generation method and device, equipment and a medium, and the method comprises the steps: constructing a search space according to a pre-constructed diffusion model, so as to enable the search space to be used for providing at least one candidate time step sequence; searching the search space according to a preset step number constraint and a preset performance condition to obtain a search result, so that performance characterization data of a model constructed based on the search result meets the preset performance condition, and a target time step sequence in the search result meets the preset step number constraint; then, the image generation model is determined by utilizing the model constructed based on the search result, so that compression processing can be performed on the diffusion model on the premise of ensuring the image generation performance as far as possible, the compressed model can generate image data at a relatively high speed, and the image generation time consumption can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model determination method, image generation method, device, equipment, and medium. Background Art

[0002] The diffusion model is an image generation model, and the diffusion model can be applied in many fields (for example, image generation, image super-resolution reconstruction, text image generation, etc.).

[0003] However, because the diffusion model divides the image generation process into multiple time steps, and each time step requires calling a denoising neural network, the image generation speed of the diffusion model is relatively slow, resulting in a relatively long image generation time consumption. Summary of the invention

[0004] The present application provides a model determination method, image generation method, device, equipment, and medium, which can effectively reduce the time consumption of image generation.

[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] The present application provides a model determination method, the method comprising:

[0007] According to the pre-constructed diffusion model, a search space is constructed, wherein the search space is used to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model;

[0008] According to a preset step number constraint and a preset performance condition, the search space is searched to obtain a search result, wherein the search result includes a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search result satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps of the reference time step sequence;

[0009] An image generation model is determined using the model constructed based on the search results.

[0010] In a possible implementation, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence, wherein the candidate network structure includes at least one feature channel of at least one network layer in the denoising neural network in the diffusion model;

[0011] The searching of the search space according to the preset step number constraint and the preset performance condition to obtain search results includes:

[0012] According to a preset step number constraint, a preset structural constraint and a preset performance condition, the search space is searched to obtain search results, wherein the search results also include a target network structure corresponding to each time step in the target time step sequence, and the target network structure satisfies the preset structural constraint.

[0013] In a possible implementation manner, the process of determining the search result includes:

[0014] Randomly sampling a number of individuals from the search space, wherein the individuals include a sampling time step sequence, and the sampling time step sequence satisfies the preset step number constraint;

[0015] Initializing a population according to the individuals;

[0016] Determining the individual ranking result corresponding to the population according to the fitness of each individual in the population; the fitness of the individual is determined according to the performance characterization data of the model constructed based on the individual;

[0017] According to the individual ranking results, the population is evolved, and the step of determining the individual ranking results corresponding to the population according to the fitness of each individual in the population is continued, until the search stop condition is reached, and the search result is determined from the population according to the preset performance condition.

[0018] In a possible implementation manner, the evolution process includes:

[0019] According to the individual sorting result, deleting individuals that meet a preset deletion condition from the population;

[0020] The time steps within at least one pair of individuals in the population are randomly exchanged, and / or the time steps within the individuals in the population are modified according to a preset probability.

[0021] In a possible implementation manner, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence;

[0022] The individual also includes a sampling network structure corresponding to each time step in the sampling time step sequence, and the sampling network structure satisfies the preset structural constraint;

[0023] The randomly exchanging time steps of at least one pair of individuals in the population comprises:

[0024] Randomly exchanging information combinations within at least one pair of individuals in the population, wherein the information combinations include time steps and network structures with corresponding relationships existing within the individuals;

[0025] The modifying of the time steps within individuals in the population according to a preset probability comprises:

[0026] The time steps and / or network structures within individuals in the population are modified according to a preset probability.

[0027] In one possible implementation, the performance characterization data is determined based on a reference image and a first generated image generated by a model constructed based on the search results.

[0028] In a possible implementation manner, the diffusion model is constructed based on a training image; the reference image includes the training image;

[0029] and / or,

[0030] The reference image includes a second generated image generated by the diffusion model.

[0031] The present application provides an image generation method, the method comprising:

[0032] Get noise data;

[0033] The noise data is processed by image generation using an image generation model to obtain a third generated image; the image generation model is determined using the model determination method provided in the present application.

[0034] The present application provides a model determination device, comprising:

[0035] A construction unit, configured to construct a search space according to a pre-constructed diffusion model, wherein the search space is configured to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model;

[0036] A search unit, configured to search the search space according to a preset step number constraint and a preset performance condition to obtain a search result, wherein the search result includes a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search result satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps of the reference time step sequence;

[0037] A determination unit is used to determine an image generation model using the model constructed based on the search results.

[0038] The present application provides an image generation device, comprising:

[0039] An acquisition unit, used for acquiring noise data;

[0040] A generating unit is used to perform image generation processing on the noise data using an image generation model to obtain a third generated image; the image generation model is determined using the model determination method provided in the present application.

[0041] The present application provides an electronic device, the device comprising: a processor and a memory;

[0042] The memory is used to store instructions or computer programs;

[0043] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the model determination method or image generation method provided in the present application.

[0044] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the model determination method or image generation method provided in the present application.

[0045] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the model determination method or image generation method provided by the present application.

[0046] Compared with the prior art, this application has at least the following advantages:

[0047] In the technical solution provided by the present application, for any pre-constructed diffusion model, a search space is first constructed based on the diffusion model, so that the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; then, based on a preset step number constraint (for example, a constraint of 10 time steps) and a preset performance condition (for example, the image generation performance reaches a preset threshold, or the image generation performance reaches the best, etc.), the search space is searched to obtain a search result, so that the performance characterization data of the model constructed based on the search result satisfies the preset performance condition, and the target time step sequence in the search result satisfies the preset step number constraint. At this time, the number of steps limited by the preset step number constraint is less than the preset step number constraint. The number of time steps of the reference time step sequence is determined so that the number of steps of the target time step sequence in the search result is smaller than the number of time steps of the reference time step sequence, thereby making the number of time steps in the model constructed based on the search result smaller than the number of time steps in the diffusion model; then, using the model constructed based on the search result, an image generation model is determined so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model, so that compression processing can be performed on the diffusion model while ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A flow chart of a model determination method provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a model determination process provided in an embodiment of the present application;

[0051] Figure 3 A flowchart of an image generation method provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the structure of a model determination device provided in an embodiment of the present application;

[0053] Figure 5A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] After research, it was found that for the diffusion model, the image generation speed of the diffusion model is related to the number of time steps involved in the generation process of the diffusion model. Specifically, if the number of time steps involved in the generation process of the diffusion model is larger, the image generation speed of the diffusion model is slower; if the number of time steps involved in the generation process of the diffusion model is smaller, the image generation speed of the diffusion model is faster. Based on this, it can be seen that the present application can increase the image generation speed by reducing the time step, thereby reducing the time consumption of image generation.

[0056] Based on the above research, in order to better reduce the time consumption of image generation, the present application provides a model determination method (that is, a model compression scheme), which may specifically include: for any pre-constructed diffusion model, first construct a search space based on the diffusion model, so that the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; then search the search space based on a preset step number constraint (for example, the constraint of 10 time steps) and a preset performance condition (for example, the image generation performance reaches a preset threshold, or the image generation performance reaches the best, etc.), and obtain search results, so that the performance characterization data of the model constructed based on the search result meets the preset performance condition, and the target time step sequence in the search result meets the preset Step number constraint, at this time, because the number of steps limited by the preset step number constraint is smaller than the number of time steps of the reference time step sequence, the number of steps of the target time step sequence in the search result is smaller than the number of time steps of the reference time step sequence, so that the number of time steps in the model constructed based on the search result is smaller than the number of time steps in the diffusion model; then, using the model constructed based on the search result, an image generation model is determined, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model, so that compression processing can be performed on the diffusion model under the premise of ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption.

[0057] In addition, the present application does not limit the execution subject of the model determination method. For example, the model determination method provided in the embodiment of the present application can be applied to a device with a data processing function such as a terminal device or a server. For another example, the model determination method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a stand-alone server, a cluster server or a cloud server.

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In order to better understand the technical solution provided by the present application, the model determination method provided by the present application is described below with reference to some drawings. Figure 1 As shown, the model determination method provided in the embodiment of the present application includes the following S101-S103. Figure 1 A flowchart of a model determination method provided in an embodiment of the present application.

[0060] S101: constructing a search space according to a pre-constructed diffusion model, where the search space is used to provide at least one candidate time step sequence, where the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model.

[0061] Among them, the diffusion model is used to perform image generation processing on the input data of the diffusion model (for example, noise data); and the present application does not limit the diffusion model. For example, it can be implemented using any existing or future trained diffusion model with image generation function (for example, a pre-trained diffusion model, etc.).

[0062] In addition, for the above diffusion model, the diffusion model may include the time step sequence required for the image generation process (e.g., [t T ,t T-1,…,t1] time step sequence) and the denoising neural network, so that the diffusion model can divide the image generation process into data processing stages corresponding to each time step according to the time step sequence, and the data processing stage corresponding to each time step in the time step sequence is implemented by calling the denoising neural network. It can be seen that when the diffusion model includes [t T ,t T-1 ,…,t1], the diffusion model can realize the image generation process by calling T denoising neural networks. The denoising neural network is used to denoise the input data of the denoising neural network, and the present application does not limit the implementation method of the denoising neural network. For example, it can be implemented by using a neural network with denoising function in any existing or future diffusion model. T is a positive integer, and T represents the length of the reference time step sequence (that is, the number of time steps).

[0063] The reference time step sequence is used to represent the time step sequence required for the image generation process of the diffusion model above (for example, [t T ,t T-1 ,…,t1] this time step sequence).

[0064] The search space refers to the search space required for compression processing of the above diffusion model (for example, Figure 2 The search space shown).

[0065] In addition, for the above search space, the search space can be determined based on the diffusion model that needs to be compressed, so that the search space is used to provide some compression results of the diffusion model (for example, compression results presented in terms of time steps and / or network structures), so that the compression model with better image generation performance can be screened out from these compression results later, thereby realizing compression processing for the diffusion model.

[0066] In addition, the present application does not limit the implementation method of the above search space. For example, in some application scenarios (for example, when compression processing is achieved by reducing the time step), the search space can be used to provide at least one candidate time step sequence. The candidate time step sequence is used to represent a time step sequence that can be sampled from the search space; and the present application does not limit the candidate time step sequence. For example, the candidate time step sequence may refer to a time step sequence sampled from the above reference time step sequence, so that the candidate time step sequence may include at least one time step in the reference time step sequence. For another example, the candidate time step sequence may refer to a time step sequence sampled from the above reference time step sequence according to a preset step number threshold, so that the number of time steps in the candidate time step sequence does not exceed the preset step number threshold. The preset step number threshold can be pre-set, for example, the preset step number threshold can be any value that is less than the number of time steps in the reference time step sequence. It can be seen that when the reference time step sequence is [t T ,t T-1 ,…,t1], the “at least one candidate time step sequence” may include at least the candidate time step sequence shown in the following formula (1).

[0067]

[0068] In the formula, represents a candidate time step sequence of length K; t ′ K represents the time step that ranks the most forward among the K time steps sampled from the reference time step sequence above, t ′ K-1 represents the second most important time step among the K time steps sampled from the reference time step sequence, ..., t1 ′ It represents the time step that is ranked last among the K time steps sampled from the reference time step sequence, t i ′ It is used to represent any time step among the K time steps sampled from the reference time step sequence; time step t i ′ +1 With time step t i ′ adjacent, and time step t i ′ +1 The corresponding arrangement position is greater than the time step t i ′ The corresponding arrangement position is at the front; [t T ,t T-1 ,…,t1] represents the reference time step sequence, K is a positive integer, K<T.

[0069] Based on the above content, it can be known that in a possible implementation, the above search space can be based on the above reference time step sequence (for example, [t T ,t T-1 ,…,t1] this time step sequence) is determined (for example, the search space includes the reference time step sequence) so that the search space can provide the time step sequence and the sampling results for the time step sequence (for example, Figure 2 As shown in [t ′ K ,t ′ K-1 ,…,t1 ′ ] this time step sequence), so that the time step sequence that is ultimately required can be determined by subsequently performing a sampling search on the search space.

[0070] It should be noted that the present application does not limit the sampling method for the candidate time step sequence above. For example, in order to better improve the sampling effect, the time step in the candidate time step sequence can refer to at least one time step sampled from the reference time step sequence above. For example, when the reference time step sequence is [t 100 ,t 99 ,…,t1], we can 100 ,t 99 ,…,t1] and randomly sample some time steps (e.g., t 74 ,t 53 ,t 21 , t6, etc.) to construct a candidate time step sequence (e.g., [t 74 ,t 53 ,t 21 ,t6] this time step sequence).

[0071] S102: Search the search space according to a preset step number constraint and a preset performance condition to obtain search results, wherein the search results include a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search results satisfies the preset performance condition, and the number of steps limited by the preset step number constraint is less than the number of time steps of the reference time step sequence.

[0072] The preset step number constraint refers to the constraint required when compressing the time step of the diffusion model mentioned above; and the number of steps limited by the preset step number constraint is less than the number of time steps of the reference time step sequence mentioned above. In addition, the preset step number constraint can be set in advance according to the application scenario. For example, when the number of time steps of the reference time step sequence is T, the preset step number constraint can be: the number of time steps of the time step sequence in the model obtained by the final compression is K (that is, the length of the time step sequence in the model obtained by the final compression is K), K is a positive integer, K<T.

[0073] The preset performance condition refers to the constraint required in terms of image generation performance when searching the above search space; and the preset performance condition can be set in advance according to the application scenario. For example, the preset performance condition can specifically be: the image generation performance reaches the maximum, or the image generation performance reaches a preset threshold.

[0074] The search results are used to represent objects (eg, time step sequences) obtained from the search space above and satisfying the two constraints of the preset step number constraint and the preset performance condition above.

[0075] In addition, the present application does not limit the above search results. For example, in some application scenarios (for example, when compression processing is achieved by reducing the time step), the search results may include a target time step sequence. The target time step sequence refers to a candidate time step sequence obtained by searching in the above search space and satisfying the above preset step number constraint and the above preset performance condition. The candidate time step sequences provided by the search space are obtained by sampling any time step from the reference time step sequence above, so that the search space can represent the time step sampling results for the reference time step sequence as comprehensively as possible, so that the search space can represent the time step compression scheme that can be selected when the time step compression is performed on the above diffusion model as comprehensively as possible, and then the target time step sequence finally searched from the search space can better represent the search for the optimal sequence under the premise of satisfying the preset step number constraint, which is conducive to improving the compression effect for the diffusion model.

[0076] Based on the above content, it can be seen that in some application scenarios (for example, when compression processing is achieved by reducing the time step), the above target time step sequence not only satisfies the preset step number constraint, but also the performance characterization data of the model constructed based on the target time step sequence meets the preset performance conditions. Among them, the "model constructed based on the target time step sequence" can be used to perform image generation processing on the input data of the model (for example, a noise data). It should be noted that the determination process of the "performance characterization data of the model constructed based on the target time step sequence" is similar to the determination process of the "performance characterization data of the model constructed based on the search results" below. For the sake of brevity, it will not be repeated here.

[0077] In addition, the present application does not limit the construction method of the "model constructed based on the target time step sequence" in the above paragraph. For example, it can be specifically: according to the target time step sequence and the denoising neural network in the above diffusion model, the "model constructed based on the target time step sequence" is constructed, so that the constructed "model constructed based on the target time step sequence" includes the target time step sequence and the denoising neural network, so that the working principle of the "model constructed based on the target time step sequence" is specifically: the "model constructed based on the target time step sequence" divides the image generation process into data processing stages corresponding to each time step according to the target time step sequence, and the data processing stage corresponding to each time step in the target time step sequence is realized by calling the denoising neural network.

[0078] Based on the above two paragraphs, it can be known that in a possible implementation, the "model constructed based on search results" in S102 above can be implemented using the "model constructed based on target time step sequence" shown in the above two paragraphs.

[0079] The above “performance characterization data of the model constructed based on the search results” is used to characterize the image generation performance of the above “model constructed based on the search results”; and the present application does not limit the determination method of the performance characterization data. For example, it can be implemented by using any existing or future method that can evaluate the image generation performance of a diffusion model, such as the FID (Fréchet Inception Distance) scoring indicator. For ease of understanding, the following is an example.

[0080] As an example, the determination process of the above “performance characterization data of the model constructed based on the search results” may specifically include the following steps 11 to 13.

[0081] Step 11: Get a reference image.

[0082] The reference image refers to the image data required for reference when performing image generation performance determination processing for a model (for example, Figure 2 reference image shown).

[0083] In addition, the present application does not limit the implementation method of the above reference image. For ease of understanding, some situations are described below.

[0084] Case 1: In some application scenarios, the image generation performance of a model can be evaluated with the help of real images. Based on this, the present application provides a possible implementation of the above reference image, in which, when the above diffusion model is constructed based on a training image, the reference image may include the training image.

[0085] Case 2: In some application scenarios, in order to ensure that the image generation performance of the compressed model is as close as possible to the image generation performance of the above diffusion model, the image generation performance of a compressed model can also be evaluated with the help of image data generated by the diffusion model. Based on this, the present application provides a possible implementation of the above reference image, in which the reference image may include a second generated image generated by the diffusion model. The second generated image refers to the image data generated by the diffusion model.

[0086] Case 3: In some application scenarios, the image generation performance of a compressed model can be evaluated with the help of real images and image data generated by the diffusion model. Based on this, the present application provides a possible implementation of the above reference image. In this implementation, when the above diffusion model is constructed based on a training image, the reference image can include the training image and a second generated image generated by the diffusion model.

[0087] In addition, the present application does not limit the number of reference images mentioned above, for example, it can be determined according to the actual application scenario. For example, when the present application uses the FID score indicator to characterize the image generation performance of a model, the number of reference images can be 50,000.

[0088] After research, it was found that, because it is only necessary to judge the relative size of the image generation performance of the two models during the search process, and there is no need to accurately determine the image generation performance of each model, in order to improve the search efficiency, a rough estimation method can be used to evaluate the image generation performance of a model. Based on this, it can be seen that the present application can speed up the search by reducing the number of reference images. It can be seen that in one possible implementation, when the present application uses the FID scoring indicator to characterize the image generation performance of a model, the number of reference images can be 5,000.

[0089] Step 12: Obtain a first generated image generated by the model constructed above based on the search results.

[0090] The first generated image refers to image data obtained by performing image generation processing on input data (eg, noise data) of the model constructed based on the search results above.

[0091] In addition, the present application does not limit the number of the first generated images mentioned above. For example, the relevant content of the "number of first generated images" is similar to the relevant content of the "number of reference images" mentioned above. For the sake of brevity, it will not be repeated here.

[0092] In addition, the present application does not limit the association relationship between the first generated image and the second generated image. For example, the two images are generated based on different noise data.

[0093] For another example, in order to better improve the performance evaluation effect, the first generated image and the second generated image can be generated based on the same noise data. It can be seen that in one possible implementation, when the reference image includes Q second generated images, and the number of the first generated images is Q, the qth second generated image is obtained by performing image generation processing on the qth noise data using the diffusion model above, and the qth first generated image is obtained by performing image generation processing on the qth noise data using the model constructed based on the search results above; q is a positive integer, q≤Q, and Q is a positive integer. It should be noted that any two of the Q noise data are different.

[0094] Step 13: According to the above reference image and the above first generated image, determine the performance characterization data of the model constructed based on the search results above.

[0095] It should be noted that the present application does not limit the implementation method of step 13. For example, it can be implemented by any existing or future method that can determine the performance characterization data of a model by comparing two image data (for example, the FID evaluation index calculation formula shown in formula (2) below).

[0096]

[0097] In the formula, Score represents the performance characterization data of the model built based on the search results above; m r represents the mean value determined based on the image features of all reference images; m g represents the mean value determined based on the image features of all first generated images; C r represents the variance determined based on the image features of all reference images; C grepresents the variance determined based on the image features of all first generated images; Tr() represents the trace of a matrix. The image feature of an image data is used to represent the image information carried by the image data; and the present application does not limit the method for determining the image feature. For example, it can be implemented by any existing or future method for extracting image features, such as the VGG (Visual Geometry Group) model.

[0098] Based on the relevant contents of steps 11 to 13 above, it can be known that, in one possible implementation, the performance characterization data of the model constructed based on the search results above is determined based on a reference image and a first generated image generated by the model constructed based on the search results, so that the performance characterization data can represent the image generation performance of the model constructed based on the search results.

[0099] In addition, the present application does not limit the process of determining the above search results. For example, it can be implemented by any existing or future method (such as evolutionary algorithm, etc.) that can search for objects that meet certain constraints in a search space.

[0100] In fact, in order to better improve the search effect, the present application also provides a possible implementation method of the process of determining the above search results. In this implementation method, when the above search space is used to provide at least one candidate time step sequence, the process of determining the search results can specifically include the following steps 21-24.

[0101] Step 21: Randomly sample a number of individuals from the search space, where the individuals include a sampling time step sequence that satisfies a preset step number constraint.

[0102] The sampled time step sequence refers to a candidate time step sequence randomly sampled from the above search space and satisfying the above preset step number constraint.

[0103] Based on the relevant content of step 21 above, it can be known that if the above search space is determined based on the above reference time step sequence, then after obtaining the search space, some individuals can be randomly sampled from the search space according to the above preset step number constraint, so that the time step sequence within each individual satisfies the preset step number constraint, so that the initial population can be constructed based on these individuals later.

[0104] It should be noted that this application does not limit the specific meaning of “several”, for example, it can mean at least two.

[0105] Step 22: Initialize the population based on the individuals above.

[0106] In the present application, after randomly sampling some individuals that meet the preset step number constraint from the above search space, these individuals can be used to form a population, so that the search process for the search space can be implemented through the evolution process for the population.

[0107] Step 23: According to the fitness of each individual in the population, determine the individual ranking result corresponding to the population; the fitness of the individual is determined based on the performance characterization data of the model constructed based on the individual.

[0108] The fitness of an individual is used to characterize the possibility of the individual being retained (or deleted); and the determination process of the fitness of the individual can be: determining the fitness of the individual according to the performance characterization data of the model constructed based on the individual. The "performance characterization data of the model constructed based on the individual" is used to characterize the image generation performance of the model constructed based on the individual, and the determination process of the "performance characterization data of the model constructed based on the individual" is similar to the determination process of the "performance characterization data of the model constructed based on the search results" above.

[0109] In addition, the present application does not limit the implementation of the step "determining the fitness of the individual according to the performance characterization data of the model constructed based on the individual" in the previous paragraph. For example, it can specifically be: directly determining the performance characterization data of the model constructed based on the individual as the fitness of the individual. For another example, it can also be: performing some processing (such as multiplying by a certain coefficient) on the performance characterization data of the model constructed based on the individual to obtain the fitness of the individual.

[0110] The above “individual sorting result corresponding to the population” is used to indicate the order in which the individuals in the population are deleted (or, the order in which the individuals are retained); and this application does not limit the “individual sorting result corresponding to the population”.

[0111] Step 24: According to the individual ranking results corresponding to the above population, the population is evolved, and the process returns to step 23 and subsequent steps until the search stop condition is reached, and the search results are determined from the population according to the preset performance conditions.

[0112] The evolution process is used to update a population; and the present application does not limit the implementation method of the evolution process, for example, it may include one or more of individual deletion processing, mutation processing and crossover processing.

[0113] In fact, in order to better improve the search effect, the present application also provides a possible implementation of the above evolutionary processing, which may specifically include one or more steps of the following steps 241 to 243.

[0114] Step 241: According to the individual sorting result corresponding to the above population, individuals that meet the preset deletion conditions are deleted from the population to implement individual deletion processing for the population.

[0115] Among them, the preset deletion condition refers to the condition that the individual to be deleted in the population must satisfy; and the present application does not limit the preset deletion condition. For example, the closer an individual is to the front of the individual ranking result, the less likely the individual is to be deleted. The preset deletion condition can specifically be: the position of the individual in the individual ranking result is later than the preset position.

[0116] In fact, in order to improve the flexibility of individual deletion processing, the present application also provides a possible implementation method of the above-mentioned preset deletion condition. Under this implementation method, the preset deletion condition can be determined based on the above-mentioned preset performance condition, so that the population individual screening process achieved based on the preset deletion condition is similar to the screening process achieved based on the preset performance condition.

[0117] Step 242: randomly exchange the time steps within at least one pair of individuals in the population to achieve crossover processing for the population.

[0118] The random exchange process may refer to exchanging some time steps involved in one individual with other time steps involved in another individual to obtain a new individual. For example, when an individual includes [t 74 ,t 53 ,t 21 ,t6] this time step sequence, and the other individual includes [t 100 ,t 87 ,t 39 ,t 10 ] this time step sequence, the [t 74 ,t 53 ,t 21 ,t6] is the time step t in this time step sequence 74 With this [t 100 ,t 87 ,t 39 ,t 10 ] The time step t in this time step sequence 87 By exchanging, we get [t 87 ,t 53 ,t 21 ,t6] new individuals in this time step sequence and / or including [t 100 ,t 74 ,t 39 ,t 10 ]A new individual for this sequence of time steps.

[0119] In addition, the present application does not limit the implementation of the above random exchange process. For example, when the population includes the first individual (for example, [t 74 ,t 53 ,t 21 ,t6] this time step sequence) and the second individual (for example, [t 100 ,t 87 ,t 39 ,t 10 ] this time step sequence), all the time steps involved in the first individual can be combined with all the time steps involved in the second individual (for example, {t 100 ,t 87 ,t 74 ,t 53 ,t 39 ,t 21 ,t 10 ,t6} this set), and then extract some time steps from the set (for example, t 87 ,t 53 ,t 21 , t6) to construct a new time step sequence (for example, [t 87 ,t 53 ,t 21 ,t6] this time step sequence) so that some time steps in the new time step sequence (for example, t 53 ,t 21 , t6) belong to the first body, and the other time steps (for example, t 87 This time step) belongs to the second individual; finally, based on the new time step sequence, a new individual is determined.

[0120] Based on the relevant content of step 242 above, it can be known that for a population (for example, a population after some individuals have been deleted), the time steps within at least one pair of individuals in the population can be randomly exchanged to obtain new individuals, and the new individuals can be added to the population to enrich the individuals in the population, thereby achieving cross processing for the population.

[0121] It should be noted that the present application does not limit the execution time of the above step 242. For example, the execution time of the above step 242 is later than the execution time of the above step 241.

[0122] Step 243: modify the time steps within individuals in the population according to a preset probability to achieve mutation processing for the population.

[0123] The preset probability refers to the probability of occurrence of a mutation required for performing individual mutations; and the preset probability can be set in advance according to the application scenario.

[0124] It should be noted that the present application does not limit the implementation method of the above step 243. For example, when the above population includes a third individual, and the third individual includes [t 88 ,t 80 ,t 60 ,t 53 ] this time step sequence, the [t 88 ,t 80 ,t 60 ,t 53 ] The time step t in this time step sequence 80 Modified to t 70 , to obtain [t 88 ,t 70 ,t 60 ,t 53 ]A new individual in this time step sequence can thus complete a mutation process for the third individual.

[0125] It should also be noted that the present application does not limit the execution time of step 243 above. For example, the execution time of step 243 is later than the execution time of step 241 above.

[0126] Based on the relevant contents of steps 241 to 243 above, it can be known that, in a possible implementation mode, after obtaining a population, individuals corresponding to some models with poor image generation performance can be deleted from the population based on the individual sorting results corresponding to the population, so that the retained individuals in the population can be used as parents; then crossover processing and / or mutation processing are performed on the individuals in the parent generation to obtain an updated population, so that the updated population includes not only the parent generation, but also new individuals obtained by crossover processing and / or mutation processing on the parent generation, so that step 23 and its subsequent steps can be continued to be executed based on the updated population to achieve the next round of processing for the population.

[0127] The above "search stop condition" refers to the condition that needs to be met when the search process for the search space is stopped; and the present application does not limit the search stop condition. For example, the search stop condition may specifically be: traversing all time step sequences in the search space that meet the preset step number constraint. For another example, the search stop condition may specifically be: the number of iterations reaches a preset number threshold.

[0128] In addition, the present application does not limit the timing of determining the above “search stop condition”. For example, in some application scenarios, the step of “determining whether the search stop condition is reached” may be performed after executing the above step 23. For another example, in other application scenarios, the step of “determining whether the search stop condition is reached” may be performed after executing the above step of “performing evolution processing on the population according to the above individual sorting results”.

[0129] In addition, the present application does not limit the implementation method of "determining search results from the population based on preset performance conditions" in the above step 24. For example, it can specifically be: based on the individual sorting results corresponding to the population, determine the individuals that meet the preset performance conditions from the population as search results.

[0130] Based on the relevant content of steps 21 to 24 above, it can be known that in some application scenarios, for a search space, after obtaining the search space, an evolutionary algorithm can be used to search the search space to obtain search results that meet certain constraints (for example, preset step constraints, preset performance conditions, etc.).

[0131] Based on the relevant content of S102 above, it can be known that after constructing the search space based on the pre-constructed diffusion model, the search space can be searched according to the preset step number constraint and the preset performance condition to obtain search results, so that the performance characterization data of the model constructed based on the search results meets the preset performance conditions, and the target time step sequence in the search results meets the preset step number constraint, so that the compression result corresponding to the diffusion model can be determined based on the search results later.

[0132] S103: Determine an image generation model using a model constructed based on the search results.

[0133] The image generation model refers to the compression result of the above diffusion model; and the image generation model is used to perform image generation processing on the input data (for example, noise data) of the image generation model.

[0134] In addition, because the number of time steps in the time step sequence in the image generation model is smaller than the number of time steps in the above reference time step sequence, the image generation speed of the image generation model is faster than that of the diffusion model.

[0135] In addition, the present application does not limit the implementation of S103. For example, it may specifically be: directly determining the model constructed based on the search results as the image generation model.

[0136] Based on the relevant contents of S101 to S103 above, it can be known that for any pre-constructed diffusion model, a search space is first constructed based on the diffusion model, so that the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; then, based on a preset step number constraint (for example, the constraint of 10 time steps) and a preset performance condition (for example, the image generation performance reaches a preset threshold, or the image generation performance reaches the best, etc.), the search space is searched to obtain search results, so that the performance characterization data of the model constructed based on the search result meets the preset performance condition, and the target time step sequence in the search result meets the preset step number constraint. At this time, the preset step number constraint is limited. The number of steps is smaller than the number of time steps of the reference time step sequence, so that the number of steps of the target time step sequence in the search result is smaller than the number of time steps of the reference time step sequence, thereby making the number of time steps in the model constructed based on the search result smaller than the number of time steps in the diffusion model; then, using the model constructed based on the search result, an image generation model is determined, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model, so that compression processing can be performed on the diffusion model under the premise of ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption.

[0137] Through research, it is found that in order to better improve the image generation speed, the diffusion model can be further compressed in terms of network structure, so that the network structure corresponding to one or more time steps in the compressed diffusion model is different from the network structure corresponding to the corresponding time step in the diffusion model before compression, and the size of the network structure corresponding to the one or more time steps in the compressed diffusion model (for example, the size in terms of depth and / or width) is smaller than the size of the network structure corresponding to the corresponding time step in the diffusion model before compression, so that the total size of the network structure of the compressed diffusion model is smaller than the total size of the network structure of the diffusion model before compression.

[0138] The study also found that for a time step sequence, the optimal network structures corresponding to different time steps in the time step sequence may be the same or different (for example, there are differences in depth and / or width, etc.).

[0139] Based on the above two paragraphs, it can be seen that in order to better improve the image generation speed, the present application also provides a possible implementation of the model determination method, which may specifically include the following steps 31 to 33.

[0140] Step 31: Based on the pre-constructed diffusion model, a search space is constructed, where the search space is used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step in the internal time step sequence in the diffusion model, where the candidate time step sequence includes at least one time step in the reference time step sequence, and the candidate network structure includes at least one feature channel of at least one network layer in the denoising neural network in the diffusion model.

[0141] For the relevant content of the diffusion model, please refer to the relevant content of the diffusion model in S101 above.

[0142] In addition, please refer to the above for the definition of search space.

[0143] In addition, the present application does not limit the implementation of the search space in step 31 above. For example, the search space can be used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step in the reference time step sequence corresponding to the diffusion model above. Among them, at least one candidate network structure corresponding to a time step refers to a network structure that can be selected when executing the data processing stage corresponding to the time step; and the present application does not limit the implementation of the "at least one candidate network structure". For example, it can include the denoising neural network in the diffusion model above and the sampling results for the denoising neural network (for example, sampling results based on depth, sampling results based on width, or sampling results based on depth + width).

[0144] It should be noted that the "sampling results based on depth" in the above paragraph refers to sampling by selecting one or more network layers from the above denoising neural network, so that the "sampling results based on depth" include at least one network layer in the denoising neural network, so that model compression processing based on depth can be achieved; the "sampling results based on width" in the above paragraph refers to sampling by selecting one or more feature channels from the denoising neural network, so that some network layers in the "sampling results based on depth" include at least one feature channel of the corresponding network layer in the denoising neural network, so that model compression processing based on width can be achieved; the "sampling results based on depth + width" in the above paragraph refers to sampling by selecting one or more network layers from the denoising neural network and selecting only one or more feature channels from these selected network layers, so that the number of network layers in the "sampling results based on depth + width" does not exceed the number of network layers in the denoising neural network, and each network layer in the "sampling results based on depth + width" includes at least one feature channel of the corresponding network layer in the denoising neural network, so that model compression processing based on depth + width can be achieved.

[0145] Based on the above content, it can be known that in a possible implementation, the above search space can be determined based on the above reference time step sequence and the denoising neural network (for example, the search space can include the reference time step sequence and the denoising neural network), so that the search space can not only provide the time step sequence and the sampling results for the time step sequence, but also provide the denoising neural network and the sampling results for each time step, so that the final time step sequence to be used and the network structure finally selected for each time step in the time step sequence can be determined by sampling and searching the search space. It can be seen that in some application scenarios, the search space can be used. Figure 2 The search space shown is implemented.

[0146] Step 32: Search the search space according to the preset step number constraint, the preset structure constraint and the preset performance condition to obtain search results, which include a target time step sequence and a target network structure corresponding to each time step in the target time step sequence, the target time step sequence satisfies the preset step number constraint, the target network structure satisfies the preset structure constraint, and the performance characterization data of the model constructed based on the search results satisfies the preset performance condition.

[0147] For details about preset step number constraints and preset performance conditions, please refer to the above text.

[0148] The preset structural constraint refers to the constraint required when compressing the network structure of the diffusion model mentioned above; and the preset structural constraint can be determined in advance based on the application scenario. For example, in some application scenarios (such as depth-based model compression processing), the preset structural constraint can specifically be: the sum of the number of network layers of the target network structure corresponding to all time steps in a time step sequence does not exceed the preset layer number threshold (such as the constraint shown in formula (3) below).

[0149]

[0150] In the formula, K represents the time series [t ′ K ,t ′ K-1 ,…,t1 ′ ] length; n i Used to represent the time step t in the time series i ′ The number of network layers of the corresponding target network structure; t i ′ Used to represent the time step sequence [t ′ K ,t ′ K-1,…,t1 ′ ] any time step in the max Indicates the preset layer number threshold.

[0151] For example, in some application scenarios (e.g., model compression processing based on width), the preset structural constraint can be specifically: the sum of the number of feature channels of each network layer in the target network structure corresponding to all time steps in a time step sequence does not exceed the preset channel number threshold (e.g., the constraint shown in formula (4) below).

[0152]

[0153] In the formula, K represents the time series [t ′ K ,t ′ K-1 ,…,t1 ′ ] length; n i Used to represent the time step t in the time series i ′ The number of network layers of the corresponding target network structure; t i ′ Used to represent the time step sequence [t ′ K ,t ′ K-1 ,…,t1 ′ ] any time step that exists in ; represents the time step t i ′ The number of feature channels of the jth network layer in the corresponding target network structure; M max Indicates the preset channel number threshold.

[0154] For example, in some application scenarios (such as model compression processing based on depth + width), the preset structural constraint can be specifically: the sum of the number of network layers of the target network structure corresponding to all time steps in a time step sequence does not exceed the preset layer threshold, and the sum of the number of feature channels of each network layer in the target network structure corresponding to all time steps in the time step sequence does not exceed the preset channel threshold.

[0155] In fact, in some application scenarios (for example, scenarios that focus on model runtime, etc.), network structure compression can be achieved by constraining runtime. Based on this, it can be seen that in a possible implementation, the above preset structural constraint can be specifically: the sum of the runtimes of the target network structure corresponding to all time steps in a time step sequence does not exceed the preset runtime threshold.

[0156] The "search results" in step 32 above are used to represent objects obtained from the search space above that satisfy the three constraints of the preset step number constraint above, the preset structure constraint above, and the preset performance condition above (for example, the time step sequence + the network structure corresponding to each time step in the time sequence).

[0157] In addition, the present application does not limit the implementation method of the search results in the above paragraph. For example, in one possible implementation method, the search results may include a target time step sequence and a target network structure corresponding to each time step in the target time step sequence. The target network structure refers to the candidate network structure finally selected for a time step in the target time step sequence above; and the target network structure corresponding to all time steps in the target time step sequence satisfies the preset structural constraints above (for example, the sum of the number of network layers of the target network structure corresponding to all time steps in the target time step sequence does not exceed the preset layer number threshold).

[0158] Based on the content in the previous paragraph, it can be seen that for the search results in the previous paragraph, in one possible implementation, the target time step sequence in the search results satisfies the preset step number constraint, the target network structure corresponding to all time steps in the target time step sequence satisfies the above preset structure constraint, and the performance characterization data of the model constructed based on the search results meets the preset performance conditions. Among them, the "model constructed based on search results" is used to perform image generation processing on the input data of the model (for example, a noise data); and the present application does not limit the construction method of the "model constructed based on search results", for example, it can be specifically: according to the target time step sequence and the target network structure corresponding to each time step in the target time step sequence, construct the "model constructed based on search results", so that the constructed "model constructed based on search results" includes the target time step sequence and the target network structure corresponding to each time step in the target time step sequence, so that the working principle of the "model constructed based on search results" is specifically: the "model constructed based on search results" divides the image generation process into data processing stages corresponding to each time step according to the target time step sequence, and the data processing stage corresponding to each time step in the target time step sequence is realized by calling the target network structure corresponding to the corresponding time step.

[0159] In addition, the present application does not limit the process of determining the search results in step 32 above. For example, it can be implemented using any existing or future method that can search for objects that meet certain constraints from a search space (for example, an evolutionary search algorithm, etc.).

[0160] In fact, in order to better improve the search effect, the present application also provides a possible implementation method of the process of determining the above search results. Under this implementation method, when the above search space is used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step in the above reference time step sequence, the process of determining the search results can specifically include the following steps 321-324.

[0161] Step 321: randomly sample a number of individuals from the search space, the individuals comprising a sampling time step sequence and a sampling network structure corresponding to each time step in the sampling time step sequence, the sampling time step sequence satisfies a preset step number constraint, and the sampling network structure satisfies a preset structure constraint.

[0162] For details about the sampling time step sequence, please refer to the above text.

[0163] The sampling network structure refers to the candidate network structure obtained by sampling a time step in the above sampling time step sequence; and the sampling network structures corresponding to all time steps in the sampling time step sequence satisfy the above preset structural constraints (for example, the sum of the number of network layers of the sampling network structures corresponding to all time steps in the sampling time step sequence does not exceed the preset layer threshold).

[0164] In addition, the present application does not limit the individual implementation methods in the above step 321. For example, it can be expressed using the following formulas (5)-(7).

[0165]

[0166] 0 <t i ′ +1 -t i ′ <t T -t1;t i ′ ∈[t T ,t T-1 ,…,t1] (6)

[0167]

[0168] In the formula, cand represents an individual sampled from the search space; represents the sampling time step sequence within the individual cand; t i ′ It is used to represent any time step in the sampling time step sequence. The time step t i ′ +1 With time step t i ′ adjacent, and time step ti ′ +1 The corresponding arrangement position is greater than the time step t i ′ The corresponding arrangement position is at the front; represents the set of sampling network structures corresponding to each time step in the sampling time step sequence within the individual cand; L k Represents the time step t in the sampling time step sequence ′ K The corresponding sampling network structure; L k-1 Represents the time step t in the sampling time step sequence ′ K-1 The corresponding sampling network structure; ...; L1 represents the time step t1 in the sampling time step sequence ′ The corresponding sampling network structure; L i represents the time step t in the sampling time step sequence within the individual cand i ′ The corresponding sampling network structure; Represents the sampling network structure L i The network layer located at the first arrangement position; Represents the sampling network structure L i The network layer located in the second arrangement position; Represents the sampling network structure L i The nth i The network layers in the arranged positions; n i Represents the sampling network structure L i The number of network layers in n i ≤The number of network layers in the denoising neural network above.

[0169] Based on the relevant content of step 321 above, it can be known that after constructing the search space based on the pre-constructed diffusion model, some individuals can be sampled from the search space according to the preset step number constraint and the preset structure constraint, so that the sampling time step sequence within the individual satisfies the preset step number constraint, and the sampling network structure corresponding to all time steps in the sampling time step sequence within the individual satisfies the preset structure constraint, so that the image generation speed of the model constructed based on the individual is faster than the image generation speed of the diffusion model.

[0170] Step 322: Initialize the population based on the above individuals.

[0171] It should be noted that the relevant content of step 322 can be found in the above step 22, and for the sake of brevity, it will not be repeated here.

[0172] Step 323: Determine the individual ranking result corresponding to the population according to the fitness of each individual in the population; the fitness of the individual is determined according to the performance characterization data of the model constructed based on the individual.

[0173] It should be noted that the relevant content of step 323 can be found in the above step 23, and for the sake of brevity, it will not be repeated here.

[0174] Step 324: According to the individual ranking results corresponding to the above population, the population is evolved, and the process returns to continue executing step 323 and subsequent steps until the search stop condition is reached, and the search results are determined from the population according to the preset performance conditions.

[0175] It should be noted that the implementation method of the "evolution processing" in step 324 is similar to the implementation method of the "evolution processing" in step 24 above. For ease of understanding, it is explained below with examples.

[0176] As an example, the “evolution process” in step 324 above may include one or more steps of steps 3241 to 3243 below.

[0177] Step 3241: Based on the individual sorting result corresponding to the above population, individuals that meet the preset deletion conditions are deleted from the population to implement individual deletion processing for the population.

[0178] It should be noted that the relevant content of step 3241 can be found in the above step 241, and for the sake of brevity, it will not be repeated here.

[0179] Step 3242: Randomly exchange information combinations within at least one pair of individuals in the population to achieve crossover processing for the population.

[0180] The information combination refers to an object that can be exchanged within an individual (e.g., a time step and the network structure corresponding to the time step); and the information combination may include the time step and the network structure that have a corresponding relationship within the individual. That is, an information combination may include a time step and the network structure corresponding to the time step.

[0181] In addition, the “random exchange process” in step 3242 above refers to exchanging some information combinations involved in one individual (e.g., a time step and its corresponding network structure) with other information combinations involved in another individual to obtain a new individual (e.g., Figure 2 The result after cross processing is shown).

[0182] In addition, the implementation of the "random exchange process" in step 3242 is similar to the implementation of the "random exchange process" in step 242 above.

[0183] Based on the relevant content of step 3242 above, it can be known that for a population (for example, a population after some individuals have been deleted), the information combination within at least one pair of individuals in the population can be randomly exchanged to obtain a new individual, and the new individual can be added to the population to enrich the individuals in the population, thereby achieving cross-processing for the population.

[0184] It should be noted that the present application does not limit the execution time of step 3242 above. For example, the execution time of step 3242 is later than the execution time of step 3241 above.

[0185] Step 3243: Modify the time steps and / or network structures within individuals in the population according to a preset probability to achieve variation processing for the population.

[0186] For the relevant content about preset probability, please refer to the above.

[0187] It should be noted that the present application does not limit the implementation method of the modification in step 3243 above. For example, a modification may only modify the time step without modifying the network structure. For another example, a modification may not modify the time step but only modify the network structure. For another example, a modification may modify both the time step and the network structure.

[0188] It should also be noted that the present application does not limit the execution time of step 3243 above. For example, the execution time of step 3243 is later than the execution time of step 3241 above.

[0189] Based on the relevant contents of steps 3241 to 3243 above, it can be known that, in a possible implementation mode, after obtaining a population, individuals corresponding to some models with poor image generation performance can be deleted from the population based on the individual sorting results corresponding to the population, so that the retained individuals in the population can be used as parents; then crossover processing and / or mutation processing are performed on the individuals in the parent generation to obtain an updated population, so that the updated population includes not only the parent generation, but also new individuals obtained by crossover processing and / or mutation processing on the parent generation, so that step 323 and its subsequent steps can be continued to be executed based on the updated population to achieve the next round of processing for the population.

[0190] For the relevant content of the “search stop condition” in step 324 above, please refer to the relevant content of the “search stop condition” in step 24 above.

[0191] Based on the relevant contents of steps 321 to 324 above, it can be known that in some application scenarios, for a search space, after obtaining the search space, an evolutionary algorithm can be used to search the search space to obtain search results that meet certain constraints (for example, preset step constraints, preset structural constraints, and preset performance conditions, etc.).

[0192] Based on the relevant content of step 32 above, it can be known that after constructing the search space based on the pre-constructed diffusion model, the search space can be searched according to the preset step number constraint, the preset structural constraint and the preset performance condition to obtain the search result, so that the performance characterization data of the model constructed based on the search result satisfies the preset performance condition, and the target time step sequence in the search result satisfies the preset step number constraint, and the target network structure corresponding to each time step in the target time step sequence satisfies the preset structural constraint, so that the compression result corresponding to the diffusion model can be determined based on the search result later.

[0193] It can be seen that, in a possible implementation, the above search space can be used to perform compression processing on the diffusion model from the two dimensions of time and space, which is specifically: the compression processing on the time dimension is achieved by searching the time step sequence from the search space, and the compression processing on the space dimension is achieved by searching the network structure corresponding to each time step from the search space. In this way, the search processing for the time step sequence and the network structure corresponding to each time step in the time step sequence can be completed with the help of the same search process, thereby avoiding the problems caused by separately executing the search processing on these two dimensions, which is beneficial to improving the model compression effect.

[0194] Step 33: Determine the image generation model using the model built based on the search results.

[0195] It should be noted that the relevant content of step 33 can be found in S103 above, and for the sake of brevity, it will not be repeated here.

[0196] Based on the relevant contents of steps 31 to 33 above, it can be known that for any pre-constructed diffusion model, a search space is first constructed based on the diffusion model, so that the search space includes the reference time step sequence and the denoising neural network, so that the search space can be used to provide at least one candidate time step sequence and at least one candidate network structure corresponding to each time step involved in these candidate time step sequences; then, based on the preset step number constraint, the preset structure constraint and the preset performance condition, the search space is searched to obtain the search result, so that the performance characterization data of the model constructed based on the search result satisfies the preset performance condition, and the target time step sequence in the search result satisfies the preset step number constraint, and the target time step sequence satisfies the preset step number constraint. The target network structure corresponding to each time step in the sequence satisfies the preset structural constraint; then, the image generation model is determined by using the model constructed based on the search result, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the scale of the network structure required to be called at each time step in the image generation model is smaller than the scale of the denoising neural network, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model. In this way, it is possible to perform multi-faceted compression processing on the diffusion model while ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption.

[0197] Based on the relevant contents of the above model determination method, the present application also provides an image generation method, which is described below in conjunction with the accompanying drawings for ease of understanding. Figure 3 As shown, the image generation method provided in the embodiment of the present application includes the following S301-S302. Figure 3 A flowchart of an image generation method provided in an embodiment of the present application.

[0198] S301: Acquire noise data.

[0199] The noise data is used to generate image data; and the present application does not limit the noise data.

[0200] S302: Perform image generation processing on the noise data using an image generation model to obtain a third generated image; the image generation model is determined using any implementation of the model determination method provided in the embodiments of the present application.

[0201] The image generation model is used to perform image data generation processing on the input data of the image generation model (for example, noise data); and the relevant content of the image generation model can be found above.

[0202] The third generated image refers to image data obtained by processing the above noise data by the image generation model.

[0203] Based on the relevant contents of S301 to S302 above, it can be known that for the image generation method provided in the embodiment of the present application, a predetermined image generation model can be used to perform image generation processing on any noise data to obtain a generated image corresponding to the noise data. Among them, because the image generation time consumption of the image generation model is relatively low and the image generation performance of the image generation model is relatively good, the image generation model can generate image data with relatively high quality at a relatively fast speed, which is conducive to reducing the image generation time consumption while ensuring the image generation quality.

[0204] In addition, the present application does not limit the execution subject of the image generation method. For example, the image generation method provided in the embodiment of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, the image generation method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).

[0205] Based on the model determination method provided in the embodiment of the present application, the embodiment of the present application also provides a model determination device. Figure 4 Explain and illustrate. Figure 4 This is a schematic diagram of the structure of a model determination device provided in an embodiment of the present application. It should be noted that for the technical details of the model determination device provided in an embodiment of the present application, please refer to the relevant content of the model determination method above.

[0206] like Figure 4 As shown, the model determination device 400 provided in the embodiment of the present application includes:

[0207] A construction unit 401 is used to construct a search space according to a pre-constructed diffusion model, wherein the search space is used to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model;

[0208] A search unit 402 is used to search the search space according to a preset step number constraint and a preset performance condition to obtain a search result, wherein the search result includes a target time step sequence, the target time step sequence satisfies the preset step number constraint, the performance characterization data of the model constructed based on the search result satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps of the reference time step sequence;

[0209] The determination unit 403 is used to determine an image generation model by using the model constructed based on the search results.

[0210] In a possible implementation, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence, wherein the candidate network structure includes at least one feature channel of at least one network layer in the denoising neural network in the diffusion model;

[0211] The search unit 402 is specifically used to search the search space according to a preset step number constraint, a preset structure constraint and a preset performance condition to obtain search results, wherein the search results also include a target network structure corresponding to each time step in the target time step sequence, and the target network structure satisfies the preset structure constraint.

[0212] In a possible implementation manner, the search unit 402 includes:

[0213] An individual sampling subunit, used for randomly sampling a number of individuals from the search space, wherein the individuals include a sampling time step sequence, and the sampling time step sequence satisfies the preset step number constraint;

[0214] A population initialization subunit, used for initializing the population according to the plurality of individuals;

[0215] A ranking determination subunit, used to determine the individual ranking result corresponding to the population according to the fitness of each individual in the population; the fitness of the individual is determined according to the performance characterization data of the model constructed based on the individual;

[0216] The population evolution subunit is used to perform evolution processing on the population according to the individual sorting result, and return to the sorting determination subunit to continue to execute the step of determining the individual sorting result corresponding to the population according to the fitness of each individual in the population.

[0217] The result determination subunit is used to determine the search result from the population according to the preset performance condition until the search stop condition is reached.

[0218] In a possible implementation manner, the population evolution subunit includes:

[0219] An individual deletion subunit, used to delete individuals that meet a preset deletion condition from the population according to the individual sorting result;

[0220] The crossover mutation subunit is used to randomly exchange the time steps within at least one pair of individuals in the population, and / or to modify the time steps within the individuals in the population according to a preset probability.

[0221] In a possible implementation manner, the search space is further used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence;

[0222] The individual also includes a sampling network structure corresponding to each time step in the sampling time step sequence, and the sampling network structure satisfies the preset structural constraint;

[0223] The crossover mutation subunit is specifically used to: randomly exchange information combinations within at least one pair of individuals in the population, wherein the information combinations include corresponding time steps and network structures existing in the individuals; and / or modify the time steps and / or network structures within the individuals in the population according to a preset probability.

[0224] In one possible implementation, the performance characterization data is determined based on a reference image and a first generated image generated by a model constructed based on the search results.

[0225] In a possible implementation manner, the diffusion model is constructed based on a training image; the reference image includes the training image; and / or the reference image includes a second generated image generated by the diffusion model.

[0226] Based on the relevant contents of the above-mentioned model determination device 400, it can be known that for the model determination device 400 provided in the embodiment of the present application, a search space is first constructed based on a pre-constructed diffusion model, so that the search space is used to provide at least one candidate time step sequence, and the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; then, based on a preset step number constraint (for example, the constraint of 10 time steps) and a preset performance condition (for example, the image generation performance reaches a preset threshold, or the image generation performance reaches the best, etc.), the search space is searched to obtain search results, so that the performance characterization data of the model constructed based on the search result satisfies the preset performance condition, and the target time step sequence in the search result satisfies the preset step number constraint. At this time, due to the preset step number constraint The limited number of steps is smaller than the number of time steps of the reference time step sequence, so that the number of steps of the target time step sequence in the search result is smaller than the number of time steps of the reference time step sequence, thereby making the number of time steps in the model constructed based on the search result smaller than the number of time steps in the diffusion model; then, using the model constructed based on the search result, an image generation model is determined, so that the number of time steps in the image generation model is smaller than the number of time steps in the diffusion model, and the image generation performance of the image generation model is almost close to the image generation performance of the diffusion model, so that compression processing can be performed on the diffusion model while ensuring the image generation performance as much as possible, so that the compressed model (that is, the image generation model) can generate image data at a faster speed, thereby effectively reducing the image generation time consumption.

[0227] Based on the image generation method provided in the embodiment of the present application, the embodiment of the present application also provides an image generation device. Figure 5 Explain and illustrate. Figure 5 This is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. It should be noted that for the technical details of the image generating device provided in an embodiment of the present application, please refer to the relevant contents of the image generating method above.

[0228] like Figure 5 As shown, the image generating device 500 provided in the embodiment of the present application includes:

[0229] An acquisition unit 501 is used to acquire noise data;

[0230] The generating unit 502 is used to perform image generation processing on the noise data by using an image generation model to obtain a third generated image; the image generation model is determined by using the model determination method according to any one of claims 1 to 7.

[0231] Based on the above-mentioned contents of the image generation device 500, it can be known that for the image generation device 500 provided in the embodiment of the present application, a predetermined image generation model can be used to perform image generation processing on any noise data to obtain a generated image corresponding to the noise data. Among them, because the image generation time consumption of the image generation model is relatively low and the image generation performance of the image generation model is relatively good, the image generation model can generate image data with relatively high quality at a relatively fast speed, which is conducive to reducing the image generation time consumption while ensuring the image generation quality.

[0232] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the model determination method provided in the embodiment of the present application or executes any implementation of the image generation method provided in the embodiment of the present application.

[0233] See also Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0234] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0235] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0236] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0237] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0238] An embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the model determination method provided in the embodiment of the present application or executes any implementation of the image generation method provided in the embodiment of the present application.

[0239] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0240] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0241] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0242] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0243] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0244] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0245] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0246] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0247] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0248] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0249] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0250] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0251] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0252] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model determination method, characterized in that: The method comprises: According to the pre-constructed diffusion model, a search space is constructed, wherein the search space is used to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in the reference time step sequence corresponding to the diffusion model; According to a preset step number constraint and a preset performance condition, the search space is searched to obtain a search result, wherein the search result includes a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search result satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps of the reference time step sequence; An image generation model is determined using the model constructed based on the search results.

2. The method according to claim 1, characterized in that The search space is also used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence, wherein the candidate network structure includes at least one feature channel of at least one network layer in the denoising neural network in the diffusion model; The searching of the search space according to the preset step number constraint and the preset performance condition to obtain search results includes: According to a preset step number constraint, a preset structural constraint and a preset performance condition, the search space is searched to obtain search results, wherein the search results also include a target network structure corresponding to each time step in the target time step sequence, and the target network structure satisfies the preset structural constraint.

3. The method according to claim 1, characterized in that The process of determining the search results includes: Randomly sampling a number of individuals from the search space, wherein the individuals include a sampling time step sequence, and the sampling time step sequence satisfies the preset step number constraint; Initializing a population according to the individuals; Determining the individual ranking result corresponding to the population according to the fitness of each individual in the population; the fitness of the individual is determined according to the performance characterization data of the model constructed based on the individual; According to the individual ranking results, the population is evolved, and the step of determining the individual ranking results corresponding to the population according to the fitness of each individual in the population is continued, until the search stop condition is reached, and the search result is determined from the population according to the preset performance condition.

4. The method according to claim 3, characterized in that The evolution process comprises: According to the individual sorting result, deleting individuals that meet a preset deletion condition from the population; The time steps within at least one pair of individuals in the population are randomly exchanged, and / or the time steps within the individuals in the population are modified according to a preset probability.

5. The method according to claim 4, characterized in that The search space is also used to provide at least one candidate network structure corresponding to each time step in the reference time step sequence; The individual also includes a sampling network structure corresponding to each time step in the sampling time step sequence, and the sampling network structure satisfies the preset structural constraint; The randomly exchanging time steps of at least one pair of individuals in the population comprises: Randomly exchanging information combinations within at least one pair of individuals in the population, wherein the information combinations include time steps and network structures with corresponding relationships existing within the individuals; The modifying of the time steps within individuals in the population according to a preset probability comprises: The time steps and / or network structures within individuals in the population are modified according to a preset probability.

6. The method according to claim 1, characterized in that The performance characterizing data is determined based on a reference image and a first generated image generated by a model constructed based on the search results.

7. The method according to claim 6, characterized in that The diffusion model is constructed based on a training image; the reference image includes the training image; and / or, The reference image includes a second generated image generated by the diffusion model.

8. An image generation method, characterized in that: The method comprises: Get noise data; The noise data is processed by image generation using an image generation model to obtain a third generated image; the image generation model is determined using the model determination method described in any one of claims 1 to 7.

9. A model determination device, characterized in that: include: A construction unit, configured to construct a search space according to a pre-constructed diffusion model, wherein the search space is configured to provide at least one candidate time step sequence, wherein the candidate time step sequence includes at least one time step in a reference time step sequence corresponding to the diffusion model; A search unit, configured to search the search space according to a preset step number constraint and a preset performance condition to obtain a search result, wherein the search result includes a target time step sequence, the target time step sequence satisfies the preset step number constraint, performance characterization data of a model constructed based on the search result satisfies the preset performance condition, and the number of steps defined by the preset step number constraint is less than the number of time steps of the reference time step sequence; A determination unit is used to determine an image generation model using the model constructed based on the search results.

10. An image generating device, characterized in that: include: An acquisition unit, used for acquiring noise data; A generating unit, configured to perform image generation processing on the noise data using an image generation model to obtain a third generated image; The image generation model is determined using the model determination method described in any one of claims 1-7.

11. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 8.

12. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 8.

Images