Image generation method and device based on random noise, equipment and storage medium

Abstract

The application discloses a kind of based on random noise's image generation method, device, equipment and storage medium, the present application is only when generating image, only need to utilize neural network model to learn the correlation matrix of noise, therefore, the training of neural network is more stable, not prone to the shortcoming of mode collapse and difficult convergence, avoid the problem that the practicality of generation confrontation network is not possessed due to not easy convergence;Meanwhile, compared to the single variance of traditional diffusion model only learning noise, the present application utilizes pixel correlation matrix as output parameter, this separable Gaussian process that considers horizontal pixel and vertical pixel correlation greatly retains the ability that model learns the correlation between image pixels, therefore, when image generation, it can generate more clear and real image, improve the problem that the correlation between pixels is ignored in prior art;Therefore, the practicability and image authenticity of the present application, compared with prior art, have been greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology based on artificial intelligence, and specifically relates to an image generation method, apparatus, device and storage medium based on random noise. Background Technology

[0002] In recent years, with the development of deep learning and artificial intelligence technologies, a large number of technologies have emerged in the field of image generation based on artificial intelligence algorithms. For example, image generation, video generation, and speech generation technologies based on generative adversarial networks and variational autoencoders are all based on these technologies. These technologies all utilize randomly sampled noise to obtain corresponding content, that is, by controlling the distribution of random sampling, they generate corresponding types of content (such as generating different kinds of images (people, cats, dogs, and other animals)). In addition, by combining natural language processing techniques with noise, language features can be coupled to generate image content corresponding to the input text. For example, inputting "an astronaut in space" can generate a corresponding image. Therefore, image generation technology based on artificial intelligence algorithms has been widely applied in different fields.

[0003] Currently, there are several image generation methods, namely Generative Adversarial Networks (GANs) and diffusion models. GANs generate images by training two independent networks (a generator network and a discriminator network). The generator network transforms random noise to generate an image, while the discriminator network judges whether the generated image is realistic. Through adversarial training, the generator network can eventually generate very realistic images when in equilibrium. However, because GANs use adversarial training, it is difficult for both networks to converge, and in most cases, the generated images are useless. Therefore, they are not practical.

[0004] The diffusion model degrades a photograph into a purely random image through a forward process (noise addition) and then restores the original photograph from the random, noisy image through a reverse process (denoising). The forward process involves adding Gaussian noise with a fixed mean and variance each time, continuously adding noise until the photograph becomes a purely random image. The reverse process learns the mean and variance of each added Gaussian noise and subtracts it until a realistic image is obtained. However, because the mean and variance of the generated Gaussian noise are fixed values, this method cannot learn the correlation between adjacent pixels in the image, thus limiting the realism of the generated image.

[0005] Given the aforementioned shortcomings, how to provide an image generation method that is practical and can improve the realism of generated images has become an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide an image generation method, apparatus, device, and storage medium based on random noise, in order to solve the problems of impracticality and low image fidelity in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Firstly, an image generation method based on random noise is provided, including:

[0009] A target neural network model is obtained, wherein the target neural network model is trained by taking noisy images of a massive number of sample images as input and noise parameters of each sample image as output. The noise parameters of any sample image in the massive number of sample images include a noise matrix, a first correlation matrix, and a second correlation matrix corresponding to the sample image. The first correlation matrix is ​​used to characterize the correlation between horizontally adjacent pixels in the sample image, and the second correlation matrix is ​​used to characterize the correlation between vertically adjacent pixels in the sample image.

[0010] A random noise image is acquired and input into the target neural network model to obtain the noise parameters of the random noise image;

[0011] A target image is generated using the target neural network model, the random noise image, and the noise parameters of the random noise image, wherein the image content of the target image is of the same type as the image content of any sample image.

[0012] Based on the above-disclosed content, this invention first uses a neural network model to learn the mapping relationship between the noisy image and noise parameters. The noise parameters include a noise matrix, a first correlation matrix, and a second correlation matrix. The first correlation matrix characterizes the correlation between horizontally adjacent pixels in the sample image, while the second correlation matrix characterizes the correlation between vertically adjacent pixels in the sample image. This is equivalent to learning the functional relationship between image noise, the noise matrix, and pixel correlation. Then, after the model training is complete and in practical use, simply inputting the acquired random noise image into the trained network yields the noise parameters of the random noise image. Finally, by continuously reconstructing the random noise image using the noise parameters of the random noise image and the trained network, a target image of the same type as the sample image can be obtained.

[0013] Through the above design, this invention only needs to learn the correlation matrix of noise using a neural network model when generating images. Therefore, the training of the neural network is more stable and less prone to mode collapse and convergence difficulties, avoiding the practicality problem caused by the poor convergence of traditional generative adversarial networks. At the same time, compared with traditional diffusion models that only learn the single variance of noise, this invention uses the pixel correlation matrix as the output parameter. This separable Gaussian process that considers the correlation between horizontal and vertical pixels greatly preserves the model's ability to learn the correlation between image pixels. Therefore, it can generate clearer and more realistic images, improving the problem of ignoring the correlation between pixels in existing technologies. Thus, the practicality and image realism of this invention are greatly improved compared with existing technologies, making it suitable for widespread application and promotion in the field of image generation technology.

[0014] In one possible design, the image noise of any sample image is obtained by superimposing noise multiple times on the sample image. Correspondingly, the target image is generated using the target neural network model, the random noise image, and the noise parameters of the random noise image, including:

[0015] Based on the noise parameters of the random noise image, the random noise image is subjected to the t-th denoising process to obtain the t-th preprocessed noise image, where the initial value of t is 1;

[0016] The t-th preprocessed noisy image is input into the target neural network model to obtain the noise parameters of the t-th preprocessed noisy image;

[0017] The random noise image is updated to the t-th preprocessed noise image, the noise parameters of the random noise image are updated to the noise parameters of the t-th preprocessed noise image, and t is incremented by 1. Then, based on the noise parameters of the random noise image, the random noise image is denoised for the t-th time until t equals T. When t equals T, the t-th preprocessed noise image is used as the target image, where T is the total number of noise superpositions for any sample image.

[0018] Based on the above disclosure, this invention discloses a specific process for generating the target image. Specifically, it utilizes the noise parameters of a random noise image to denoise the random noise image, obtaining a pre-processed noise image. Then, the pre-processed noise image is input into a target neural network to obtain the corresponding noise parameters. Next, the pre-processed noise image is used as a random noise image, and the corresponding noise parameters are used to denoise the random noise image again. This process is repeated continuously until the number of denoising operations reaches the total number of noise superpositions, at which point the target image is obtained. Therefore, this invention essentially uses a target neural network to predict the denoising parameters for each iteration, allowing the predicted denoising parameters to be used to continuously denoise the random noise image. After the iterations are completed, the target image is obtained.

[0019] In one possible design, based on the noise parameters of the random noise image, the random noise image is subjected to the t-th denoising process to obtain the t-th preprocessed noise image, including:

[0020] Based on the noise parameters of the random noise image, and according to the following formula (1), the random noise image is denoised for the tth time to obtain the tth preprocessed noise image;

[0021]

[0022] In the above formula (1), x t Let x′ represent the t-th preprocessed noisy image. t Represents a random noise image, β t Σ represents the step size weight of any sample image at the t-th noise superposition. s Σ represents the first correlation matrix in the noise parameters corresponding to the random noise image. n ω represents the second correlation matrix in the noise parameters corresponding to the random noise image, and ω represents the noise matrix in the noise parameters corresponding to the random noise image.

[0023] In one possible design, the loss function of the target neural network model is:

[0024] L=||ε T -Σ′ s ω′Σ′ n || (2)

[0025] In equation (2) above, L represents the loss function, Σ′ s Σ′ represents the first correlation matrix in the noise parameters corresponding to any given sample image. nLet ω' represent the second correlation matrix in the noise parameters corresponding to any given sample image, and let ε represent the noise matrix in the noise parameters corresponding to any given sample image. T The actual image random noise is represented by any sample image, and the actual image random noise is obtained from the noise image of any sample image.

[0026] Based on the above-disclosed content, this invention decomposes the generation of Gaussian noise into a combination of two correlation matrices and a noise matrix. The advantage of this structure is that the noise covariance matrix is ​​no longer a simple diagonal matrix. Thus, complex noise correlation features can be modeled. Compared with the past noise generation that only used a diagonal matrix as the covariance matrix, this invention greatly increases the correlation complexity of noise without significantly increasing the computational complexity. This allows the model to retain the ability to learn the correlation between image pixels, thereby further improving the realism of image generation when the model is used.

[0027] In one possible design, the method before obtaining the target neural network model includes:

[0028] A large number of sample images are acquired, and noise is superimposed multiple times on each sample image in the large number of sample images to obtain a noisy image for each sample image. The total number of noise superpositions for any sample image is determined based on the complexity of the target image.

[0029] Using the noisy image of each sample image as input and the noise parameters of each sample image as output, a neural network model is trained to obtain the target neural network model after training is completed.

[0030] Based on the above disclosure, in this invention, the total number of noise superpositions on the sample image during model training is determined by the complexity of the image to be generated. In other words, the number of iterations of the model's input data is determined by the complexity of the target image. Furthermore, when denoising the random noise image subsequently, the number of denoising iterations is the same as the total number of noise superpositions. Thus, this invention can adjust the number of noise superpositions and denoising iterations according to the complexity of the generated image. Therefore, even when dealing with complex images, the process can still guarantee the realism of the generated image and will not limit the realism and size of the generated image due to the complexity of the model.

[0031] In one possible design, noise is superimposed multiple times on each sample image from a massive dataset to obtain a noisy image for each sample image, including:

[0032] For any sample image, obtain the step size weight at the t-th noise superposition, and based on the step size weight at the t-th noise superposition, perform the t-th noise superposition processing on the sample image to obtain the t-th noise superposition image;

[0033] Increment t by 1 and update any sample image to the t-th noise superimposed image. Then, based on the step size weight at the t-th noise superimposition, perform the t-th noise superimposition process on any sample image again until t equals T. When t equals T, the T-th noise superimposed image is used as the noise image of any sample image, where T represents the total number of noise superimpositions.

[0034] In one possible design, based on the step size weight at the t-th noise superposition, the t-th noise superposition process is performed on any sample image to obtain the t-th noise-superposition image, including:

[0035] Based on the step size weight of the t-th noise superposition, and using the following formula (3), the t-th noise superposition process is performed on any sample image to obtain the t-th noise superposition image;

[0036]

[0037] In the above formula (3), x″′ t Let x″ represent the t-th noise-stacked image. t-1 Let β represent any sample image. t ε represents the step size weight of any sample image at the t-th noise superposition. t Let represent the random noise in any sample image at the t-th noise superposition, where t = 1, 2, ..., T.

[0038] Secondly, an image generation apparatus based on random noise is provided, comprising:

[0039] The model acquisition unit is used to acquire a target neural network model, wherein the target neural network model is trained by taking noisy images of a massive number of sample images as input and noise parameters of each sample image as output. The noise parameters of any sample image in the massive number of sample images include a noise matrix, a first correlation matrix, and a second correlation matrix corresponding to the sample image. The first correlation matrix is ​​used to characterize the correlation between horizontally adjacent pixels in the sample image, and the second correlation matrix is ​​used to characterize the correlation between vertically adjacent pixels in the sample image.

[0040] A noise parameter generation unit is used to acquire a random noise image and input the random noise image into the target neural network model to obtain the noise parameters of the random noise image;

[0041] An image generation unit is used to generate a target image using the target neural network model, the random noise image, and the noise parameters of the random noise image, wherein the image content of the target image is of the same type as the image content of any sample image.

[0042] Thirdly, another image generation apparatus based on random noise is provided. Taking an electronic device as an example, it includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the image generation method based on random noise as described in the first aspect or any possible design of the first aspect.

[0043] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the image generation method based on random noise as described in the first aspect or any possible design of the first aspect.

[0044] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the image generation method based on random noise as described in the first aspect or any possible design of the first aspect.

[0045] Beneficial effects:

[0046] (1) When generating images, this invention only needs to use a neural network model to learn the correlation matrix of noise. Therefore, the training of the neural network is more stable and less prone to mode collapse and difficulty in convergence, thus avoiding the problem of traditional generative adversarial networks being impractical due to difficulty in convergence. At the same time, compared with the traditional diffusion model that only learns the single variance of noise, this invention uses the pixel correlation matrix as the output parameter. This separable Gaussian process that considers the correlation between horizontal and vertical pixels greatly preserves the model's ability to learn the correlation between image pixels. Therefore, when generating images, clearer and more realistic images can be generated, improving the problem of ignoring the correlation between pixels in the prior art. As a result, the practicality and image realism of this invention are greatly improved compared with the prior art, making it suitable for widespread application and promotion in the field of image generation technology.

[0047] (2) The present invention decomposes the generation of Gaussian noise into a combination of two correlation matrices and a noise matrix. The advantage of this structure is that the noise covariance matrix is ​​no longer a simple diagonal matrix. Thus, complex noise correlation features can be modeled. Compared with the noise generation that only used a diagonal matrix as the covariance matrix in the past, the correlation complexity of noise is greatly increased without significantly increasing the computational complexity. This allows the model to retain the ability to learn the correlation between image pixels, thereby improving the realism of image generation when the model is used.

[0048] (3) The present invention can adjust the number of noise superposition and denoising times according to the complexity of the generated image. In this way, when faced with complex images, the process can still guarantee the realism of the generated image and will not limit the realism and size of the generated image due to the complexity of the model, thereby further improving the realism of the generated image. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the steps of an image generation method based on random noise provided in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of the structure of an image generation device based on random noise provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0053] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0054] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0055] Example:

[0056] See Figure 1 As shown, the image generation method based on random noise provided in this embodiment only needs to learn the correlation matrix of noise using a neural network model when generating images. Therefore, the training of the neural network is more stable and less prone to mode collapse and difficulty in convergence. At the same time, the correlation matrix between image pixels is learned using the neural network model and applied to the subsequent image generation process. Furthermore, the total number of noise superpositions during model training and the number of denoising iterations during image generation can be adjusted according to the complexity of the generated image. Thus, compared with the prior art, this method considers the impact of pixel correlation on image realism and the impact of model complexity on realism, and solves the aforementioned problems by adding the pixel correlation matrix and adjusting the number of iterations. As a result, clearer and more realistic images can be generated, which is more suitable for large-scale application and promotion in the field of image generation. In this embodiment, the method can be executed on the image generation end, which can be, but is not limited to, a personal computer (PC), tablet computer, or smartphone. It is understood that the aforementioned execution subject does not constitute a limitation on the embodiments of this application. Accordingly, the operation steps of this method can be, but are not limited to, the steps S1 to S3 below.

[0057] S1. Obtain the target neural network model, wherein the target neural network model is trained using noisy images of a massive number of sample images as input and noise parameters of each sample image as output. The noise parameters of any sample image in the massive number of sample images include the noise matrix, the first correlation matrix, and the second correlation matrix corresponding to the sample image. The first correlation matrix is ​​used to characterize the correlation between horizontally adjacent pixels in the sample image, and the second correlation matrix is ​​used to characterize the correlation between vertically adjacent pixels in the sample image. In specific applications, this embodiment is equivalent to using a neural network model to learn the mapping relationship between the noise of the image and the corresponding noise matrix and pixel correlation matrix, so that the correlation of image pixels can be added to the image generation process in actual use, thereby improving the realism of the generated image. Optionally, one of the training processes of the target neural network model provided below may be, but is not limited to, as shown in steps S01 and S02 below.

[0058] S01. Acquire a massive number of sample images, and perform multiple noise superpositions on each sample image in the massive number of sample images to obtain a noise image for each sample image. The total number of noise superpositions for any sample image is determined based on the complexity of the target image. In this embodiment, the type of sample image is determined based on the type of image to be generated. For example, if the image to be generated (i.e., the target image) is a landscape image, then the collected sample images are all of the landscape image type. Similarly, if the image to be generated is a puppy image, then the collected sample images are of the puppy image type. Of course, other types will not be elaborated on. Optionally, the number of sample images is at least 10,000. It is understood that the number of sample images can be specifically set according to actual use and is not limited to the examples mentioned above. In addition, in this embodiment, the total number of noise superpositions is determined based on the complexity of the target image (i.e., the image to be generated). It can be, but is not limited to, preset by personnel according to the image complexity at the image generation end, and its size is usually between 100 and 200.

[0059] Furthermore, before performing multiple noise superpositions on each sample image, this embodiment also includes a data preprocessing step, such as scaling, averaging, etc., to process all sample images to the same size, and to adjust each pixel of each sample image to the range of (-1,1); thereby, the format of the sample images can be unified.

[0060] After the format of all sample images is unified, noise superposition processing can be performed on each sample image. In this embodiment, since the noise superposition process is the same for each sample image, the following uses any sample image as an example to illustrate the noise superposition process in detail, which may be, but is not limited to, the steps S01 a and S01 b below.

[0061] S01 a. For any sample image, obtain the step size weight at the t-th noise superposition, and based on the step size weight at the t-th noise superposition, perform the t-th noise superposition processing on the sample image to obtain the t-th noise superposition image; In specific applications, since it has been explained above that the total number of noise superpositions is known, the step size weight at each noise superposition can be preset in the image generation end, and in actual use, the corresponding step size weight can be selected according to the number of noise superpositions; Furthermore, the following formula (3) can be used to perform the t-th noise superposition processing on the sample image.

[0062]

[0063] In the above formula (3), x t "′" represents the t-th noise-stacked image, x t " -1 Let β represent any sample image. t ε represents the step size weight of any sample image at the t-th noise superposition. t Let ε represent the random noise in any sample image at the t-th noise superposition, where t = 1, 2, ..., T; in this embodiment, ε t It is obtained based on the mean and variance of the image at the t-th noise superposition, that is, ε at the first noise superposition. t It is obtained based on the mean and variance of any sample image. In the second noise superposition, it is obtained based on the mean and variance of the first noise-superimposed image. Thus, ε t The noise is constantly changing during the superposition process, and when the noise superposition is complete, the final image random noise can be obtained.

[0064] After completing the t-th noise superposition, the next noise superposition can be performed based on the t-th noise superposition image, and this process is repeated until the preset total number of noise superpositions is reached. The cyclic process is shown in step S01b below.

[0065] S01b. Increment t by 1 and update any sample image to the t-th noise superimposed image, and re-perform the t-th noise superimposed processing on any sample image based on the step size weight at the t-th noise superimposed time, until t equals T, so that when t equals T, the T-th noise superimposed image is used as the noise image of any sample image, where T represents the total number of noise superimposed times.

[0066] The following example illustrates steps S01a and S01b: In the first noise superposition, the step size weight and random noise of the image are used, and the first noise superposition image is obtained based on the aforementioned formula (3); then, in the second noise superposition, the step size weight and random noise of the image are used, and the first noise superposition image is processed by the aforementioned formula (3) to obtain the second noise superposition image. In this way, the image obtained from the previous superposition is continuously used to perform noise superposition until the number of superpositions reaches the total number of noise superpositions, and the noise image of any sample can be obtained; of course, the noise superposition process of each other sample image is the same as the noise superposition process of any sample image mentioned above, and will not be repeated here.

[0067] After obtaining the noise image of each sample image, the noise images can be used to form a training set, and the neural network model can be trained using this set, as shown in step S02 below.

[0068] S02. Using the noisy image of each sample image as input and the noise parameters of each sample image as output, train the neural network model to obtain the target neural network model after training. In this embodiment, the neural network model can be, but is not limited to, the U-Net network, and the loss function used is:

[0069] L=||ε T -Σ′ s ω′Σ′ n || (2)

[0070] In equation (2) above, L represents the loss function, Σ′ s Σ′ represents the first correlation matrix in the noise parameters corresponding to any given sample image. n Let ω' represent the second correlation matrix in the noise parameters corresponding to any given sample image, and let ε represent the noise matrix in the noise parameters corresponding to any given sample image. T The actual image random noise is represented by any sample image, and the actual image random noise is obtained from the noise image of any sample image.

[0071] In practical applications, each time a noisy image of a sample image is input into the neural network model, the neural network model will output the noise parameters corresponding to the sample image. At the same time, the noisy image of the input sample image is obtained by multiple noise superpositions. Therefore, the random noise of the image corresponding to the last noise superposition is used as the actual random noise of the input sample image. Then, using the aforementioned equation (2), the loss function value during this training can be calculated. Based on this principle, a loss function can be obtained for each input noisy image. Finally, the convergence of the model can be determined by the loss function value.

[0072] Therefore, through the detailed explanation of the model training process described above, this embodiment decomposes the generation of Gaussian noise into a combination of two correlation matrices and a noise matrix. In this way, complex noise correlation features can be modeled. Compared with the past noise generation that only used a diagonal matrix as the covariance matrix, the correlation complexity of noise is greatly increased without significantly increasing the computational complexity. This allows the model to retain the ability to learn the correlation between image pixels, thereby improving the realism of image generation when the model is used.

[0073] After the model is trained, it can be used to predict the noise parameters of random noise, and then the target image can be generated using the predicted noise parameters. The generation process of the target image is shown in steps S2 and S3 below.

[0074] S2. Obtain a random noise image and input the random noise image into the target neural network model to obtain the noise parameters of the random noise image; in specific applications, the noise parameters of the random noise image include the noise matrix, the first correlation matrix and the second correlation matrix of the random noise image; then, the aforementioned three matrices can be used to continuously remove noise from the random noise image to restore the target image, as shown in step S3 below.

[0075] S3. Using the target neural network model, the random noise image, and the noise parameters of the random noise image, generate a target image, wherein the image content of the target image is of the same type as the image content of any sample image. In specific applications, step S3 is equivalent to using the noise parameters to continuously denoise the random noise image. That is, each time noise is denoised, a new noise image is obtained. Then, the new noise image is input into the target neural network to obtain new noise parameters. Next, the new noise parameters are used to denoise the new noise image to obtain a completely new noise image. At this time, the above process is repeated until the number of denoising times equals the total number of noise superpositions. The image obtained by the last denoising is the target image. Optionally, the specific process of denoising can be, but is not limited to, the steps S31 to S33 below.

[0076] S31. Based on the noise parameters of the random noise image, the random noise image is subjected to the t-th denoising process to obtain the t-th preprocessed noise image, where the initial value of t is 1; in specific applications, the random noise image can be subjected to the t-th denoising process according to the following formula (1) to obtain the t-th preprocessed noise image.

[0077]

[0078] In the above formula (1), x t Let x represent the t-th preprocessed noisy image. t ' represents a random noise image, β t Σ represents the step size weight of any sample image at the t-th noise superposition. s Σ represents the first correlation matrix in the noise parameters corresponding to the random noise image. n ω represents the second correlation matrix in the noise parameters corresponding to the random noise image, and ω represents the noise matrix in the noise parameters corresponding to the random noise image.

[0079] After calculating the t-th preprocessed noise image based on the aforementioned formula (1), it can be input into the target neural network model to obtain its corresponding noise parameters. Then, the t-th preprocessed noise image is used as a random noise image, and the noise parameters of the t-th preprocessed noise image are used as the noise parameters of the random noise image. Step S31 is then executed again to complete the second denoising process. This principle is repeated continuously to complete multiple denoising processes. The cyclic process is shown in steps S32 and S33 below.

[0080] S32. Input the t-th preprocessed noise image into the target neural network model to obtain the noise parameters of the t-th preprocessed noise image.

[0081] S33. Update the random noise image to the t-th preprocessed noise image, update the noise parameters of the random noise image to the noise parameters of the t-th preprocessed noise image, increment t by 1, and re-perform denoising processing on the random noise image based on the noise parameters of the random noise image until t equals T, so that when t equals T, the t-th preprocessed noise image is used as the target image, where T is the total number of noise superpositions for any sample image.

[0082] The following example illustrates steps S31 to S33: First, the random noise image is denoised for the first time using the formula (1) to obtain the first pre-processed noise image. Then, the first pre-processed noise image is input into the target neural network to obtain the noise parameters of the first pre-processed noise image. At this time, the random noise image is updated to the first pre-processed noise image, and the noise parameters of the random noise image are updated to the noise parameters of the first pre-processed noise image. Then, the new noise parameters (i.e., the noise parameters of the first pre-processed noise image) are used to perform a second denoising process on the new random noise image (i.e., the first pre-processed noise image) to obtain the second pre-processed noise image. Similarly, the random noise image and noise parameters are updated (i.e., the random noise image is the second pre-processed noise image, and the corresponding noise parameters are updated to the noise parameters of the second pre-processed noise image). The updated noise parameters are used to perform a third denoising process on the updated random noise image. This process is repeated until the number of denoising operations equals the total number of noise superpositions. When the number of denoising operations equals the total number of noise superpositions, the target image is obtained.

[0083] In this embodiment, since the number of denoising iterations is equal to the total number of noise superpositions of the input noisy image during model training, and the total number of noise superpositions is determined by the complexity of the image to be generated, this method can adjust the number of iterations according to the complexity of the generated image when generating the image. Therefore, when facing complex images, this process can still guarantee the realism of the generated image and will not limit the realism and size of the generated image due to the complexity of the model.

[0084] Therefore, through the image generation method based on random noise described in detail in steps S1 to S3 above, this invention only needs to use a neural network model to learn the correlation matrix of noise when generating images. Therefore, the training of the neural network is more stable and less prone to mode collapse and difficulty in convergence. At the same time, the neural network model learns the correlation matrix between image pixels and applies the pixel correlation matrix to the subsequent image generation process. Furthermore, the total number of noise superpositions during model training and the number of denoising steps during image generation can be adjusted according to the complexity of the generated image. Thus, compared with the prior art, this invention considers the impact of pixel correlation on image realism and the impact of model complexity on realism, and can generate clearer and more realistic images, making it more suitable for large-scale application and promotion in the field of image generation.

[0085] like Figure 2 As shown, the second aspect of this embodiment provides a hardware apparatus for implementing the image generation method based on random noise described in the first aspect of the embodiment, comprising:

[0086] The model acquisition unit is used to acquire a target neural network model, wherein the target neural network model is trained by taking noisy images of a massive number of sample images as input and noise parameters of each sample image as output. The noise parameters of any sample image in the massive number of sample images include a noise matrix, a first correlation matrix, and a second correlation matrix corresponding to the sample image. The first correlation matrix is ​​used to characterize the correlation between horizontally adjacent pixels in the sample image, and the second correlation matrix is ​​used to characterize the correlation between vertically adjacent pixels in the sample image.

[0087] The noise parameter generation unit is used to acquire a random noise image and input the random noise image into the target neural network model to obtain the noise parameters of the random noise image.

[0088] An image generation unit is used to generate a target image using the target neural network model, the random noise image, and the noise parameters of the random noise image, wherein the image content of the target image is of the same type as the image content of any sample image.

[0089] The working process, working details and technical effects of the device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0090] like Figure 3 As shown, the third aspect of this embodiment provides another image generation apparatus based on random noise. Taking the apparatus as an electronic device as an example, it includes: a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the image generation method based on random noise as described in the first aspect of the embodiment.

[0091] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0092] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0093] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0094] The fourth aspect of this embodiment provides a storage medium storing instructions for an image generation method based on random noise as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the image generation method based on random noise as described in the first aspect.

[0095] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0096] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0097] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the image generation method based on random noise as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0098] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image generation method based on random noise, characterized in that, include: A target neural network model is obtained, wherein the target neural network model is trained by taking noisy images of a massive number of sample images as input and noise parameters of each sample image as output. The noise parameters of any sample image in the massive number of sample images include a noise matrix, a first correlation matrix, and a second correlation matrix corresponding to the sample image. The first correlation matrix is ​​used to characterize the correlation between horizontally adjacent pixels in the sample image, and the second correlation matrix is ​​used to characterize the correlation between vertically adjacent pixels in the sample image. A random noise image is acquired and input into the target neural network model to obtain the noise parameters of the random noise image; A target image is generated using the target neural network model, the random noise image, and the noise parameters of the random noise image, wherein the image content of the target image is of the same type as the image content of any sample image.

2. The method according to claim 1, characterized in that, The image noise of any sample image is obtained by superimposing noise multiple times on the sample image. Correspondingly, generating the target image using the target neural network model, the random noise image, and the noise parameters of the random noise image includes: Based on the noise parameters of the random noise image, the random noise image is subjected to the t-th denoising process to obtain the t-th preprocessed noise image, where the initial value of t is 1; The t-th preprocessed noisy image is input into the target neural network model to obtain the noise parameters of the t-th preprocessed noisy image; The random noise image is updated to the t-th preprocessed noise image, the noise parameters of the random noise image are updated to the noise parameters of the t-th preprocessed noise image, and t is incremented by 1. Then, based on the noise parameters of the random noise image, the random noise image is denoised for the t-th time until t equals T. When t equals T, the t-th preprocessed noise image is used as the target image, where T is the total number of noise superpositions for any sample image.

3. The method according to claim 2, characterized in that, Based on the noise parameters of the random noise image, the random noise image is subjected to the t-th denoising process to obtain the t-th preprocessed noise image, including: Based on the noise parameters of the random noise image, and according to the following formula (1), the random noise image is denoised for the tth time to obtain the tth preprocessed noise image; In the above formula (1), x t Let x represent the t-th preprocessed noisy image. t ' represents a random noise image, β t Σ represents the step size weight of any sample image at the t-th noise superposition. s Σ represents the first correlation matrix in the noise parameters corresponding to the random noise image. n ω represents the second correlation matrix in the noise parameters corresponding to the random noise image, and ω represents the noise matrix in the noise parameters corresponding to the random noise image.

4. The method according to claim 1, characterized in that, The loss function of the target neural network model is: L=e T -S′ s as' n (2) In equation (2) above, L represents the loss function, Σ′ s Σ′ represents the first correlation matrix in the noise parameters corresponding to any given sample image. n Let ω' represent the second correlation matrix in the noise parameters corresponding to any given sample image, and let ε represent the noise matrix in the noise parameters corresponding to any given sample image. T The actual image random noise is represented by any sample image, and the actual image random noise is obtained from the noise image of any sample image.

5. The method according to claim 1, characterized in that, Before obtaining the target neural network model, the method includes: A large number of sample images are acquired, and noise is superimposed multiple times on each sample image in the large number of sample images to obtain a noisy image for each sample image. The total number of noise superpositions for any sample image is determined based on the complexity of the target image. Using the noisy image of each sample image as input and the noise parameters of each sample image as output, a neural network model is trained to obtain the target neural network model after training is completed.

6. The method according to claim 5, characterized in that, Multiple noise superpositions are performed on each sample image in a massive dataset to obtain a noisy image for each sample image, including: For any sample image, obtain the step size weight at the t-th noise superposition, and based on the step size weight at the t-th noise superposition, perform the t-th noise superposition processing on the sample image to obtain the t-th noise superposition image; Increment t by 1 and update any sample image to the t-th noise superimposed image. Then, based on the step size weight at the t-th noise superimposition, perform the t-th noise superimposition process on any sample image again until t equals T. When t equals T, the T-th noise superimposed image is used as the noise image of any sample image, where T represents the total number of noise superimpositions.

7. The method according to claim 6, characterized in that, Based on the step size weight at the t-th noise superposition, the t-th noise superposition process is performed on any sample image to obtain the t-th noise superposition image, including: Based on the step size weight of the t-th noise superposition, and using the following formula (3), the t-th noise superposition process is performed on any sample image to obtain the t-th noise superposition image; In the above formula (3), x t "′" represents the t-th noise-stacked image, x t " -1 Let β represent any sample image. t ε represents the step size weight of any sample image at the t-th noise superposition. t Let represent the random noise in any sample image at the t-th noise superposition, where t = 1, 2, ..., T.

8. An image generation device based on random noise, characterized in that, include: The model acquisition unit is used to acquire a target neural network model, wherein the target neural network model is trained by taking noisy images of a massive number of sample images as input and noise parameters of each sample image as output. The noise parameters of any sample image in the massive number of sample images include a noise matrix, a first correlation matrix, and a second correlation matrix corresponding to the sample image. The first correlation matrix is ​​used to characterize the correlation between horizontally adjacent pixels in the sample image, and the second correlation matrix is ​​used to characterize the correlation between vertically adjacent pixels in the sample image. A noise parameter generation unit is used to acquire a random noise image and input the random noise image into the target neural network model to obtain the noise parameters of the random noise image; An image generation unit is used to generate a target image using the target neural network model, the random noise image, and the noise parameters of the random noise image, wherein the image content of the target image is of the same type as the image content of any sample image.

9. An electronic device, characterized in that, include: A memory, a processor, and a transceiver are sequentially connected in communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the image generation method based on random noise as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores instructions that, when executed on a computer, perform the image generation method based on random noise as described in any one of claims 1 to 7.

Images