Image generation method and device and computer

Abstract

The invention discloses an image generation method and the like, which are used for generating a high-quality image. The method comprises the following steps: acquiring a target vector; respectively inputting the target vector into a first generator and a second generator to correspondingly generate a first sub-graph and a second sub-graph, the first generator being obtained by training an initially configured first generative adversarial network GAN by a server according to a low-frequency image and a first random noise variable satisfying normal distribution, and the second generator being obtained by training an initially configured second generative adversarial network GAN according to a second random noise variable satisfying normal distribution; the second generator is obtained by training an initially configured second generative adversarial network (GAN) by the server according to the high-frequency image and a second random noise variable satisfying normal distribution, and the frequency of the low-frequency image is lower than that of the high-frequency image; and synthesizing the first sub-image and the second sub-image to obtain a target image.

Description

[0001] This application is a divisional application. The original application has the application number 202010695936.6 and the original application date is July 17, 2020. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more particularly to a method, apparatus, and computer for image generation. Background Technology

[0003] Image generation is one of the most important research areas in computer vision, and it is applied to related technologies such as image inpainting, image classification, and virtual reality. In the development of autonomous driving technology, the diversity of generated scenes and the preservation of scene objects are two distinct technical challenges. One reason is that the complexity of scenes makes learning the mapping between various attribute variables and the high-dimensional representation of the image one of the unsolved problems in academia. Another reason is the significant variations in pixel values ​​caused by lighting, scale, and occlusion in outdoor scene images; compared to the very robust recognition performance of humans, existing algorithms still have a long way to go in this regard.

[0004] Current image generation technology has achieved certain results in the field of neural networks, especially Generative Adversarial Networks (GANs), which have achieved the best performance in this task. A GAN consists of at least one generator and one discriminator. The generator is a network structure that uses random noise variables to produce images; ideally, the generated images should be very similar to real images. The discriminator is a metric network used to distinguish between real and generated images. GANs improve their performance through game-like learning between the generator and the discriminator, thereby generating high-quality images from the input variables when performance requirements are met.

[0005] However, the biggest drawback of existing generative adversarial networks is the instability of the generation process, which leads to low image quality generated by generative adversarial networks. Summary of the Invention

[0006] This application provides an image generation method, apparatus, computer, storage medium, chip system, etc., for improving image generation quality using GAN technology.

[0007] In a first aspect, this application provides an image generation method, which may include: obtaining a target vector; inputting the target vector into a first generator and a second generator respectively to generate a first sub-image and a second sub-image, wherein the first generator is obtained by a server training an initially configured first generative adversarial network (GAN) based on a low-frequency image and a first random noise variable that satisfies a normal distribution, and the second generator is obtained by the server training an initially configured second generative adversarial network (GAN) based on a high-frequency image and a second random noise variable that satisfies a normal distribution, wherein the frequency of the low-frequency image is lower than the frequency of the high-frequency image; and synthesizing the first sub-image and the second sub-image to obtain a target image.

[0008] In some possible implementations of the first aspect, the method may further include: acquiring the low-frequency image and the high-frequency image; acquiring the first random noise variable and the second random noise variable; setting the low-frequency image and the high-frequency image as training samples for the first GAN and the second GAN, respectively; training the first GAN using the low-frequency image and the first random noise variable to obtain the first generator; and training the second GAN using the high-frequency image and the second random noise variable to obtain the second generator.

[0009] In some possible implementations of the first aspect, acquiring the low-frequency image and the high-frequency image may include: acquiring an original image; performing wavelet transform processing on the original image to obtain the low-frequency image and the high-frequency image. Synthesizing the first sub-image and the second sub-image to obtain the target image may include: synthesizing the first sub-image and the second sub-image using inverse wavelet transform processing to obtain the target image.

[0010] In some possible implementations of the first aspect, performing wavelet transform processing on the original image to obtain the low-frequency image and the high-frequency image may include: performing discrete wavelet transform processing on the original image to obtain at least one low-frequency image and at least one high-frequency image containing K resolutions, wherein the Q-th resolution corresponds to M Q A low-frequency image and N Q A high-frequency image, K, M Q N Q All values ​​are positive integers, Q = 1, 2, 3...K; the first GAN is trained using the low-frequency image and the first random noise variable to obtain the first generator, which may include: using the M at the Qth resolution. Q A low-frequency image and the first random noise variable relative to the initially configured S Q Train a low-frequency GAN to obtain S Q A low-frequency generator, where S QThe second generator is an integer greater than or equal to 1; the second GAN is trained using the high-frequency image and the second random noise variable to obtain the second generator, which may include: using the N at the Q-th resolution. Q The high-frequency image and the second random noise variable are compared with the Wth image in the initial configuration. Q Train a high-frequency GAN to obtain W Q A high-frequency generator, where W Q The target vector is an integer greater than or equal to 1; the target vector is input into the first generator and the second generator respectively to generate the first subgraph and the second subgraph respectively, which may include: inputting the target vector into the first generator and the second generator respectively to generate the first subgraph and the second subgraph respectively. Low-frequency generator and A high-frequency generator is input to the target vector to obtain... A low-frequency generated subgraph and A high-frequency sub-image is generated; the first sub-image and the second sub-image are synthesized using wavelet transform processing to obtain the target image, which may include: processing the first sub-image and the second sub-image using discrete wavelet inverse transform processing. A low-frequency generated subgraph and the The target image is obtained by synthesizing a high-frequency generated sub-image.

[0011] In some possible implementations of the first aspect, the process of training any generator further includes: using the output of any one or more other generators as the input of the generator, wherein the other one or more generators include any one or more generators other than the generator among the low-frequency generator and the high-frequency generator.

[0012] In some possible implementations of the first aspect, any two random noise variables, the first random noise variable and the second random noise variable, are orthogonal.

[0013] In some possible implementations of the first aspect, the M Q Each low-frequency image may include a first low-frequency image, the N Q The high-frequency image may include a first high-frequency image, a second high-frequency image, and a third high-frequency image. The first low-frequency image may include low-frequency information of the original image in the vertical and horizontal directions. The first high-frequency image may include low-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The second high-frequency image may include high-frequency information of the original image in the vertical direction and low-frequency information in the horizontal direction. The third high-frequency image may include high-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The M image at the Qth resolution... Q A low-frequency image and the first random noise variable relative to the initially configured S Q Train a low-frequency GAN to obtain SQ A low-frequency generator, comprising: utilizing the M at the Qth resolution Q The first low-frequency GAN is trained using the first low-frequency image and the first random noise variable to obtain the Q-th low-frequency generator; the N-th low-frequency generator is then used at the Q-th resolution. Q The high-frequency image and the second random noise variable are compared with the Wth image in the initial configuration. Q Train a high-frequency GAN to obtain W Q A high-frequency generator is provided, comprising: training an initially configured Q-th first high-frequency GAN using the first high-frequency image at the Q-th resolution and a third random noise variable to obtain a Q-th first high-frequency generator; training an initially configured Q-th second high-frequency GAN using the second high-frequency image at the Q-th resolution and a fourth random noise variable to obtain a Q-th second high-frequency generator; and training an initially configured Q-th third high-frequency GAN using the third high-frequency image at the Q-th resolution and a fifth random noise variable to obtain a Q-th third high-frequency generator; the step of respectively sending... Low-frequency generator and A high-frequency generator is input to the target vector to obtain... A low-frequency generated subgraph and The process involves generating K high-frequency sub-graphs, including: inputting the target vector into K low-frequency generators, K first high-frequency generators, K second high-frequency generators, and K third high-frequency generators respectively, to obtain K low-frequency generated sub-graphs, K first high-frequency generated sub-graphs, K second high-frequency generated sub-graphs, and K third high-frequency generated sub-graphs; and employing discrete wavelet inverse transform processing on the... A low-frequency generated subgraph and the The target image is obtained by synthesizing the K low-frequency generated sub-images, the K first high-frequency generated sub-images, the K second high-frequency generated sub-images, and the K third high-frequency generated sub-images using discrete wavelet inverse transform processing.

[0014] In some possible implementations of the first aspect, acquiring the original image; performing discrete cosine transform processing on the original image to obtain the low-frequency image and the high-frequency image; and synthesizing the first sub-image and the second sub-image to obtain the target image may include: synthesizing the first sub-image and the second sub-image by performing inverse discrete cosine transform processing to obtain the target image.

[0015] In some possible implementations of the first aspect, acquiring the original image; performing Fourier transform processing on the original image to obtain the low-frequency image and the high-frequency image; and synthesizing the first sub-image and the second sub-image to obtain the target image may include: synthesizing the first sub-image and the second sub-image by using inverse Fourier transform processing to obtain the target image.

[0016] In some possible implementations of the first aspect, the method further includes: superimposing the target image with images generated by other generators to obtain the final target image, wherein the superposition can be a weighted combination. It should be noted that the other generators can be any type of generator in the prior art, and these generators will also participate in the training process. Weight adjustment factor It can learn independently based on the dataset, and different datasets are used for different scenarios. The values ​​are different.

[0017] Secondly, this application provides an apparatus for image generation. This apparatus can be a computer, which can be a terminal device or a server. For example, the computer can be a smartphone, smart TV (or smart screen), virtual reality device, augmented reality device, mixed reality device, or in-vehicle device (including devices used in assisted driving and autonomous driving), etc., devices that require high image quality. The apparatus can also be considered as a software program, executed by one or more processors to implement its functions. The apparatus can also be considered as hardware, including multiple functional circuits for implementing its functions. Furthermore, the apparatus can be considered as a combination of software and hardware.

[0018] The device includes a transceiver unit for acquiring a target vector; and a processing unit for inputting the target vector into a first generator and a second generator respectively to generate a first sub-image and a second sub-image. The first generator is obtained by training a first generative adversarial network (GAN) with an initial configuration based on a low-frequency image and a first random noise variable that follows a normal distribution. The second generator is obtained by training a second generative adversarial network (GAN) with an initial configuration based on a high-frequency image and a second random noise variable that follows a normal distribution. The frequency of the low-frequency image is lower than the frequency of the high-frequency image. The first sub-image and the second sub-image are then synthesized to obtain the target image.

[0019] In some possible implementations of the second aspect, the transceiver unit is further configured to acquire the low-frequency image, the high-frequency image, the first random noise variable, and the second random noise variable; the processing unit is further configured to set the low-frequency image and the high-frequency image as training samples for the first GAN and the second GAN, respectively; train the first GAN using the low-frequency image and the first random noise variable to obtain the first generator; and train the second GAN using the high-frequency image and the second random noise variable to obtain the second generator.

[0020] In some possible implementations of the second aspect, the transceiver unit is specifically used to acquire the original image; the processing unit is specifically used to perform wavelet transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse wavelet transform processing to obtain the target image.

[0021] In some possible implementations of the second aspect, the processing unit is specifically configured to perform discrete wavelet transform processing on the original image to obtain at least one low-frequency image and at least one high-frequency image containing K resolutions, wherein the Q-th resolution corresponds to M. Q A low-frequency image and N Q A high-frequency image, K, M Q N Q All are positive integers, Q = 1, 2, 3...K; using the M at the Qth resolution... Q A low-frequency image and the first random noise variable relative to the initially configured S Q Train a low-frequency GAN to obtain S Q A low-frequency generator, where S Q It is an integer greater than or equal to 1; using the N at the Qth resolution. Q The high-frequency image and the second random noise variable are compared with the Wth image in the initial configuration. Q Train a high-frequency GAN to obtain W Q A high-frequency generator, where W Q For integers greater than or equal to 1; respectively to Low-frequency generator and A high-frequency generator is input to the target vector to obtain... A low-frequency generated subgraph and A high-frequency generated sub-graph; processed using inverse discrete wavelet transform. A low-frequency generated subgraph and the The target image is obtained by synthesizing a high-frequency generated sub-image.

[0022] In some possible implementations of the second aspect, the processing unit is used during the training of any generator to: take the output of any one or more other generators as the input of the generator, wherein the other one or more generators include any one or more generators other than the generator among the low-frequency generator and the high-frequency generator.

[0023] In some possible implementations of the second aspect, any two random noise variables, the first random noise variable and the second random noise variable, are orthogonal.

[0024] In some possible implementations of the second aspect, the M Q Each low-frequency image may include a first low-frequency image, the N Q The high-frequency image may include a first high-frequency image, a second high-frequency image, and a third high-frequency image. The first low-frequency image may include low-frequency information of the original image in the vertical and horizontal directions. The first high-frequency image may include low-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The second high-frequency image may include high-frequency information of the original image in the vertical direction and low-frequency information in the horizontal direction. The third high-frequency image may include high-frequency information of the original image in the vertical and horizontal directions. Specifically, the processing unit is used to utilize the M at the Qth resolution. Q The first low-frequency GAN is trained using the first low-frequency image and the first random noise variable to obtain the Qth low-frequency generator; the Qth high-frequency GAN in the initial configuration is trained using the first high-frequency image at the Qth resolution and the third random noise variable to obtain the Qth high-frequency generator; the Qth high-frequency GAN in the initial configuration is trained using the second high-frequency image at the Qth resolution and the fourth random noise variable to obtain the Qth high-frequency generator; the Qth high-frequency GAN in the initial configuration is trained using the third high-frequency image at the Qth resolution and the fifth random noise variable. The Qth third high-frequency GAN is trained to obtain the Qth third high-frequency generator; the target vector is input into K low-frequency generators, K first high-frequency generators, K second high-frequency generators and K third high-frequency generators respectively to obtain K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images and K third high-frequency generated sub-images; the K low-frequency generated sub-images, the K first high-frequency generated sub-images, the K second high-frequency generated sub-images and the K third high-frequency generated sub-images are synthesized by discrete wavelet inverse transform to obtain the target image.

[0025] In some possible implementations of the second aspect, the transceiver unit is specifically used to acquire the original image; the processing unit is specifically used to perform discrete cosine transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse discrete cosine transform processing to obtain the target image.

[0026] In some possible implementations of the second aspect, the transceiver unit is specifically used to acquire the original image; the processing unit is specifically used to perform Fourier transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse Fourier transform processing to obtain the target image.

[0027] In some possible implementations of the second aspect, the method further includes a superposition unit for superimposing the target image with images generated by other generators to obtain a final target image. This superposition can be a weighted combination. It should be noted that the other generators can be any type of generator in the prior art, and these generators will also participate in the training process.

[0028] A third aspect of this application provides a computer for image generation, which may include: a processor, a memory, and a transceiver; the transceiver is used to communicate with a device other than the computer; the memory is used to store instruction code; when the processor executes the instruction code, it causes the computer to perform the method as described in the first aspect and any one of the first aspects.

[0029] A fourth aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect and any one of the first aspects.

[0030] A fifth aspect of this application provides a computer program product that may include instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect and any one of the first aspects.

[0031] A sixth aspect of this application provides a chip system including an interface and a processing circuit. The chip system acquires a software program through the interface and executes the software program through the processing circuit to implement the method as described in the first aspect and any one of the first aspects.

[0032] A seventh aspect of this application provides a chip system including one or more functional circuits, the one or more functional circuits being used to implement the method as described in the first aspect and any one of the first aspects.

[0033] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: After training the first and second generators, the computer inputs the target variable into each generator, generating a first sub-image and a second sub-image respectively. These sub-images are then combined to obtain the target image. Since the first generator is trained on an initially configured first GAN using a first random noise variable and a low-frequency image, and the second generator is trained on an initially configured second GAN using a second random noise variable and a high-frequency image, the generated first and second sub-images are also low-frequency and high-frequency images, respectively. It should be noted that, according to the definition of image frequency, high-frequency images better reflect the detailed information of an image, such as the contour information of various main features, while low-frequency information better reflects the main information of an image, such as grayscale and color information. In this scheme, generating low-frequency and high-frequency images separately allows for better preservation of the detailed and main information of the target image during the generation process, thus ensuring that the generated target image has better quality. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A schematic diagram of the structure of an existing generative adversarial network; Figure 2 This is a schematic diagram of the existing process for image generation using GAN technology. Figure 3 This is a schematic diagram of the structure of an existing convolutional neural network; Figure 4 This is a schematic diagram of another existing convolutional neural network structure; Figure 5 A schematic diagram of an embodiment of an image generation method provided in this application; Figure 6 A schematic diagram illustrating an embodiment of a system architecture provided in this application; Figure 7 This is a schematic diagram of another embodiment of an image generation method provided in this application. Figure 8 A schematic diagram illustrating another system architecture embodiment provided for the purposes of this application; Figure 9A schematic diagram of one embodiment of a server provided in this application; Figure 10 This is a schematic diagram of another embodiment of a server provided in this application. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0037] In recent years, artificial intelligence and deep learning have become household names. Generally speaking, deep learning models can be divided into discriminative models and generative models. Due to the invention of algorithms such as back propagation (BP) and dropout, discriminative models have developed rapidly. However, generative models have developed slowly due to the difficulty of modeling them, until the invention of GANs in recent years, which has revitalized this field. With the rapid development of GANs in both theory and models, they are finding increasingly in-depth applications in computer vision, natural language processing, and human-computer interaction, and are continuously expanding into other fields.

[0038] Among them, such as Figure 1 As shown, Figure 1 This is a schematic diagram of the GAN structure, whose basic structure includes a generator and a discriminator. Inspired by zero-sum games in game theory, GAN technology views the generation problem as an adversarial game between two networks: the discriminator and the generator. The generator uses given noise (generally uniformly or normally distributed) to produce synthetic data, while the discriminator distinguishes the generator's output from real data. The former attempts to generate data closer to reality, while the latter attempts to more perfectly distinguish between real and generated data. Thus, the two networks improve through this adversarial process, and continue to compete after improvement. The data obtained by the generator becomes increasingly perfect, approximating real data, thereby enabling the generation of desired data (images, sequences, videos, etc.).

[0039] Specifically, let's take its application in image processing as an example. The existing workflow for image generation using GAN technology can be found by referring to... Figure 2 The flowchart shown below provides a brief description of each step.

[0040] S201. Initial GAN ​​Configuration on Server: When using GAN for image generation, it is necessary to initially configure GAN on the server. The generator and discriminator in the initial configuration may have weak performance and need to be trained.

[0041] S202. Server obtains random noise variables and original images: After the initial configuration of GAN on the server, at least one random noise variable and at least one original image can be input into GAN.

[0042] S203. The server uses the original image as a training sample and trains the GAN using random noise variables and the original image: After obtaining the random noise variables and the original image, the server sets the original image as the training sample for the initially configured GAN and uses the generator in the GAN to transform the random noise variables into generated images that deceive the discriminator. Then, the server randomly selects one image from the original image and the generated image as input and transmits it to the discriminator. The discriminator is essentially similar to a binary classifier. After receiving the image transmitted by the generator, it discriminates the received image, determining whether it comes from the original image or the image generated by the generator, and derives the probability value that the image is the original image. After each probability value is calculated, the GAN can calculate the loss function corresponding to the generator and discriminator based on the probability value, and use the backpropagation algorithm to perform gradient backpropagation, updating the parameters of the discriminator and generator sequentially according to the loss function. Specifically, when updating the discriminator and generator, an alternating iterative update strategy is adopted, that is, first fix the generator and update the parameters of the discriminator, then fix the discriminator again and update the parameters of the generator. After updating the parameters of the discriminator and generator, the generator's "forgery" capability and the discriminator's "fake detection" capability can be further improved. GAN performs a "generate-discriminate-update" process multiple times, ultimately enabling the discriminator to accurately determine whether an image is the original image, and the generator's probability distribution function, generated using a first random noise variable, approximates the probability distribution function of the original image. At this point, the discriminator cannot determine whether the image it transmits is real or fake, thus achieving Nash equilibrium between the generator and discriminator. When Nash equilibrium is reached, GAN training is complete.

[0043] S204. When GAN training is complete, the server removes the discriminator from the initially configured GAN and retains the generator: When GAN training is complete, the generator in the initially configured GAN meets the set performance requirements. At this time, the server can remove the discriminator network from the GAN and retain the generator as an image generation model.

[0044] S205. Server obtains target variables: After the server trains the GAN and obtains the generator after training, it obtains the target variables when it needs to generate the target image.

[0045] S206. The server uses the trained generator to process the target variable to obtain the target image: After obtaining the target variable, the server inputs it into the generator, which then processes it to generate the target image. In practical applications, the target variable can be a random noise variable obtained from external input or generated by the server itself, or it can be a specific variable containing the image feature information to be generated. Specifically, for example, if the original image consists of multiple real-world landscape images, and the target variable is a random noise variable, the final output target image can be a composite image with a style similar to the original images; if the target variable contains the image feature information to be generated (e.g., the image elements need to include mountains and their outlines), the final output target image can be a composite image containing image feature information and with a style similar to the original images.

[0046] In the earliest GAN theory, it wasn't required that both the generator and discriminator be neural networks; they only needed to be able to fit the corresponding generation and discrimination functions. However, due to the excellent fitting and expressive capabilities of neural networks, with the development of GANs, the networks for both the generator and discriminator are now mostly implemented using neural networks. Specifically, when applied to image processing, a more powerful improvement to GAN is the Deep Convolutional Generative Adversarial Network (DCGAN). In DCGAN, the discriminator uses a convolutional neural network (CNN), while the generator uses an anti-CNN.

[0047] The discriminator uses a convolutional neural network, a deep neural network with a convolutional structure. It's a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network where each neuron responds to overlapping regions in the input image.

[0048] like Figure 3 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0049] Convolutional / pooling layers 120: Convolutional layers: like Figure 3The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0050] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0051] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0052] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0053] Pooling layer: Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 3 Layers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0054] Neural network layer 130: After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 3 As shown in 131, 132 to 13n) and output layer 140, these multi-layer hidden layers, also known as fully connected layers, contain parameters that can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0055] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (such as...), the loss function is applied. Figure 3 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 3 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0056] It should be noted that, as Figure 3 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 4 The multiple convolutional / pooling layers shown are used in parallel, and the extracted features are all input into the neural network layer 130 for processing.

[0057] The generator corresponds to the discriminator, and it uses a deconvolutional neural network. In the generator's deconvolutional neural network, the operation performed is deconvolution, or transpose convolution.

[0058] The above provides a brief description of the current process for image generation using GAN technology. It can be seen that image generation using GAN can include two processes: training the GAN and using it. However, currently, when using GAN technology for image processing, the difficulty in training deep neural networks and the instability of the training process often make it impossible to guarantee the quality of the generated target image.

[0059] Based on the above description, this application provides an image generation method for generating high-quality images. Specifically, after training a first generator and a second generator, the server inputs target variables into the first generator and the second generator respectively, generating a first sub-image and a second sub-image accordingly. Since the first generator is pre-trained using a first random noise variable and a low-frequency image on an initially configured first GAN, and the second generator is pre-trained using a second random noise variable and a high-frequency image on an initially configured second GAN, the corresponding first and second sub-images generated are also low-frequency and high-frequency images, respectively. Then, the first and second sub-images are synthesized to obtain the target image.

[0060] It's important to note that image frequency, also known as spatial frequency, refers to the number of sinusoidal modulation cycles of the brightness and darkness of an image or stimulus graphic per degree of viewing angle, measured in cycles per degree. It reflects the spatial variation of pixel grayscale values. Specifically, if an image has a flat grayscale distribution, such as an image of a wall, its low-frequency components are stronger, while its high-frequency components are weaker. Conversely, if an image exhibits drastic grayscale variations, such as a satellite map with crisscrossing ravines, its high-frequency components will be relatively stronger, while its low-frequency components will be weaker. Therefore, low-frequency images better reflect the main information of an image, such as the color and grayscale information of its main features, while high-frequency images better reflect the detailed information, such as the contour edges of its main features. Thus, generating the first and second sub-images and then synthesizing them can better preserve both the main and detailed information of the target image, resulting in a higher quality target image.

[0061] The following is combined with Figure 5 , Figure 5 An embodiment of image generation provided in this application is illustrated, including: S501, The server acquires low-frequency images, high-frequency images, a first random noise variable, and a second random noise variable.

[0062] In one specific embodiment, the server is initially configured with a first GAN and a second GAN. Before training the first GAN and the second GAN, it is necessary to acquire at least one low-frequency image, at least one high-frequency image, a first random noise variable, and a second random noise variable. The high-frequency image has a higher frequency than the low-frequency image, and the vector lengths of the first and second random noise variables can be the same and both satisfy a normal distribution. The low-frequency and high-frequency images can be input from external devices or obtained by the server by decomposing the acquired original image. When decomposing the original image, a single original image can be decomposed into one or more low-frequency and high-frequency images.

[0063] It should be noted that the terms "first" and "second" appearing in this application are only for distinguishing concepts and do not limit the order; sometimes, depending on the context, "first" may include "second" and "third," or similar other cases. In addition, the concepts modified by "first" and "second" are not limited to only one, but can be one or more.

[0064] The images acquired in the above process include both low-frequency and high-frequency images. However, it should be noted that this is not limited to images with only two frequencies. In practical applications, more frequency types can be set as needed. For example, images can be divided into low-frequency, mid-frequency, and high-frequency images, with the three frequencies increasing sequentially. Four, five, or more categories can also be set, which can be pre-defined.

[0065] In one specific embodiment, the server acquires the original image and performs a decomposition operation on the original image to obtain at least one low-frequency image and at least one high-frequency image corresponding to each original image.

[0066] In one specific embodiment, after acquiring the original images, the server can decompose each original image to obtain at least one low-frequency image and at least one high-frequency image corresponding to each original image. The methods used to decompose the original images can be varied, such as Fourier transform, discrete cosine transform, wavelet transform, etc., but are not limited to these. This application can also use other methods to decompose the original images. The number of low-frequency and high-frequency images, as well as their frequencies, can be preset; the specific number and frequency settings are not limited in this embodiment.

[0067] In one specific embodiment, the number of GANs initially configured on the server is related to the set resolution and / or the number of image frequency categories. Specifically, it can be K = P * Q, where K is the number of GANs initially configured, P is the number of resolution categories, and Q is the number of image frequency categories. Therefore, when decomposing the original image, it can be divided according to the set resolution and image frequency.

[0068] S502. The server uses the first random noise variable and the low-frequency image to train the first GAN with the initial configuration to obtain the first generator.

[0069] In one specific embodiment, the server sets low-frequency images as training samples for the first GAN and inputs a first random noise variable into the first GAN to train it. Specifically, the training process for the first GAN is as described above. Figure 2 The relevant descriptions in step S203 are similar and will not be repeated here.

[0070] When training is complete, the server removes the discriminator from the first GAN and retains the generator of the first GAN, which is also the first generator.

[0071] S503. The server trains the initially configured second GAN using the second random noise variable and the high-frequency image to obtain the second generator. Note that the second random noise variable should be orthogonal to the first random noise variable.

[0072] In one specific embodiment, the server sets the high-frequency image as the real image of the first GAN, that is, as a training sample for the second GAN, and inputs a second random noise variable into the second GAN to train the second GAN. Specifically, the training process of the second GAN is as described above. Figure 2The relevant descriptions in step S203 are similar and will not be repeated here.

[0073] When training is complete, the server removes the discriminator from the second GAN and retains the generator of the second GAN, which is also the second generator.

[0074] In one specific embodiment, after the server obtains the original image, it can decompose the original image using discrete wavelet transform processing to obtain at least one low-frequency image and at least one high-frequency image containing K resolutions, with the K resolutions decreasing sequentially. The Q-th resolution corresponds to M... Q A low-frequency image and N Q A high-frequency image, K, M Q and N Q All are positive integers, Q = 1, 2, 3... or K. M at different resolutions... Q and N Q The values ​​can be the same or different, and the user can set them in advance.

[0075] The following explanation uses the training process at the Q-th resolution as an example; other resolutions can refer to this example.

[0076] Obtain M at the Qth resolution Q A low-frequency image and N Q After a high-frequency image, the server first uses the first random noise variable (e.g., M) Q (a random noise variable) and M Q A low-frequency image for the initial configuration S Q A low-frequency GAN is trained, and upon completion of training, S is obtained. Q A low-frequency generator; the server utilizes a second random noise variable (e.g., N). Q (a random noise variable) and N Q A high-frequency image against the initial configuration W Q A high-frequency GAN is trained, and W is obtained upon completion of training. Q A high-frequency generator.

[0077] It should be noted that S Q and W Q All are integers greater than or equal to 1, S Q and W Q The values ​​can be the same or different. For example, at one resolution, there can be one low-frequency GAN and one high-frequency GAN, resulting in one low-frequency generator and one high-frequency generator; as another example, at one resolution, there can be any number of low-frequency GANs and any number of high-frequency GANs, resulting in any number of corresponding low-frequency generators and any number of high-frequency generators.

[0078] During the training process, S Q A low-frequency generator and W Q The high-frequency generators are not completely independent; the output of each generator can potentially serve as input information for the others. For example, in the first iteration, the first low-frequency generator receives only random noise as input; the second low-frequency generator receives random noise as input, along with the output of the first generator; the third low-frequency generator receives random noise as input, along with the outputs of the first and second generators, and so on, until the Sth low-frequency generator... Q The input to the low-frequency generator has random noise and previous S Q -1 is the output of the low-frequency generator, and so on. The input of the first high-frequency generator is random noise and the previous S. Q The output of a low-frequency generator ， This continues until the Wth Q The input to the high-frequency generator is random noise and the previous S... Q+ W Q -1 generator output (including low-frequency generator and high-frequency generator).

[0079] Similarly, in the second iteration, each generator (regardless of high or low frequency) is fed with random noise and the output of the previous generator as input. This process continues for several iterations.

[0080] In other embodiments, when selecting the output of a previous generator as the input of the current generator, it is not necessary to select the output of all previous generators as in this embodiment. Instead, the output of any one or more previous generators may be selected. The specific generators selected can be set according to specific needs, and this application does not limit them.

[0081] In addition, all the random noise vectors input to the generators during training should be pairwise orthogonal. Orthogonalization techniques are needed to orthogonalize the random noise vectors to ensure their independence.

[0082] The above describes the training process at the Q-th resolution; other resolutions can refer to this example.

[0083] After the server trains on both low-frequency and high-frequency images at K different resolutions, it obtains... A low-frequency generator and A high-frequency generator. Here... The low-frequency generator is the first generator, here... The high-frequency generator is the second generator.

[0084] It should be noted that there is no necessary execution order between steps S502 and S503. Step S502 can be executed first or step S503 can be executed first, which will not be elaborated here.

[0085] S504. The server inputs the target variable into the first generator and the second generator respectively, and generates the first subgraph and the second subgraph accordingly.

[0086] In one specific embodiment, after obtaining the first generator and the second generator, when a high-quality image needs to be generated, the server inputs a target vector into the first generator and the second generator, respectively. The target variable can be a random noise variable obtained from external input or generated by the server itself, and can also include output information from other generators, or a specific variable containing image feature information to be generated. Specifically, for example, if the original image consists of multiple real-world landscape images, and the target variable is a random noise variable, the final output target image can be a composite image with a style similar to the original image; if the target variable contains image feature information to be generated (e.g., image elements need to include mountains and their outlines), the final output target image can be a composite image containing image feature information and with a style similar to the original image.

[0087] It should be noted that since the first generator is trained using low-frequency images as training samples and the second generator is trained using high-frequency images as training samples, the first sub-image generated by the first generator is still a low-frequency image, and the second sub-image generated by the second generator is still a high-frequency image.

[0088] S505 The server combines the first sub-image and the second sub-image to obtain the target image.

[0089] In a specific embodiment, after the server obtains the first sub-image and the second sub-image, it synthesizes the first sub-image and the second sub-image to obtain the target image. Various methods can be used for synthesis, such as inverse wavelet transform, inverse Fourier transform, and inverse discrete cosine transform. Specifically, the methods used to synthesize the first sub-image and the second sub-image are common techniques in the prior art and will not be elaborated upon in this embodiment.

[0090] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: After training the first and second generators, the server inputs the target variable into each generator, generating a first sub-image and a second sub-image respectively. These sub-images are then combined to obtain the target image. Since the first generator is trained on an initially configured first GAN using a first random noise variable and a low-frequency image, and the second generator is trained on an initially configured second GAN using a second random noise variable and a high-frequency image, the generated first and second sub-images are respectively low-frequency and high-frequency images. It should be noted that, according to the definition of image frequency, high-frequency images better reflect the detailed information of an image, such as the contour information of various main features, while low-frequency information better reflects the main information of an image, such as grayscale and color information. In this scheme, generating low-frequency and high-frequency images separately allows for better preservation of the detailed and main information of the target image during the generation process, thus ensuring better quality of the generated target image.

[0091] The above Figure 5 The schematic diagram of the embodiment shown provides a brief description of this solution. The following is an explanation using a specific application.

[0092] The following reference Figure 6 , Figure 6 This is a schematic diagram of a system architecture provided in an embodiment of this application. Figure 6 As shown, in a specific embodiment, the server can be divided into software and hardware components. The software component consists of program code contained in the AI ​​data storage system and deployed on the server hardware. This program code may include a discrete wavelet transform image decomposition module, a GAN-generated sub-image module, and a discrete wavelet inverse transform synthesized image module. The hardware component includes host storage and memory (GPU, FPGA, dedicated chip), with the host storage specifically including a physical image storage device and a generated image storage device.

[0093] Based on the above Figure 6 The system architecture is as follows, refer to the following: Figure 7 , Figure 7 Another embodiment of an image generation method provided in this application may include: S701, The server retrieves the original image.

[0094] In this embodiment, the server can acquire the original image input from the outside and store it in the real image storage device in the host storage.

[0095] S702. The server decomposes the original image using discrete wavelet transform to obtain at least one low-frequency image and at least one high-frequency image with K resolutions. The Q-th resolution corresponds to a first low-frequency image, a first high-frequency image, a second high-frequency image, and a third high-frequency image. The first low-frequency image includes low-frequency information of the original image in the vertical and horizontal directions. The first high-frequency image includes low-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The second high-frequency image includes high-frequency information of the original image in the vertical direction and low-frequency information in the horizontal direction. The third high-frequency image includes high-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. Q = 1, 2, 3, ..., K.

[0096] In one specific embodiment, after the server acquires the original image, it retrieves the original image from the real image storage device and decomposes the original image using a discrete wavelet transform image decomposition module. After decomposing an original image, at least one low-frequency image and at least one high-frequency image at K different resolutions can be obtained. Each of the K resolutions corresponds to a first low-frequency image, a first high-frequency image, a second high-frequency image, a third high-frequency image, and a fourth high-frequency image. Specifically, the decomposition process can be described as follows: The discrete wavelet transform can be represented as a tree consisting of low-pass and high-pass filters. The matrix representation of an image is x[2m, 2n], where 2m and 2n are the height and width of the image. Its two-dimensional discrete wavelet decomposition process can be described as follows: First, one-dimensional wavelet transform (1D-DWT) is performed on each row of the original image using the following formulas (1) and (2), where g[k] is a low-pass filter that filters out the high-frequency part of the input signal and outputs the low-frequency part, and h[k] is a high-pass filter that filters out the low-frequency part of the input signal and outputs the high-frequency information, thereby obtaining the low-frequency component L and high-frequency component H of the original image in the horizontal direction, where k represents the size of the filter window.

[0097] (1) (2) Then, 1D-DWT is performed on each column of the low-frequency component data L and the high-frequency component data H in the horizontal direction of the original image, as shown in formulas (3)-(6), to obtain the low-frequency component LL in the horizontal and vertical directions of the original image, which is the first low-frequency image; the high-frequency component HL in the horizontal and vertical directions, which is the first high-frequency image; the low-frequency component LH in the horizontal and vertical directions, which is the second high-frequency image; and the high-frequency component HH in the horizontal and vertical directions, which is the third high-frequency image.

[0098] (3) (4) (5) (6) When decomposing the original image using discrete wavelet transform, the resolution of the generated low-frequency and high-frequency images can also be controlled.

[0099] S703. The server uses the first low-frequency image at the Qth resolution and the first random noise variable to train the Qth low-frequency GAN in the initial configuration, and obtains the Qth low-frequency generator.

[0100] In one specific embodiment, the server obtains a first low-frequency image and a first random noise variable at the Qth resolution, and uses this first low-frequency image and the first random noise variable to train the Qth low-frequency GAN to obtain the Qth low-frequency generator. Specifically, this training process can be referred to... Figure 2 The description of step S203 shown will not be repeated here.

[0101] S704. The server uses the first high-frequency image at the Qth resolution and the third random noise variable to train the Qth first high-frequency GAN in the initial configuration, and obtains the Qth first high-frequency generator.

[0102] In one specific embodiment, the server obtains a first high-frequency image and a third random noise variable at the Qth resolution, and uses this first high-frequency image and the first random noise variable to train the Qth first high-frequency GAN to obtain the Qth first high-frequency generator. Specifically, this training process can be referred to... Figure 2 The description of step S203 shown will not be repeated here.

[0103] S705. The server uses the second high-frequency image at the Qth resolution and the fourth random noise variable to train the Qth second high-frequency GAN in the initial configuration, and obtains the Qth second high-frequency generator.

[0104] In one specific embodiment, the server obtains a first random noise variable of a first high-frequency image at the Qth resolution, and uses this first random noise variable to train the Qth second high-frequency GAN to obtain the Qth second high-frequency generator. Specifically, this training process can be referred to... Figure 2 The description of step S203 shown will not be repeated here.

[0105] S706. The server uses the third high-frequency image at the Qth resolution and the fifth random noise variable to train the Qth third high-frequency GAN in the initial configuration, and obtains the Qth third high-frequency generator.

[0106] In one specific embodiment, the server obtains a first high-frequency image and a first random noise variable at the Qth resolution, and uses this first high-frequency image and the first random noise variable to train the Qth third high-frequency GAN to obtain the Qth third high-frequency generator. Specifically, this training process can be referred to... Figure 2 The description of step S203 shown will not be repeated here.

[0107] It should be noted that during the generator training process in steps S703-S706, the output of any one or more other generators can be used as the input of the currently trained generator.

[0108] In a specific embodiment, at a defined resolution, the system architecture diagrams of the low-frequency GAN, the first high-frequency GAN, the second high-frequency GAN, and the third high-frequency GAN can be referred to... Figure 8 The diagram shown is as follows. Figure 8 As shown, G1 and D1 are the generator and discriminator of the low-frequency GAN, respectively; G2 and D2 are the generator and discriminator of the first high-frequency GAN, respectively; G3 and D3 are the generator and discriminator of the second high-frequency GAN, respectively; and G4 and D4 are the generator and discriminator of the third high-frequency GAN, respectively. After the server obtains the original image, it uses the VGG19 network module to obtain the corresponding real image features. VGG19 is a type of convolutional neural network.

[0109] S707: The server inputs the target vector into K low-frequency generators, K first high-frequency generators, K second high-frequency generators, and K third high-frequency generators respectively, and obtains K low-frequency generated subgraphs, K first high-frequency generated subgraphs, K second high-frequency generated subgraphs, and K third high-frequency generated subgraphs.

[0110] In one specific embodiment, after the server obtains K low-frequency generators, K first high-frequency generators, K second high-frequency generators, and K third high-frequency generators, it inputs the target vector into each generator to generate K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images, and K third high-frequency generated sub-images respectively. The image parameters (resolution and frequency) of each generated sub-image are consistent with the parameters of the generator's training samples.

[0111] S708 The server uses discrete wavelet transform to synthesize K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images, and K third high-frequency generated sub-images to obtain the target image.

[0112] In one specific embodiment, the server generates K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images, and K third high-frequency generated sub-images, and synthesizes the generated sub-images to obtain the target image.

[0113] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: After training K first generators, K first high-frequency generators, K second high-frequency generators, and K third high-frequency generators, the server inputs the target variable into each of these generators, generating K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images, and K third high-frequency generated sub-images respectively. These sub-images are then combined to obtain the target image.

[0114] It should be noted that the generators are not isolated; the output of each generator can be used as the input of the others, and they can be chained together in a loop. The combined generators produce images of better quality.

[0115] Since the K first generators, K first high-frequency generators, K second high-frequency generators, and K third high-frequency generators are all trained using images with different resolutions and frequencies as training samples, the corresponding K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images, and K third high-frequency generated sub-images also have different resolutions and frequencies, meaning they carry the information mainly expressed by the different image parameters. Therefore, when generating the target image, the detailed and main information of the target image can be better preserved, improving the quality of the generated image.

[0116] Furthermore, the target image is superimposed on images generated by other generators to obtain the final target image; this superposition can be a weighted combination. It should be noted that the other generators can be any type of generator in the prior art, and these generators will also participate in the training process. Weight adjustment factor It can learn independently based on the dataset, and different datasets are used for different scenarios. The values ​​are different.

[0117] The following reference Figure 9 , Figure 9 A schematic diagram of an embodiment of a server provided in this application includes: Transceiver unit 901 is used to acquire the target vector; Processing unit 902 is used to input the target vector into a first generator and a second generator respectively, and generate a first sub-image and a second sub-image accordingly. The first generator is obtained by the server training an initially configured first generative adversarial network (GAN) based on a low-frequency image and a first random noise variable that satisfies a normal distribution. The second generator is obtained by the server training an initially configured second generative adversarial network (GAN) based on a high-frequency image and a second random noise variable that satisfies a normal distribution. The frequency of the low-frequency image is lower than the frequency of the high-frequency image. The first sub-image and the second sub-image are then synthesized to obtain the target image.

[0118] It should be noted that the number of the first random noise variables and the number of the second random noise variables correspond to the number of the first generator and the number of the second generator, respectively. However, the random noise variables need to be orthogonal, which requires specific orthogonalization techniques.

[0119] In one specific embodiment The transceiver unit 901 is also used to acquire the low-frequency image, the high-frequency image, the first random noise variable, and the second random noise variable; The processing unit 902 is further configured to set the low-frequency image and the high-frequency image as training samples for the first GAN and the second GAN, respectively; train the first GAN using the low-frequency image and the first random noise variable to obtain the first generator; and train the second GAN using the high-frequency image and the second random noise variable to obtain the second generator.

[0120] It should be noted that the generators of the first GAN and the second GAN are connected in series. That is, the output of the generator of the first GAN will be combined with the second random noise variable as the input of the second generator. The combination method is not limited here, and vice versa.

[0121] In one specific embodiment The transceiver unit 901 is specifically used to acquire the original image; The processing unit 902 is specifically used to perform wavelet transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse wavelet transform processing to obtain the target image.

[0122] In one specific embodiment Processing unit 902 is specifically used to perform discrete wavelet transform processing on the original image to obtain at least one low-frequency image and at least one high-frequency image containing K resolutions, wherein the Q-th resolution corresponds to M. Q A low-frequency image and N Q A high-frequency image, K, M Q N QAll are positive integers, Q = 1, 2, 3...K; using this M at the Qth resolution Q The Qth low-frequency GAN in the initial configuration is trained using the first low-frequency image and the first random noise variable to obtain the Qth low-frequency generator; the N at the Qth resolution is then used. Q The Q-th high-frequency GAN is trained using the high-frequency image and the second random noise variable to obtain the Q-th high-frequency generator; the target vector is input into the K low-frequency generators and the K high-frequency generators respectively to obtain the K low-frequency generated sub-images and the K high-frequency generated sub-images; the K low-frequency generated sub-images and the K high-frequency generated sub-images are synthesized by using discrete wavelet inverse transform to obtain the target image.

[0123] It should be noted that at each resolution, the input of each generator can be composed of random noise and the outputs of the other generators. In addition, the random noises are mutually orthogonal.

[0124] In one specific embodiment, the M Q The N low-frequency images include a first low-frequency image. Q The high-frequency image includes a first high-frequency image, a second high-frequency image, and a third high-frequency image. The first low-frequency image includes low-frequency information of the original image in the vertical and horizontal directions. The first high-frequency image includes low-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The second high-frequency image includes high-frequency information of the original image in the vertical direction and low-frequency information in the horizontal direction. The third high-frequency image includes high-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. Processing unit 902 is specifically configured to train an initially configured Q-th first high-frequency GAN using the first high-frequency image at the Q-th resolution and the second random noise variable to obtain a Q-th first high-frequency generator; train an initially configured Q-th second high-frequency GAN using the second high-frequency image at the Q-th resolution and the second random noise variable to obtain a Q-th second high-frequency generator; train an initially configured Q-th third high-frequency GAN using the third high-frequency image at the Q-th resolution and the second random noise variable to obtain a Q-th third high-frequency generator; input the target vector into K low-frequency generators, K first high-frequency generators, K second high-frequency generators, and K third high-frequency generators respectively to obtain K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images, and K third high-frequency generated sub-images; and synthesize the K low-frequency generated sub-images, K first high-frequency generated sub-images, K second high-frequency generated sub-images, and K third high-frequency generated sub-images using discrete wavelet transform processing to obtain the target image.

[0125] In one specific embodiment The transceiver unit 901 is specifically used to acquire the original image; The processing unit 902 is specifically used to perform discrete cosine transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse discrete cosine transform processing to obtain the target image.

[0126] In one specific embodiment The transceiver unit 901 is specifically used to acquire the original image; The processing unit 902 is specifically used to perform Fourier transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse Fourier transform processing to obtain the target image.

[0127] Furthermore, it may also include an overlay unit, which is used to overlay the target image with images generated by other generators to obtain the final target image. The overlay can be a weighted combination. It should be noted that the other generators can be any type of generator in the prior art, and these generators will also participate in the training process. Weight adjustment factor It can learn independently based on the dataset, and different datasets are used for different scenarios. The values ​​are different.

[0128] The following reference Figure 10 , Figure 10 Another embodiment of a server provided in this application is illustrated in the following diagram: Processor 1010, memory 1020, and transceiver 1030; Transceiver 1030 is used to communicate with devices outside the server; Memory 1020 is used to store instruction codes; Processor 1010 is used to execute the instruction code, causing the server to perform actions such as... Figure 5 or Figure 7 The method as described in any of the embodiments shown.

[0129] This application also provides a computer storage medium, characterized in that the medium stores instructions that, when executed on a computer, cause the computer to perform actions such as... Figure 5 or Figure 7 The method as described in any of the embodiments shown.

[0130] This application also provides a computer program product, characterized in that it includes instructions, which, when executed on a computer, cause the computer to perform the following... Figure 5 or Figure 7 The method as described in any of the embodiments shown.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for image generation, characterized in that, include: The target vector is input into the first generator and the second generator respectively, and the first sub-image and the second sub-image are generated accordingly. The first generator is obtained by training a first generative adversarial network (GAN) based on a low-frequency image and a first random noise variable. The second generator is obtained by training a second generative adversarial network (GAN) based on a high-frequency image and a second random noise variable. The frequency of the low-frequency image is lower than the frequency of the high-frequency image. The first sub-image and the second sub-image are combined to obtain the target image.

2. The method according to claim 1, characterized in that, The method further includes: Acquire the low-frequency image and the high-frequency image; Obtain the first random noise variable and the second random noise variable; The low-frequency image and the high-frequency image are respectively set as training samples for the first GAN and the second GAN; The first GAN is trained using the low-frequency image and the first random noise variable to obtain the first generator; The second GAN is trained using the high-frequency image and the second random noise variable to obtain the second generator.

3. The method according to claim 2, characterized in that, The acquisition of the low-frequency image and the high-frequency image includes: Obtain the original image; The original image is subjected to wavelet transform processing to obtain the low-frequency image and the high-frequency image; The process of synthesizing the first sub-image and the second sub-image to obtain the target image includes: The first sub-image and the second sub-image are synthesized by inverse wavelet transform to obtain the target image.

4. The method according to claim 3, characterized in that, Performing wavelet transform on the original image to obtain the low-frequency image and the high-frequency image includes: The original image is subjected to discrete wavelet transform processing to obtain at least one low-frequency image and at least one high-frequency image containing K resolutions, wherein the Q-th resolution corresponds to M. Q A low-frequency image and N Q A high-frequency image, K, M Q N Q All are positive integers, Q = 1, 2, 3...K; The first generator is obtained by training the first GAN using the low-frequency image and the first random noise variable, including: Using the M at the Qth resolution Q A low-frequency image and the first random noise variable pair S Q Train a low-frequency GAN to obtain S Q A low-frequency generator, where S Q An integer greater than or equal to 1; The second GAN is trained using the high-frequency image and the second random noise variable to obtain the second generator, comprising: Using the N at the Qth resolution Q The high-frequency image and the second random noise variable are paired with the Wth high-frequency image. Q Train a high-frequency GAN to obtain W Q A high-frequency generator, where W Q An integer greater than or equal to 1; The target vector is input into the first generator and the second generator respectively to generate the first subgraph and the second subgraph, respectively, including: To each Low-frequency generator and A high-frequency generator is input to the target vector to obtain... A low-frequency generated subgraph and A high-frequency generated subgraph; The target image is obtained by synthesizing the first sub-image and the second sub-image using wavelet transform processing, including: The discrete wavelet inverse transform method is used to process the above. A low-frequency generated subgraph and the The target image is obtained by synthesizing a high-frequency generated sub-image.

5. The method according to claim 4, characterized in that, The training of any generator also includes: The output of any one or more other generators is used as the input of this generator, and the other one or more generators include any one or more generators other than this generator among low-frequency generators and high-frequency generators.

6. The method according to any one of claims 2-5, characterized in that, Any two random noise variables in the first random noise variable and the second random noise variable are orthogonal.

7. The method according to any one of claims 4-6, characterized in that, The M Q The N low-frequency images include a first low-frequency image. Q The high-frequency image includes a first high-frequency image, a second high-frequency image, and a third high-frequency image. The first low-frequency image includes low-frequency information of the original image in the vertical and horizontal directions. The first high-frequency image includes low-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The second high-frequency image includes high-frequency information of the original image in the vertical direction and low-frequency information in the horizontal direction. The third high-frequency image includes high-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The M under the Qth resolution Q A low-frequency image and the first random noise variable pair S Q Train a low-frequency GAN to obtain S Q A low-frequency generator, including: Using the M at the Qth resolution Q The first low-frequency GAN is trained using the first low-frequency image and the first random noise variable to obtain the Qth low-frequency generator. The N using the Qth resolution Q The high-frequency image and the second random noise variable are paired with the Wth high-frequency image. Q Train a high-frequency GAN to obtain W Q A high-frequency generator, including: The first high-frequency image at the Qth resolution and the third random noise variable are used to train the Qth first high-frequency GAN to obtain the Qth first high-frequency generator; The second high-frequency image at the Qth resolution and the fourth random noise variable are used to train the Qth second high-frequency GAN to obtain the Qth second high-frequency generator; The Qth third high-frequency GAN is trained using the third high-frequency image at the Qth resolution and the fifth random noise variable to obtain the Qth third high-frequency generator; The respective directions Low-frequency generator and A high-frequency generator is input to the target vector to obtain... A low-frequency generated subgraph and A number of high-frequency generated subgraphs, including: The target vector is input into K low-frequency generators, K first high-frequency generators, K second high-frequency generators, and K third high-frequency generators respectively to obtain K low-frequency generated sub-graphs, K first high-frequency generated sub-graphs, K second high-frequency generated sub-graphs, and K third high-frequency generated sub-graphs; The method of using discrete wavelet inverse transform is used to process the... A low-frequency generated subgraph and the The target image is obtained by synthesizing several high-frequency generated sub-images, including: The target image is obtained by synthesizing the K low-frequency generated sub-images, the K first high-frequency generated sub-images, the K second high-frequency generated sub-images, and the K third high-frequency generated sub-images using discrete wavelet inverse transform processing.

8. The method according to claim 2, characterized in that, The method further includes: Obtain the original image; The original image is subjected to discrete cosine transform processing to obtain the low-frequency image and the high-frequency image; The process of synthesizing the first sub-image and the second sub-image to obtain the target image includes: The first sub-image and the second sub-image are synthesized by using the inverse discrete cosine transform to obtain the target image.

9. The method according to claim 2, characterized in that, The method further includes: Obtain the original image; The original image is subjected to Fourier transform processing to obtain the low-frequency image and the high-frequency image; The process of synthesizing the first sub-image and the second sub-image to obtain the target image includes: The first sub-image and the second sub-image are synthesized by using inverse Fourier transform to obtain the target image.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: superimposing the target image with images generated by other generators to obtain the final target image.

11. An apparatus for image generation, characterized in that, include: The processing unit is configured to input the target vector into a first generator and a second generator respectively, and generate a first sub-image and a second sub-image accordingly. The first generator is obtained by training a first generative adversarial network (GAN) based on a low-frequency image and a first random noise variable, and the second generator is obtained by training a second generative adversarial network (GAN) based on a high-frequency image and a second random noise variable. The frequency of the low-frequency image is lower than the frequency of the high-frequency image. The first sub-image and the second sub-image are then synthesized to obtain the target image.

12. The apparatus according to claim 11, characterized in that, The transceiver unit is also used to acquire the low-frequency image, the high-frequency image, the first random noise variable, and the second random noise variable; The processing unit is further configured to set the low-frequency image and the high-frequency image as training samples for the first GAN and the second GAN, respectively; and to train the first GAN using the low-frequency image and the first random noise variable to obtain the first generator. The second GAN is trained using the high-frequency image and the second random noise variable to obtain the second generator.

13. The apparatus according to claim 12, characterized in that, The transceiver unit is specifically used to acquire the original image; The processing unit is specifically used to perform wavelet transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse wavelet transform processing to obtain the target image.

14. The apparatus according to claim 13, characterized in that, The processing unit is specifically used to perform discrete wavelet transform processing on the original image to obtain at least one low-frequency image and at least one high-frequency image containing K resolutions, wherein the Q-th resolution corresponds to M. Q A low-frequency image and N Q A high-frequency image, K, M Q N Q All are positive integers, Q = 1, 2, 3...K; using the M at the Qth resolution... Q A low-frequency image and the first random noise variable pair S Q Train a low-frequency GAN to obtain S Q A low-frequency generator, where S Q It is an integer greater than or equal to 1; using the N at the Qth resolution. Q The high-frequency image and the second random noise variable are paired with the Wth high-frequency image. Q Train a high-frequency GAN to obtain W Q A high-frequency generator, where W Q For integers greater than or equal to 1; respectively to Low-frequency generator and A high-frequency generator is input to the target vector to obtain... A low-frequency generated subgraph and A high-frequency generated sub-graph; processed using inverse discrete wavelet transform. A low-frequency generated subgraph and the The target image is obtained by synthesizing a high-frequency generated sub-image.

15. The apparatus according to claim 14, characterized in that, The processing unit is used during the training of any generator to: take the output of any one or more other generators as the input of the generator, wherein the other one or more generators include any one or more generators other than the generator in the low-frequency generator and the high-frequency generator.

16. The apparatus according to any one of claims 12-15, characterized in that, Any two random noise variables in the first random noise variable and the second random noise variable are orthogonal.

17. The apparatus according to any one of claims 14-16, characterized in that, The M Q The N low-frequency images include a first low-frequency image. Q The high-frequency image includes a first high-frequency image, a second high-frequency image, and a third high-frequency image. The first low-frequency image includes low-frequency information of the original image in the vertical and horizontal directions. The first high-frequency image includes low-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction. The second high-frequency image includes high-frequency information of the original image in the vertical direction and low-frequency information in the horizontal direction. The third high-frequency image includes high-frequency information of the original image in the vertical direction and high-frequency information in the horizontal direction.

18. The apparatus according to claim 12, characterized in that, The transceiver unit is specifically used to acquire the original image; The processing unit is specifically used to perform discrete cosine transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse discrete cosine transform processing to obtain the target image.

19. The apparatus according to claim 12, characterized in that, The transceiver unit is specifically used to acquire the original image; The processing unit is specifically used to perform Fourier transform processing on the original image to obtain the low-frequency image and the high-frequency image; and to synthesize the first sub-image and the second sub-image by using inverse Fourier transform processing to obtain the target image.

20. A computer, characterized in that, include: Processor, memory, and transceiver; The transceiver is used to communicate with devices other than the server; The memory is used to store instruction codes; When the processor executes the instruction code, it causes the server to perform the method as described in any one of claims 1-10.

21. A computer storage medium, characterized in that, The medium stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 10.

22. A computer program product, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 10.

23. A chip system, characterized in that, The chip system includes an interface and processing circuitry. It acquires a software program through the interface and executes the software program through the processing circuitry to implement the method described in any one of claims 1-10.

24. A chip system, characterized in that, It includes one or more functional circuits, said one or more functional circuits being used to implement the method as described in any one of claims 1-10.

Images