Image generation method, device, electronic device and storage medium

Abstract

The present application provides an image generation method, device, electronic device and storage medium; the method comprises: using a generator in a generative adversarial network to generate an original image; performing discriminative processing on the original image by a discriminator in the generative adversarial network to obtain initial error information of the original image, and adjusting parameters of the generator and an offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator; then, using the adjusted offset module to obtain offset information of the original image on at least one scale based on the initial error information, and triggering the discriminator to adjust the initial error information based on the offset information on at least one scale to obtain adjusted error information; finally, triggering the adjusted generator to generate a target image that meets preset conditions in terms of target attributes according to the adjusted error information, thereby improving the accuracy of the image generated by the generator.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and specifically to an image generation method, device, electronic device and storage medium. Background Art

[0002] As one of the methods of deep learning, the Generative Adversarial Network is different from ordinary neural networks. It consists of two main networks, one is the generative network or generator, and the other is the discriminative network or discriminator. The generator is used to generate images, and the discriminator is used to discriminate the images input into the discriminator. The core logic of the Generative Adversarial Network is that the generator and the discriminator confront and compete with each other, so that the generator can generate images that the discriminator cannot distinguish between true and false.

[0003] In the process of research and practice of the prior art, the inventors of the present application discovered that the generative adversarial network has the phenomenon of unstable discriminator discrimination, which causes the accuracy of the image generated by the generator to be low. Summary of the invention

[0004] The present application relates to artificial intelligence technology, and specifically to an image generation method, device, electronic device and computer-readable storage medium.

[0005] The present application provides an image generation method, including:

[0006] Use the generator in the generative adversarial network to generate the original image;

[0007] The discriminator in the generative adversarial network performs discriminative processing on the original image to obtain initial error information of the original image, wherein the initial error information represents the error between the original image and a preset reference image in a target attribute;

[0008] Adjusting parameters of a generator and an offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator;

[0009] Using the adjusted offset module to obtain offset information of the original image on at least one scale based on the initial error information;

[0010] Based on the offset information on the at least one scale, trigger the discriminator to adjust the initial error information to obtain adjusted error information;

[0011] The adjusted generator is triggered according to the adjusted error information to generate a target image that meets a preset condition on the target attribute.

[0012] Accordingly, the embodiment of the present application further provides an image generating device, including:

[0013] A generating unit, used for generating an original image using a generator in a generative adversarial network;

[0014] A discriminant processing unit, configured to perform discriminant processing on the original image through a discriminator in the generative adversarial network to obtain initial error information of the original image, wherein the initial error information represents an error between the original image and a preset reference image in a target attribute;

[0015] A parameter adjustment unit, configured to adjust parameters of a generator and an offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator;

[0016] an acquiring unit, configured to acquire offset information of the original image at at least one scale based on the initial error information by using the adjusted offset module;

[0017] A first triggering unit, configured to trigger the discriminator to adjust the initial error information based on the offset information on the at least one scale to obtain adjusted error information;

[0018] The second triggering unit is used to trigger the adjusted generator to generate a target image that meets a preset condition in terms of the target attribute according to the adjusted error information.

[0019] In one embodiment, the determination processing unit may include:

[0020] A convolution feature extraction subunit, used to perform convolution feature extraction on the original image using the discriminator to obtain convolution feature information of the original image;

[0021] A discriminating subunit, configured to discriminate the original image based on the convolution feature information of the original image and the preset reference image, and obtain a discriminating result of the original image;

[0022] A determination subunit is used to determine the initial error information of the original image according to the discrimination result.

[0023] In one embodiment, the convolutional feature extraction subunit may be used to:

[0024] Performing feature sampling on the original image using at least one convolution kernel to obtain sampling feature information after feature sampling of the at least one convolution kernel;

[0025] The sampled feature information after each convolution kernel feature sampling is fused to obtain the convolution feature information of the original image.

[0026] In one embodiment, the acquiring unit may include:

[0027] a scale transformation subunit, configured to perform at least one scale transformation on the original image based on the initial error information by using the adjusted offset module to obtain a transformed image after at least one scale transformation;

[0028] The obtaining subunit is used to calculate the offset information between the transformed image after each scale transformation and the original image by using the adjusted offset module, so as to obtain the offset information of the original image in at least one scale.

[0029] In one embodiment, the second trigger unit may include:

[0030] a triggering subunit, configured to trigger the adjusted generator to update the original image according to the adjusted error information when the adjusted error information satisfies a preset error condition, so as to obtain a target image that meets the preset condition in terms of the target attribute;

[0031] The execution subunit is used to, when the adjusted error information does not meet the preset error condition, execute the step of adjusting the parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain the adjusted offset module and the adjusted generator.

[0032] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the method provided in various optional modes of the above-mentioned aspect.

[0033] An embodiment of the present application also provides a network device, including a memory and a processor; the memory stores multiple instructions, and the processor loads the instructions in the memory to execute the steps in any image generation method provided in the embodiment of the present invention.

[0034] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, implements any image generation method provided in the embodiments of the present application.

[0035] The embodiment of the present application can use a generator in a generative adversarial network to generate an original image; use a discriminator in the generative adversarial network to perform discriminative processing on the original image to obtain initial error information of the original image, and adjust parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator; then, use the adjusted offset module to obtain offset information of the original image on at least one scale based on the initial error information, and trigger the discriminator to adjust the initial error information based on the offset information on at least one scale to obtain adjusted error information; finally, trigger the adjusted generator according to the adjusted error information to generate a target image that meets preset conditions in terms of target attributes, thereby improving the accuracy of the image generated by the generator. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1a is a scene schematic diagram of the image generation method provided in an embodiment of the present application;

[0038] Figure 1b is a flow chart of an image generation method provided in an embodiment of the present application;

[0039] Figure 1c is a schematic diagram of a semantic segmentation image provided in an embodiment of the present application;

[0040] Figure 1d It is a convolution schematic diagram provided in an embodiment of the present application;

[0041] Figure 1e is a schematic diagram of an RGB image provided in an embodiment of the present application;

[0042] Figure 1f is another schematic diagram of an RGB image provided in an embodiment of the present application;

[0043] Figure 1g It is a schematic diagram of convolution feature extraction provided in an embodiment of the present application;

[0044] Figure 1h is another convolution schematic diagram provided in an embodiment of the present application;

[0045] Figure 2a is another flow chart of the image generation method provided in an embodiment of the present application;

[0046] Figure 2bIt is a schematic diagram of the generator network structure provided in an embodiment of the present application;

[0047] Figure 3a It is a structural diagram of an application login device provided in an embodiment of the present application;

[0048] Figure 3b is another structural schematic diagram of the application login device provided in an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the structure of the terminal provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0051] The embodiment of the present application proposes an image generation device, which can be integrated in a computer device, and the computer device can include at least one of a terminal and a server. The terminal can be a smart phone, a tablet computer, a laptop or a personal computer (PC), a smart wearable device, etc. The server can be an intercommunication server between multiple heterogeneous systems or a background server of a content interaction system, or an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc.

[0052] For example, Figure 1a The image generation device can be integrated on a computer device such as a terminal or a server, and the computer device can use a generator in a generative adversarial network to generate an original image; then, the original image is discriminated by a discriminator in the generative adversarial network to obtain initial error information of the original image, and the parameters of the generator and the offset module in the generative adversarial network are adjusted according to the initial error information to obtain an adjusted offset module and an adjusted generator; next, the adjusted offset module is used to obtain the offset information of the original image on at least one scale based on the initial error information, and based on the offset information on at least one scale, the discriminator is triggered to adjust the initial error information to obtain the adjusted error information; finally, the adjusted generator is triggered according to the adjusted error information to generate a target image that meets preset conditions in terms of target attributes.

[0053] The following are detailed descriptions respectively. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0054] The embodiments of the present application will be described from the perspective of an image generating apparatus, which may be integrated into a computer device, which may be a server or a terminal.

[0055] like Figure 1b A method for generating an image is provided, and the specific process includes:

[0056] 101. Use the generator in the generative adversarial network to generate the original image.

[0057] Among them, Generative Adversarial Networks (GAN) is one of the methods of deep learning and one of the most promising artificial intelligence methods in recent years.

[0058] Among them, artificial intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, machines are given the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0059] Among them, deep learning, as an implementation technology of artificial intelligence, follows bionics and originates from the study of neurons and neural networks. It can simulate the way human neural networks transmit and receive signals, thereby achieving the purpose of learning the human way of thinking. Based on the research of deep learning, artificial neural networks (Artificial Neural Networks), as a technology that simulates the neural networks and functions of biological brains through mathematical models, are widely used in various fields due to their effectiveness.

[0060] In biology, the most basic structural and functional unit of the biological brain neural network (i.e., the biological brain nervous system) is the neuron. Among them, the neuron consists of two parts: the cell body and the protrusion. The cell body is composed of the nucleus, cell membrane and cytoplasm, and has the function of connecting and integrating input information and transmitting information. The protrusion has dendrites and axons. Among them, the dendrites are short and have many branches. They extend and protrude directly from the cell body to form a tree-like shape. Their function is to receive impulses from the axons of other neurons and transmit them to the cell body. The axon is long and has few branches. It is a slender protrusion with uniform thickness. It often starts from the axon hillock. Its function is to receive external stimuli and then transmit them from the cell body.

[0061] When neurons transmit information to each other, the cell body, as the main body of the neuron, can control the concentration of ions in the cell fluid through the cell membrane, so that the ion concentration inside and outside the cell is different, resulting in a resting potential that is negative inside and positive outside. When the cell body receives a signal from the dendrites, it can control the intracellular ion concentration through the cell membrane to generate new processing signals. These signals will be transmitted along the axon and all the way to the synapse of the neuron, thereby transmitting the new signal to the dendrite of the next neuron.

[0062] Artificial neural network (often referred to as neural network (NN)) is to abstract the human brain neural network from the perspective of information processing, build a model, and form different networks according to different connection methods. Neural network is a mathematical model composed of a large number of nodes (equivalent to neurons in the human brain neural network) connected to each other. Each node represents a specific output function, called an activation function (Activation, Function). The connection between each two nodes represents a weighted value for the signal passing through the connection, called the weight, which is equivalent to the memory of the human brain neural network. The neural network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy. The output of the neural network also varies according to the network's connection method, weight value and activation function.

[0063] Neural networks have spawned dozens of mathematical models, such as Convolutional Neural Networks (CNN), De-Convolutional Neural Networks (DN), Deep Convolutional Inverse Graphics Networks (DCIGN), UNet, and so on.

[0064] GAN is one of the methods of deep learning. Different from ordinary neural networks, GAN consists of two main networks: a generator or generator network and a discriminator or discriminator network. The core logic of GAN is that the generator and the discriminator compete with each other.

[0065] The generator can be a neural network, which is used to generate content. For example, the generator can generate a picture, a text, or a video.

[0066] The discriminator can also be a neural network, and its function is to discriminate the content input into the discriminator. For example, taking a picture as an example, the goal of the discriminator is to determine whether the picture input into the discriminator is a picture generated by the generator or a real picture.

[0067] Among them, the training process of the GAN network is the process in which the generator and the discriminator confront and compete with each other.

[0068] In one embodiment, the generator may be any mathematical model in a neural network, for example, a CNN network, a DN network, a DCIGN network, a UNet network, etc.

[0069] In one embodiment, the original image may include various images generated by the generator. For example, the original image may include an image initially generated by the generator, an image generated by the generator in the process of confrontation and game between the generator and the discriminator, and the like.

[0070] The images generated by the generator may include images of various color modes. A color mode may be a model that represents a certain color in digital form, or a way to record the color of an image. Common color space modes include the three primary colors (Red Green and Blue, RGB) color mode, the Red Green Blue and Alpha (RGBA) color mode, and the like.

[0071] For example, the original image may include an image in RGB color mode, an image in RGBA color mode, and so on.

[0072] Among them, the images generated by the generator can also include various types of images.

[0073] For example, the images generated by the generator can include binary images, grayscale images, true color RGB images, semantic segmentation images, and so on.

[0074] Among them, binary images include images whose matrix elements of the image matrix are composed of only two values, 0 or 1, where "0" represents black and "1" represents white. Grayscale images include images whose matrix elements of the image matrix have a value range of [0,256]. True color RGB images include images that use a combination of the three primary colors of red, green, and blue to represent the color of each pixel. Semantic segmentation images include images in which the machine automatically segments the object area from the image and recognizes the content therein.

[0075] In semantic segmentation, “semantics” refers to the content of the image, which is the understanding of the meaning of the image. For example, Figure 1c As shown in the image 1011 in , the semantics is a person riding a motorcycle. Segmentation refers to segmenting different objects in the image from the perspective of pixels and labeling each pixel in the original image. For example, Figure 1c As shown, image 1012 is a semantic segmentation image obtained after semantic segmentation.

[0076] In addition, the original image may also include images in various formats. For example, the image generated by the generator may include images in bitmap (BMP) format, Joint Photographic Expert Group (JPEG) format, Portable Network Graphics (PNG) format, etc.

[0077] In one embodiment, when a generator is used to generate an image, a random noise may be input into the generator, and the image may be generated by the random noise. For example, a multi-dimensional vector noise that obeys a uniform distribution may be input into the generator, and then the generator may obtain feature information of the noise based on the vector information of the input noise, and continuously generate and optimize the details of the image based on the feature information, and finally generate an image.

[0078] For example, when using a generator to generate semantic segmentation images, the generator can be set to a UNet network architecture, and then random noise can be input into the UNet network. The UNet network can then generate semantic segmentation images based on the input random noise, and so on.

[0079] 102. The original image is discriminated by the discriminator in the generative adversarial network to obtain initial error information of the original image. The initial error information represents the error between the original image and the preset reference image in the target attribute.

[0080] The error information may include the error between the image generated by the generator and the preset reference image in terms of the target attribute, and the initial error information may include the error between the original image and the preset reference image in terms of the target attribute, wherein the initial error information may be the initial state of the error information.

[0081] The target attribute may include various properties of the image. For example, the target attribute may include similarity between images, pixel values, saturation, grayscale values, etc. Therefore, the initial error information may include the difference between the original image and the preset reference image in properties such as similarity, pixel values, saturation, grayscale values, etc.

[0082] Optionally, the error information may include multiple forms of expression. For example, the initial error information may be a numerical value, a probability value, a vector, and the like. For example, when the initial error information includes the similarity between the original image and the preset reference image, the initial error information may be a probability value; when the initial error information includes the difference in pixel values ​​between the original image and the preset reference image, the initial error information may be a numerical value; for another example, when the initial error information includes the difference in pixel values, saturation, and other attributes between the original image and the preset reference image, the initial error information may be a vector, and the like.

[0083] The preset reference image may include various images that meet preset conditions in terms of target attributes. For example, when the discriminator discriminates images based on similarity, the preset reference image may be a reference object for the discriminator to discriminate images. For another example, when the discriminator discriminates images based on pixel values, the preset reference image may be an image whose pixel values ​​meet preset conditions, such as an image with high pixel values, etc. In addition, the preset reference image may also be an image whose attributes such as pixel values ​​and saturation meet preset conditions, etc.

[0084] Among them, various reference images can be collected on the Internet or obtained through shooting tools such as cameras and mobile phones, etc.

[0085] In one embodiment, the discriminator may be a convolutional neural network. Therefore, when the discriminator is used to discriminate the original image, the discriminator may be used to first perform convolution feature extraction on the original image input to the discriminator to obtain feature information of the original image. Then, the original image is discriminated based on the convolution feature information of the original image and a preset reference image, thereby determining initial discriminant error information of the original image. Specifically, the step of "performing discriminant processing on the original image by the discriminator in the generative adversarial network to obtain initial error information of the original image" includes:

[0086] The discriminator is used to extract convolution features of the original image to obtain the convolution feature information of the original image;

[0087] The original image is discriminated based on the convolution feature information of the original image and the preset reference image to obtain a discrimination result of the original image;

[0088] The initial error information of the original image is determined according to the discrimination result.

[0089] When performing convolution feature extraction on the original image, at least one convolution kernel may be used to sample the original image to obtain sampling information corresponding to each convolution kernel, and then the sampling information corresponding to each convolution kernel is fused to obtain feature information of the original image. Specifically, the step of "using the convolution layer of the discriminator to extract features from the original image to obtain feature information of the original image" includes:

[0090] Performing feature sampling on the original image using at least one convolution kernel to obtain sampling feature information after feature sampling of at least one convolution kernel;

[0091] The sampled feature information after each convolution kernel feature sampling is fused to obtain the convolution feature information of the original image.

[0092] The convolution kernel may include several convolution units, and each convolution unit may sample the projection area of ​​the convolution kernel on the image. Figure 1d 1013 in the figure may be a presentation form of a convolution kernel, wherein 1014 may be one of the convolution units in the convolution kernel.

[0093] In one embodiment, when performing feature sampling on an image, at least one convolution kernel may be used to perform feature sampling on the image according to the properties of the image.

[0094] For example, when the original image is an RGB image, that is, the original image is obtained by changing the three color channels of red, green, and blue and superimposing them on each other to obtain the image color, wherein the pixels of the three components of the image are red, green, and blue. Figure 1e As shown, the original image 1018 is an image in RGB mode, which includes pixels of three components: red, green, and blue, which are image 1015, image 1016, and image 1017 respectively.

[0095] For a computer, each component of an image is presented in the form of pixels, and pixels are small squares that make up an image, wherein each of these small squares has a clear position and an assigned color value. Figure 1f As shown, from the perspective of the human visual system, the image of the red component is a face image, while for a computer, the red image is a series of pixels, each of which is a numerical value.

[0096] When performing convolution feature extraction on the original image in RGB mode, at least three convolution kernels can be used to perform feature sampling on the original image. Each convolution kernel corresponds to an image of a color component. For example, Figure 1g As shown, each color component has its corresponding convolution kernel, and each convolution kernel has 9 convolution units.

[0097] In one embodiment, the convolution unit in each convolution kernel may correspond to a preset function, and then each convolution kernel may also correspond to a preset function. When the convolution kernel is used to perform feature sampling on an image, the image may be first sampled using the convolution unit, and then the collected image may be processed using the preset function corresponding to the convolution unit, thereby obtaining sampling feature information after the convolution unit feature sampling. Then the convolution kernel may also use its corresponding preset function to process the sampling feature information after the convolution unit feature sampling, thereby obtaining the sampling feature information after the convolution kernel feature sampling.

[0098] For example, Figure 1g As shown, the preset function of the convolution unit in the blue component convolution kernel can be a multiplication function, while the preset function of the blue component convolution kernel can be an addition function. Therefore, after the convolution unit in the blue component convolution kernel performs feature sampling on the image of the blue component, the data corresponding to the convolution unit can be multiplied by the data obtained by the sampling, so as to obtain the sampling feature information after the convolution unit feature sampling.

[0099] Then, after obtaining the sampling feature information after the convolution unit feature sampling, the convolution kernel can process the sampling feature information after the convolution unit feature sampling according to its preset function, so as to obtain the sampling feature information after the convolution unit feature sampling.

[0100] Among them, when using at least one convolution kernel to perform feature sampling on an image, the preset function corresponding to each convolution kernel may be the same or different; the preset function corresponding to the convolution unit in each convolution kernel may also be the same or different.

[0101] After each convolution kernel samples the image, the sampling information corresponding to each convolution kernel can be fused to obtain the feature information of the original image. There are many ways to fuse. For example, the sampling information corresponding to each convolution kernel can be added to obtain the feature information of the original image. For another example, the sampling information corresponding to each convolution kernel can be weighted averaged to obtain the feature information of the original image, and so on.

[0102] For example, Figure 1g As shown, the sampling feature information corresponding to each convolution kernel can be added to obtain the convolution feature information of the image.

[0103] In one embodiment, in practice, the pixels of the original image generated by the generator are very high. For example, when the original image generated by the generator is 12*16 inches, its pixel points may include 3264*2488; for another example, when the original image generated by the generator is 18*24 inches, its pixel points may include 4536*3024. Therefore, when the discriminator performs convolution feature extraction on the original image, it may use multiple convolution kernel groups (hereinafter referred to as convolution layers) to extract features from the original image. Each convolution layer may include multiple convolution kernels to perform feature sampling on the original image.

[0104] For example, five convolutional layers may be used to perform convolutional feature extraction on the original image, wherein each convolutional layer may include three convolutional kernels to perform feature sampling on the original image, thereby obtaining convolutional feature information of the original image, and so on.

[0105] After the feature information of the original image is acquired, the original image may be identified based on the feature information of the original image and a preset reference image to obtain an identification result of the original image.

[0106] In one embodiment, when the preset reference image is stored in the discriminator in the form of an image, the discriminator can be used to extract features of the preset reference image, and the feature information of the preset reference image can be stored in the discriminator. At this time, when the original image is discriminated based on the feature information of the original image and the preset reference image, the feature information of the original image and the feature information of the preset reference image can be compared, and the initial error information of the original image can be determined according to the comparison result.

[0107] Among them, the preset loss function in the discriminator can be used to calculate the feature difference value between the feature information of the original image and the feature information of the preset reference image, and then the calculated feature difference value can be used as the initial error information of the original image.

[0108] The preset loss function includes a function used to quantify the gap between information. For example, the preset loss function may include a function that quantifies the gap between feature information of an original image and feature information of a preset reference image. For example, the preset loss function may include a 0-1 loss function, a square loss function, a logarithmic loss function, a cross entropy loss function, an exponential loss function, and the like.

[0109] In one embodiment, the discriminator may be first trained using a preset reference image so that the discriminator has a discrimination standard. Then, the original image may be discriminated using the same process as the training process of the discriminator to obtain a discrimination result of the original image. Then, the discrimination result is compared with the discrimination standard to obtain initial error information of the original image, and so on.

[0110] 103. Adjust parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator.

[0111] After obtaining the error information of the original image, the parameters of the generator and the offset module in the generative adversarial network can be adjusted according to the initial error information to obtain an adjusted offset module and an adjusted generator.

[0112] The offset module includes a module that can generate offset information. For example, the offset module can be an image converter, wherein the image converter includes several image conversion algorithms. Various image conversions can be achieved through the offset module, such as size conversion, shape conversion, geometric conversion, orthogonal conversion, etc.

[0113] For example, when the image is resized, the offset module may include a linear algorithm, which may include several linear transformation functions; when the image is transformed in shape, the offset module may include an image transformation algorithm, which may include several nonlinear transformation functions; when the image is orthogonally transformed, the offset module may include an orthogonal transformation algorithm, which includes several orthogonal transformation functions, and so on.

[0114] In one embodiment, when adjusting the parameters of the offset module in the generative adversarial network according to the initial error information, the parameters of several functions in the module may be adjusted to obtain an adjusted offset module.

[0115] In which, since the generator in the adversarial generative network can be a neural network, and since the neural network can be a mathematical model, the operation model is composed of a large number of nodes connected to each other, each node represents a specific output function, and the connection between every two nodes represents a weighted value for the signal passing through the connection. Therefore, when adjusting the parameters of the generator, the parameters and / or weighted values ​​of each function in the generator can be adjusted.

[0116] 104. Use an adjusted offset module to obtain offset information of the original image in at least one scale based on the initial error information.

[0117] The scale may include various attribute information of the image, wherein the attribute information may include the shape, size, etc. of the image.

[0118] The offset information may include the degree of transformation of the original image. For example, when the original image is scaled, the offset information may include the degree of transformation between the scaled image and the original image. For example, when the original image is resized, the offset information may include the degree of resize between the scaled image and the original image, and so on.

[0119] The offset information may be a numerical value, a percentage value, a vector, a matrix, etc.

[0120] After obtaining the initial error information of the original image and the adjusted offset module, the adjusted offset module may be used to obtain the offset information of the original image on at least one scale based on the initial error information.

[0121] In one embodiment, the adjusted offset module may be used to scale the original image based on the initial error information to obtain a scaled image, and then calculate the offset information between the scaled image and the original image. Specifically, the step of "using the adjusted offset module to obtain the offset information of the original image at at least one scale based on the initial error information" includes:

[0122] Using the adjusted offset module to perform at least one scale transformation on the original image based on the initial error information to obtain a transformed image after at least one scale transformation;

[0123] The offset information between the transformed image and the original image after each scale transformation is calculated by using the adjusted offset module to obtain the offset information of the original image in at least one scale.

[0124] Among them, scale transformation can include image size transformation, shape transformation, geometric transformation, orthogonal transformation, etc.

[0125] In one embodiment, since the offset module may be an image converter, wherein the image converter includes several image conversion algorithms. For example, when performing size conversion on an image, the offset module may include a linear algorithm, wherein the linear algorithm may include several linear conversion functions; when performing shape conversion on an image, the offset module may include an image conversion algorithm, wherein the image conversion algorithm may include several nonlinear conversion functions; when performing orthogonal conversion on an image, the offset module may include an orthogonal conversion algorithm, wherein the orthogonal conversion algorithm includes several orthogonal conversion functions, and so on.

[0126] Therefore, when the original image is scaled based on the initial error information using the offset module, the original image and the initial error information may be input into the offset module, and then the offset module scales the original image based on the initial error information.

[0127] For example, it is assumed that the offset module performs a size conversion on an image, and the initial error information is a numerical value. When the initial error information is input into the offset module, if the initial error information is a value indicating that the error between the original image and the preset reference image in the target attribute is large, the offset module will adjust the size of the original image according to the initial error information. Specifically, the original image can be enlarged or reduced by using the linear transformation function preset in the offset module, and so on.

[0128] When the original image is scaled, the original image may be scaled at least once. For example, when the original image is resized, the original image may be scaled multiple times, and then each resized image is obtained.

[0129] In addition, when the original image is scaled, the original image may be scaled in various forms. For example, the original image may be scaled and transformed. For another example, when the original image is scaled multiple times, some of the scales may be scaled, some of the scales may be transformed, and so on.

[0130] After obtaining the transformed image after each scale transformation, the transformation degree between the transformed image and the original image can be calculated, thereby obtaining the offset information of the original image. The transformation degree of the scale transformation between the transformed image and the original image can be calculated according to the image scale transformation method.

[0131] For example, when the image is resized, the size difference between the transformed image and the original image can be calculated to determine the offset information. For example, the size of the transformed image can be subtracted from the size of the original image and then the absolute value can be taken to determine the offset information.

[0132] For another example, when an image is transformed in shape, the image recognition technology can be used to calculate the degree of transformation of the scale transformation between the transformed image and the original image. For example, when the original image is transformed from a rectangle to a rounded rectangle, the image recognition technology can be used to calculate the degree of transformation of the original image from a rectangle to a rounded rectangle.

[0133] 105. Based on the offset information on at least one scale, trigger the discriminator to adjust the initial error information to obtain adjusted error information.

[0134] After the offset information is obtained, the discriminator may be triggered to adjust the initial error information based on the offset information, thereby obtaining the adjusted error information.

[0135] In one embodiment, when the original image is processed by a convolution kernel to obtain feature information of the original image, and finally the original image is discriminated based on the convolution feature information of the original image and a preset reference image to determine the initial discriminant error information of the original image, the convolution kernel can be scaled based on the offset information, and then the discriminator including the adjusted convolution kernel is used to discriminate the original image to obtain the adjusted error information. Specifically, the step of "triggering the discriminator to adjust the initial error information based on the offset information on at least one scale to obtain the adjusted error information" includes:

[0136] Based on the offset information at each scale, the convolution kernel is scaled to obtain the adjusted convolution kernel;

[0137] The discriminator including the adjusted convolution kernel is used to perform discriminative processing on the original image to obtain adjusted error information.

[0138] Among them, since the scale of the image includes various attribute information, such as the shape and size of the image, etc. The offset information generated by the offset module can also include offset information of various attributes. For example, when the original image is resized, the offset information may include the degree of size transformation between the image after the scale transformation and the original image; when the image is changed in shape, the offset information may include the degree of shape transformation between the image after the scale transformation and the original image; since different offset information corresponds to different scale adjustment methods, when the convolution kernel is scaled based on the offset information on the scale, the scale adjustment method can be first determined according to the offset information, and then the convolution kernel is scaled based on the scale adjustment method. Specifically, the step of "scale-adjusting the convolution kernel based on the offset information on each scale to obtain the adjusted convolution kernel" includes:

[0139] According to the offset information, determine the way to adjust the convolution kernel scale;

[0140] The convolution kernel is resized based on the resizing method to obtain the adjusted convolution kernel.

[0141] For example, after obtaining the offset information, it can be determined whether the offset information represents the degree of size transformation between the scale-transformed image and the original image or the degree of shape transformation between the scale-transformed image and the original image, etc.

[0142] Then, the convolution kernel scale adjustment method is determined based on the offset information. For example, when the offset information indicates the degree of size transformation between the scale-transformed image and the original image, a linear function can be used to linearly transform the sampling method of each sampling point in the convolution kernel based on the offset. For example, Figure 1hThe original sampling method and the deformed sampling method 1. When a linear transformation is performed on the sampling method of each sampling point in the convolution kernel, the sampling point of each convolution kernel is no longer just a rectangular sampling, but a divergent sampling.

[0143] For another example, when the offset information indicates the degree of shape transformation between the scale-transformed image and the original image, a nonlinear function can be used to perform a nonlinear transformation on the sampling method of each sampling point in the convolution kernel. Figure 1h The original sampling method and the deformed sampling method 2, when the sampling method of each sampling point in the convolution kernel is nonlinearly transformed, the sampling point of each convolution kernel is no longer just a rectangular sampling, but can also be a deformed sampling.

[0144] In one embodiment, a mapping table may be established, which may indicate a mapping relationship between the offset information and the convolution kernel scale adjustment method. Then, when determining the convolution kernel scale adjustment method according to the offset information, the mapping table may be queried for determination.

[0145] Finally, the convolution kernel can be resized based on the determined resizing method to obtain an adjusted convolution kernel.

[0146] After the adjusted convolution kernel is generated, the discriminator of the adjusted convolution kernel can be used to perform discriminative processing on the original image to obtain adjusted error information.

[0147] For example, after generating the adjusted convolution kernel, the adjusted convolution kernel can be used to perform feature sampling on the original image, thereby obtaining convolution feature information of the adjusted convolution kernel. The original image is then judged based on the convolution feature information, and the adjusted error information of the original image is determined based on the judgment result.

[0148] Since the convolution kernel is adjusted according to the offset information, the sampling method of the convolution kernel is no longer single and fixed, but can be adjusted according to the error between the image and the preset reference image in the target attribute, thereby improving the accuracy of the convolution feature information of the image. When the accuracy of the convolution feature information of the image is improved, the discriminant ability of the discriminator can be improved. Since in the generative adversarial network, the discriminator and the generator are in a state of mutual confrontation and mutual game, the improvement of the discriminant ability can prompt the generator to improve its generation ability, so that the target image that meets the preset conditions in terms of target attributes can be generated.

[0149] 106. Triggering the adjusted generator to generate a target image that meets preset conditions in terms of target attributes according to the adjusted error information.

[0150] After the adjusted error information is obtained, the adjusted error information can be used to trigger an adjusted generator to generate a target image that meets preset conditions in terms of target attributes.

[0151] Before using the adjusted error information to trigger the adjusted generator to generate a target image that meets the preset conditions in terms of target attributes, it can be determined whether the adjusted error information meets the preset error conditions. If not, the adjusted error information is used to trigger the adjusted generator to generate a target image that meets the preset conditions in terms of target attributes. Specifically, the step of "triggering the adjusted generator to generate a target image that meets the preset conditions in terms of target attributes according to the adjusted error information" includes:

[0152] When the adjusted error information meets the preset error condition, triggering the adjusted generator to update the original image according to the adjusted error information to obtain a target image that meets the preset condition in terms of target attributes;

[0153] When the adjusted error information does not meet the preset error condition, the step of adjusting the parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain the adjusted offset module and the adjusted generator is performed.

[0154] Since the error information may include multiple forms of expression, for example, the error information may be a numerical value, a probability value, or a vector, etc. Therefore, the preset error condition may be set according to the form of the error information.

[0155] For example, when the error information is a numerical value, the preset error condition may also be a threshold value. In this case, when the adjusted error information is within the range of the preset threshold value, the adjusted error information satisfies the preset error condition. For another example, when the error information is a probability value, the preset error condition may also be a probability value. The preset error condition may be satisfied when the adjusted error information is greater than or equal to the probability value, or may be satisfied when the adjusted error information is less than or equal to the probability value, and so on.

[0156] When the adjusted error information satisfies a preset error condition, the adjusted generator may be triggered according to the adjusted error information to update the original image, so as to obtain a target image that meets the preset condition in terms of target attributes.

[0157] When the adjusted error information does not meet the preset error condition, the generator and the offset module in the generative adversarial network are adjusted according to the initial error information to obtain the adjusted offset module and the adjusted generator. For example, the initial error information can be updated to the adjusted error information, and then the parameters of the generator and the offset module in the generative adversarial network are adjusted again according to the adjusted error information to obtain the adjusted offset module and the adjusted generator and the following steps. This is repeated until the adjusted error information meets the preset error condition.

[0158] The image generation method introduced above can use the generator in the generative adversarial network to generate the original image; the discriminator in the generative adversarial network performs discriminative processing on the original image to obtain the initial error information of the original image, and adjusts the parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain the adjusted offset module and the adjusted generator; then, the adjusted offset module is used to obtain the offset information of the original image on at least one scale based on the initial error information, and based on the offset information on at least one scale, the discriminator is triggered to adjust the initial error information to obtain the adjusted error information; finally, the adjusted generator is triggered according to the adjusted error information to generate a target image that meets preset conditions in terms of target attributes, thereby improving the accuracy of the image generated by the generator.

[0159] In addition, since the convolution kernel is adjusted according to the offset information, the sampling method of the convolution kernel is no longer single and fixed, but can be adjusted according to the error between the image and the preset reference image in the target attribute, thereby improving the accuracy of the convolution feature information of the image. When the accuracy of the convolution feature information of the image is improved, the discriminant ability of the discriminator can be improved. Since in the generative adversarial network, the discriminator and the generator are in a state of mutual confrontation and mutual game, the improvement of the discriminant ability can prompt the generator to improve its generation ability, so that the target image that meets the preset conditions in terms of target attributes can be generated.

[0160] According to the method described in the above embodiment, the following is further described in detail with examples.

[0161] The present embodiment of the application will take the example of an image generating device integrated on a computer device to introduce the method of the present embodiment of the application.

[0162] like Figure 2a As shown, an image generation method, the specific process is as follows:

[0163] 201. The computer device uses a generator in a generative adversarial network to generate an original image.

[0164] The generator can be a UNet network, for example, Figure 2bAs shown, the original image can be a semantic segmentation image.

[0165] from Figure 2b It can be seen that the two main features of the UNet network structure are: U-shaped network structure and skip layer connection. The UNet network structure is a symmetrical network structure, which includes two paths on the left and right. The path on the left can be regarded as an encoder, or it can be called an upsampling processing path. It includes five convolution submodules, each of which includes two convolution layers and a ReLU layer. The convolution layer structure here is unified as a 3×3 convolution kernel. After each submodule, there is a downsampling layer implemented by maximum pooling. The convolution submodule is used to extract features, and the maximum pooling layer is used to reduce the dimension. The resolution of the output feature image after each maximum pooling layer is halved. The feature map output by the last convolution submodule is not subjected to maximum pooling and is directly input to the decoder on the right. The path on the right can be regarded as a decoder, or it can be called a downsampling processing path. It includes a structure that is basically symmetrical with the encoder. It performs 3×3 convolution and upsampling on the input feature map to gradually repair the details and spatial dimensions of the object. In addition, feature fusion is also used in the network, such as Figure 2b As shown by the dotted arrows in the figure, the features of the previous down-sampling network are spliced ​​and fused with the features of the subsequent up-sampling through skip-layer connections to obtain more accurate context information and achieve better segmentation results. The semantic segmentation model finally outputs a segmentation map of the target object, in which the pixel value of each pixel can be a label representing its category.

[0166] exist Figure 2b In the UNet network structure shown, the solid arrow to the right represents a 3×3 convolution layer + activation layer (ReLU), which is used for feature extraction. The hollow arrow pointing downward is a pooling layer, which is used to reduce the dimension; the hollow arrow pointing upward is an upsampling layer, which is used to restore the dimension; the hollow arrow pointing right is a 1×1 convolution layer, which is used to output the result; the dotted arrow pointing right in the middle is a concatenation layer, which copies and concatenates the features of the previous layer to the corresponding next layer through skip-layer connection for feature fusion, thereby retaining more information. In this network structure, the image is continuously reduced in size when extracting features, and when it is reduced to a certain size, it starts to be upsampled to expand the size, and finally outputs the segmentation map of the target object, in which each pixel contains the label of its category.

[0167] Use Figure 2b The UNet network structure shown in the figure can input random noise into the UNet network, and then the UNet network can generate a semantic segmentation image based on the input random noise.

[0168] 202. The computer device performs discriminative processing on the original image through the discriminator in the generative adversarial network to obtain initial error information of the original image, and the initial error information represents the error between the original image and the preset reference image in the target attribute.

[0169] When the discriminator is used to discriminate the original image, the discriminator can first be used to perform convolution feature extraction on the original image input to the discriminator to obtain feature information of the original image. Then, the original image is discriminated based on the convolution feature information of the original image and a preset reference image, thereby determining initial discrimination error information of the original image.

[0170] Among them, when performing convolution feature extraction on the original image, at least one convolution kernel can be used to sample the original image to obtain sampling information corresponding to each convolution kernel, and then the sampling information corresponding to each convolution kernel is fused to obtain the feature information of the original image.

[0171] The discriminator may be a convolutional neural network. Specifically, the network structure of the convolutional neural network may be as shown in Table 1.

[0172]

[0173] Table 1: Structure of the discriminator

[0174] The structure of the discriminator indicates that the discriminator includes 5 convolutional layers. The structure of the convolution kernel in each convolutional layer indicates how many convolution units there are in the convolution kernel, and 4*4 indicates that there are 16 convolution units in the convolution kernel, and the 16 convolution units are arranged in the form of a 4*4 matrix. The number of channels may include the number of convolution kernels in the convolution layer. The step size may include the sampling interval of each sampling when the convolution kernel is sampled.

[0175] Among them, each convolution layer in the discriminator is not independent of each other, but interconnected. The feature information extracted by each convolution layer will be used as the input of the next convolution layer. For example, the feature information extracted by convolution layer 1 will be used as the input of convolution layer 2, and convolution layer 2 will extract the feature information extracted by convolution layer 1. Among them, since the neural network simulates the neural network of the biological brain, when the biological brain obtains external information, the external information is complex and diverse, so the brain will generate the law that best conforms to the distribution of external information based on the external information. The process of the brain generating this law is not rigid and step-by-step, but flexible. Therefore, after obtaining the feature information, each convolution layer can flexibly transform the feature information so that the feature information extracted by the feature extraction is more in line with the law of the image, and this flexible transformation can be achieved through the excitation function.

[0176] 203. The computer device adjusts parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator.

[0177] For example, when adjusting the parameters of the offset module in the generative adversarial network according to the initial error information, the parameters of several functions in the module can be adjusted to obtain an adjusted offset module.

[0178] For example, when the generator is a UNet network structure, the parameters and / or weight values ​​of each function in the UNet network structure may be adjusted.

[0179] 204. The computer device uses the adjusted offset module to obtain offset information of the original image in at least one scale based on the initial error information.

[0180] For example, when performing a size transformation on an image, the offset module may include a linear algorithm, wherein the linear algorithm may include several linear transformation functions;

[0181] For another example, when performing shape transformation on an image, the offset module may include an image transformation algorithm, wherein the image transformation algorithm may include several nonlinear transformation functions.

[0182] 205. The computer device triggers the discriminator to adjust the initial error information based on the offset information on at least one scale to obtain adjusted error information.

[0183] For example, the convolution kernel can be resized based on the offset information, and then the discriminator including the adjusted convolution kernel is used to perform discriminative processing on the original image to obtain adjusted error information.

[0184] When the convolution kernel is scaled based on the offset information, the scale adjustment method of the convolution kernel can be determined according to the offset information, and then the convolution kernel is scaled according to the scale adjustment method to obtain the adjusted convolution kernel.

[0185] For example, when the offset information represents the degree of size transformation between the scale-transformed image and the original image, a linear function can be used to linearly transform the sampling method of each sampling point in the convolution kernel according to the offset.

[0186] For another example, when the offset information represents the degree of shape transformation between the image after the scale transformation and the original image, a nonlinear function can be used to perform a nonlinear transformation on the sampling method of each sampling point in the convolution kernel, and so on.

[0187] Finally, the convolution kernel can be resized based on the determined resizing method to obtain an adjusted convolution kernel.

[0188] After the adjusted convolution kernel is generated, the discriminator of the adjusted convolution kernel can be used to perform discriminative processing on the original image to obtain adjusted error information.

[0189] 206. The computer device triggers the adjusted generator to generate a target image that meets preset conditions in terms of target attributes according to the adjusted error information.

[0190] Wherein, when the error information may be a numerical value, the preset error condition may be a threshold value.

[0191] Among them, when the adjusted error information does not meet the preset error condition, the initial error information can be updated to the adjusted error information, and then the parameters of the generator and the offset module in the generative adversarial network are adjusted again according to the adjusted error information to obtain the adjusted offset module and the adjusted generator and the next steps. Through such repeated cycles, until the adjusted error information meets the preset error condition.

[0192] In one embodiment, in order to make the adjusted error information meet the preset error condition more quickly, a momentum mechanism (also called momentum method) may be introduced in the loop process.

[0193] For example, a momentum mechanism may be introduced in updating the initial error information to the adjusted error information, or in scaling the convolution kernel, or in an offset module that obtains the offset information of the original image on at least one scale based on the initial error information, so that the adjusted error information can meet the preset error condition more quickly.

[0194] Among them, the momentum mechanism is an optimization algorithm, which is similar to optimization algorithms such as gradient descent, and can quickly find the optimal solution in the model loop iteration.

[0195] In an embodiment of the present application, a computer device may use a generator in a generative adversarial network to generate an original image; the computer device may obtain initial error information of the original image by performing discriminative processing on the original image by a discriminator in the generative adversarial network, and adjust parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator; then, the computer device uses the adjusted offset module to obtain offset information of the original image on at least one scale based on the initial error information, and triggers the discriminator to adjust the initial error information based on the offset information on at least one scale to obtain adjusted error information; finally, the computer device triggers the adjusted generator to generate a target image that meets preset conditions in terms of target attributes according to the adjusted error information, thereby improving the accuracy of the image generated by the generator.

[0196] In order to better implement the image generation method provided in the embodiment of the present application, an image generation device is also provided in one embodiment, and the image generation device can be integrated in a terminal or a server. The meanings of the terms are the same as those in the above-mentioned image generation method, and the specific implementation details can refer to the description in the method embodiment.

[0197] In one embodiment, an image generating device is provided, which can be integrated into a computer device, such as Figure 3a As shown, the image generation device includes: a generation unit 301, a discrimination processing unit 302, a parameter adjustment unit 303, an acquisition unit 304, a first trigger unit 305, and a second trigger unit 306, which are specifically as follows:

[0198] A generating unit 301 is used to generate an original image using a generator in a generative adversarial network;

[0199] A discriminant processing unit 302 is used to perform discriminant processing on the original image through the discriminator in the generative adversarial network to obtain initial error information of the original image, wherein the initial error information represents the error between the original image and a preset reference image in a target attribute;

[0200] A parameter adjustment unit 303 is used to adjust the parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator;

[0201] An acquiring unit 304 is configured to acquire offset information of the original image at at least one scale based on the initial error information by using the adjusted offset module;

[0202] A first triggering unit 305 is configured to trigger the discriminator to adjust the initial error information based on the offset information on the at least one scale to obtain adjusted error information;

[0203] The second triggering unit 306 is configured to trigger the adjusted generator to generate a target image that meets a preset condition in terms of the target attribute according to the adjusted error information.

[0204] In one embodiment, if Figure 3b As shown, the determination processing unit 302 may include:

[0205] A convolution feature extraction subunit 3021 is used to perform convolution feature extraction on the original image using the discriminator to obtain convolution feature information of the original image;

[0206] The discriminating subunit 3022 is used to discriminate the original image based on the convolution feature information of the original image and the preset reference image to obtain a discriminating result of the original image;

[0207] The determination subunit 3023 is used to determine the initial error information of the original image according to the discrimination result.

[0208] In one embodiment, the convolution feature extraction subunit 3021 may be used to:

[0209] Performing feature sampling on the original image using at least one convolution kernel to obtain sampling feature information after feature sampling of the at least one convolution kernel;

[0210] The sampled feature information after each convolution kernel feature sampling is fused to obtain the convolution feature information of the original image.

[0211] In one embodiment, if Figure 3b As shown, the acquisition unit 304 may include:

[0212] The scale transformation subunit 3041 is used to perform at least one scale transformation on the original image based on the initial error information by using the adjusted offset module to obtain a transformed image after at least one scale transformation;

[0213] The obtaining subunit 3042 is used to calculate the offset information between the transformed image after each scale transformation and the original image by using the adjusted offset module to obtain the offset information of the original image in at least one scale.

[0214] In one embodiment, if Figure 3b As shown, the second trigger unit 306 may include:

[0215] The trigger subunit 3061 is configured to trigger the adjusted generator to update the original image according to the adjusted error information when the adjusted error information satisfies a preset error condition, so as to obtain a target image that meets the preset condition in terms of the target attribute;

[0216] The execution subunit 3062 is used to, when the adjusted error information does not meet the preset error condition, execute the step of adjusting the parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain the adjusted offset module and the adjusted generator.

[0217] In an embodiment of the present application, the generation unit 301 can be used to generate a generator in a generative adversarial network to generate an original image; the discriminant processing unit 302 performs discriminant processing on the original image to obtain initial error information of the original image, and the parameter adjustment unit 302 is used to adjust the parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator; then, the acquisition unit 304 uses the adjusted offset module to obtain the offset information of the original image on at least one scale based on the initial error information; based on the offset information on at least one scale, the first trigger unit 305 triggers the discriminator to adjust the initial error information to obtain the adjusted error information; finally, the second trigger unit 306 triggers the adjusted generator to generate a target image that meets preset conditions in terms of target attributes according to the adjusted error information, thereby improving the accuracy of the image generated by the generator.

[0218] The present application also provides a computer device, which may include a terminal or a server. For example, the computer device may be used as an image generation terminal, which may be a mobile phone, a tablet computer, etc.; for another example, the computer device may be a server, such as an image generation server. Figure 4 As shown, it shows a schematic diagram of the structure of the terminal involved in the embodiment of the present application, specifically:

[0219] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0220] The processor 401 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, it executes various functions of the computer device and processes data, thereby performing overall detection of the computer device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0221] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0222] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0223] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0224] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically, in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0225] Use the generator in the generative adversarial network to generate the original image;

[0226] The discriminator in the generative adversarial network performs discriminative processing on the original image to obtain initial error information of the original image, wherein the initial error information represents the error between the original image and a preset reference image in a target attribute;

[0227] Adjusting parameters of a generator and an offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator;

[0228] Using the adjusted offset module to obtain offset information of the original image on at least one scale based on the initial error information;

[0229] Based on the offset information on the at least one scale, trigger the discriminator to adjust the initial error information to obtain adjusted error information;

[0230] The adjusted generator is triggered according to the adjusted error information to generate a target image that meets a preset condition on the target attribute.

[0231] The above is a detailed introduction to an image generation method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image generation method, characterized in that: include: Use the generator in the generative adversarial network to generate the original image; The discriminator in the generative adversarial network performs discriminative processing on the original image to obtain initial error information of the original image, wherein the initial error information represents the error between the original image and a preset reference image in a target attribute; Adjusting parameters of a generator and an offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator; Using the adjusted offset module to obtain offset information of the original image on at least one scale based on the initial error information; Based on the offset information, the convolution kernel in the discriminator is scaled, and the original image is discriminated by using the adjusted discriminator to obtain adjusted error information; The adjusted generator is triggered according to the adjusted error information to generate a target image that meets a preset condition on the target attribute.

2. The image generation method according to claim 1, characterized in that: The discriminant processing of the original image by the discriminator in the generative adversarial network to obtain initial error information of the original image includes: Using the discriminator to perform convolution feature extraction on the original image to obtain convolution feature information of the original image; The original image is discriminated based on the convolution feature information of the original image and the preset reference image to obtain a discrimination result of the original image; The initial error information of the original image is determined according to the discrimination result.

3. The image generation method according to claim 2, characterized in that: The step of extracting convolutional features from the original image using the discriminator to obtain convolutional feature information of the original image includes: Performing feature sampling on the original image using at least one convolution kernel to obtain sampling feature information after feature sampling of the at least one convolution kernel; The sampled feature information after each convolution kernel feature sampling is fused to obtain the convolution feature information of the original image.

4. The image generation method according to claim 3, characterized in that: The convolution kernel in the discriminator is scaled based on the offset information, and the original image is discriminated by the adjusted discriminator to obtain adjusted error information, including: Based on the offset information at each scale, the convolution kernel is scaled to obtain an adjusted convolution kernel; The original image is discriminated by using a discriminator including the adjusted convolution kernel to obtain the adjusted error information.

5. The image generation method according to claim 4, characterized in that: The step of adjusting the scale of the convolution kernel based on the offset information at each scale to obtain an adjusted convolution kernel includes: Determining a method for adjusting the scale of the convolution kernel according to the offset information; The convolution kernel is scaled based on the scale adjustment method to obtain an adjusted convolution kernel.

6. The image generation method according to claim 1, characterized in that: The adopting the adjusted offset module to obtain the offset information of the original image at at least one scale based on the initial error information includes: Using the adjusted offset module to perform at least one scale transformation on the original image based on the initial error information to obtain a transformed image after at least one scale transformation; The adjusted offset module is used to calculate the offset information between the transformed image after each scale transformation and the original image, so as to obtain the offset information of the original image in at least one scale.

7. The image generation method according to claim 1, characterized in that: Triggering the adjusted generator to generate a target image that meets a preset condition on the target attribute according to the adjusted error information includes: When the adjusted error information satisfies a preset error condition, triggering the adjusted generator to update the original image according to the adjusted error information to obtain a target image that meets the preset condition in terms of the target attribute; When the adjusted error information does not meet the preset error condition, the step of adjusting parameters of the generator and the offset module in the generative adversarial network according to the initial error information to obtain the adjusted offset module and the adjusted generator is performed.

8. An image generating device, characterized in that: include: A generating unit, used for generating an original image using a generator in a generative adversarial network; A discriminant processing unit, configured to perform discriminant processing on the original image through a discriminator in the generative adversarial network to obtain initial error information of the original image, wherein the initial error information represents an error between the original image and a preset reference image in a target attribute; A parameter adjustment unit, configured to adjust parameters of a generator and an offset module in the generative adversarial network according to the initial error information to obtain an adjusted offset module and an adjusted generator; an acquiring unit, configured to acquire offset information of the original image at at least one scale based on the initial error information by using the adjusted offset module; A first trigger unit is used to adjust the scale of the convolution kernel in the discriminator based on the offset information, and use the adjusted discriminator to perform discriminative processing on the original image to obtain adjusted error information; The second triggering unit is used to trigger the adjusted generator to generate a target image that meets a preset condition in terms of the target attribute according to the adjusted error information.

9. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the operation in the image generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image generating method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the image generation method described in any one of claims 1 to 7.

Images