Image generation method and device
By generating feature images and then generating target images, the problem of unstableness when the generative adversarial network model is solved, the image details and stability are improved, and the probability of color distortion is reduced.
Patent Information
- Application Number
- CN201910498731.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-06-10
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-06-10
Smart Images

Figure CN112070853B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, apparatus, computer equipment and readable storage medium. Background Art
[0002] In the field of website and web design, the demand for image materials is very large. If each image material is designed and constructed separately, it will generate a very large workload and the efficiency will be extremely low. Therefore, artificial intelligence technology is often used to automatically generate image materials.
[0003] The existing methods for automatically generating image materials basically input random normally distributed Gaussian noise into a generative adversarial network (GAN) model, and the GAN model directly outputs the complete image material.
[0004] However, in actual application scenarios, when the image material is required to have more image details, Gaussian noise is directly generated into a complete image material through the generative adversarial network model. However, due to the large number of details in the complete image, it is difficult to directly generate a complete image from noise, resulting in the generated image material being unstable and the image material being more likely to have color distortion.
[0005] Application Contents
[0006] In view of the above problems, the present application is proposed to provide an image generation method, apparatus, computer device and readable storage medium that overcome the above problems or at least partially solve the above problems.
[0007] According to one aspect of the present application, there is provided an image generation method, comprising:
[0008] Get noise data;
[0009] Obtaining a characteristic image according to the noise data and a first data set; wherein the first data set is used to generate the characteristic image according to the noise data;
[0010] A target image is obtained according to the feature image and a second data set; wherein the second data set is used to generate the target image according to the feature image.
[0011] According to another aspect of the present application, there is provided an image generation method, comprising:
[0012] Get noise data;
[0013] Generate a feature image based on the noise data and a first neural network, wherein the feature image includes a texture feature image or an edge feature image;
[0014] According to the feature image, based on a second neural network, a target image is generated.
[0015] According to another aspect of the present application, there is provided an image generating device, comprising:
[0016] A first acquisition module, used for acquiring noise data;
[0017] A first generating module, configured to obtain a characteristic image according to the noise data and a first data set; wherein the first data set is used to generate the target characteristic image according to the noise data;
[0018] The second generating module is used to obtain a target image according to the feature image and a second data set; wherein the second data set is used to generate the target image according to the feature image.
[0019] According to another aspect of the present application, there is provided an image generating device, the device comprising:
[0020] A second acquisition module is used to acquire noise data;
[0021] A third generating module, configured to generate a feature image based on the noise data and the first neural network, wherein the feature image includes a texture feature image or an edge feature image;
[0022] The fourth generating module is used to generate a target image according to the feature image based on the second neural network.
[0023] According to another aspect of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements one or more of the above methods when executing the computer program.
[0024] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, one or more of the above methods are implemented.
[0025] According to the embodiment of the present application, noise data can be directly generated into a feature image through the first data set, and because the picture composition of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image. This makes the process of generating the feature image from the first data set more concise, and also reduces the difficulty of optimizing the relevant parameters in the first data set, so that the probability of the first data set generating a feature image with a relatively stable picture is greatly improved. Furthermore, the embodiment of the present application reuses the second data set to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the probability of color distortion in the target image is reduced, thereby improving the picture quality of the target image.
[0026] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the optional embodiments below. The accompanying drawings are only for the purpose of illustrating the optional embodiments and are not to be considered as limiting the present application. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0028] Figure 1 A system architecture diagram of an image generation method provided by the present application is shown;
[0029] Figure 2 An actual scene diagram of an image generation method provided by the present application is shown;
[0030] Figure 3 A flowchart of an image generation method according to an embodiment of the present application is shown;
[0031] Figure 4 A specific scene diagram of an image generation method according to an embodiment of the present application is shown;
[0032] Figure 5 A flowchart of an image generation method according to an embodiment of the present application is shown;
[0033] Figure 6 A flowchart showing the specific steps of an image generation method according to an embodiment of the present application is shown;
[0034] Figure 7 A specific application scenario diagram of another image generation method according to an embodiment of the present application is shown;
[0035] Figure 8 A flowchart of another image generation method embodiment according to an embodiment of the present application is shown;
[0036] Fig. 9 A flowchart showing the specific steps of another image generation method embodiment according to an embodiment of the present application is shown;
[0037] Fig.10 A structural block diagram of an image generation method according to an embodiment of the present application is shown;
[0038] Fig.11 A structural block diagram of another image generation method embodiment according to an embodiment of the present application is shown;
[0039] Fig.12 An exemplary system is shown that can be used to implement the various embodiments described in this disclosure. DETAILED DESCRIPTION
[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0041] In order to enable those skilled in the art to better understand the present application, the concepts involved in the present application are explained below:
[0042] Noise data refers to data in which there are errors or anomalies (deviations from expected values). In the embodiment of the present application, the noise data may be Gaussian noise. Gaussian noise refers to a type of noise whose probability density function obeys a Gaussian distribution (i.e., a normal distribution). Gaussian noise is also a common noise in digital images. Common Gaussian noises include fluctuation noise, cosmic noise, thermal noise, shot noise, and the like. In the embodiment of the present application, Gaussian noise may exist in the form of a noise image and may be processed by the first data set into a feature image of a preset size. Of course, in the embodiment of the present application, other types of noise may also be used, such as Rayleigh noise, gamma noise, and the like.
[0043] Feature images mainly include texture feature images and edge feature images. Among them, texture feature images describe the surface properties of the scene corresponding to the image or image area, reflect the visual characteristics of homogeneous phenomena in the image, and embody the surface structure organization and arrangement properties of the object surface in the image with slow or periodic changes. The two elements that constitute the texture feature image are as follows:
[0044] (1) Texture primitives: The most basic unit in an image is a pixel. A collection of pixels with a certain shape and size, such as round spots, block spots, and floral patterns, is called a texture primitive.
[0045] (2) Texture feature images are composed of texture primitives. The density, periodicity, and directionality of the texture primitives can greatly change the appearance of the image.
[0046] Edge feature images are used to represent edges or discontinuous areas with obvious changes in an image. Since the edge is the boundary between different areas in an image, an edge image can be a binary image. The purpose of edge detection is to capture areas with sharp changes in brightness. Ideally, edge detection of an image can produce an edge feature image composed of a series of continuous curves, which is used to represent the boundaries of an object.
[0047] It should be noted that the edge feature image and texture feature image included in a complete image can be paired. For example, if the final requirement is to obtain image material of leaves, the generated feature image can include the edge feature image and the texture feature image of the leaves.
[0048] Simply put, the edge feature image can be the boundary line of the object in the image, and the texture feature image can be the arrangement attribute of the surface structure organization of the object in the area formed between the boundary lines in the image. In the embodiment of the present application, the edge feature image and the texture feature image can exist in the form of an edge feature image and a texture feature image, and a complete image material can be obtained by combining an edge feature image and a corresponding texture feature image.
[0049] The first data set and the second data set may respectively include one or more mathematical models. A mathematical model is a scientific or engineering model constructed using mathematical logic methods and mathematical language. A mathematical model is a mathematical structure that is generally or approximately expressed using mathematical language with reference to the characteristics or quantitative dependencies of a certain system of things. This mathematical structure is a pure relational structure of a certain system depicted with the help of mathematical symbols. A mathematical model can be one or a group of algebraic equations, differential equations, difference equations, integral equations or statistical equations and their combinations, which quantitatively or qualitatively describe the mutual relationship or causal relationship between the variables of the system. In addition to mathematical models described by equations, there are also models described by other mathematical tools, such as algebra, geometry, topology, mathematical logic, etc. Mathematical models describe the behavior and characteristics of the system rather than the actual structure of the system.
[0050] The first data set is used to generate the feature image according to the noise data. The second data set is used to generate the target image according to the feature image. In practical applications, the machine learning method used by the first data set may include a generative adversarial network model, and the machine learning method used by the second data set may include a convolutional neural network (CNN), etc.
[0051] Reference Figure 1 , which shows a system architecture diagram of an image generation method provided by an embodiment of the present application, wherein the first data set may include a generative adversarial network model, and the second data set may include a convolutional neural network model. The generative adversarial network model is mainly used to generate a feature image based on noise data, and the convolutional neural network model is mainly used to generate a target image based on the feature image.
[0052] Specifically, the generative adversarial network model may include a generator (the first sub-data set in this application) and a discriminator (the second sub-data set in this application). The generative adversarial network model is a deep learning model. The generative adversarial network model produces preferred outputs through the mutual game learning of (at least) two modules in its framework: the generator and the discriminator.
[0053] First, a sample feature image set can be determined so that the discriminator can call the sample feature images in the sample feature image set. After that, the acquired noise data can be input into the generator in the form of an image. The generator can generate a candidate image through the noise data. Due to the limitation of the generator's own parameters, the candidate image has two situations: "true" or "false". The "true" situation means that the candidate image is highly similar to the sample feature image, and the "false" situation means that the candidate image is highly dissimilar to the sample feature image. In order to ultimately generate better quality image materials, a better generator should continuously optimize its own parameters so that the probability of generating a "true" feature image is greater than the first set threshold.
[0054] In an embodiment of the present application, the discriminator can be used to determine whether the candidate image output by the generator is "true" or "false". Specifically, the candidate image output by the generator can be matched with the sample feature image to determine whether the candidate image and the sample feature image are highly similar. If they are highly similar, the candidate image is determined to be "true" and is a feature image; if they are highly dissimilar, the candidate image is determined to be "false" and is not a feature image. In practical applications, the sample feature image and the candidate image can be sampled respectively, and then input into the discriminator. The discriminator determines whether the candidate image is true based on the two sampling results. After multiple rounds of iterative training, the generator can deceive the discriminator to achieve the purpose of improving the output quality of the generator. At the same time, the discriminator will gradually improve its own performance in multiple rounds of iterative training to achieve the purpose of accurately judging the "true" or "false" of the candidate image, so that the probability that the discriminator judges that the candidate image is indeed a true feature image or a false feature image is greater than the second set threshold.
[0055] Furthermore, the feature image determined by the discriminator can be further input into the second data set, and the convolutional neural network model included in the second data set maps the feature image into a complete target image. The generation process is similar to combining an edge feature image and a texture feature image in the feature image to obtain a complete image material, such as Figure 1 The target image in .
[0056] For example, refer to Figure 2 , which shows an actual scene diagram of an image generation method provided by an embodiment of the present application. If the target image 30 of the leaf type is required to be generated in the end, some sample feature image sets including edge feature images 10 and texture feature images 20 of the leaf can be collected. In the edge feature image 10, the boundary of the leaf divides the leaf into multiple edge regions 11. In the texture feature image 20, there is an arrangement 21 of the surface structure organization of the leaf in each edge region 11. After the noise data of the random Gaussian distribution is input into the first data set, the generator or the noise data is generated as the initial feature image, and the discriminator matches these initial feature images with the edge feature image 10 and the texture feature image 20 of the leaf in the sample feature image set respectively. If there is any matching situation, the candidate image is determined to be "true", and finally the "true" feature image is input into the second data set. After the convolutional neural network model in the second data set combines the edge feature image 10 of the leaf and the texture feature image 20 of the leaf included in the "true" feature image, the target image 30 of the leaf is generated.
[0057] In an embodiment of the present application, noise data can be directly generated into a feature image through the first data set, and because the picture composition of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image. This makes the process of generating feature images by the generative adversarial network model more concise, and reduces the difficulty of optimizing relevant parameters in the generative adversarial network model, so that the probability of the first data set generating a feature image with a relatively stable picture is greatly improved. Furthermore, the embodiment of the present application reuses the convolutional neural network model to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the relatively stable feature image is input into another convolutional neural network model to construct the target image. The construction process is relatively simple, which reduces the probability of color distortion in the target image and improves the picture quality of the target image.
[0058] Reference Figure 3 , shows a flow chart of an image generation method according to an embodiment of the present application, the method may specifically include the following steps:
[0059] Step 101: Obtain noise data.
[0060] In an embodiment of the present application, the noise data may be Gaussian noise, which refers to a type of noise whose probability density function obeys Gaussian distribution (i.e., normal distribution). In a specific implementation of an embodiment of the present application, a randn (random normal distribution) function may be used to obtain Gaussian noise. The randn function is a function that generates a random number or matrix of a standard normal distribution, and belongs to a Matrix Laboratory (MATLAB, Matrix Laboratory) function. The operation of obtaining random Gaussian noise data may be implemented through the open source randn function interface provided by MATLAB. Of course, the embodiment of the present application may also use noise data of other digital images, and the embodiment of the present invention does not limit it.
[0061] It should be noted that in the embodiment of the present application, in order to facilitate the processing of noise data in subsequent steps, the noise data can be converted into a noise image. Specifically, the output pixels are obtained by adding the default input pixels to the normally distributed random array in the noise data, and then the output pixels are filled in a picture of a preset size to obtain a noise image of a preset size.
[0062] Step 102: Obtain a characteristic image according to the noise data and a first data set; wherein the first data set is used to generate the characteristic image according to the noise data.
[0063] In an embodiment of the present application, a first data set and a second data set can be pre-set; the first data set may include a generative adversarial network model, which is mainly used to generate feature images based on noise data. Specifically, the generative adversarial network model may include a generator and a discriminator. The generative adversarial network model is a deep learning model. The generative adversarial network model generates a preferred output through the mutual game learning of (at least) two modules in its framework: the generator and the discriminator, so that the noise data is generated into a feature image.
[0064] Specifically, first, a sample feature image set can be determined so that the discriminator can call the sample feature images in the sample feature image set. Afterwards, the acquired normally distributed noise data can be input into the generator in the form of an image, and the generator can generate a candidate image through the noise data. In the embodiment of the present application, the discriminator can be used to determine whether the candidate image output by the generator is "true" or "false", specifically, the candidate image output by the generator is matched with the sample feature image to determine whether the candidate image and the sample feature image are highly similar. If they are highly similar, the candidate image is determined to be "true" and is a feature image; if they are highly dissimilar, the candidate image is determined to be "false", and the candidate image is not a feature image. In the embodiment of the present application, the noise data is directly used to generate a feature image with less image details, rather than generating a complete image with more image details from the noise data, which can simplify the process of generating feature images from the first data set and reduce the difficulty of optimizing the relevant parameters in the first data set, so that the first data set can generate a feature image with a relatively stable picture.
[0065] Step 103, obtaining a target image according to the feature image and a second data set; wherein the second data set is used to generate the target image according to the feature image.
[0066] In this step, the feature image determined as "true" by the discriminator can be further input into the second data set, and the convolutional neural network model included in the second data set generates the feature image into a complete target image. The generation process is similar to combining an edge feature image and a texture feature image in the feature image to obtain a complete target image. Since the picture of the feature image is relatively stable, the probability of color distortion in the target image is reduced, and the picture quality of the target image is improved.
[0067] For example, refer to Figure 4, shows a specific example diagram of an image generation method according to the present application, wherein four edge feature images a1, b1, c1, d1 with different edges and four texture feature images a2, b2, c2, d2 with different textures corresponding to the four edge feature images are obtained according to the first data set, and then a1-a2, b1-b2, c1-c2, d1-d2 are input into the second data set to obtain four corresponding different target images a3, b3, c3, d3.
[0068] In summary, an image generation method provided by an embodiment of the present application includes: obtaining noise data; obtaining a feature image according to the noise data and a first data set; wherein the first data set is used to generate a feature image according to the noise data; obtaining a target image according to the feature image and a second data set; wherein the second data set is used to generate a target image according to the feature image. The noise data can be directly generated into a feature image through the first data set, and since the picture composition of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image, so that the process of generating the feature image from the first data set is more concise, and the difficulty of optimizing the relevant parameters in the first data set is also reduced, so that the probability of generating a feature image with a relatively stable picture from the first data set is greatly improved. Further, the embodiment of the present application reuses the second data set to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the relatively stable feature image is input into another second data set to construct the target image, and the construction process is relatively simple, so that the probability of color distortion in the target image is reduced, and the picture quality of the target image is improved.
[0069] Reference Figure 5 , shows a specific flow chart of an image generation method according to an embodiment of the present application, the method may specifically include the following steps:
[0070] Step 201, obtaining a second sample feature image.
[0071] In an embodiment of the present application, before applying the first data set to generate noise data into a feature image, the first data set can be trained using training data to adjust relevant parameters in the first data set to a better level so that the first data set can generate a feature image of better quality.
[0072] Specifically, the training data can use the second sample feature image. Assuming that the target image ultimately required to be generated is an image of leaves, the method for obtaining the second sample feature image can specifically include: collecting a large number of complete leaf images, and extracting edge feature images and texture feature images of these leaf images, and determining these extracted edge feature images and texture feature images as the second sample feature image.
[0073] In practical applications, the second sample feature image may include: an edge feature image and / or a texture feature image.
[0074] The edge feature image can be obtained by edge detection of the complete image, which is also called step detection. For example, if the image to be generated is leaves, edge detection can be performed on images of various leaves to obtain the image.
[0075] Texture feature images can be obtained through structural methods. The structural method is a spatial domain method. Its basic idea is that complex textures can be composed of some simple texture primitives that are repeatedly arranged and combined in a certain regular form. There are two key points in structural description: one is to determine the texture primitives; the other is to establish arrangement rules. A texture primitive is a set of connected pixels characterized by a set of attributes. In order to describe the texture using the structural method, on the basis of obtaining the texture primitives, it is also necessary to establish rules for arranging them in reverse order. The arrangement rules and methods can be defined by formal grammar. Through the above structural method, its texture feature image can be obtained from the above complete image. For example, if the generated image is a leaf, the image of various leaves can be obtained by edge detection.
[0076] It should be noted that if both edge feature images and texture feature images are to be used at the same time, the above extraction process can be performed on a complete image to obtain its edge feature image and texture feature image. In the embodiment of the present application, in this case, the edge feature image and the texture feature image are combined in pairs, and the combination corresponds to the complete image from which the feature is extracted. For example, for a leaf image A, the edge feature image A1 and the texture feature image A2 are extracted respectively according to the above method, then A1 and A2 are combined in pairs, and A1 and A2 correspond to A. Of course, edge feature images and texture feature images can also be generated in other ways, which are not limited by the embodiments of the present invention.
[0077] Step 202: training the first data set according to the second sample feature image.
[0078] In an embodiment of the present application, the first data set may be a generative adversarial network model, including: a first sub-data set (generator) and a second sub-data set (discriminator). The embodiment of the present application may solve the loss function for the first data set through the second sample feature image and the first data set. The loss function is a function that maps the value of a random event or its related random variable to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function to achieve parameter optimization of the model. Using the loss function for the first data set, the parameters of the generator and the discriminator can be optimized to achieve the purpose of training the first data set. It is worth noting that the generative adversarial network model is a system in which the optimization minimum value is not fixed. The training process of the generative adversarial network model does not aim to minimize the loss value, but aims to achieve a dynamic balance between the loss values of the generator and the discriminator by using the loss function.
[0079] Specifically, in one implementation, assuming that the training of the first data set may include:
[0080] In the first step, the discriminator is trained using the second sample feature image labeled with "true" or "false", that is, the second sample feature image is input, and the value of "true" or "false" output by the discriminator is different from the value of "true" or "false" originally labeled by the second sample feature image, and the loss function of the discriminator is obtained, and the weights of the relevant parameters in the discriminator are adjusted according to the loss function, so that the value of "true" or "false" output by the discriminator is as close as possible to the value of "true" or "false" originally labeled by the second sample feature image. For example, if a second sample feature image labeled as "true" is input into the discriminator, but the output result is "false", the weight parameter of the function that is judged as "false" in the discriminator can be correspondingly reduced; if a second sample feature image labeled as "false" is input into the discriminator, but the output result is "true", the weight parameter of the function that is judged as "true" in the discriminator can be correspondingly reduced. In addition, the training process can go through multiple rounds of iterations, and each round of iteration involves calculating the loss function and adjusting the relevant parameters of the discriminator according to the loss function until the relevant parameters of the discriminator are adjusted to the optimal level.
[0081] The second step is to train the generator using the second sample feature image, such as inputting random Gaussian distributed noise data into the generator to obtain the initial image output by the generator, calculating the similarity between the initial image and the second sample feature image, obtaining the loss value between the two, and constructing the loss function corresponding to the generator based on the loss value, and using the loss function to adjust the relevant weight parameters of the generator so that the loss value generated by the subsequent iterative process is smaller. In addition, the training process can go through multiple rounds of iterations, and each round of iteration goes through the process of calculating the loss value and adjusting the relevant weight parameters of the generator according to the loss function until the relevant weight parameters of the generator are adjusted to the optimal value. Of course, in the actual training process, the above noise data can be extracted from the second sample feature image, and then the noise data can be used as the input feature of the generator and the output feature of the corresponding second sample feature, and then the above training process is carried out.
[0082] The third step is to fit the loss function of the generator and the loss function of the discriminator into a target loss function, and use the target loss function to dynamically adjust the parameters of the generator and the discriminator, and observe in real time the degree of fit between the actual output distribution of the generator and the discriminator and the preset sample distribution. The higher the degree of fit, the better the training effect of the first data set. Therefore, the parameters of the generator and the discriminator with the highest degree of fit can be determined as the final parameters of the generator and the discriminator to complete the training of the first data set.
[0083] Step 203: Acquire a sample target image corresponding to the second sample feature image.
[0084] In the embodiment of the present application, the second sample feature image can be obtained by extracting the feature image from the sample target image. For example, if the second sample feature image is the edge feature image and the texture feature image corresponding to the leaf image, the sample target image can be the leaf image.
[0085] In practical applications, for example, the second data set is a convolutional neural network model, and input images and output images are required for training. If the sample feature image is extracted from the sample target image, and the sample target image can be an existing image material, then the sample feature image can be used as the input of the convolutional neural network model, and the sample target image can be used as the output of the convolutional neural network model for training. For example, the aforementioned (A1, A2) is input into the convolutional neural network model, the image output by the convolutional neural network model is compared with A, the loss function is calculated, and then the parameters of the convolutional neural network model are adjusted.
[0086] Step 204: training the second data set according to the second sample feature image and the sample target image.
[0087] In an embodiment of the present application, the second data set may be a convolutional neural network model. Using the second sample feature image and the sample target image, training the second data set may specifically include:
[0088] The first step is to input the second sample feature image into the second data set to obtain an initial complete image output by the second data set.
[0089] The second step is to calculate the difference between the initial complete image and the second sample feature image, and obtain a loss function based on the difference according to the difference.
[0090] The third step is to adjust the parameters of the second data set by using the loss function through multiple rounds of iterations, so that the difference between the initial complete image output by the second data set and the second sample feature image is as small as possible. Specifically, there are many ways to adjust the parameters of the second data set through the loss function. In one implementation, the parameters of the second data set can be adjusted using a gradient back propagation algorithm based on the loss function.
[0091] It should be noted that steps 201-204 are the presetting process of the first data set and the second data set, which can be performed in advance, and the training of the first data set and the second data set can be in any order.
[0092] Step 205: Acquire noise data.
[0093] This step may specifically refer to the above step 101, which will not be described in detail here.
[0094] Step 206: Obtain a feature image according to the noise data and a first data set; wherein the first data set is used to generate the feature image according to the noise data.
[0095] Optionally, the feature image includes at least one of a texture feature image and an edge feature image.
[0096] In the embodiment of the present application, the edge feature image may be the boundary line of an object in the image, and the texture feature image may be the color filled in the area formed between the boundary lines in the image. In the embodiment of the present application, the edge feature image and the texture feature image may exist in the form of an edge feature image and a texture feature image, and a complete image material may be obtained by combining an edge feature image and a corresponding texture feature image.
[0097] In general, the feature image needs to include both the texture feature image and the edge feature image to achieve the purpose of combining the edge feature image with a corresponding texture feature image to obtain a complete image material. In addition, in some other cases, the feature image may also include only the texture feature image or only the edge feature image. For example, when the second data set includes a classifier for identifying the classification of an image, the texture feature image or the edge feature image can be input into the second data set, and the classification category of the texture feature image or the edge feature image can be output.
[0098] Optionally, in a specific implementation of the embodiment of the present application, the first data set includes: a first sub-data set and a second sub-data set, wherein the first sub-data set is used to generate a candidate image according to the noise data, and the second sub-data set is used to identify the feature image from the candidate image; the step 206 may specifically include:
[0099] Sub-step 2061: Generate the candidate image according to the noise data and the first sub-data set.
[0100] In an embodiment of the present application, if the first sub-data set (generator) has been trained, after the noise data is input into the first sub-data set, the first sub-data set will output candidate images, which are likely to include a large number of feature images.
[0101] Sub-step 2062: determine the feature image in the candidate image through the second sub-data set.
[0102] In this step, due to the limitation of the training process of the first sub-data set, the effect of the first sub-data set outputting 100% of the feature images cannot be achieved, so it is necessary to screen out non-feature images in the candidate images. In the embodiment of the present application, the candidate images can be input into the second sub-data set (discriminator), and the discriminator can screen out non-feature images in the candidate images.
[0103] Optionally, in a specific implementation of an embodiment of the present application, the second sub-data set includes: a first sample feature image; the sub-step 2062 can be specifically implemented by determining, based on the second sub-data set, a candidate image whose similarity with the first sample feature image is greater than or equal to a preset similarity threshold as the feature image.
[0104] In this step, the first sample feature image may be completely identical to, partially identical to, or completely different from the second sample feature image. The first sample feature image and the second sample feature image may belong to the same image category, and this application does not impose any limitation on this.
[0105] Specifically, the second sub-data set can call the first sample feature image, and calculate the similarity between the first sample feature image and the input feature image, and determine the candidate image whose similarity with the first sample feature image is greater than or equal to a preset similarity threshold as a real image that is highly similar to the first sample feature image, that is, determined as a feature image.
[0106] Step 207, obtaining a target image according to the feature image and the second data set; wherein the second data set is used to generate the target image according to the feature image.
[0107] This step may specifically refer to the above step 103, which will not be described in detail here.
[0108] In summary, an image generation method provided by an embodiment of the present application includes: obtaining noise data; obtaining a feature image according to the noise data and a first data set; wherein the first data set is used to generate a feature image according to the noise data; obtaining a target image according to the feature image and a second data set; wherein the second data set is used to generate a target image according to the feature image. The noise data can be directly generated into a feature image through the first data set, and because the picture composition of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image, which makes the process of generating the feature image from the first data set more concise, and also reduces the difficulty of optimizing the relevant parameters in the first data set, so that the probability of generating a feature image with a relatively stable picture from the first data set is greatly improved. Furthermore, the embodiment of the present application reuses the second data set to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the probability of color distortion in the target image is reduced, and the picture quality of the target image is improved.
[0109] Reference Figure 6 , shows a specific flow chart of another image generation method embodiment according to an embodiment of the present application, the method may specifically include the following steps:
[0110] Step 301, obtaining noise data.
[0111] This step may refer to the above step 101 and will not be described again here.
[0112] Step 302: obtaining a feature image corresponding to a target category according to the noise data and the first data set.
[0113] In this step, the target category may be determined. Specifically, there may be multiple ways to determine the target category.
[0114] Among them, the target category can be a category of a certain object that a specific user wants. For example, if the user wants an image of leaves, leaves can be the target category, and if the user wants an image of fish, fish can be the target category. The specific target category can be selected according to user needs, and the embodiments of the present application do not limit it. When training the first data set and the second data set, the second sample feature image of the corresponding target category and the sample target image can be used to perform the above training process. For example, for the leaf category, by collecting a large number of leaf images, and constructing the feature images extracted from the leaf images as leaf sample feature images, the leaf sample feature images are used to train the first data set, so that when the noise data is input into the first data set, the leaf feature image corresponding to the leaf target category can be obtained. Similarly, the corresponding first data set and second data set can also be trained for categories such as fish. In practical applications, different target categories can correspond to different first data sets and second data sets.
[0115] For example, in one implementation, referring to Figure 7 , shows a specific application scenario diagram of another image generation method provided by the present application, which provides two target category data sets, namely, the first data set of target category A, the second data set of target category A, and the first data set of target category B, the second data set of target category B. Among them, the target category can be specified according to demand. Assuming that the user determines the target category A on the client, for example, the user selects the target category A in the menu of the terminal page, or enters the target category A in the terminal page, and then sends an image generation request to the server, then after the server receives the request, it can obtain the target category A from the request, and can select the data set corresponding to the target category A, such as the first data set of target category A, the second data set of target category A. Then input the noise data to the first data set of target category A to obtain the feature image of target category A. Assuming that the service provider provides the first data set and the second data set of the leaf category, then
[0116] In another implementation, the target category may not be set, that is, by collecting random images, and constructing the feature images extracted from the random images as second sample feature images, and using the second sample feature images to train the first data set, when the noise data is input into the first data set, a random feature image can be obtained.
[0117] Step 303: Obtain a target image corresponding to the target category according to the feature image and the second data set.
[0118] In this step, refer to Figure 7, the feature image of target category A is input into the second data set of target category A to perform the aforementioned target image generation process to obtain the target image of target category A.
[0119] In another implementation, the target category may not be set, that is, when noise data is input into the first data set, a random feature image may be obtained, and then the random feature image is input into the second data structure to obtain a random target image.
[0120] In summary, the embodiment of the present application can directly generate noise data into a feature image through the first data set, and because the picture composition of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image, which makes the process of generating the feature image from the first data set more concise, and also reduces the difficulty of optimizing the relevant parameters in the first data set, so that the probability of the first data set generating a feature image with a relatively stable picture is greatly improved. Furthermore, the embodiment of the present application reuses the second data set to directly generate the feature image into a complete target image. Because the picture of the feature image is relatively stable, the probability of color distortion in the target image is reduced, and the picture quality of the target image is improved. In addition, target images of corresponding categories can be generated for different target categories, and the generated images are more accurate.
[0121] Reference Figure 8 , shows a flow chart of an image generation method according to an embodiment of the present application, the method may specifically include the following steps:
[0122] Step 401, obtaining noise data.
[0123] This step may specifically refer to the above step 101, which will not be described in detail here.
[0124] Step 402: Generate a feature image based on the first neural network according to the noise data, wherein the feature image includes a texture feature image or an edge feature image.
[0125] Neural Networks (NNs) can also be called connection models. Neural networks are an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This type of network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.
[0126] In an embodiment of the present application, the first neural network may include a generative adversarial network model, which is mainly used to generate a feature image based on noise data. Specifically, the generative adversarial network model may include a generator and a discriminator. The generative adversarial network model is a deep learning model. The generative adversarial network model generates a preferred output through the mutual game learning of (at least) two modules in its framework: a generator and a discriminator, so that the noise data is generated as a feature image.
[0127] Specifically, first, a set of sample feature images can be determined so that the discriminator can call the sample feature images in the set of sample feature images. After that, the noise data obtained, such as the normal distribution, can be input into the generator in the form of an image, and the generator can generate a candidate image through the noise data. In the embodiment of the present application, the discriminator can be used to determine whether the candidate image output by the generator is "true" or "false", specifically, the candidate image output by the generator is matched with the sample feature image to determine whether the candidate image and the sample feature image are highly similar. If they are highly similar, the candidate image is determined to be "true" and is a feature image; if they are highly dissimilar, the candidate image is determined to be "false", and the candidate image is not a feature image. In the embodiment of the present application, the noise data is directly used to generate a feature image with less image details, rather than generating a complete image with more image details from the noise data, which can simplify the process of generating feature images by the first neural network and reduce the difficulty of optimizing the relevant parameters in the first neural network, so that the first neural network can generate feature images with relatively stable pictures. Among them, the feature image is at least one of the texture feature image and edge feature image described in the aforementioned embodiment.
[0128] Step 403: Generate a target image based on the feature image and a second neural network.
[0129] Specifically, the second neural network can be a convolutional neural network. The convolutional neural network includes a feature extractor composed of a convolutional layer and a subsampling layer. In the convolutional layer of the convolutional neural network, a neuron is only connected to some neurons in the adjacent layers. In a convolutional layer of the convolutional neural network, it usually contains several feature planes (featureMaps). Each feature plane is composed of some neurons arranged in a rectangular shape. The neurons in the same feature plane share weights, and the shared weights here are convolution kernels. The convolution kernel can be initialized in the form of a random decimal matrix. During the training process of the network, the convolution kernel will learn to obtain the weights that meet the requirements. The direct benefit of sharing weights (convolution kernels) is to reduce the connection between the layers of the network and reduce the risk of overfitting. Subsampling is also called pooling, for example, there are two forms of mean subsampling (meanpooling) and maximum subsampling (max pooling). Subsampling can be regarded as a special convolution process. Convolution and subsampling greatly simplify the complexity of the model, reduce the parameters of the model, and improve the processing ability of the convolutional neural network in the image field.
[0130] In this step, the feature image output by the first neural network can be further input into the second neural network, and the second neural network generates the feature image into a complete target image. The generation process is similar to combining an edge feature image and a texture feature image in the feature image to obtain a complete target image. Since the feature image is relatively stable, the probability of color distortion in the target image is reduced, thereby improving the image quality of the target image.
[0131] In summary, an image generation method provided by an embodiment of the present application includes: obtaining noise data; generating a feature image based on the first neural network according to the noise data, wherein the feature image includes a texture feature image or an edge feature image; generating a target image based on the second neural network according to the feature image. The noise data can be directly generated into a feature image by the first neural network, and because the picture composition of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image, which makes the process of generating the feature image by the first neural network more concise, and also reduces the difficulty of optimizing the relevant parameters in the first neural network, so that the probability of the first neural network generating a feature image with a relatively stable picture is greatly improved. Further, the embodiment of the present application reuses the second neural network to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the relatively stable feature image is input into another second neural network to construct the target image. The construction process is relatively simple, which reduces the probability of color distortion in the target image and improves the picture quality of the target image.
[0132] Reference Fig. 9 , shows a specific flow chart of an image generation method according to an embodiment of the present application, the method may specifically include the following steps:
[0133] Step 501, obtaining noise data.
[0134] This step may specifically refer to the above step 101, which will not be described in detail here.
[0135] Step 502: Generate the candidate image based on the noise data and the first sub-neural network.
[0136] In an embodiment of the present application, if the first sub-neural network (such as a generator) has been trained, after the noise data is input into the first sub-neural network, the first sub-neural network will output a candidate image, which is likely to include a large number of feature images.
[0137] It should be noted that the candidate image may include at least one of an edge feature image and a texture feature image, for example, an edge feature image, or a texture feature image, or both an edge feature image and a texture feature image.
[0138] Optionally, step 502 may specifically include:
[0139] Sub-step 5021, based on the fully connected layer of the first sub-neural network, convert the noise data into a first feature map.
[0140] In the embodiment of the present application, the fully connected layers (FC) play the role of "classifier" in the entire neural network, that is, the fully connected layers play the role of mapping the learned "distributed feature representation" to the sample label space. In actual use, the fully connected layer can be implemented by a convolution operation, and each neuron in the fully connected layer is fully connected to all neurons in the adjacent layer. The fully connected layer can integrate local information with category distinction in the convolution layer or the pooling layer.
[0141] In each convolution layer of the neural network, the data exists in three-dimensional form. The three-dimensional data can be regarded as many two-dimensional images stacked together, each of which can be called a feature map. In the neural network, the feature map can exist in the form of a feature matrix. For example, if the data is a color image, the data can include 3 feature maps, namely, a feature map reflecting the red channel, a feature map reflecting the green channel, and a feature map reflecting the blue channel. There will be several convolution kernels between the convolution layers of the neural network. Each feature map output by the previous convolution layer is convolved with each convolution kernel of the current convolution layer, and a feature map for input to the next convolution layer will be generated.
[0142] The first sub-neural network can be a generator in a generative adversarial network. The purpose of the generator is to process the aforementioned noise data into a feature image through a convolution operation, so the generator needs to combine more features into a feature image.
[0143] Therefore, in this step, the noise data can first be converted into a first feature map through the fully connected layer of the first sub-neural network, thereby obtaining a preliminary feature map.
[0144] It should be noted that, when edge and texture features are used simultaneously, the first feature map may include two feature maps, that is, the first feature map may include a first feature map corresponding to an edge feature image, and a first feature map corresponding to a texture feature image.
[0145] Sub-step 5022: Based on the convolution layer of the first sub-neural network, perform a first convolution process on the first feature map to obtain the candidate image.
[0146] In this step, multiple convolution layers can be preset in the first sub-neural network, and each convolution layer can perform a first convolution operation once. Based on the multiple convolution layers of the first sub-neural network, the first feature map can be subjected to a preset number of first convolution processing to realize multiple rounds of iterative convolution of the first feature map. The size of the first feature map can be increased through the first convolution processing. The purpose is to extract features from the first feature map corresponding to the noise data through the first convolution processing, and combine these features to obtain a first feature map with an increased size. The first feature map with an increased size contains more features than the original first feature map. Until the size of the first feature map reaches a first preset size threshold after a preset number of first convolution processing, the first feature map is determined as the candidate image.
[0147] Specifically, the preset number can be set to be no less than 10 times, that is, the number of convolutional layers is no less than 10 layers. The larger the preset number, the higher the accuracy of the candidate image, but the more time and computing resources consumed. Therefore, the preset number can be controlled according to actual needs to achieve the target effect. In addition, the feature map output by the next convolutional layer is increased in size by multiples compared to the feature map output by the previous convolutional layer.
[0148] It should be noted that the candidate image may include two images, that is, the candidate image may include a candidate image corresponding to an edge feature image and a candidate image corresponding to a texture feature image.
[0149] Specifically, sub-step 5022 may include:
[0150] Sub-step 50221: In the first sub-neural network, a first convolution process is performed on the first feature map to obtain the first candidate feature map with an increased size.
[0151] In this step, the first feature map output by the fully connected layer can be further input into the convolutional layer located at the next layer of the fully connected layer. In the convolutional layer, the first feature map can be subjected to a first convolution process to obtain a first candidate feature map with an increased size.
[0152] Specifically, each convolutional layer in the first sub-neural network is composed of several convolutional units, and the parameters of each convolutional unit are optimized by the back propagation algorithm. The purpose of the convolution operation in the convolutional layer is to extract different features of the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features. In this step, the size of the first feature map is increased by the first convolution process. The purpose is to extract features from the first feature map corresponding to the noise data through the first convolution process, and combine these features to obtain a first candidate feature map with an increased size. The first candidate feature map with an increased size contains more features than the original first feature map.
[0153] It should be noted that before performing the first convolution processing on the first feature map, the first feature map may also be subjected to batch normalization (BN) processing so that the mean of the first feature map is 0 and the variance is 1. The idea of batch normalization processing is to normalize the input of the current layer so that their mean is 0 and the variance is 1. The benefit of batch normalization processing is that it can accelerate convergence, and the convolutional neural network with batch normalization processing is very little affected by weight initialization, has very good stability, and has a good effect on improving convolution performance.
[0154] In addition, after the convolution layer performs the first convolution processing on the first feature map, the first candidate feature map after convolution can also be activated by a preset activation function, thereby increasing the number of activated neurons in the first sub-neural network. For example, the linear rectifier function (Rectified Linear Unit, ReLU) can be used as the activation function. Compared with the traditional neural network activation function, the ReLU function has the following advantages: 1. Biological principle: In a neural network using the ReLU function, about 50% of the neurons are in an activated state. 2. More efficient gradient descent and back propagation: avoids the problems of gradient explosion and gradient disappearance. 3. Simplified calculation process: There is no influence of other complex activation functions such as exponential functions; at the same time, the dispersion of activity reduces the overall calculation cost of the neural network.
[0155] Sub-step 50222, performs a first convolution process on the first candidate feature map for a preset number of times to obtain the first candidate feature map with an increased size, until the size of the first candidate feature map reaches a first preset size threshold, and the first candidate feature map is determined as the candidate image.
[0156] In this step, the first candidate feature map with an increased size obtained in sub-step 50221 can be further input into the next convolution layer, and the next convolution layer further increases the size of the first candidate feature map through the first convolution process, the purpose of which is to extract more features from the first candidate feature map through the first convolution process, and combine more features to obtain a first candidate feature map with a further increased size, and the first candidate feature map with a further increased size contains more features. And the first candidate feature map with a further increased size can be further input into the next convolution layer for the first convolution process, until after multiple rounds of iterative convolution, when the size of the first candidate feature map reaches the first preset size threshold, the first candidate feature map is determined as the candidate image.
[0157] The number of iterations of the multi-round iterative convolution can be set to no less than 10 times, for example. The greater the number of iterations, the higher the accuracy of the candidate image, but the more time and computing resources consumed. Therefore, the number of iterations can be controlled according to actual needs to achieve the target effect. In addition, the size of the feature map output by the next convolution layer is increased by multiples compared to the feature map output by the previous convolution layer.
[0158] It should be noted that, in this step, before performing the first convolution processing on the first candidate feature map, the first candidate feature map may also be batch normalized so that the mean of the first candidate feature map is 0 and the variance is 1. In addition, after the convolution layer performs the first convolution processing on the first candidate feature map, the convolved first candidate feature map may also be activated by a preset activation function, thereby increasing the number of activated neurons in the first sub-neural network.
[0159] Step 503: determine the feature image in the candidate image according to the second sub-neural network.
[0160] In this step, due to the limitation of the training process of the first sub-neural network, the effect of the first sub-neural network outputting 100% of the feature images cannot be achieved, so it is necessary to screen out non-feature images in the candidate images. In the embodiment of the present application, the candidate images can be input into the second sub-neural network (discriminator), and the discriminator screens out non-feature images in the candidate images.
[0161] It should be noted that, in the embodiment of the present application, the feature image may include two images, that is, the feature image may include a feature image corresponding to the edge feature image, and a feature image corresponding to the texture feature image.
[0162] Optionally, the second sub-neural network includes: a sample feature image; step 503 may specifically include:
[0163] Sub-step 5031, based on the fully connected layer of the second sub-neural network, convert the candidate image into a second feature map.
[0164] In the embodiment of the present application, fully connected layers (FC) play the role of "classifier" in the entire neural network. In actual use, fully connected layers can be implemented by convolution operations, and each neuron in the fully connected layer is fully connected to all neurons in the adjacent layer. The fully connected layer can integrate local information with category distinction in the convolution layer or the pooling layer.
[0165] The second sub-neural network can be a discriminator in a generative adversarial network. The purpose of the discriminator is to filter out feature images from the candidate images output by the generator, so the discriminator needs to obtain the target feature of the candidate image through a convolution operation. The target feature can accurately reflect the image properties of the candidate image. The discriminator can calculate the similarity between the target feature and the sample feature image, so as to determine the candidate image corresponding to the target feature whose similarity with the sample feature image is greater than or equal to the preset similarity threshold as the feature image according to the size of the similarity.
[0166] Therefore, in this step, the noise data can first be converted into a second feature map through the fully connected layer of the second sub-neural network, thereby obtaining a preliminary feature map.
[0167] It should be noted that the second feature map may include two feature maps, that is, the second feature map may include a second feature map corresponding to an edge feature image, and a second feature map corresponding to a texture feature image.
[0168] Sub-step 5032: Based on the convolution layer of the second sub-neural network, perform a second convolution process on the second feature map to obtain a second candidate feature map.
[0169] In this step, multiple convolution layers can be preset in the second sub-neural network, and each convolution layer can perform a second convolution operation once. Based on the multiple convolution layers of the second sub-neural network, the second feature map can be subjected to a preset number of second convolution processing to achieve multiple rounds of iterative convolution of the second feature map. The size of the second feature map can be reduced through the second convolution processing to obtain a second candidate feature map. The purpose is to extract the target features that can accurately reflect the candidate image from the second feature map through the second convolution processing, and filter out non-target features other than the target features. The reduced-size second candidate feature map contains fewer but more accurate features than the original second feature map.
[0170] The preset number of times can be set to be no less than 10 times. The larger the preset number of times, the higher the accuracy of the target feature, but the more time and computing resources consumed. Therefore, the preset number of times can be controlled according to actual needs to achieve the target effect. In addition, the feature map output by the next convolution layer is reduced in size by multiples compared to the feature map output by the previous convolution layer.
[0171] Optionally, sub-step 5032 may specifically include:
[0172] Sub-step 50321: In the second sub-neural network, convolution processing is performed on the second feature map to obtain the second candidate feature map with reduced size.
[0173] In this step, the second feature map output by the fully connected layer can be further input into the convolutional layer located at the next layer of the fully connected layer. In the convolutional layer, the second feature map can be subjected to a second convolution process to obtain a second candidate feature map with a reduced size.
[0174] Specifically, the size of the second feature map is reduced through the convolution operation. The purpose is to extract the target feature from the second feature map through the second convolution processing, and filter out non-target features other than the target feature. Because irrelevant non-target features are filtered out, a second candidate feature map with reduced size is obtained. The second candidate feature map with reduced size contains fewer but more accurate features than the original second feature map.
[0175] It should be noted that before performing the second convolution processing on the second feature map, the second feature map can also be batch normalized so that the mean of the first feature map is 0 and the variance is 1. In addition, after the convolution layer performs the second convolution processing on the second feature map, the convolved second candidate feature map can also be activated by a preset activation function, thereby increasing the number of activated neurons in the second sub-neural network.
[0176] Sub-step 50322, performs a second convolution process on the second candidate feature map for a preset number of times to obtain the second candidate feature map with reduced size, until the size of the second candidate feature map reaches a second preset size threshold, and then calculates the similarity between the second candidate feature map and the sample feature image.
[0177] The second preset size threshold is smaller than the first preset size threshold.
[0178] In this step, the second candidate feature map with a reduced size obtained in sub-step 50321 can be further input into the next convolution layer, and the next convolution layer further reduces the size of the second candidate feature map through convolution operation. The purpose is to filter out more irrelevant non-target features from the second candidate feature map through the second convolution process, extract fewer but more accurate target features, and obtain a second candidate feature map with a further reduced size. The second candidate feature map with a further reduced size contains fewer features than the second candidate feature map obtained in sub-step 50321. And the second candidate feature map with a further reduced size can be further input into the next convolution layer for convolution operation until the size of the second candidate feature map reaches a smaller second preset size threshold after multiple rounds of iterative convolution. When the size of the second candidate feature map reaches the smaller second preset size threshold, the irrelevant non-target features in the second candidate feature map are almost completely filtered out. Therefore, when calculating the similarity between the second candidate feature map and the sample feature image, the interference of irrelevant non-target features can be reduced, and the calculation accuracy can be improved.
[0179] The number of iterations of multiple rounds of iterative convolution is generally not less than 10. The greater the number of iterations, the higher the accuracy of the target features, but the more time and computing resources consumed. Therefore, the number of iterations can be controlled according to actual needs to achieve the target effect. In addition, the feature map output by the next convolution layer is reduced in size by multiples compared to the feature map output by the previous convolution layer.
[0180] It should be noted that, in this step, before performing the second convolution processing on the second candidate feature map, the second candidate feature map may also be batch normalized so that the mean of the second candidate feature map is 0 and the variance is 1. In addition, after the convolution layer performs the second convolution processing on the second candidate feature map, the convolved second candidate feature map may also be activated by a preset activation function, thereby increasing the number of activated neurons in the first sub-neural network.
[0181] Sub-step 5033, determining the similarity between the second candidate feature map and the sample feature image.
[0182] In this step, when the size of the second candidate feature map reaches a smaller second preset size threshold, irrelevant non-target features in the second candidate feature map are almost completely screened out. Therefore, when calculating the similarity between the second candidate feature map and the sample feature image, the interference of irrelevant non-target features can be reduced and the calculation accuracy can be improved.
[0183] Sub-step 5034, determining the candidate image corresponding to the second candidate feature image whose similarity with the sample feature image is greater than or equal to a preset similarity threshold as the feature image.
[0184] In this step, the second candidate feature map may be completely identical to the sample feature image, may be partially identical to the sample feature image, or may be completely different from the sample feature image. The second candidate feature map and the sample feature image may belong to the same image category, and this application does not impose any limitation on this.
[0185] Specifically, the second sub-neural network can call the sample feature image, and calculate the similarity between the sample feature image and the input second candidate feature map, and determine the candidate image corresponding to the second candidate feature map whose similarity with the sample feature image is greater than or equal to a preset similarity threshold as a real image that is highly similar to the sample feature image, that is, determined as a feature image.
[0186] Step 504: Generate a target image based on the feature image and a second neural network.
[0187] This step may be specifically referred to as step 403 above, which will not be described in detail here.
[0188] Optionally, step 504 may specifically include:
[0189] Sub-step 5041, based on the fully connected layer of the second neural network, converting the feature image into a third feature map.
[0190] In the embodiment of the present application, the fully connected layer plays the role of a "classifier" in the entire neural network, that is, the fully connected layer plays the role of mapping the learned "distributed feature representation" to the sample label space. In actual use, the fully connected layer can be implemented by a convolution operation, and each neuron in the fully connected layer is fully connected to all neurons in the adjacent layer. The fully connected layer can integrate local information with category distinction in the convolution layer or the pooling layer.
[0191] The second neural network can be a convolutional neural network. The purpose of the convolutional neural network is to process the feature image into a target image through a convolution operation, so the generator needs to first extract the precise target features in the feature image, filter out irrelevant non-target features, and then combine the precise target features into a target image.
[0192] Therefore, in this step, the noise data can first be converted into a third feature map through the fully connected layer of the second neural network, thereby obtaining a preliminary feature map.
[0193] It should be noted that the third feature map may include two feature maps, that is, the third feature map may include a third feature map corresponding to an edge feature image, and a third feature map corresponding to a texture feature image.
[0194] Sub-step 5042: Based on the first convolutional layer in the second neural network, perform a third convolution process on the third feature map to obtain a third candidate feature map.
[0195] In this step, the third feature map output by the fully connected layer can be further input into the convolution layer located at the next layer of the fully connected layer. In the convolution layer, the third feature map can be subjected to a third convolution process to obtain a third candidate feature map with a reduced size.
[0196] Specifically, the size of the third feature map is reduced through the convolution operation. The purpose is to extract the target feature from the third feature map through the third convolution processing, and filter out non-target features other than the target feature. Because irrelevant non-target features are filtered out, a third candidate feature map with a reduced size is obtained. The third candidate feature map with a reduced size contains fewer but more accurate features than the original third feature map.
[0197] It should be noted that before performing the third convolution processing on the third feature map, the third feature map may also be batch normalized so that the mean of the third feature map is 0 and the variance is 1. In addition, after the convolution layer performs the third convolution processing on the third feature map, the convolved third candidate feature map may be activated by a preset activation function, thereby increasing the number of activated neurons in the first sub-neural network.
[0198] Furthermore, the size-reduced third candidate feature map can be further input into the next convolution layer, and the size of the third candidate feature map can be further reduced by the next convolution layer through convolution operation. The purpose is to filter out more irrelevant non-target features from the third candidate feature map through the third convolution process, extract fewer but more accurate target features, and obtain a third candidate feature map with a further reduced size. And the size-reduced third candidate feature map can be further input into the next convolution layer for convolution operation, until after multiple rounds of iterative convolution, the size of the third candidate feature map reaches a smaller third preset size threshold, and the third convolution process is stopped.
[0199] The number of iterations of the multi-round iterative convolution can be set to no less than 10 times. The greater the number of iterations, the higher the accuracy of the target features, but the more time and computing resources consumed. Therefore, the number of iterations can be controlled according to actual needs to achieve the target effect. In addition, the feature map output by the next convolution layer is reduced in size by multiples compared to the feature map output by the previous convolution layer.
[0200] Sub-step 5043: Based on the second convolutional layer in the second neural network, perform a fourth convolution process on the third candidate feature map to obtain a target image.
[0201] In this step, since the accurate target features in the feature image have been obtained through the third convolution process and irrelevant non-target features have been screened out, the third candidate feature map whose size reaches the third preset size threshold can be further input into the next convolution layer, and the fourth convolution process is performed in the next convolution layer to obtain a third candidate feature map with an increased size. The purpose is to combine the target features through the fourth convolution process to obtain a third candidate feature map with an increased size.
[0202] It should be noted that before performing the fourth convolution processing on the third candidate feature map, the third candidate feature map may also be batch normalized so that the mean of the third candidate feature map is 0 and the variance is 1. In addition, after the convolution layer performs the fourth convolution processing on the third candidate feature map, the convolved third candidate feature map may be activated by a preset activation function, thereby increasing the number of activated neurons in the first sub-neural network.
[0203] It should be noted that, when the third feature map includes a third feature map corresponding to an edge feature image and a third feature map corresponding to a texture feature image, the third candidate feature map includes a third candidate feature map corresponding to an edge feature image and a third candidate feature map corresponding to a texture feature image.
[0204] In the process of performing the fourth convolution processing on the third candidate feature map, the third candidate feature map corresponding to the edge feature image and the third candidate feature map corresponding to the texture feature image can be merged to obtain the target image.
[0205] Furthermore, the obtained third candidate feature map with increased size is further input into the next convolution layer, and the next convolution layer further increases the size of the third candidate feature map through convolution operation, the purpose of which is to extract more features from the third candidate feature map through the fourth convolution process, and combine more features to obtain a third candidate feature map with further increased size. And the third candidate feature map with further increased size can be further input into the next convolution layer for convolution operation, until after multiple rounds of iterative convolution, the size of the third candidate feature map reaches a larger fourth preset size threshold, and the third candidate feature map with a size reaching the fourth preset size threshold is determined as the target image.
[0206] In the case where the third candidate feature map includes: a third candidate feature map corresponding to an edge feature image, and a third candidate feature map corresponding to a texture feature image, when the sizes of the two third candidate feature maps reach a fourth preset size threshold, the two third candidate feature maps whose sizes reach the fourth preset size threshold can be merged to obtain a target image.
[0207] The number of iterations of the multi-round iterative convolution can be no less than 10. The greater the number of iterations, the higher the accuracy of the target image, but the more time and computing resources consumed. Therefore, the number of iterations can be controlled according to actual needs to achieve the target effect. In addition, the size of the feature map output by the next convolution layer is increased by multiples compared to the feature map output by the previous convolution layer.
[0208] It should be noted that, in this step, before performing the fourth convolution processing on the third candidate feature map, the third feature map may also be batch normalized so that the mean of the third candidate feature map is 0 and the variance is 1. In addition, after the convolution layer performs the fourth convolution processing on the third candidate feature map, the convolved third candidate feature map may be activated by a preset activation function, thereby increasing the number of activated neurons in the first sub-neural network.
[0209] Optionally, the feature image includes: a texture feature image and an edge feature image, and the scheme of sub-steps 5041 to 5043 can also be specifically implemented by the scheme of sub-steps A1 to A7 below.
[0210] Sub-step A1: based on the fully connected layer of the second neural network, convert the texture feature image into a first sub-feature map.
[0211] Sub-step A2: based on the fully connected layer of the second neural network, convert the edge feature image into a second sub-feature map.
[0212] In the case where the feature image includes a texture feature image and an edge feature image, the fully connected layer of the second neural network can convert the texture feature image into a first sub-feature map, and convert the edge feature image into a second sub-feature map.
[0213] Sub-step A3: Based on the first convolutional layer in the second neural network, perform a third convolution process on the first sub-feature map to obtain a first sub-candidate feature map.
[0214] Sub-step A4: Based on the first convolution layer in the second neural network, perform a third convolution process on the second sub-feature map to obtain a second sub-candidate feature map.
[0215] In the case where the feature image includes a texture feature image and an edge feature image, a first sub-feature map corresponding to the texture feature image and a second sub-feature map corresponding to the edge feature image are obtained.
[0216] After obtaining the first sub-feature map and the second sub-feature map, a first sub-candidate feature map after the size of the first sub-feature map is reduced, and a second sub-candidate feature map after the size of the second sub-feature map is reduced is obtained through a third convolution operation. The first sub-candidate feature map includes accurate features that can reflect the texture feature image, and the second sub-candidate feature map includes accurate features that can reflect the edge feature image.
[0217] Sub-step A5, based on the second convolutional layer in the second neural network, according to the fourth convolution operation, merge the first sub-candidate feature map and the second sub-candidate feature map to obtain the target image.
[0218] After obtaining the first sub-candidate feature map and the first sub-candidate feature map, a third sub-candidate feature map after the size of the first sub-candidate feature map is increased, and a fourth sub-candidate feature map after the size of the second sub-candidate feature map is increased is obtained through the fourth convolution operation, and the third sub-candidate feature map is merged with the fourth sub-candidate feature map to generate a target image, that is, the second neural network generates a complete target image through the edge feature image and the texture feature image.
[0219] In summary, an image generation method provided by an embodiment of the present application includes: obtaining noise data; generating a feature image based on the first neural network according to the noise data, wherein the feature image includes a texture feature image or an edge feature image; generating a target image based on the second neural network according to the feature image. The noise data can be directly generated into a feature image by the first neural network, and because the picture composition of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image, which makes the process of generating the feature image by the first neural network more concise, and also reduces the difficulty of optimizing the relevant parameters in the first neural network, so that the probability of the first neural network generating a feature image with a relatively stable picture is greatly improved. Further, the embodiment of the present application reuses the second neural network to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the relatively stable feature image is input into another second neural network to construct the target image. The construction process is relatively simple, which reduces the probability of color distortion in the target image and improves the picture quality of the target image.
[0220] Reference Fig.10 , shows a structural block diagram of an image generating device according to an embodiment of the present application, and the device may specifically include:
[0221] A first acquisition module 601 is used to acquire noise data;
[0222] A first generating module 602 is used to obtain a feature image according to the noise data and a first data set; wherein the first data set is used to generate the target feature image according to the noise data;
[0223] Optionally, the first data set includes: a first sub-data set and a second sub-data set, wherein the first sub-data set is used to generate a candidate image according to the noise data, and the second sub-data set is used to identify the feature image from the candidate image;
[0224] The first generating module 602 includes:
[0225] A generating submodule, configured to generate the candidate image according to the noise data and the first sub-data set;
[0226] A determination submodule is used to determine the feature image in the candidate image through the second sub-data set.
[0227] Optionally, the second sub-data set includes: a first sample feature image; the determination submodule is also used to determine, based on the second sub-data set, a candidate image whose similarity with the first sample feature image is greater than or equal to a preset similarity threshold as the feature image.
[0228] Optionally, the first generating module 602 includes:
[0229] The first category acquisition module is used to obtain a feature image corresponding to a target category according to the noise data and the first data set.
[0230] The second generating module 603 is used to obtain a target image according to the feature image and a second data set; wherein the second data set is used to generate the target image according to the feature image.
[0231] Optionally, the second generating module 603 includes:
[0232] The category determination module is used to obtain a target image corresponding to the target category according to the feature image and the second data set.
[0233] Optionally, the device further comprises:
[0234] A first sample first acquisition module, used for acquiring a second sample feature image;
[0235] The first training module is used to train the first data set according to the second sample feature image.
[0236] A second sample first acquisition module, used to acquire a sample target image corresponding to the second sample feature image;
[0237] The second training module is used to train the second data set according to the second sample feature image and the sample target image.
[0238] Optionally, the feature image includes at least one of a texture feature image and an edge feature image.
[0239] In summary, an image generation device provided by an embodiment of the present application includes: obtaining noise data; obtaining a feature image according to the noise data and a first data set; wherein the first data set is used to generate a feature image according to the noise data; obtaining a target image according to the feature image and a second data set; wherein the second data set is used to generate a target image according to the feature image. The noise data can be directly generated into a feature image through the first data set, and since the picture structure of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image, so that the process of generating the feature image from the first data set is more concise, and the difficulty of optimizing the relevant parameters in the first data set is also reduced, so that the probability of generating a feature image with a relatively stable picture from the first data set is greatly improved. Further, the embodiment of the present application reuses the second data set to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the relatively stable feature image is input into another second data set to construct the target image, and the construction process is relatively simple, so that the probability of color distortion in the target image is reduced, and the picture quality of the target image is improved.
[0240] Reference Fig.11 , shows a structural block diagram of an image generating device according to an embodiment of the present application, and the device may specifically include:
[0241] The second acquisition module 701 is used to acquire noise data;
[0242] A second generating module 702 is used to generate a feature image based on the noise data and the first neural network, wherein the feature image includes a texture feature image or an edge feature image;
[0243] Optionally, the first neural network includes: a first sub-neural network and a second sub-neural network; the second generation module 702 includes:
[0244] A first sub-neural network submodule, configured to generate the candidate image according to the noise data and the first sub-neural network;
[0245] Optionally, the first sub-neural network submodule is further used to convert the noise data into a first feature map based on the fully connected layer of the first sub-neural network;
[0246] Based on the convolution layer of the first sub-neural network, a first convolution process is performed on the first feature map to obtain the candidate image.
[0247] The second sub-neural network submodule is used to determine the feature image in the candidate image according to the second sub-neural network.
[0248] Optionally, the second sub-neural network includes: a sample feature image; a second sub-neural network submodule, further used to convert the candidate image into a second feature map based on a fully connected layer of the second sub-neural network;
[0249] Based on the convolution layer of the second sub-neural network, performing a second convolution process on the second feature map to obtain a second candidate feature map;
[0250] Determining the similarity between the second candidate feature map and the sample feature image;
[0251] The candidate image corresponding to the second candidate feature image whose similarity with the sample feature image is greater than or equal to a preset similarity threshold is determined as the feature image.
[0252] The second generating module 703 is used to generate a target image based on the feature image and a second neural network.
[0253] Optionally, the second generating module 703 is further used to convert the feature image into a third feature map based on the fully connected layer of the second neural network;
[0254] Based on the first convolution layer in the second neural network, performing a third convolution process on the third feature map to obtain a third candidate feature map;
[0255] Based on the second convolutional layer in the second neural network, a fourth convolution process is performed on the third candidate feature map to obtain a target image.
[0256] Optionally, the feature image includes: a texture feature image and an edge feature image, and the second generating module 703 is further used to convert the texture feature image into a first sub-feature image based on the fully connected layer of the second neural network;
[0257] Based on the fully connected layer of the second neural network, converting the edge feature image into a second sub-feature map;
[0258] Based on the first convolution layer in the second neural network, performing a third convolution process on the first sub-feature map to obtain a first sub-candidate feature map;
[0259] Based on the first convolution layer in the second neural network, performing a third convolution process on the second sub-feature map to obtain a second sub-candidate feature map;
[0260] Based on the second convolution layer in the second neural network, and according to the fourth convolution operation, the first sub-candidate feature map and the second sub-candidate feature map are merged to obtain the target image.
[0261] In summary, an image generation device provided by an embodiment of the present application includes: obtaining noise data; obtaining a feature image according to the noise data and a first data set; wherein the first data set is used to generate a feature image according to the noise data; obtaining a target image according to the feature image and a second data set; wherein the second data set is used to generate a target image according to the feature image. The noise data can be directly generated into a feature image through the first data set, and since the picture structure of the feature image is relatively simple, the image details in the feature image are greatly reduced compared to the final target image, so that the process of generating the feature image from the first data set is more concise, and the difficulty of optimizing the relevant parameters in the first data set is also reduced, so that the probability of generating a feature image with a relatively stable picture from the first data set is greatly improved. Further, the embodiment of the present application reuses the second data set to directly generate the feature image into a complete target image. Since the picture of the feature image is relatively stable, the relatively stable feature image is input into another second data set to construct the target image, and the construction process is relatively simple, so that the probability of color distortion in the target image is reduced, and the picture quality of the target image is improved.
[0262] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0263] Embodiments of the present disclosure may be implemented as a system using any suitable hardware, firmware, software, or any combination thereof to perform the desired configuration. Fig.12 An exemplary system (or device) 1600 that can be used to implement various embodiments described in the present disclosure is schematically illustrated.
[0264] For one embodiment, Fig.12 An exemplary system 1600 is shown having one or more processors 1602, a system control module (chip set) 1604 coupled to at least one of the processor(s) 1602, a system memory 1606 coupled to the system control module 1604, a non-volatile memory (NVM) / storage device 1608 coupled to the system control module 1604, one or more input / output devices 1610 coupled to the system control module 1604, and a network interface 1612 coupled to the system control module 1606.
[0265] Processor 1602 may include one or more single-core or multi-core processors, and processor 1602 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.) In some embodiments, system 1600 can be used as a browser as described in the embodiments of the present application.
[0266] In some embodiments, system 1600 may include one or more computer-readable media (e.g., system memory 1606 or NVM / storage device 1608) having instructions and one or more processors 1602 configured to execute the instructions in conjunction with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.
[0267] For one embodiment, the system control module 1604 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1602 and / or any suitable device or component in communication with the system control module 1604 .
[0268] The system control module 1604 may include a memory controller module to provide an interface to the system memory 1606. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0269] The system memory 1606 may be used, for example, to load and store data and / or instructions for the system 1600. For one embodiment, the system memory 1606 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the system memory 1606 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
[0270] For one embodiment, system control module 1604 may include one or more input / output controllers to provide interfaces to NVM / storage device 1608 and input / output device(s) 1610 .
[0271] For example, NVM / storage 1608 may be used to store data and / or instructions. NVM / storage 1608 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).
[0272] NVM / storage device 1608 may include storage resources that are physically part of the device on which system 1600 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1608 may be accessed via input / output device(s) 1610 over a network.
[0273] (One or more) input / output devices 1610 may provide an interface for the system 1600 to communicate with any other appropriate device, and the input / output device 1610 may include a communication component, an audio component, a sensor component, etc. The network interface 1612 may provide an interface for the system 1600 to communicate through one or more networks, and the system 1600 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof for wireless communication.
[0274] For one embodiment, at least one of the processor(s) 1602 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the system control module 1604. For one embodiment, at least one of the processor(s) 1602 may be packaged together with the logic of one or more controllers of the system control module 1604 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1602 may be integrated on the same die with the logic of one or more controllers of the system control module 1604. For one embodiment, at least one of the processor(s) 1602 may be integrated on the same die with the logic of one or more controllers of the system control module 1604 to form a system on chip (SoC).
[0275] In various embodiments, system 1600 may be, but is not limited to: a browser, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, system 1600 may have more or fewer components and / or a different architecture. For example, in some embodiments, system 1600 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.
[0276] Wherein, if the display includes a touch panel, the display screen can be implemented as a touch screen display to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
[0277] An embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a terminal device, the terminal device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0278] In one example, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as in an embodiment of the present application when executing the computer program.
[0279] In one example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, one or more methods of the embodiments of the present application are implemented.
[0280] The embodiments of the present application disclose an image generation method and device. Example 1 includes an image generation method, including:
[0281] Get noise data;
[0282] Obtaining a characteristic image according to the noise data and a first data set; wherein the first data set is used to generate the characteristic image according to the noise data;
[0283] A target image is obtained according to the feature image and a second data set; wherein the second data set is used to generate the target image according to the feature image.
[0284] Example 2 includes the method of Example 1, the first data set includes: a first sub-data set and a second sub-data set, wherein the first sub-data set is used to generate a candidate image according to the noise data, and the second sub-data set is used to identify the feature image from the candidate image;
[0285] The step of obtaining a characteristic image according to the noise data and the first data set includes:
[0286] generating the candidate image according to the noise data and the first sub-data set;
[0287] The feature image in the candidate image is determined by using the second sub-data set.
[0288] Example 3 includes the method of Example 2, wherein the second sub-data set includes: a first sample feature image;
[0289] The determining the feature image in the candidate image by using the second sub-data set comprises:
[0290] According to the second sub-data set, a candidate image whose similarity to the first sample feature image is greater than or equal to a preset similarity threshold is determined as the feature image.
[0291] Example 4 includes the method of Example 1, further comprising:
[0292] Acquire a second sample feature image;
[0293] The first data set is trained according to the second sample feature image.
[0294] Example 5 includes the method of Example 1, further comprising:
[0295] Acquire a sample target image corresponding to the second sample feature image;
[0296] The second data set is trained according to the second sample feature image and the sample target image.
[0297] Example 6 includes the method of Example 1, wherein obtaining a feature image according to the noise data and the first data set comprises:
[0298] Obtaining a feature image corresponding to a target category according to the noise data and the first data set;
[0299] The step of obtaining a target image according to the feature image and the second data set comprises:
[0300] According to the feature image and the second data set, a target image corresponding to the target category is obtained.
[0301] Example 7 includes the method described in any one of Examples 1-6, characterized in that the feature image includes: at least one of a texture feature image and an edge feature image.
[0302] Example 8: A method for generating an image, comprising:
[0303] Get noise data;
[0304] Generate a feature image based on the noise data and a first neural network, wherein the feature image includes a texture feature image or an edge feature image;
[0305] According to the feature image, based on a second neural network, a target image is generated.
[0306] Example 9. According to the method of Example 8, the first neural network includes: a first sub-neural network and a second sub-neural network;
[0307] The step of generating a feature image based on the noise data and a first neural network includes:
[0308] Generate the candidate image according to the noise data and the first sub-neural network;
[0309] The feature image in the candidate image is determined according to the second sub-neural network.
[0310] Example 10. According to the method of Example 9, generating the candidate image according to the noise data and the first sub-neural network includes:
[0311] Based on the fully connected layer of the first sub-neural network, converting the noise data into a first feature map;
[0312] Based on the convolution layer of the first sub-neural network, a first convolution process is performed on the first feature map to obtain the candidate image.
[0313] Example 11. According to the method of Example 9, the second sub-neural network includes: a sample feature image;
[0314] The determining the feature image in the candidate image according to the second sub-neural network comprises:
[0315] Based on the fully connected layer of the second sub-neural network, converting the candidate image into a second feature map;
[0316] Based on the convolution layer of the second sub-neural network, performing a second convolution process on the second feature map to obtain a second candidate feature map;
[0317] Determining the similarity between the second candidate feature map and the sample feature image;
[0318] The candidate image corresponding to the second candidate feature image whose similarity with the sample feature image is greater than or equal to a preset similarity threshold is determined as the feature image.
[0319] Example 12: According to the method of Example 8, generating a target image based on the feature image and a second neural network includes:
[0320] Based on the fully connected layer of the second neural network, converting the feature image into a third feature map;
[0321] Based on the first convolution layer in the second neural network, performing a third convolution process on the third feature map to obtain a third candidate feature map;
[0322] Based on the second convolutional layer in the second neural network, a fourth convolution process is performed on the third candidate feature map to obtain a target image.
[0323] Example 13. According to the method of Example 12, the feature image includes: a texture feature image and an edge feature image, and the fully connected layer based on the second neural network converts the feature image into a third feature map, including:
[0324] Based on the fully connected layer of the second neural network, converting the texture feature image into a first sub-feature map;
[0325] Based on the fully connected layer of the second neural network, converting the edge feature image into a second sub-feature map;
[0326] The step of performing a third convolution process on the third feature map based on the first convolution layer in the second neural network to obtain a third candidate feature map includes:
[0327] Based on the first convolution layer in the second neural network, performing a third convolution process on the first sub-feature map to obtain a first sub-candidate feature map;
[0328] Based on the first convolution layer in the second neural network, performing a third convolution process on the second sub-feature map to obtain a second sub-candidate feature map;
[0329] The step of performing a fourth convolution process on the third candidate feature map based on the second convolution layer in the second neural network to obtain a target image includes:
[0330] Based on the second convolution layer in the second neural network, and according to the fourth convolution operation, the first sub-candidate feature map and the second sub-candidate feature map are merged to obtain the target image.
[0331] Example 14: An image generating device, the device comprising:
[0332] A first acquisition module, used for acquiring noise data;
[0333] A first generating module, configured to obtain a characteristic image according to the noise data and a first data set; wherein the first data set is used to generate the target characteristic image according to the noise data;
[0334] The second generating module is used to obtain a target image according to the feature image and a second data set; wherein the second data set is used to generate the target image according to the feature image.
[0335] Example 15: An image generating device, the device comprising:
[0336] A second acquisition module is used to acquire noise data;
[0337] A third generating module, configured to generate a feature image based on the noise data and the first neural network, wherein the feature image includes a texture feature image or an edge feature image;
[0338] The fourth generating module is used to generate a target image according to the feature image based on the second neural network.
[0339] Example 16. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, one or more methods of Examples 1-7 are implemented.
[0340] Example 17. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements one or more of the methods of Examples 1-7.
[0341] Example 18. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, one or more methods of Examples 8-13 are implemented.
[0342] Example 19. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements one or more of the methods of Examples 8-13.
[0343] Although certain embodiments are for the purpose of illustration and description, various alternatives, and / or, equivalent implementations, or calculations to achieve the same purpose of the embodiments shown and described implementations do not depart from the scope of the present application. The present application is intended to cover any modifications or changes of the embodiments discussed herein. Therefore, it is apparent that the embodiments described herein are limited only by the claims and their equivalents.
Claims
1. An image generation method, characterized in that: include: Get noise data; Obtain a feature image according to the noise data and a first data set; wherein the first data set includes a generative adversarial network model, and the first data set is used to generate the feature image according to the noise data; A target image is obtained according to the feature image and a second data set; wherein the second data set includes a convolutional neural network model, and the second data set is used to generate the target image according to the feature image.
2. The method according to claim 1, characterized in that: The first data set includes: a first sub-data set and a second sub-data set, wherein the first sub-data set includes a generator, the first sub-data set is used to generate a candidate image according to the noise data, and the second sub-data set includes a discriminator, the second sub-data set is used to identify the feature image from the candidate image; The step of obtaining a characteristic image according to the noise data and the first data set includes: generating the candidate image according to the noise data and the first sub-data set; The feature image in the candidate image is determined by using the second sub-data set.
3. The method according to claim 2, characterized in that The second sub-data set includes: a first sample feature image; The determining the feature image in the candidate image by using the second sub-data set comprises: According to the second sub-data set, a candidate image whose similarity to the first sample feature image is greater than or equal to a preset similarity threshold is determined as the feature image.
4. The method according to claim 1, characterized in that: Also includes: Acquire a second sample feature image; The first data set is trained according to the second sample feature image.
5. The method according to claim 4, characterized in that Also includes: Acquire a sample target image corresponding to the second sample feature image; The second data set is trained according to the second sample feature image and the sample target image.
6. The method according to claim 1, characterized in that The step of obtaining a characteristic image according to the noise data and the first data set includes: Obtaining a feature image corresponding to a target category according to the noise data and the first data set; The step of obtaining a target image according to the feature image and the second data set comprises: According to the feature image and the second data set, a target image corresponding to the target category is obtained.
7. The method according to any one of claims 1 to 6, characterized in that: The feature image includes at least one of a texture feature image and an edge feature image.
8. An image generation method, characterized in that: include: Get noise data; Generate a feature image based on the noise data and a first neural network, wherein the feature image includes a texture feature image or an edge feature image; According to the feature image, based on a second neural network, a target image is generated.
9. The method according to claim 8, characterized in that The first neural network includes: a first sub-neural network and a second sub-neural network; The step of generating a feature image based on the noise data and a first neural network includes: Generate a candidate image according to the noise data and the first sub-neural network; The feature image in the candidate image is determined according to the second sub-neural network.
10. The method according to claim 9, characterized in that The step of generating the candidate image according to the noise data and the first sub-neural network comprises: Based on the fully connected layer of the first sub-neural network, converting the noise data into a first feature map; Based on the convolution layer of the first sub-neural network, a first convolution process is performed on the first feature map to obtain the candidate image.
11. The method according to claim 9, characterized in that The second sub-neural network includes: a sample feature image; The determining the feature image in the candidate image according to the second sub-neural network comprises: Based on the fully connected layer of the second sub-neural network, converting the candidate image into a second feature map; Based on the convolution layer of the second sub-neural network, performing a second convolution process on the second feature map to obtain a second candidate feature map; Determining the similarity between the second candidate feature map and the sample feature image; The candidate image corresponding to the second candidate feature image whose similarity with the sample feature image is greater than or equal to a preset similarity threshold is determined as the feature image.
12. The method according to claim 8, characterized in that The step of generating a target image based on the feature image and a second neural network includes: Based on the fully connected layer of the second neural network, converting the feature image into a third feature map; Based on the first convolution layer in the second neural network, performing a third convolution process on the third feature map to obtain a third candidate feature map; Based on the second convolutional layer in the second neural network, a fourth convolution process is performed on the third candidate feature map to obtain a target image.
13. The method according to claim 12, characterized in that The feature image includes: a texture feature image and an edge feature image, and the fully connected layer based on the second neural network converts the feature image into a third feature map, including: Based on the fully connected layer of the second neural network, converting the texture feature image into a first sub-feature map; Based on the fully connected layer of the second neural network, converting the edge feature image into a second sub-feature map; The step of performing a third convolution process on the third feature map based on the first convolution layer in the second neural network to obtain a third candidate feature map includes: Based on the first convolution layer in the second neural network, performing a third convolution process on the first sub-feature map to obtain a first sub-candidate feature map; Based on the first convolution layer in the second neural network, performing a third convolution process on the second sub-feature map to obtain a second sub-candidate feature map; The step of performing a fourth convolution process on the third candidate feature map based on the second convolution layer in the second neural network to obtain a target image includes: Based on the second convolution layer in the second neural network, and according to the fourth convolution operation, the first sub-candidate feature map and the second sub-candidate feature map are merged to obtain the target image.
14. An image generating device, characterized in that: The device comprises: A first acquisition module, used for acquiring noise data; A first generating module, configured to obtain a feature image according to the noise data and a first data set; wherein the first data set includes a generative adversarial network model, and the first data set is used to generate the feature image according to the noise data; The second generating module is used to obtain the target image according to the feature image and the second data set; wherein the second data set includes a convolutional neural network model, and the second data set is used to generate the target image according to the feature image.
15. An image generating device, characterized in that: The device comprises: A second acquisition module is used to acquire noise data; A third generating module, configured to generate a feature image based on the noise data and the first neural network, wherein the feature image includes a texture feature image or an edge feature image; The fourth generating module is used to generate a target image according to the feature image based on the second neural network.
16. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to one or more of claims 1-7 is implemented.
17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to one or more of claims 1 to 7 is implemented.
18. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to one or more of claims 8 to 13 is implemented.
19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to one or more of claims 8 to 13 is implemented.
Citation Information
Patent Citations
Method for automatically filling structure information and texture information of hole area of image scene
CN102324102A
Image processing method and device
CN106910166A
