Deep space exploration image generation method and system based on generative adversarial network

Abstract

The invention discloses a deep space exploration image generation method and system based on a generative adversarial network, and the method comprises the steps: obtaining a basic data set, carrying out the sampling from Gaussian distribution, obtaining noise data, and constructing a generative adversarial network model; in the training process, the main discriminator forces the distribution output by the generator to be close to the distribution of the basic data set, the auxiliary discriminator forces the distribution output by the generator to be close to the distribution of the real images of the corresponding category, and a trained generative adversarial network model is obtained. According to the invention, a double-discriminator structure of the main discriminator and the auxiliary discriminator can promote the output distribution of different categories of generators to converge to the real distribution of the category of data, thereby alleviating the problem that the adversarial generative network is liable to have an over-fitting phenomenon during training in a category imbalance training set. And independently generating different types of deep space detection images through the trained generator, wherein the deep space detection images are used for constructing a balanced deep space detection image data set with sufficient sample size and multiple types.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep space exploration image generation method and system based on a generative adversarial network. Background Art

[0002] In recent years, artificial intelligence technologies represented by machine learning have developed rapidly, providing good support for building new spacecraft control modes and developing intelligent deep space exploration systems. Deep learning is a new type of learning algorithm based on deep neural networks. It has attracted widespread attention in the field of machine learning because its weight pre-training and layer-by-layer training methods effectively avoid the local minimum problem and gradient vanishing problem of traditional training algorithms. The performance of deep neural networks depends greatly on whether there is a training set consisting of a large number of training samples in the training stage, so that the network can efficiently learn a high-precision model from it. However, due to the particularity of the visible light images in the deep space environment and the confidentiality of the mission, it is difficult to systematically obtain a large number of samples to construct a basic data set.

[0003] The traditional adversarial generative network uses a deep neural network as the basic unit of the model, and constructs two sub-networks, namely the generator G (Generator) and the discriminator D (Discriminator). The generator G is used to generate false samples, while the discriminator D is used to distinguish the authenticity of the samples. For the above sub-networks, the adversarial generative network adopts the idea of ​​adversarial game in training. Among them, the generator G needs to continuously update its parameters and generate more real data to achieve the purpose of deceiving the discriminator D, while the discriminator D needs to accurately judge whether the sample is from the real data set or the generated sample. In theory, when the training process finally converges to the Nash equilibrium, the output sample of the generator will be real enough. Although the traditional adversarial generative network has a good effect on a data set with few and balanced data categories, it ignores the learning of categories with only a few samples in a data set with many and unbalanced data categories, which is easy to cause overfitting. Therefore, it is difficult for the traditional adversarial generative network to effectively generate deep space exploration images covering all categories, and it is difficult to solve the problem that the real image samples of deep space exploration are small in number, many in categories and unbalanced. Summary of the invention

[0004] The present invention provides a deep space exploration image generation method and system based on a generative adversarial network, which is used to solve the technical problem that the existing generative adversarial network is difficult to effectively generate deep space exploration images covering various categories, and further difficult to solve the technical problem that the real image samples of deep space exploration are small in number, multi-category and unbalanced.

[0005] A first aspect of the present invention provides a deep space exploration image generation method based on a generative adversarial network, comprising:

[0006] Obtain a basic data set and sample noise data from a Gaussian distribution, and construct a generative adversarial network model; the basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images of deep space exploration, where n is a positive integer; the generative adversarial network model consists of n categories of generators, a main discriminator, and an auxiliary discriminator;

[0007] The adversarial generative network model is trained by using the basic data set and the noise data obtained by sampling; during the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real image of its corresponding category, thereby obtaining a trained adversarial generative network model;

[0008] The trained classifiers are used to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

[0009] Specifically, the basic data set consists of n image data subsets, and each of the image data subsets includes real images of corresponding categories; the training process of the adversarial generative network model specifically includes:

[0010] Initialize the parameters of each generator, main discriminator and auxiliary discriminator;

[0011] Inputting m noise data sampled from Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set; wherein m is a positive integer;

[0012] The basic data set is sampled to obtain m real image data as the second data set; the n image data subsets are sampled to obtain m real image data of n categories as the third data set;

[0013] Inputting the first data set and the second data set into the main discriminator for discrimination, calculating the objective function of the main discriminator, and obtaining the gradient value of the main discriminator to update the parameters of the main discriminator;

[0014] Performing data enhancement processing on the first data set and the second data set, inputting the data enhanced into the auxiliary discriminator for discrimination, and calculating the objective function of the auxiliary discriminator to obtain the gradient value of the auxiliary discriminator to update the parameters of the auxiliary discriminator;

[0015] Resample the Gaussian distribution to obtain m data and input them into each category generator to generate m generated image data of different categories; based on the m generated images of different categories, calculate the objective functions of the generators of different categories in turn to obtain the gradient value of each generator to update the parameters of each generator;

[0016] Jump to the step of inputting the m noise data sampled from the Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set until the number of training iterations of the adversarial generative network model meets the preset number of iterations.

[0017] Specifically, the process of performing data enhancement processing on the first data set and the second data set includes:

[0018] The image data in the first data set and the second data set are converted into grayscale format, and the edge detection is performed on the image data by using a Sobel operator to obtain edge intensity information associated with the first data set and the second data set respectively;

[0019] Converting image data in the first data set and the second data set to the frequency domain respectively, and mapping the associated edge strength information to the frequency domain;

[0020] The frequency domain data is filtered using a low-pass filter, and the filtered frequency domain data is inversely transformed into the time domain to obtain enhanced image data associated with the first data set and the second data set, respectively.

[0021] Specifically, the Sobel operator is based on 3 3. The Sobel convolution kernel dynamically processes the image data; the Sobel operator is divided into two directions: horizontal direction and vertical direction;

[0022] Among them, the convolution kernel G of the Sobel operator in the horizontal direction x It is expressed as:

[0023]

[0024] The convolution kernel G of the Sobel operator in the vertical direction y It is expressed as:

[0025]

[0026] The Sobel operator uses 3 The Sobel convolution kernel of 3 performs convolution operations on the image data in the horizontal and vertical directions to obtain the gradient amplitude and gradient direction of the image data;

[0027] The calculation method of the gradient amplitude G of the image data is expressed as:

[0028]

[0029] Among them, the weight Kernel of the Sobel convolution kernel is expressed as:

[0030]

[0031] Where: p is a dynamic parameter, which indicates the probability that the discriminator D correctly distinguishes the real image. It is adaptively adjusted during the training process according to the degree of overfitting of the discriminator. p = 0 indicates that it is not in an overfitting state, and p = 1 indicates a completely overfitting state. E represents the expected value, and sign(·) is a sign function used to calculate the number of positive and negative loss results. Represents input data Output result in the discriminator.

[0032] Specifically, the overall objective function of the adversarial generative network model is expressed as:

[0033]

[0034] Where: G i represents the generator of the i-th category, D represents the main discriminator, and D aux represents the auxiliary discriminator, n represents the total number of categories; V1 (G i , D) represents the generator G of the i-th category i With the objective function of the main discriminator D, V2(G i , D aux ) represents the generator G of category i i With auxiliary discriminator D aux The objective function of , ;

[0035] Among them, the objective function of the main discriminator is calculated as follows:

[0036]

[0037] Where: ) represents the basic data set, Represents real image data The discrimination result in the main discriminator, represents a Gaussian distribution, ) represents the generated image data generated by the generator of the i-th category The discrimination result in the main discriminator;

[0038] The objective function of the auxiliary discriminator is calculated as follows:

[0039]

[0040] Where: ) represents the image data subset of the i-th category, Represents the real image data of the i-th category The discrimination result in the auxiliary discriminator, Represents the generated image data generated by the generator of the i-th category The discrimination results in the auxiliary discriminator;

[0041] The objective function of the generator for each category is calculated as follows:

[0042]

[0043] Where: Represents the influence weight of the auxiliary discriminator on the i-th category generator.

[0044] Specifically, the generator consists of a mapping network and a synthesis network;

[0045] The mapping network is composed of multiple fully connected layers;

[0046] The synthesis network consists of an upsampling layer, an adaptive instance normalization module, a depthwise separable convolutional layer and a CBL module.

[0047] Specifically, the main discriminator and the auxiliary discriminator are both composed of multiple convolutional layers, Relu activation layers, tanh activation layers and fully connected layers.

[0048] A second aspect of the present invention provides a deep space exploration image generation system based on a generative adversarial network, comprising:

[0049] A construction module is used to obtain a basic data set and sample noise data from a Gaussian distribution, and to construct a generative adversarial network model; the basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images of deep space exploration, where n is a positive integer; the generative adversarial network model consists of n categories of generators, a main discriminator, and an auxiliary discriminator;

[0050] A training module is used to train the adversarial generative network model through the basic data set and the noise data obtained by sampling; during the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real image of its corresponding category, so as to obtain a trained adversarial generative network model;

[0051] The generation module is used to use the trained classifiers to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

[0052] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above-described methods for generating deep space exploration images are implemented.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of any of the above-described methods for generating deep space exploration images are implemented.

[0054] It can be seen from the above technical solutions that the present invention has the following advantages:

[0055] The present invention provides a deep space exploration image generation method and system based on a generative adversarial network, wherein the method comprises: obtaining a basic data set and sampling noise data from a Gaussian distribution, and constructing a generative adversarial network model; the basic data set is composed of image data subsets of n categories, and each image data subset includes a plurality of real images of deep space exploration, where n is a positive integer; the generative adversarial network model is composed of generators of n categories and a main discriminator and an auxiliary discriminator; the generative adversarial network model is trained by the basic data set and the noise data obtained by sampling; during the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real image of its corresponding category, so as to obtain a trained generative adversarial network model; and each trained classifier is used to generate deep space exploration images of different categories to supplement the basic data set, so as to obtain a complete deep space exploration image data set.

[0056] In the present invention, the dual discriminator structure of the main discriminator plus the auxiliary discriminator can cause the output distribution of generators of different categories to converge to the true distribution of the category data, thereby alleviating the problem of overfitting in the training of adversarial generative networks in category-imbalanced training sets. The trained generators independently generate deep space exploration images of different categories to construct a deep space exploration image dataset with sufficient sample size, multiple categories and balance, thereby solving the problem that the existing adversarial generative networks are difficult to effectively generate deep space exploration images covering various categories, and thus it is difficult to solve the technical problem that the real image samples of deep space exploration are small in number, multiple in categories and unbalanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 A flowchart of a method for generating deep space exploration images based on a generative adversarial network provided by an embodiment of the present invention;

[0059] Figure 2A schematic diagram of a StyleGAN generator structure provided in an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a discriminator structure provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the structure of a generative adversarial network model provided in an embodiment of the present invention.

[0062] Figure 5 A schematic diagram of a data enhancement processing strategy provided by an embodiment of the present invention;

[0063] Figure 6 A schematic diagram of the structure of a deep space exploration image generation system based on a generative adversarial network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention provide a method and system for generating deep space exploration images based on a generative adversarial network, which are used to solve the technical problem that the existing generative adversarial network is difficult to effectively generate deep space exploration images covering various categories, and further difficult to solve the technical problem that the real image samples of deep space exploration are small in number, multi-category and unbalanced.

[0065] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] See also Figure 1 The first aspect of the present invention provides a deep space exploration image generation method based on a generative adversarial network, comprising:

[0067] Step 101, obtain a basic data set and sample noise data from a Gaussian distribution, and construct a generative adversarial network model; the generative adversarial network model consists of n categories of generators, a main discriminator, and an auxiliary discriminator.

[0068] It should be noted that the basic data set is an existing basic deep space exploration image data set, which contains different categories of sample data such as Earth near-orbit images, deep space planetary images, and deep space spacecraft images. Its overall distribution satisfies the requirement of multiple and unbalanced categories, that is, the distribution of samples of all categories conforms to the real distribution of samples of that category. Assume that the basic data set consists of image data subsets of n categories, and each image data subset includes multiple real images of deep space exploration, and n is a positive integer.

[0069] Step 102, training the adversarial generative network model using the basic data set and the sampled noise data; during the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real image of its corresponding category, thereby obtaining a trained adversarial generative network model.

[0070] In the present invention, the generator can use the generator structure of StyleGAN. Figure 2 , the generator consists of a mapping network and a synthesis network; the mapping network consists of multiple fully connected layers; the synthesis network consists of an upsampling layer, an adaptive instance normalization module, a depth-separable convolutional layer and a CBL module.

[0071] The implementation process of the generator specifically includes: 1) The dimension is 512 1) After being normalized, the latent vector Z is input into a mapping network consisting of 8 fully connected layers (FC), which decouples the latent vector to obtain an intermediate latent vector W that can express the deep image features; 2) In the synthesis network of the generator, a 4 4 The third-order tensor of 512 is taken as input, and the resolution is gradually increased by upsampling and convolution operations, eventually reaching 512 512 3) Deconvolution is used to achieve upsampling, and depthwise separable convolution (DWConv) and CBL modules are used to achieve feature extraction; the CBL module consists of 1 1 convolution layer (Conv), normalization layer (BN), activation layer (Leaky Relu); before each upsampling and after the convolution operation, the AdaIN (Adaptive Instance Normalization) module is used to accurately control the style information and retain the key information of the image.

[0072] Among them, the implementation of AdaIN is as follows:

[0073] Expand W through a learnable affine transformation A (two-layer fully connected layer) to (scaling factor) and (bias factor); extract feature maps along the channel dimension And normalize it; use the following formula for weighted summation so that W affects the global information of the image, while the key information of the generated samples is retained by the upsampling and convolution layers.

[0074] Among them, the formula for weighted summation so that W affects the global information of the picture is expressed as follows:

[0075]

[0076] Where: Representation feature map The mean of Representation feature map The variance value of .

[0077] In the present invention, the main discriminator ( ) and the auxiliary discriminator () share the same architecture, which is a bit simpler to set up compared to the generator architecture. See Figure 3 , both the main discriminator and the auxiliary discriminator are composed of multiple convolutional layers, Relu activation layers, tanh activation layers and fully connected layers. In the last layer, the tanh function is used to generate a scalar, which represents the probability of a real image rather than a generated image.

[0078] In the adversarial generative network model of the present invention, the connections between different types of generators and the main discriminator and the auxiliary discriminator are independent of each other. Generator i is taken as an example to illustrate the connections between the generator and the main discriminator and the auxiliary discriminator respectively:

[0079] The connection between generator i and the main discriminator: The present invention inputs the images of category i generated by generator i and the real images of the basic data set into the main discriminator. The task of the main discriminator is to judge the generated images of category i as false and the images in the basic data set as true; and for generator i, it is expected that the output image is as real as possible (that is, it is judged as true by the main discriminator). During the training process, generator i and the main discriminator compete with each other until they converge to the Nash equilibrium point. At this time, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set.

[0080] The connection between generator i and the auxiliary discriminator: The present invention inputs the i-category images generated by generator i together with the real i-category images in the basic data set into the auxiliary discriminator. The task of the auxiliary discriminator is to judge the generated i-category images as false and the real i-category images as true; and for generator i, it is expected that the output image is as real as possible (that is, it is judged as true by the auxiliary discriminator). During the training process, generator i and the auxiliary discriminator compete with each other until they converge to the Nash equilibrium point. At this time, the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real data of its corresponding category.

[0081] From the perspective of probability distribution, the data in a certain data set all conform to a certain potential distribution, and the function of the generator network can be regarded as: completing the mapping from the known potential distribution to the target data distribution. Under the dual discriminator structure of the present invention, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set; the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real data of its corresponding category. The network parameters are corrected by continuously using the error of the deep generator on the data set.

[0082] Therefore, when the dual discriminator structure faces unbalanced data, the main discriminator will ensure that the generated image is as close to the real image as possible, increasing the overall authenticity of the generated model. The auxiliary discriminator focuses on evaluating whether the generated image meets the characteristics of its specific category, ensuring that each category of images is properly represented, and prompting the generator to pay more attention to the minority class during the generation process. This prevents the model from only optimizing the generation effect of the majority class, thereby overcoming the technical problem of the difficulty of training the traditional single discriminator system on the category imbalanced dataset.

[0083] In addition, in practical applications, other discriminator algorithms such as Markov discriminator can be replaced, and other constraints can be introduced to improve the effect, as long as the generated effect can achieve the ideal effect.

[0084] See also Figure 4 As shown in Table 1, the training process of the adversarial generative network model is described in detail below. The present invention adopts a fixed alternating training method of the discriminator-generator, that is, when iterating one network, the remaining networks are regarded as known quantities.

[0085] like Figure 4 As shown in the figure, the adversarial generative network model has multiple independently working generators G, each of which generates images of one category in the basic data set, that is, the number of generators is determined by the number of sample categories in the basic data set; the generator accepts a set of latent vectors (Z) sampled from the prior distribution as the input of the network, and Z obeys the probability density function of the Gaussian distribution; where the probability density function of the Gaussian distribution is expressed as: , represents the mean, Represents variance.

[0086] Main Discriminator and auxiliary discriminator is the discriminator of StyleGAN, which is used to calculate the binary cross entropy loss of the input image. and The output of is averaged and fused to guide the updated parameters of the generator G. The adaptive structure filter (ASF) uses the Sobel edge feature extraction operator to adaptively extract the target structure features to assist in the training process. train.

[0087] Table 1 Training algorithm of adversarial generative network model

[0088]

[0089] The training process includes steps S1 to S7:

[0090] Step S1, initialize the parameters of each generator, main discriminator and auxiliary discriminator.

[0091] In step S2, m noise data sampled from the Gaussian distribution are input into each category generator to generate m generated image data of different categories as the first data set; wherein m is a positive integer.

[0092] It can be understood that random sampling from a Gaussian distribution obtains a random noise sequence as the input of the generator; assuming that Sampling to obtain m noise data , input m noise data into each category generator to obtain n types of outputs containing m data, that is, .

[0093] Step S3, sampling the basic data set to obtain m real image data as the second data set; sampling the n image data subsets to obtain m real image data of n categories as the third data set.

[0094] Assume that m data are sampled from the basic data set , and m data are obtained by sampling each of n image data subsets .

[0095] Then, the overall objective function of the adversarial generative network model constructed by the present invention is shown in formula (1):

[0096] (1)

[0097] Where: G i represents the generator of the i-th category, D represents the main discriminator, and D aux represents the auxiliary discriminator, n represents the total number of categories; V1 (G i , D) represents the generator G of the i-th category i With the objective function of the main discriminator D, V2(G i , D aux ) represents the generator G of category i i With auxiliary discriminator D aux The objective function of .

[0098] Among them, V1 (Gi , D) and V2 (G i , D aux ) are shown in formulas (2) and (3) respectively:

[0099] (2)

[0100] (3)

[0101] According to formulas (1)-(3), the main discriminator D and the auxiliary discriminator D can be obtained respectively. The objective function of .

[0102] Among them, the calculation method of the objective function of the main discriminator is shown in formula (4):

[0103] (4)

[0104] Where: ) represents the basic data set, Represents real image data The discrimination result in the main discriminator, represents a Gaussian distribution, ) represents the generated image data generated by the generator of the i-th category The discrimination results in the main discriminator.

[0105] The calculation method of the objective function of the auxiliary discriminator is shown in formula (5):

[0106] (5)

[0107] Where: ) represents the image data subset of the i-th category, Represents the real image data of the i-th category The discrimination result in the auxiliary discriminator, Represents the generated image data generated by the generator of the i-th category The discrimination results in the auxiliary discriminator.

[0108] In the actual training process of the model, the training of the generator is often more difficult. In order to flexibly adjust the influence of the main discriminator and the auxiliary discriminator on each generator, the objective function of each category of the generator is shown in formula (6):

[0109] The objective function of the generator for each category is calculated as follows:

[0110] (6)

[0111] Where: Represents the influence weight of the auxiliary discriminator on the i-th category generator.

[0112] Step S4, inputting the first data set and the second data set into the main discriminator for discrimination, calculating the objective function of the main discriminator, obtaining the gradient value of the main discriminator, and updating the parameters of the main discriminator.

[0113] It is understandable that the data label in the basic data set can be 1, while the data label of the generated image data generated by the generator can be 0. Therefore, during training, it is hoped that the output of the main discriminator for the data in the basic data set is as 1 as possible, while the output of the generated image data is as 0 as possible. The difference between the label value and the discrimination probability value is used as the back propagation of the main discriminator to provide error information to optimize the main discriminator.

[0114] Then in step S4, the first data set and the second data set are substituted into formula (4) to maximize the objective function and obtain the gradient value of the main discriminator: , so as to use the back-propagation algorithm to provide error information to update the parameters of the main discriminator.

[0115] Step S5, performing data enhancement processing on the first data set and the second data set, and inputting the data enhanced into the auxiliary discriminator for discrimination, and calculating the objective function of the auxiliary discriminator to obtain the gradient value of the auxiliary discriminator to update the parameters of the auxiliary discriminator.

[0116] Similarly, in step S5, the first data set and the second data set after data enhancement are substituted into formula (5) to maximize the objective function and obtain the gradient value of the auxiliary discriminator: , to use the back-propagation algorithm to provide error information to update the parameters of the main discriminator.

[0117] It should be noted that deep space exploration images usually have complex structures, high noise, and uneven illumination. Deep space exploration images also contain a lot of details and complex structures, such as galaxies, stars, and nebulae. These structures are more obvious in the frequency domain, and edge features are particularly important. The data enhancement techniques used by traditional adversarial generative networks, such as geometric transformation, feature interpolation, and color jitter, only focus on the texture and features of the image in the pixel space, and tend to ignore the structural information of the image. When processing deep space images with rich edge features, it is impossible to effectively enhance the edges and details in the image, resulting in the generated deep space image structure being unclear. In addition, deep space images are often accompanied by high noise, and taking pixel space transformations will further amplify the noise, affecting the quality of the generated image.

[0118] Based on this, the present invention adopts a frequency domain-based data enhancement method to enhance the image data. Different from the pixel space-based data enhancement method adopted by the traditional adversarial generation network, the frequency domain space pays more attention to the edge and structural features of the image, which can make up for the feature information missed by the traditional adversarial generation network; at the same time, the present invention also utilizes an adaptive dynamic data enhancement strategy based on the Sobel operator, and completes the image structure feature extraction task with intensity parameters through the Sobel operator convolution kernel with directional edge filtering effect.

[0119] For details, please refer to Figure 5 , the process of performing data enhancement processing on the first data set and the second data set includes: 1) converting the image data in the first data set and the second data set into grayscale format, and performing edge detection on the image data through the Sobel operator to obtain edge strength information associated with the first data set and the second data set respectively. 2) converting the image data in the first data set and the second data set into the frequency domain respectively, and mapping the associated edge strength information into the frequency domain; 3) filtering the frequency domain data using a low-pass filter, inversely transforming the filtered frequency domain data into the time domain, and obtaining enhanced image data associated with the first data set and the second data set respectively.

[0120] It is understandable that converting image data into grayscale format can simplify the subsequent processing process and improve the processing speed. At the same time, in edge detection, the use of adaptive Sobel convolution kernel can not only identify the edge strength in the image, but also ensure the structural characteristics of the feature map to a certain extent. Using a low-pass filter to filter the frequency domain data can enhance the high-frequency components of the image or suppress the low-frequency components, thereby improving the clarity of the image or reducing noise.

[0121] In the present invention, the Sobel operator is based on 3 3 Sobel convolution kernel dynamically processes the image data; the Sobel operator is divided into two directions: horizontal and vertical.

[0122] Among them, the convolution kernel G of the Sobel operator in the horizontal direction x It is expressed as:

[0123] (7)

[0124] The convolution kernel G of the Sobel operator in the vertical direction y It is expressed as:

[0125] (8)

[0126] The Sobel operator uses 3 The Sobel convolution kernel of 3 performs convolution operations on the image data in the horizontal and vertical directions to obtain the gradient amplitude and gradient direction of the image data;

[0127] The calculation method of the gradient amplitude G of image data is expressed as:

[0128] (9)

[0129] Among them, the weight Kernel of the Sobel convolution kernel is expressed as:

[0130] (10)

[0131] (11)

[0132] Where: p is a dynamic parameter, which indicates the probability that the discriminator D correctly distinguishes the real image. It is adaptively adjusted during the training process according to the degree of overfitting of the discriminator. p = 0 indicates that it is not in an overfitting state, and p = 1 indicates a completely overfitting state. E represents the expected value, which refers to the statistical average of the output random variable. sign(·) is a sign function used to calculate the number of positive and negative loss results. Represents the input sample The output result in the discriminator is The closer it is to 1, the higher the probability that the discriminator believes that the input sample is a real image; conversely, The closer it is to 0, the more the discriminator thinks the input sample is fake.

[0133] In addition, in practical applications, it is possible to consider combining deep learning techniques with frequency domain filtering, such as using convolutional neural networks (CNNs) to learn the optimal frequency domain filters.

[0134] It can be understood that during the training of the discriminator in step S4 and step S5, the parameters of the generator are fixed; and during the training of the generator in step S6, the parameters of the main discriminator and the auxiliary discriminator are fixed.

[0135] Step S6, resample the Gaussian distribution to obtain m noise data and input them into each category generator to generate m generated image data of different categories; based on the m generated image data of different categories, calculate the objective functions of the generators of different categories in turn to obtain the gradient value of each generator to update the parameters of each generator.

[0136] In step S6, during the training process of the generator, it is necessary to resample the Gaussian distribution to obtain m noise data and input them into each category generator to generate m generated image data of different categories. In the process of training the generator, only the generated image data generated by the generator are used to input these data into the main discriminator and the auxiliary discriminator for discrimination. The discriminator generates errors (the discriminator may determine that some generated image data is true), minimizes the objective function, and obtains the gradient value of each generator, so that the generator adjusts its own parameters. Among them, the algorithm steps of training are: Substitute the m generated image data of each category into formula (6) in turn to obtain the gradient value of each generator , thereby updating the parameters of each generator.

[0137] Step S7, jump to the step of inputting m noise data sampled from the Gaussian distribution into each category generator, generating m generated image data of different categories as the first data set, until the number of training iterations of the adversarial generative network model meets the preset number of iterations.

[0138] Step 103, using each trained classifier to generate deep space exploration images of different categories to supplement the basic data set, and obtain a complete deep space exploration image data set.

[0139] It should be noted that the trained classifiers are used to generate various categories of deep space exploration images, and a deep space exploration image dataset with sufficient sample size, multiple categories and balance is constructed to solve the problem of multiple and unbalanced categories in the basic dataset.

[0140] The deep space exploration image generation method based on the adversarial generative network provided in this application has the following advantages:

[0141] 1. In view of the problem that traditional adversarial generative networks cannot effectively generate images of various categories, the present invention adopts a dual discriminator structure to alleviate the problem that adversarial generative networks are prone to overfitting in the training of category-imbalanced training sets. At the same time, the dual discriminator structure ensures that the output distribution of each generator conforms to the actual distribution of its corresponding category data, solving the problem of multiple and unbalanced data categories.

[0142] 2. In view of the problem that images generated by traditional generative adversarial networks have missing structures and contours, an adaptive dynamic data enhancement strategy based on the Sobel operator is adopted. The original image is projected into the frequency domain, and implicit data augmentation of the image is realized from the frequency domain space to enrich the image data and solve the problem of insufficient image clarity.

[0143] In the present invention, a basic multi-category unbalanced deep space exploration data set is used as a training set to train the constructed adversarial generative network model, and finally a plurality of trained generators are used to generate different categories of deep space exploration image data. At the same time, a data enhancement method is designed to project the limited data of deep space training data into the frequency domain, thereby increasing the implicit data of the image and improving the quality of the generated image. And through the structure of the main discriminator plus the auxiliary discriminator, the output distribution of generators of different categories can be made to converge to the true distribution of the data of that category, solving the problem of multiple and unbalanced training set categories.

[0144] See also Figure 6 The second aspect of the present invention provides a deep space exploration image generation system based on a generative adversarial network.

[0145] The construction module 201 is used to obtain a basic data set and sample noise data from a Gaussian distribution, and to construct a generative adversarial network model; the basic data set is composed of n categories of image data subsets, and each image data subset includes a plurality of real images of deep space exploration, and n is a positive integer; the generative adversarial network model is composed of n categories of generators, a main discriminator, and an auxiliary discriminator;

[0146] The training module 202 is used to train the adversarial generative network model through the basic data set and the noise data obtained by sampling. During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real image of its corresponding category, so as to obtain a trained adversarial generative network model.

[0147] The generation module 203 is used to use the trained classifiers to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

[0148] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above deep space exploration image generation methods are implemented.

[0149] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above deep space exploration image generation methods.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0155] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep space exploration image generation method based on a generative adversarial network, characterized in that: include: Obtain the basic data set and sample the noise data from the Gaussian distribution, and build an adversarial generative network model; The basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images of deep space exploration, n is a positive integer; The adversarial generative network model consists of n types of generators, main discriminators and auxiliary discriminators; Training a generative adversarial network model using the basic data set and the sampled noise data; During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real images of its corresponding category, thus obtaining a trained adversarial generative network model; The trained classifiers are used to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

2. The deep space exploration image generation method according to claim 1, characterized in that: The training process of the adversarial generative network model specifically includes: Initialize the parameters of each generator, main discriminator and auxiliary discriminator; Inputting m noise data sampled from Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set; wherein m is a positive integer; The basic data set is sampled to obtain m real image data as the second data set; the n image data subsets are sampled to obtain m real image data of n categories as the third data set; Inputting the first data set and the second data set into the main discriminator for discrimination, calculating the objective function of the main discriminator, and obtaining the gradient value of the main discriminator to update the parameters of the main discriminator; Performing data enhancement processing on the first data set and the second data set, and inputting the data enhanced into the auxiliary discriminator for discrimination, and calculating the objective function of the auxiliary discriminator to obtain the gradient value of the auxiliary discriminator to update the parameters of the auxiliary discriminator; Resample the Gaussian distribution to obtain m noise data and input them into each category generator to generate m generated image data of different categories; based on the m generated images of different categories, calculate the objective functions of the generators of different categories in turn to obtain the gradient value of each generator to update the parameters of each generator; Jump to the step of inputting the m noise data sampled from the Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set until the number of training iterations of the adversarial generative network model meets the preset number of iterations.

3. The deep space exploration image generation method according to claim 2, characterized in that: The process of performing data enhancement processing on the first data set and the second data set includes: The image data in the first data set and the second data set are converted into grayscale format, and the edge detection is performed on the image data by using a Sobel operator to obtain edge intensity information associated with the first data set and the second data set respectively; Converting image data in the first data set and the second data set to the frequency domain respectively, and mapping the associated edge strength information to the frequency domain; The frequency domain data is filtered using a low-pass filter, and the filtered frequency domain data is inversely transformed into the time domain to obtain enhanced image data associated with the first data set and the second data set, respectively.

4. The deep space exploration image generation method according to claim 3, characterized in that: The Sobel operator is based on 3 3. The Sobel convolution kernel dynamically processes the image data; the Sobel operator is divided into two directions: horizontal and vertical. Among them, the convolution kernel G of the Sobel operator in the horizontal direction x It is expressed as: The convolution kernel G of the Sobel operator in the vertical direction y It is expressed as: The Sobel operator uses 3 The Sobel convolution kernel of 3 performs convolution operations on the image data in the horizontal and vertical directions to obtain the gradient amplitude and gradient direction of the image data; The calculation method of the gradient amplitude G of the image data is expressed as: Among them, the weight Kernel of the Sobel convolution kernel is expressed as: Where: p is a dynamic parameter, which indicates the probability that the discriminator D correctly distinguishes the real image. It is adaptively adjusted during the training process according to the degree of overfitting of the discriminator. p = 0 indicates that it is not in an overfitting state, and p = 1 indicates a completely overfitting state. E represents the expected value, and sign(·) is a sign function used to calculate the number of positive and negative loss results. Represents the input sample Output result in the discriminator.

5. The deep space exploration image generation method according to claim 2, characterized in that: The overall objective function of the adversarial generative network model is expressed as: Where: G i represents the generator of the i-th category, D represents the main discriminator, and D aux represents the auxiliary discriminator, n represents the total number of categories; V1 (G i , D) represents the generator G of the i-th category i With the objective function of the main discriminator D, V2 (G i , D aux ) represents the generator G of category i i With auxiliary discriminator D aux The objective function of , ; Among them, the objective function of the main discriminator is calculated as follows: Where: ) represents the basic data set, Represents real image data The discrimination result in the main discriminator, represents a Gaussian distribution, ) represents the generated image data generated by the generator of the i-th category The discrimination result in the main discriminator; The objective function of the auxiliary discriminator is calculated as follows: Where: ) represents the image data subset of the i-th category, Represents the real image data of the i-th category The discrimination result in the auxiliary discriminator, Represents the generated image data generated by the generator of the i-th category The discrimination results in the auxiliary discriminator; The objective function of the generator for each category is calculated as follows: Where: Represents the influence weight of the auxiliary discriminator on the i-th category generator.

6. The deep space exploration image generation method according to claim 1, characterized in that: The generator consists of a mapping network and a synthesis network; The mapping network is composed of multiple fully connected layers; The synthesis network consists of an upsampling layer, an adaptive instance normalization module, a depthwise separable convolutional layer and a CBL module.

7. The deep space exploration image generation method according to claim 1, characterized in that: The main discriminator and the auxiliary discriminator are both composed of multiple convolutional layers, Relu activation layers, tanh activation layers and fully connected layers.

8. A deep space exploration image generation system based on a generative adversarial network, characterized in that: include: A construction module is used to obtain the basic data set and sample the noise data from the Gaussian distribution, and build an adversarial generative network model; The basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images of deep space exploration, n is a positive integer; The adversarial generative network model consists of n types of generators, main discriminators and auxiliary discriminators; A training module, used for training a generative adversarial network model using the basic data set and the noise data obtained by sampling; During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real images of its corresponding category, thus obtaining a trained adversarial generative network model; The generation module is used to use the trained classifiers to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the deep space exploration image generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the deep space exploration image generation method as described in any one of claims 1 to 7 are implemented.

Images