A deep space exploration image generation method and system based on generative adversarial network

Abstract

The present invention discloses a method and system for generating deep space exploration images based on a generative adversarial network. The method comprises: obtaining a basic data set, sampling from a Gaussian distribution to obtain noise data, and constructing a generative adversarial network model; during the training process, a main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and an auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real image of its corresponding category, thereby obtaining a trained generative adversarial network model. In the present invention, the dual discriminator structure of the main discriminator plus the auxiliary discriminator can force the output distribution of generators of different categories to converge to the real distribution of the data of that category, thereby alleviating the problem of overfitting in the training of the generative adversarial network in a category-imbalanced training set. The trained generator independently generates deep space exploration images of different categories, which is used to construct a deep space exploration image dataset with sufficient sample size, multiple categories and balance.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep space exploration image generation method and system based on a generative adversarial network. Background Art

[0002] In recent years, artificial intelligence technologies, particularly machine learning, have rapidly developed, providing valuable support for building new spacecraft control modes and developing intelligent deep-space exploration systems. Deep learning, a novel learning algorithm based on deep neural networks, has attracted widespread attention in the field of machine learning because its weight pre-training and layer-by-layer training methods effectively avoid the local minimum and vanishing gradient problems of traditional training algorithms. The performance of deep neural networks depends heavily on the availability of a large training set consisting of training samples during the training phase, enabling the network to efficiently learn a highly accurate model. However, due to the specific nature of the visible light images of deep space environments and the confidentiality of the mission, it is difficult to systematically obtain a large number of samples to construct a basic dataset.

[0003] Traditional GANs use deep neural networks as the basic model unit and construct two sub-networks: the generator G (G) and the discriminator D (D). The generator G is used to generate fake samples, while the discriminator D is used to distinguish between genuine and fake samples. For these sub-networks, GANs employ an adversarial game approach during training. Generator G must continuously update its parameters to generate more real data in order to deceive the discriminator D, while the discriminator D must accurately determine whether samples are from real datasets or generated ones. Theoretically, when the training process converges to a Nash equilibrium, the generator's output samples will be sufficiently realistic. While traditional GANs perform well on datasets with a small number of balanced categories, they neglect learning for categories with only a few examples in datasets with a large number of unbalanced categories, which can easily lead to overfitting. Therefore, traditional GANs struggle to effectively generate deep space exploration images covering all categories, and thus struggle to address the problem of a small number of real deep space exploration images, with a large number of categories and an imbalanced distribution. Summary of the Invention

[0004] The present invention provides a deep space exploration image generation method and system based on a generative adversarial network, which is used to solve the technical problem that existing generative adversarial networks are difficult to effectively generate deep space exploration images covering various categories, and further difficult to solve the technical problem that real image samples of deep space exploration are small in number, multi-category and unbalanced.

[0005] A first aspect of the present invention provides a method for generating deep space exploration images based on a generative adversarial network, comprising:

[0006] Obtain a basic data set and sample noise data from a Gaussian distribution, and construct a generative adversarial network model; the basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images from deep space exploration, where n is a positive integer; the generative adversarial network model consists of n categories of generators, a main discriminator, and an auxiliary discriminator;

[0007] The adversarial generative network model is trained using the basic dataset and the sampled noise data. During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic dataset, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real images of its corresponding category, thereby obtaining a trained adversarial generative network model.

[0008] The trained classifiers are used to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

[0009] Specifically, the basic data set consists of n image data subsets, and each image data subset includes real images of the corresponding category; the training process of the adversarial generative network model specifically includes:

[0010] Initialize the parameters of each generator, main discriminator and auxiliary discriminator;

[0011] Input m noise data sampled from Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set; where m is a positive integer;

[0012] The basic data set is sampled to obtain m real image data as the second data set; the n image data subsets are sampled to obtain m real image data of n categories as the third data set;

[0013] Inputting the first data set and the second data set into the main discriminator for discrimination, calculating the objective function of the main discriminator, and obtaining the gradient value of the main discriminator to update the parameters of the main discriminator;

[0014] Performing data augmentation processing on the first data set and the second data set, inputting the data augmented data into the auxiliary discriminator for discrimination, calculating the objective function of the auxiliary discriminator, obtaining the gradient value of the auxiliary discriminator, and updating the parameters of the auxiliary discriminator;

[0015] Resample the Gaussian distribution to obtain m data and input them into each category generator to generate m generated image data of different categories; based on the m generated images of different categories, calculate the objective function of the generator of different categories in turn, obtain the gradient value of each generator, and update the parameters of each generator;

[0016] Jump to the step of inputting the m noise data sampled from the Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set until the number of training iterations of the adversarial generative network model meets the preset number of iterations.

[0017] Specifically, the process of performing data enhancement processing on the first data set and the second data set includes:

[0018] Converting the image data in the first data set and the second data set into grayscale format, and performing edge detection on the image data using a Sobel operator to obtain edge strength information associated with the first data set and the second data set respectively;

[0019] Converting the image data in the first dataset and the second dataset into the frequency domain respectively, and mapping the associated edge strength information into the frequency domain;

[0020] The frequency domain data is filtered using a low-pass filter, and the filtered frequency domain data is inversely transformed into the time domain to obtain enhanced image data associated with the first data set and the second data set, respectively.

[0021] Specifically, the Sobel operator is based on 3 3. The Sobel convolution kernel dynamically processes the image data; the Sobel operator is divided into two directions: horizontal and vertical operators;

[0022] Among them, the convolution kernel G of the Sobel operator in the horizontal direction x Expressed as:

[0023]

[0024] The convolution kernel G of the Sobel operator in the vertical direction y Expressed as:

[0025]

[0026] The Sobel operator uses 3 The Sobel convolution kernel of 3 performs convolution operations on the image data in the horizontal and vertical directions to obtain the gradient amplitude and gradient direction of the image data;

[0027] The calculation method of the gradient amplitude G of the image data is expressed as:

[0028]

[0029] Among them, the weight Kernel of the Sobel convolution kernel is expressed as:

[0030]

[0031] Where p is a dynamic parameter that represents the probability that the discriminator D correctly distinguishes real images. It is adaptively adjusted during training as the discriminator overfits. p = 0 indicates no overfitting, and p = 1 indicates complete overfitting. E represents the expected value, and sign(·) is a sign function used to calculate the number of positive and negative loss results. Represents input data Output result in the discriminator.

[0032] Specifically, the overall objective function of the adversarial generative network model is expressed as:

[0033]

[0034] Where: G i represents the generator of the i-th category, D represents the main discriminator, D aux represents the auxiliary discriminator, n represents the total number of categories; V1 (G i , D) represents the generator G of the i-th category i With the objective function of the main discriminator D, V2 (G i , D aux ) represents the generator G of category i i With auxiliary discriminator D aux The objective function of , ;

[0035] Among them, the objective function of the main discriminator is calculated as follows:

[0036]

[0037] Where: ) represents the basic data set, Represents real image data The discrimination result in the main discriminator, represents a Gaussian distribution, ) represents the generated image data generated by the generator of category i The discrimination result in the main discriminator;

[0038] The objective function of the auxiliary discriminator is calculated as follows:

[0039]

[0040] Where: ) represents the image data subset of category i, Represents the real image data of category i The discrimination results in the auxiliary discriminator, Represents the generated image data generated by the generator of category i The discrimination results in the auxiliary discriminator;

[0041] The objective function of the generator for each category is calculated as follows:

[0042]

[0043] Where: Represents the influence weight of the auxiliary discriminator on the i-th category generator.

[0044] Specifically, the generator consists of a mapping network and a synthesis network;

[0045] The mapping network is composed of multiple fully connected layers;

[0046] The synthesis network consists of an upsampling layer, an adaptive instance normalization module, a depthwise separable convolutional layer and a CBL module.

[0047] Specifically, the main discriminator and the auxiliary discriminator are both composed of multiple convolutional layers, Relu activation layers, tanh activation layers and fully connected layers.

[0048] A second aspect of the present invention provides a deep space exploration image generation system based on a generative adversarial network, comprising:

[0049] A construction module is used to obtain a basic data set, sample noise data from a Gaussian distribution, and construct a generative adversarial network model; the basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images from deep space exploration, where n is a positive integer; the generative adversarial network model consists of n categories of generators, a main discriminator, and an auxiliary discriminator;

[0050] A training module is used to train a generative adversarial network model using the basic dataset and the sampled noise data. During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic dataset, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of real images of its corresponding category, thereby obtaining a trained generative adversarial network model.

[0051] The generation module is used to use the trained classifiers to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

[0052] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above-described methods for generating deep space exploration images are implemented.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for generating deep space exploration images.

[0054] It can be seen from the above technical solutions that the present invention has the following advantages:

[0055] The present invention provides a deep space exploration image generation method and system based on a generative adversarial network, wherein the method comprises: obtaining a basic data set and sampling noise data from a Gaussian distribution, and constructing a generative adversarial network model; the basic data set is composed of n categories of image data subsets, and each image data subset includes a plurality of real images of deep space exploration, where n is a positive integer; the generative adversarial network model is composed of n categories of generators, a main discriminator, and an auxiliary discriminator; the generative adversarial network model is trained using the basic data set and the sampled noise data; during the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real images of its corresponding category, thereby obtaining a trained generative adversarial network model; and each trained classifier is used to generate deep space exploration images of different categories to supplement the basic data set, thereby obtaining a complete deep space exploration image data set.

[0056] In the present invention, the dual discriminator structure of the main discriminator plus the auxiliary discriminator can make the output distribution of generators of different categories converge to the true distribution of the category data, thereby alleviating the problem of overfitting in the training of the adversarial generative network in the category-imbalanced training set. The trained generator independently generates deep space exploration images of different categories, which are used to construct a deep space exploration image dataset with sufficient sample size, multiple categories and balance, thereby solving the technical problem that the existing adversarial generative network is difficult to effectively generate deep space exploration images covering various categories, and thus it is difficult to solve the technical problem that the real image samples of deep space exploration are small in number, multiple in categories and unbalanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A flowchart of a method for generating deep space exploration images based on a generative adversarial network provided by an embodiment of the present invention;

[0059] Figure 2A schematic diagram of the StyleGAN generator structure provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a discriminator structure provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the structure of the generative adversarial network model provided in an embodiment of the present invention.

[0062] Figure 5 A schematic diagram of a data enhancement processing strategy provided by an embodiment of the present invention;

[0063] Figure 6 A schematic structural diagram of a deep space exploration image generation system based on a generative adversarial network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention provide a method and system for generating deep space exploration images based on a generative adversarial network, which is used to solve the technical problem that existing generative adversarial networks are difficult to effectively generate deep space exploration images covering various categories, and further difficult to solve the technical problem that real image samples of deep space exploration are small in number, numerous in categories and unbalanced.

[0065] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0066] See also Figure 1 The first aspect of the present invention provides a method for generating deep space exploration images based on a generative adversarial network, comprising:

[0067] Step 101: obtain a basic data set and sample noise data from a Gaussian distribution, and construct a generative adversarial network model; the generative adversarial network model consists of n categories of generators, a main discriminator, and an auxiliary discriminator.

[0068] It should be noted that the basic dataset is an existing dataset of basic deep space exploration images. It contains sample data from different categories, such as Earth near-orbit images, deep space planetary images, and deep space spacecraft images. Its overall distribution satisfies the requirement of being multi-category and unbalanced, meaning that the distribution of samples from all categories conforms to the true distribution of samples in that category. Assume that the basic dataset consists of n subsets of image data from each category, and each subset includes multiple real deep space exploration images, where n is a positive integer.

[0069] Step 102: Train the GAN model using the basic dataset and the sampled noise data. During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic dataset, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real images of the corresponding category, thereby obtaining a trained GAN model.

[0070] In this invention, the generator can use the generator structure of StyleGAN. Figure 2 , the generator consists of a mapping network and a synthesis network; the mapping network consists of multiple fully connected layers; the synthesis network consists of an upsampling layer, an adaptive instance normalization module, a depth-separable convolutional layer and a CBL module.

[0071] The implementation process of the generator specifically includes: 1) The dimension is 512 1) The latent vector Z is normalized and then input into a mapping network composed of 8 fully connected layers (FC). The mapping network decouples the latent vector to obtain an intermediate latent vector W that can express the deep image features. 2) The synthesis network of the generator is composed of a 4 4 The third-order tensor of 512 is taken as input, and the resolution is gradually increased by upsampling and convolution operations, eventually reaching 512. 512 3); 3) Deconvolution is used to achieve upsampling, and depthwise separable convolution (DWConv) and CBL modules are used to achieve feature extraction; the CBL module consists of 1 1. It consists of a convolutional layer (Conv), a normalization layer (BN), and an activation layer (Leaky Relu). Before each upsampling and after the convolution operation, the AdaIN (Adaptive Instance Normalization) module is used to precisely control style information and retain key image information.

[0072] Among them, the implementation of AdaIN is as follows:

[0073] Expand W through a learnable affine transformation A (two-layer fully connected layer) to (scaling factor) and (bias factor); extract feature maps along the channel dimension And normalize it; use the following formula for weighted summation so that W affects the global information of the image, while the key information of the generated sample is retained by the upsampling and convolution layers.

[0074] The formula for weighted summation so that W affects the global information of the image is expressed as follows:

[0075]

[0076] Where: Representation feature map The mean of Representation feature map The variance value of .

[0077] In the present invention, the main discriminator ( ) and the auxiliary discriminator () share the same architecture, which is a bit simpler to set up compared to the generator architecture. See Figure 3 , both the main discriminator and the auxiliary discriminator are composed of multiple convolutional layers, ReLU activation layers, tanh activation layers and fully connected layers. In the last layer, the tanh function is used to generate a scalar that represents the probability of a real image rather than a generated image.

[0078] In the adversarial generative network model of the present invention, the connections between different types of generators and the main discriminator and auxiliary discriminator are independent of each other. Taking generator i as an example, the connections between the generator and the main discriminator and the auxiliary discriminator are explained:

[0079] The connection between generator i and the main discriminator: The present invention inputs the images of category i generated by generator i and the real images of the basic data set into the main discriminator. The task of the main discriminator is to judge the generated images of category i as false and the images in the basic data set as true; and for generator i, it is expected that the output images are as real as possible (that is, they are judged as true by the main discriminator). During the training process, generator i and the main discriminator compete with each other until they converge to the Nash equilibrium point. At this point, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic data set.

[0080] The connection between generator i and the auxiliary discriminator: The present invention inputs the i-category images generated by generator i together with the real i-category images in the basic data set into the auxiliary discriminator. The task of the auxiliary discriminator is to judge the generated i-category images as false and the real i-category images as true; and for generator i, it is expected that the output images are as real as possible (that is, they are judged as true by the auxiliary discriminator). During the training process, generator i and the auxiliary discriminator compete with each other until they converge to the Nash equilibrium point. At this time, the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real data of its corresponding category.

[0081] From the perspective of probability distribution, the data in a dataset all conform to a certain underlying distribution, and the function of the generator network can be seen as completing the mapping from the known underlying distribution to the target data distribution. Under the dual discriminator structure of the present invention, the main discriminator forces the distribution of the generator output to be close to the distribution of the underlying dataset; the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real data of its corresponding category. The network parameters are corrected by continuously using the errors of the deep generator on the dataset.

[0082] Therefore, when the dual-discriminator architecture is faced with imbalanced data, the main discriminator ensures that the generated images are as close to real images as possible, increasing the overall realism of the generated model. The auxiliary discriminator, on the other hand, focuses on evaluating whether the generated images conform to the characteristics of their specific categories, ensuring that images of each category are appropriately represented. This encourages the generator to pay more attention to the minority class during the generation process, preventing the model from optimizing only for the majority class. This overcomes the technical problem of traditional single-discriminator systems being difficult to train on class-imbalanced datasets.

[0083] In addition, in practical applications, other discriminator algorithms such as Markov discriminator can be replaced, and other constraints can be introduced to improve the effect, as long as the generation effect can achieve the ideal effect.

[0084] See also Figure 4 As shown in Table 1, the following is a detailed description of the training process of the adversarial generative network model. The present invention adopts a fixed alternating training method of the discriminator-generator, that is, when iterating one network, the remaining networks are regarded as known quantities.

[0085] like Figure 4 As shown in the figure, the adversarial generative network model has multiple independently working generators G, each of which generates images of one category in the basic dataset, that is, the number of generators is determined by the number of sample categories in the basic dataset; the generator accepts a set of latent vectors (Z) sampled from the prior distribution as the input of the network, and Z obeys the probability density function of the Gaussian distribution; where the probability density function of the Gaussian distribution is expressed as: , represents the mean, Represents variance.

[0086] Main Discriminator and auxiliary discriminator is the discriminator of StyleGAN, which is used to calculate the binary cross entropy loss of the input image. and The output of the average fusion is used to guide the updated parameters of the generator G. The adaptive structure filter (ASF) uses the Sobel edge feature extraction operator to adaptively extract the target structure features to assist in the training process. train.

[0087] Table 1 Training algorithm for the adversarial generative network model

[0088]

[0089] The training process includes steps S1 to S7:

[0090] Step S1: Initialize the parameters of each generator, main discriminator and auxiliary discriminator.

[0091] In step S2, m noise data sampled from the Gaussian distribution are input into each category generator to generate m generated image data of different categories as the first data set; wherein m is a positive integer.

[0092] It can be understood that random sampling from Gaussian distribution can obtain a random noise sequence as the input of the generator; assuming that random sampling from Gaussian distribution Sampling m noise data , input m noise data into each category generator to obtain n types of output containing m data, that is .

[0093] Step S3: sampling the basic data set to obtain m real image data as the second data set; sampling the n image data subsets to obtain m real image data of n categories as the third data set.

[0094] Assume that m data are sampled from the basic data set , and m data are obtained by sampling from n image data subsets .

[0095] Then, the overall objective function of the adversarial generative network model constructed by the present invention is shown in formula (1):

[0096] (1)

[0097] Where: G i represents the generator of the i-th category, D represents the main discriminator, D aux represents the auxiliary discriminator, n represents the total number of categories; V1 (G i , D) represents the generator G of the i-th category i With the objective function of the main discriminator D, V2 (G i , D aux ) represents the generator G of category i i With auxiliary discriminator D aux The objective function.

[0098] Among them, V1 (Gi , D) and V2 (G i , D aux ) are shown in formulas (2) and (3) respectively:

[0099] (2)

[0100] (3)

[0101] According to formulas (1)-(3), the main discriminator D and the auxiliary discriminator D can be obtained respectively. The objective function.

[0102] Among them, the calculation method of the objective function of the main discriminator is shown in formula (4):

[0103] (4)

[0104] Where: ) represents the basic data set, Represents real image data The discrimination result in the main discriminator, represents a Gaussian distribution, ) represents the generated image data generated by the generator of category i The discrimination results in the main discriminator.

[0105] The objective function of the auxiliary discriminator is calculated as shown in formula (5):

[0106] (5)

[0107] Where: ) represents the image data subset of category i, Represents the real image data of category i The discrimination results in the auxiliary discriminator, Represents the generated image data generated by the generator of category i The discrimination results in the auxiliary discriminator.

[0108] In the actual training process of the model, the training of the generator is often more difficult. In order to flexibly adjust the influence of the main discriminator and the auxiliary discriminator on each generator, the objective function of each category of the generator is shown in formula (6):

[0109] The objective function of the generator for each category is calculated as follows:

[0110] (6)

[0111] Where: Represents the influence weight of the auxiliary discriminator on the i-th category generator.

[0112] Step S4: input the first data set and the second data set into the main discriminator for discrimination, calculate the objective function of the main discriminator, obtain the gradient value of the main discriminator, and update the parameters of the main discriminator.

[0113] It is understandable that the data label in the basic data set can be 1, while the data label of the generated image data generated by the generator can be 0. Therefore, during training, it is hoped that the output of the main discriminator for the data in the basic data set is as 1 as possible, and the output of the generated image data is as 0 as possible. The difference between the label value and the discrimination probability value is used as the back propagation of the main discriminator to provide error information to optimize the main discriminator.

[0114] Then in step S4, the first data set and the second data set are substituted into formula (4) to maximize the objective function and obtain the gradient value of the main discriminator , to use the back-propagation algorithm to provide error information to update the parameters of the main discriminator.

[0115] Step S5: perform data enhancement processing on the first data set and the second data set, input the data after data enhancement processing into the auxiliary discriminator for discrimination, calculate the objective function of the auxiliary discriminator, obtain the gradient value of the auxiliary discriminator, and update the parameters of the auxiliary discriminator.

[0116] Similarly, in step S5, the first data set and the second data set that have undergone data enhancement are substituted into formula (5) to maximize the objective function and obtain the gradient value of the auxiliary discriminator: , to use the back-propagation algorithm to provide error information to update the parameters of the main discriminator.

[0117] It's important to note that deep space exploration images often feature complex structures, high noise levels, and uneven illumination. They also contain a wealth of detail and complex structures, such as galaxies, stars, and nebulae. These structures are more pronounced in the frequency domain, making edge features particularly important. Traditional data augmentation techniques used in generative adversarial networks, such as geometric transformations, feature interpolation, and color dithering, focus solely on texture and features in the pixel space, tending to overlook structural information. Furthermore, when processing deep space images rich in edge features, they are unable to effectively enhance edges and details, resulting in a lack of clarity in the generated deep space images. Furthermore, deep space images are often highly noisy, and applying pixel-space transformations will further amplify this noise, compromising the quality of the generated images.

[0118] Based on this, the present invention adopts a frequency domain-based data enhancement method to enhance image data. Unlike the pixel space-based data enhancement method adopted by traditional adversarial generative networks, the frequency domain space pays more attention to the edge and structural features of the image, which can make up for the feature information missed by traditional adversarial generative networks; at the same time, the present invention also utilizes an adaptive dynamic data enhancement strategy based on the Sobel operator, and completes the image structure feature extraction task with intensity parameters through the Sobel operator convolution kernel with directional edge filtering effect.

[0119] For details, please refer to Figure 5 The process of performing data enhancement processing on the first dataset and the second dataset includes: 1) converting the image data in the first dataset and the second dataset into grayscale format, performing edge detection on the image data using the Sobel operator, and obtaining edge intensity information associated with the first dataset and the second dataset, respectively. 2) converting the image data in the first dataset and the second dataset into the frequency domain, respectively, and mapping the associated edge intensity information into the frequency domain; 3) filtering the frequency domain data using a low-pass filter, and inversely transforming the filtered frequency domain data into the time domain, respectively, to obtain enhanced image data associated with the first dataset and the second dataset.

[0120] It's understandable that converting image data to grayscale format simplifies subsequent processing and increases processing speed. Furthermore, using an adaptive Sobel convolution kernel during edge detection can both identify edge strength within an image and, to a certain extent, preserve the structural characteristics of the feature map. Using a low-pass filter to filter frequency-domain data can enhance high-frequency components or suppress low-frequency components, thereby improving image clarity or reducing noise.

[0121] In the present invention, the Sobel operator is based on 3 3 Sobel convolution kernel dynamically processes the image data; the Sobel operator is divided into two directions: horizontal and vertical operators;

[0122] Among them, the convolution kernel G of the Sobel operator in the horizontal direction x Expressed as:

[0123] (7)

[0124] The convolution kernel G of the Sobel operator in the vertical direction y Expressed as:

[0125] (8)

[0126] The Sobel operator uses 3 The Sobel convolution kernel of 3 performs convolution operations on the image data in the horizontal and vertical directions to obtain the gradient amplitude and gradient direction of the image data;

[0127] The calculation method of the gradient magnitude G of image data is expressed as:

[0128] (9)

[0129] Among them, the weight Kernel of the Sobel convolution kernel is expressed as:

[0130] (10)

[0131] (11)

[0132] Where p is a dynamic parameter, which represents the probability that the discriminator D correctly distinguishes real images. It is adaptively adjusted during training as the discriminator overfits. p = 0 indicates no overfitting, and p = 1 indicates complete overfitting. E represents the expected value, which refers to the statistical average of the output random variable. sign(·) is a sign function used to calculate the number of positive and negative loss results. Represents the input sample The output result in the discriminator is, The closer it is to 1, the higher the probability that the discriminator believes the input sample is a real image; otherwise, The closer it is to 0, the more the discriminator thinks the input sample is fake.

[0133] In addition, in practical applications, it is possible to consider combining deep learning technology with frequency domain filtering, such as using convolutional neural networks (CNNs) to learn the optimal frequency domain filters.

[0134] It can be understood that during the training of the discriminator in steps S4 and S5, the parameters of the generator are fixed; and during the training of the generator in step S6, the parameters of the main discriminator and the auxiliary discriminator are fixed.

[0135] Step S6: resample the Gaussian distribution to obtain m noise data and input them into each category generator to generate m generated image data of different categories; based on the m generated image data of different categories, calculate the objective functions of the generators of different categories in turn to obtain the gradient value of each generator to update the parameters of each generator.

[0136] In step S6, during the training process of the generator, it is necessary to resample the Gaussian distribution to obtain m noise data and input them into each category generator to generate m generated image data of different categories. In the process of training the generator, only the generated image data generated by the generator are used, and these data are input into the main discriminator and the auxiliary discriminator for discrimination. The discriminator generates errors (the discriminator may determine that some generated image data is true), minimizes the objective function, and obtains the gradient value of each generator, so that the generator adjusts its own parameters. Among them, the training algorithm steps are: substitute the m generated image data of each category into formula (6) in turn to obtain the gradient value of each generator , thereby updating the parameters of each generator.

[0137] Step S7, jump to the step of inputting the m noise data sampled from the Gaussian distribution into each category generator, generating m generated image data of different categories as the first data set, until the number of training iterations of the adversarial generative network model meets the preset number of iterations.

[0138] Step 103: Use the trained classifiers to generate deep space exploration images of different categories to supplement the basic data set, thereby obtaining a complete deep space exploration image data set.

[0139] It should be noted that the trained classifiers are used to generate deep space exploration images of various categories, and a deep space exploration image dataset with sufficient sample size, multiple categories and balance is constructed to solve the problem of multiple and unbalanced categories in the basic dataset.

[0140] This application provides a deep space exploration image generation method based on a generative adversarial network, which has the following advantages:

[0141] 1. To address the problem that traditional adversarial generative networks cannot effectively generate images of various categories, the present invention adopts a dual discriminator structure to alleviate the problem of overfitting in the training of adversarial generative networks in category-imbalanced training sets. At the same time, the dual discriminator structure ensures that the output distribution of each generator conforms to the true distribution of its corresponding category data, solving the problem of multiple and unbalanced data categories.

[0142] 2. To address the problem of missing structure and contour in images generated by traditional adversarial networks, an adaptive dynamic data enhancement strategy based on the Sobel operator is adopted. The original image is projected into the frequency domain, and implicit data augmentation of the image is achieved from the frequency domain space to enrich the image data and solve the problem of insufficient image clarity.

[0143] In this paper, a basic multi-category imbalanced deep space exploration dataset is used as a training set for a constructed generative adversarial network model. Ultimately, multiple trained generators are used to generate deep space exploration image data of different categories. A data augmentation method is also designed to project the limited deep space training data into the frequency domain, thereby increasing the implicit image data and improving the quality of the generated images. Furthermore, a main discriminator plus auxiliary discriminator structure is used to force the output distribution of generators of different categories to converge to the true distribution of data for that category, thus resolving the problem of a multi-category and imbalanced training set.

[0144] See also Figure 6 The second aspect of the present invention provides a deep space exploration image generation system based on a generative adversarial network.

[0145] Construction module 201 is used to obtain a basic data set and sample noise data from a Gaussian distribution, and construct a generative adversarial network model; the basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images of deep space exploration, where n is a positive integer; the generative adversarial network model consists of n categories of generators, a main discriminator, and an auxiliary discriminator;

[0146] A training module 202 is configured to train a GAN model using a base dataset and sampled noise data. During the training process, the primary discriminator forces the distribution of the generator output to be close to the distribution of the base dataset, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of real images of the corresponding category, thereby obtaining a trained GAN model.

[0147] The generation module 203 is used to use the trained classifiers to generate deep space exploration images of different categories to supplement the basic data set and obtain a complete deep space exploration image data set.

[0148] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above deep space exploration image generation methods are implemented.

[0149] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods for generating deep space exploration images.

[0150] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0153] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0155] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep space exploration image generation method based on a generative adversarial network, characterized in that: include: Obtain the basic data set and sample noise data from the Gaussian distribution, and build a generative adversarial network model; The basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images of deep space exploration, where n is a positive integer; The adversarial generative network model consists of n categories of generators, main discriminators and auxiliary discriminators; Training a generative adversarial network model using the basic data set and the sampled noise data; During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic dataset, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real images of its corresponding category, resulting in a trained adversarial generative network model; Use the trained classifiers to generate deep space exploration images of different categories to supplement the basic dataset and obtain a complete deep space exploration image dataset; The training process of the adversarial generative network model specifically includes: Initialize the parameters of each generator, main discriminator and auxiliary discriminator; Input m noise data sampled from Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set; where m is a positive integer; The basic data set is sampled to obtain m real image data as the second data set; the n image data subsets are sampled to obtain m real image data of n categories as the third data set; Inputting the first data set and the second data set into the main discriminator for discrimination, calculating the objective function of the main discriminator, and obtaining the gradient value of the main discriminator to update the parameters of the main discriminator; Performing data augmentation processing on the first data set and the second data set, inputting the data augmented data into the auxiliary discriminator for discrimination, calculating the objective function of the auxiliary discriminator, obtaining the gradient value of the auxiliary discriminator, and updating the parameters of the auxiliary discriminator; Resample the Gaussian distribution to obtain m noise data and input them into each category generator to generate m generated image data of different categories; based on the m generated images of different categories, calculate the objective function of the generator of different categories in turn, obtain the gradient value of each generator, and update the parameters of each generator; Jump to the step of inputting the m noise data sampled from the Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set, until the number of training iterations of the adversarial generative network model meets the preset number of iterations; The process of performing data enhancement processing on the first data set and the second data set includes: Converting the image data in the first data set and the second data set into grayscale format, and performing edge detection on the image data using a Sobel operator to obtain edge strength information associated with the first data set and the second data set respectively; Converting the image data in the first dataset and the second dataset into the frequency domain respectively, and mapping the associated edge strength information into the frequency domain; The frequency domain data is filtered using a low-pass filter, and the filtered frequency domain data is inversely transformed into the time domain to obtain enhanced image data associated with the first data set and the second data set, respectively.

2. The deep space exploration image generation method according to claim 1, characterized in that: The Sobel operator dynamically processes the image data based on a 3*3 Sobel convolution kernel; the Sobel operator is divided into two directions: horizontal and vertical. Among them, the convolution kernel G of the Sobel operator in the horizontal direction x Expressed as: ； The convolution kernel G of the Sobel operator in the vertical direction y Expressed as: ； The Sobel operator uses a 3*3 Sobel convolution kernel to perform convolution operations on the image data in the horizontal and vertical directions to obtain the gradient amplitude and gradient direction of the image data; The calculation method of the gradient amplitude G of the image data is expressed as: ； Among them, the weight Kernel of the Sobel convolution kernel is expressed as: ； ； Where p is a dynamic parameter that represents the probability that the discriminator D correctly distinguishes real images. It is adaptively adjusted during training as the discriminator overfits. p = 0 indicates no overfitting, and p = 1 indicates complete overfitting. E represents the expected value, and sign(·) is a sign function used to calculate the number of positive and negative loss results. Represents the input sample Output result in the discriminator.

3. The deep space exploration image generation method according to claim 1, characterized in that: The overall objective function of the adversarial generative network model is expressed as: ； Where: G i represents the generator of the i-th category, D represents the main discriminator, D aux represents the auxiliary discriminator, n represents the total number of categories; V1 (G i , D) represents the generator G of the i-th category i With the objective function of the main discriminator D, V2 (G i , D aux ) represents the generator G of category i i With auxiliary discriminator D aux The objective function of , ; Among them, the objective function of the main discriminator is calculated as follows: ； Where: ) represents the basic data set, Represents real image data The discrimination result in the main discriminator, represents a Gaussian distribution, ) represents the generated image data generated by the generator of category i The discrimination result in the main discriminator; The objective function of the auxiliary discriminator is calculated as follows: ； Where: ) represents the image data subset of category i, Represents the real image data of category i The discrimination results in the auxiliary discriminator, Represents the generated image data generated by the generator of category i The discrimination results in the auxiliary discriminator; The objective function of the generator for each category is calculated as follows: ； Where: Represents the influence weight of the auxiliary discriminator on the i-th category generator.

4. The deep space exploration image generation method according to claim 1, characterized in that: The generator consists of a mapping network and a synthesis network; The mapping network is composed of multiple fully connected layers; The synthesis network consists of an upsampling layer, an adaptive instance normalization module, a depthwise separable convolutional layer and a CBL module.

5. The deep space exploration image generation method according to claim 1, characterized in that: The main discriminator and the auxiliary discriminator are both composed of multiple convolutional layers, ReLU activation layers, tanh activation layers and fully connected layers.

6. A deep space exploration image generation system based on a generative adversarial network, characterized in that: include: A construction module is used to obtain the basic data set and sample noise data from the Gaussian distribution, and build an adversarial generative network model; The basic data set consists of n categories of image data subsets, and each image data subset includes multiple real images of deep space exploration, where n is a positive integer; The adversarial generative network model consists of n categories of generators, main discriminators and auxiliary discriminators; A training module, configured to train a generative adversarial network model using the basic data set and sampled noise data; During the training process, the main discriminator forces the distribution of the generator output to be close to the distribution of the basic dataset, and the auxiliary discriminator forces the distribution of the generator output to be close to the distribution of the real images of its corresponding category, resulting in a trained adversarial generative network model; The generation module is used to use the trained classifiers to generate deep space exploration images of different categories to supplement the basic dataset and obtain a complete deep space exploration image dataset; The training process of the adversarial generative network model specifically includes: Initialize the parameters of each generator, main discriminator and auxiliary discriminator; Input m noise data sampled from Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set; where m is a positive integer; The basic data set is sampled to obtain m real image data as the second data set; the n image data subsets are sampled to obtain m real image data of n categories as the third data set; Inputting the first data set and the second data set into the main discriminator for discrimination, calculating the objective function of the main discriminator, and obtaining the gradient value of the main discriminator to update the parameters of the main discriminator; Performing data augmentation processing on the first data set and the second data set, inputting the data augmented data into the auxiliary discriminator for discrimination, calculating the objective function of the auxiliary discriminator, obtaining the gradient value of the auxiliary discriminator, and updating the parameters of the auxiliary discriminator; Resample the Gaussian distribution to obtain m noise data and input them into each category generator to generate m generated image data of different categories; based on the m generated images of different categories, calculate the objective function of the generator of different categories in turn, obtain the gradient value of each generator, and update the parameters of each generator; Jump to the step of inputting the m noise data sampled from the Gaussian distribution into each category generator to generate m generated image data of different categories as the first data set, until the number of training iterations of the adversarial generative network model meets the preset number of iterations; The process of performing data enhancement processing on the first data set and the second data set includes: Converting the image data in the first data set and the second data set into grayscale format, and performing edge detection on the image data using a Sobel operator to obtain edge strength information associated with the first data set and the second data set respectively; Converting the image data in the first dataset and the second dataset into the frequency domain respectively, and mapping the associated edge strength information into the frequency domain; The frequency domain data is filtered using a low-pass filter, and the filtered frequency domain data is inversely transformed into the time domain to obtain enhanced image data associated with the first data set and the second data set, respectively.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the deep space exploration image generation method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the deep space exploration image generation method according to any one of claims 1 to 5 are implemented.

Images