A quality assessment method and device for SAR image simulation using a deep generative model

Abstract

The present invention discloses a quality assessment method for a deep generative model simulated SAR image, comprising: obtaining original sample sets of various categories using real SAR target slice data; inputting the original sample sets into a preset deep generative model for each category to obtain a simulated sample set; mixing the original sample sets and simulated sample sets of the category according to different preset ratios and keeping the number of samples the same to obtain multiple mixed sample sets; obtaining the total original sample sets and the total simulated sample sets of all categories, and for each preset ratio, combining the mixed sample sets of all categories of the preset ratio into a total mixed sample set; based on the total original sample set, the total mixed sample sets of each preset ratio and the total simulated sample set, obtaining a mixed recognition rate curve corresponding to a preset classification model, and using the mixed recognition rate curve to assess the quality of the simulated SAR image. The present invention can accurately determine whether the simulated SAR image has a positive impact on the change in recognition rate, and judge the application capability of the simulated SAR image.

Description

Technical Field

[0001] The present invention belongs to the field of signal and information processing, and in particular relates to a quality assessment method and device for a deep generative model simulated SAR image. Background Art

[0002] Synthetic Aperture Radar Automatic Target Recognition (SAR-ATR) is one of the classic problems in synthetic aperture radar (SAR) technology. A variety of deep learning methods have been applied in SAR-ATR. Deep learning is data-driven. When training deep neural networks, a large amount of training data can improve the performance of the network. However, the acquisition of SAR data is time-consuming and laborious, especially for non-cooperative targets, it is difficult to obtain specific SAR data. Due to the limitations of the data acquisition process, obtaining enough SAR images under various conditions is the key to breaking through the bottleneck of deep learning applications in this field. At present, using simulated SAR images as training samples to expand the real SAR training data set is a widely used method in the model training process. Since the quality of simulated SAR images will directly affect the training effect of models such as SAR target recognition and then affect the recognition accuracy, it is of great significance to evaluate the quality of simulated SAR images.

[0003] Traditional SAR image simulation is based on electromagnetic theory calculations and precise CAD target models. The quality of the simulated image is related to the surface conditions of the CAD model, and some electromagnetic properties of the target cannot be directly measured, which significantly reduces the quality of the simulated SAR image. In recent years, deep generative models have been widely used for data enhancement, such as VAE (Variational autoencoder, a generative model based on deep learning of variational ideas), etc. Using deep generative models to generate simulated SAR images has become a trend in the industry.

[0004] Image Quality Assessment (IQA) is a method to measure data similarity by analyzing relevant features of images, including subjective methods and objective methods. Subjective methods are directly performed by humans, which is simple and direct, but takes a long time and is costly. Objective methods can automatically calculate data indicators and are therefore widely used. Due to the inherent imaging mode and speckle noise of SAR, the similarity evaluation method of traditional optical images cannot be applied to SAR images. Existing objective evaluation methods for the quality of simulated SAR images are mostly based on image or feature evaluation, which evaluates certain indicators of simulated SAR images and real SAR images by quantitatively calculating them, such as applying statistical mean square error or correlation coefficient methods to contours, shadows, textures, structures, etc. to obtain similarity. This image or feature-based evaluation method can reflect the difference level between the two SAR images, but it cannot reflect the application ability of the simulated SAR images obtained by the deep generative model in tasks such as target recognition, and it is impossible to know its use effect. Moreover, in previous work, the real SAR images and simulated SAR images generated by different deep generative models were trained together. However, adding simulated SAR images to the training dataset will cause the number of training samples to change, so it is difficult to accurately determine whether the simulated SAR images have a positive impact on the change in recognition rate. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a quality assessment method, device, electronic device and computer-readable storage medium for deep generative model simulated SAR images.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for quality assessment of a deep generative model simulated SAR image, the method comprising:

[0008] Using multiple real SAR target slice data with known categories, the original sample sets of each category are obtained;

[0009] For each category, all sample data in the original sample set of the category are input into the preset deep generation model to obtain multiple simulated SAR image data with the same number of input data, which together constitute the simulated sample set of the category;

[0010] For each category, the sample data in the original sample set and the simulated sample set of the category are mixed according to different preset proportions, and the number of samples in each generated mixed sample set is kept the same as that of the original sample set of the category, so as to obtain multiple mixed sample sets of the category;

[0011] The original sample sets of all categories are combined into a total original sample set, the simulated sample sets of all categories are combined into a total simulated sample set, and for each preset ratio, the mixed sample sets of all categories under the preset ratio are combined into a total mixed sample set of the preset ratio;

[0012] Based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples, a mixed recognition rate curve corresponding to the preset classification model is obtained, and the mixed recognition rate curve is used to evaluate the quality of the simulated SAR image generated by the preset depth generation model.

[0013] In one embodiment of the present invention, the method of obtaining original sample sets of each category by using a plurality of real SAR target slice data of known categories includes:

[0014] For each category, all real SAR target slice data of the category are respectively subjected to complex data modulo calculation, and all the obtained real SAR target slice data after modulo calculation are combined into the original sample set of the category.

[0015] In one embodiment of the present invention, the preset depth generation model includes:

[0016] GAN model and its variant models.

[0017] In one embodiment of the present invention, for each category, the sample data in the original sample set and the simulated sample set of the category are mixed according to different preset ratios, and the number of samples in each generated mixed sample set is kept the same as that of the original sample set of the category, to obtain multiple mixed sample sets of the category, including:

[0018] For each category, the sample data in the original sample set of the category is reduced by 10% in turn, and the sample data in the simulated sample set of the category is used to supplement the missing samples each time, so as to obtain multiple mixed sample sets of the category.

[0019] In one embodiment of the present invention, the mixed recognition rate curve corresponding to the preset classification model is obtained based on the original sample set, the mixed sample set of each preset ratio and the simulated sample set, including:

[0020] The total set of original samples, the total set of mixed samples of each preset ratio, and the total set of simulated samples are respectively used as training sets of the preset classification model for model training, and after each training is completed, the preset test set is used to classify the SAR target to obtain the corresponding recognition rate;

[0021] The mixed recognition rate curve is obtained by using the proportion of the simulated SAR image data in the training set and the obtained recognition rate.

[0022] In one embodiment of the present invention, the hybrid recognition rate curve is obtained by using the proportion of the simulated SAR image data in the training set and the obtained recognition rate, including:

[0023] The mixed recognition rate curve is drawn using the proportion of the simulated SAR image data in the training set as the horizontal axis and the recognition rate as the vertical axis using the corresponding data values; wherein the proportion of the simulated SAR image data in the horizontal axis increases successively.

[0024] In one embodiment of the present invention, the step of evaluating the quality of the simulated SAR image generated by the preset depth generation model using the hybrid recognition rate curve includes:

[0025] Determine whether, in the mixed recognition rate curve, when the proportion of the simulated SAR image data is a preset simulated SAR image proportion, the corresponding recognition rate is greater than a preset recognition rate threshold; if so, evaluate whether the quality of the simulated SAR image generated by the preset depth generation model meets the requirements.

[0026] In a second aspect, an embodiment of the present invention provides a quality assessment device for a deep generative model simulated SAR image, the device comprising:

[0027] The original sample set generation module is used to obtain the original sample sets of each category by using a plurality of real SAR target slice data of known categories;

[0028] A simulation sample set generation module is used to input all sample data in the original sample set of each category into a preset depth generation model to obtain the same number of simulated SAR image data to form a simulation sample set of the category;

[0029] A mixed sample set generation module is used to mix the sample data in the original sample set and the simulated sample set of each category according to different preset proportions, and keep the number of samples in each generated mixed sample set the same as that of the original sample set of the category, so as to obtain multiple mixed sample sets of the category;

[0030] A sample total set generation module, used to combine original sample sets of all categories into an original sample total set, combine simulated sample sets of all categories into a simulated sample total set, and for each preset ratio, combine mixed sample sets of all categories under the preset ratio into a mixed sample total set of the preset ratio;

[0031] The quality assessment module is used to obtain a mixed recognition rate curve corresponding to a preset classification model based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples, and use the mixed recognition rate curve to evaluate the quality of the simulated SAR image generated by the preset depth generation model.

[0032] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to implement the steps of quality assessment of the deep generation model simulated SAR image provided by the embodiment of the present invention when executing the program stored in the memory.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of quality assessment of the deep generation model simulated SAR image provided by the embodiment of the present invention are implemented.

[0036] In the solution provided by the embodiment of the present invention, the sample data volume of the mixed sample sets of each preset ratio is the same as that of the original sample set and the simulated sample set. Therefore, the influence of the change in the sample data volume on the recognition rate can be eliminated, so as to simply obtain the influence of the change in the number of simulated SAR images in the sample set on the recognition rate. Therefore, the problem that it is difficult to accurately judge whether the change in the number of training data sets has a positive impact on the change in the recognition rate can be avoided. Moreover, the mixed recognition rate curve designed by the embodiment of the present invention can be displayed to intuitively reflect the above-mentioned degree of influence, which is convenient for personnel to observe. Moreover, the embodiment of the present invention associates the simulated SAR image with the recognition rate, which can be used to judge the application ability of the simulated SAR image in scenarios such as SAR target recognition, and realize its utility evaluation.

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic flow chart of a method for evaluating the quality of SAR images simulated by a deep generative model provided in an embodiment of the present invention;

[0039] Figure 2 An example diagram of multiple training sets provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of four real SAR vehicle slice data provided by an embodiment of the present invention;

[0041] Figure 4 The embodiment of the present invention is directed to Figure 3 ,Simulated SAR images generated using three preset depth generation models;

[0042] Figure 5A schematic diagram of a hybrid recognition rate curve provided by an embodiment of the present invention;

[0043] Figure 6 A schematic diagram of the structure of a quality assessment device for SAR image simulation using a deep generative model provided by an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In order to solve the problems existing in the prior art when performing quality assessment on simulated SAR images generated by a deep generative model, embodiments of the present invention provide a quality assessment method, device, electronic device and computer-readable storage medium for deep generative model simulated SAR images.

[0047] It should be noted that the execution subject of the quality assessment method of the deep generative model simulation SAR image provided in the embodiment of the present invention can be a quality assessment device of the deep generative model simulation SAR image, and the device can be run in an electronic device. Among them, the electronic device can be a server or a terminal device, of course, but it is not limited to this.

[0048] In a first aspect, an embodiment of the present invention provides a method for evaluating the quality of SAR images simulated by a deep generative model. Figure 1 As shown, the following steps may be included:

[0049] S1, using multiple real SAR target slice data of known categories, obtain the original sample set of each category.

[0050] In an embodiment of the present invention, a real SAR image is obtained from the echo data of a SAR system. A real SAR image may include one or more targets. The target category may be a ship, a tank, a car, a building, etc. For example, in an optional implementation, the real SAR target slice data is vehicle target data. Each real SAR target slice data comes from a real SAR image and is the data corresponding to a single target in a real SAR image.

[0051] The process of obtaining a real SAR target slice data from a real SAR image may include:

[0052] For this real SAR image, according to the marking information of each target therein, the target slice image corresponding to each target is extracted, that is, the real SAR target slice data corresponding to each target is obtained.

[0053] The target marking information may be the position information of the target in the real SAR image, such as the coordinate value of the rectangular frame containing the target. When extracting the real SAR target slice data corresponding to the target, it may be marked according to its known category. In addition, the size of all target slice images may be set to a consistent size to ensure that the data dimensions of each real SAR target slice data are the same.

[0054] In the embodiment of the present invention, for each category, a plurality of real SAR target slice data can be obtained, and the number of real SAR target slice data of each category may be different.

[0055] Those skilled in the art can understand that the target slice data is complex data. To facilitate subsequent data processing, S1 may specifically include the following steps:

[0056] For each category, all real SAR target slice data of the category are respectively subjected to complex data modulo calculation, and all the obtained real SAR target slice data after modulo calculation are combined into the original sample set of the category.

[0057] Among them, the complex data modulus calculation belongs to the existing technology and will not be described in detail here.

[0058] For ease of understanding, the original sample set of category i can be represented as X i , where i is a natural number representing the category.

[0059] S2, for each category, all sample data in the original sample set of the category are input into a preset deep generation model to obtain a plurality of simulated SAR image data with the same number of input data, which together constitute a simulated sample set of the category.

[0060] In an embodiment of the present invention, the preset deep generative model may be any existing deep generative model, such as VAE.

[0061] For category i, its original sample set X i As input data, the preset depth generation model is input, and the multiple simulated SAR image data output constitute a simulated sample set of category i, which can be expressed as Y i In addition, in the embodiment of the present invention, in order to ensure that the number of training samples used by the preset classification model in subsequent training is the same each time, so as to evaluate the recognition rate under the same standard, X i and Y iThe number of sample data is the same.

[0062] To facilitate understanding of the solution, the full text uses three categories as examples. It is assumed that the embodiment of the present invention uses X1 (containing 100 sample data) to obtain Y1 (containing 100 sample data); uses X2 (containing 200 sample data) to obtain Y2 (containing 200 sample data); and uses X3 (containing 300 sample data) to obtain Y3 (containing 300 sample data).

[0063] S3, for each category, the sample data in the original sample set and the simulated sample set of the category are mixed according to different preset proportions, and the number of samples in each generated mixed sample set is kept the same as that of the original sample set of the category, so as to obtain multiple mixed sample sets of the category.

[0064] In this step, multiple sample mixing is performed for category i (i = 1, 2, 3). The jth sample mixing of category i is to select a j sample data, and select b from the simulation sample set of category i j sample data, the number of sample data in the two parts is greater than a j b j is a preset ratio S j , where j is a natural number greater than 0 indicating the number of times. The two parts of sample data are mixed to obtain the j-th mixed sample set of category i, which can be expressed as X ij , but each time the mixed sample set X ij , in terms of the number of sample data, it is still the same as the original sample set X of this category i And the simulation sample set Y of this category i same.

[0065] For example, the mixed sample set obtained by category 1 is X 11 ,X 12 ,...,X 1j ; The mixed sample set obtained by category 2 is X 21 ,X 22 ,...,X 2j ; The mixed sample set obtained by category 3 is X 31 ,X 32 ,...,X 3j .

[0066] For category i, multiple sample mixtures correspond to the preset ratio S j It can be 1 / 2, 1 / 3, etc. In order to facilitate subsequent processing, for different categories, the preset ratio S of the jth sample mixture is j are the same.

[0067] In order to facilitate the subsequent display of the preset ratio S j The relationship between the changing trend and the recognition rate can make the preset ratio of multiple sample mixtures change in an increasing or decreasing manner. For example, in an optional implementation, for each category, the sample data in the original sample set and the simulated sample set of the category are mixed according to different preset ratios, and the number of samples in each generated mixed sample set is kept the same as that of the original sample set of the category, to obtain multiple mixed sample sets of the category, including:

[0068] For each category, the sample data in the original sample set of the category is reduced by 10% in turn, and the sample data in the simulated sample set of the category is used to supplement the missing samples each time, so as to obtain multiple mixed sample sets of the category.

[0069] For example, for each mixed sample set obtained in each category, the preset proportion S of sample data belonging to the original sample set and the simulated sample set is j They are 9 / 1, 8 / 2, 7 / 3, …, 1 / 9, and a total of 9 mixed sample sets are obtained.

[0070] Of course, the ratio of 10% in the above implementation may also be changed to a value such as 20%. In this case, it is understandable that the number of the obtained mixed sample sets will change.

[0071] S4, the original sample sets of all categories are combined into a total original sample set, the simulated sample sets of all categories are combined into a total simulated sample set, and for each preset ratio, the mixed sample sets of all categories under the preset ratio are combined into a total mixed sample set of the preset ratio.

[0072] For the above example, the original sample set X1 of category 1, the original sample set X2 of category 2, and the original sample set X3 of category 3 are combined into the original sample set X′. The simulated sample set Y1 of category 1, the simulated sample set Y2 of category 2, and the simulated sample set Y3 of category 3 are combined into the simulated sample set Y′. For the preset ratio S1=9 / 1, the mixed sample set X1 of category 1 is combined into the simulated sample set Y′. 11 , mixed sample set X of category 2 21 and the mixed sample set X of category 3 31 The total set X′ of mixed samples with a preset ratio S1 i1 , ... Similarly, for the preset ratio S9 = 1 / 9, the mixed sample set X of category 1 19 , mixed sample set X of category 2 29 and the mixed sample set X of category 3 39 The total set X′ of mixed samples constituting the preset ratio S9 i9 .

[0073] That is to say, for the above example, the obtained sample sets include: the original sample set X′, the mixed sample set X′ with a preset ratio S1 i1 , the total set of mixed samples X′ with a preset ratio S2 i2 , ..., the total set of mixed samples X′ with a preset ratio S9 i9 , and the total set of simulation samples Y′, a total of 11 sample sets, where the number of samples in each sample set is the same, which is 600 data.

[0074] S5, based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples, obtain a mixed recognition rate curve corresponding to the preset classification model, and use the mixed recognition rate curve to evaluate the quality of the simulated SAR image generated by the preset depth generation model.

[0075] Specifically, this step may include:

[0076] S51, the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples are respectively used as training sets of the preset classification model for model training, and after each training is completed, the preset test set is used to classify the SAR target to obtain the corresponding recognition rate.

[0077] The sample data in the preset test set and training set are similar. Each training set is iterated multiple times, and the optimal recognition rate is retained as the result of this step. The Epoch can be set to 100, etc.

[0078] See also Figure 2 , Figure 2 An example diagram of multiple training sets provided by an embodiment of the present invention. A square represents a training set, where the white part is the image in the original sample set and the gray part is the simulated SAR image. It can be understood that Figure 2 From left to right, they are X', X' i1 To X′ i9 , and Y′.

[0079] S52, obtaining the mixed recognition rate curve by using the proportion of the simulated SAR image data in the training set and the obtained recognition rate.

[0080] In the embodiment of the present invention, a curve of the relationship between the proportion of the simulated SAR image data in the training set and the obtained recognition rate is defined as a hybrid recognition rate curve (HRR).

[0081] In an optional implementation manner, S52 may include:

[0082] The mixed recognition rate curve is drawn using the proportion of the simulated SAR image data in the training set as the horizontal axis and the recognition rate as the vertical axis using the corresponding data values; wherein the proportion of the simulated SAR image data in the horizontal axis increases successively.

[0083] It can be understood that in this implementation, since the training set is the original sample set X′, the mixed sample set X′ with a preset ratio S1, i1 , the total set of mixed samples X′ with a preset ratio S2 i2 , ..., the total set of mixed samples X′ with a preset ratio S9 i9 , and the total set of simulated samples Y′, the proportion of the simulated SAR image data in each training set is 0%, 10%, 20%, 30%, ..., 100% in turn. These values ​​are distributed on the horizontal axis in sequence. The recognition rate values ​​obtained by training the above training sets for SAR target classification are distributed on the vertical axis in order from small to large. The horizontal and vertical axis data of the same training set can be plotted as a scatter point, and the scatter points of each training set are plotted into a curve to obtain a mixed recognition rate curve.

[0084] Of course, it is also feasible to use the proportion of the simulated SAR image data in the training set as the vertical axis and the recognition rate as the horizontal axis, and to draw the mixed recognition rate curve using corresponding data values.

[0085] After the hybrid recognition rate curve is obtained, it can be used to evaluate the quality of the simulated SAR image generated by the preset depth generation model. For example, the hybrid recognition rate curves corresponding to multiple preset depth generation models can be obtained at the same time, the same horizontal axis value can be determined, and the size of the vertical axis value can be analyzed. It can be understood that the recognition rate when the simulated SAR image is not added is the recognition rate of the original sample set. At this time, if the recognition rate in a hybrid recognition rate curve is closer to the recognition rate of the original sample set, that is, the curve fluctuates less, it means that the quality of the simulated SAR image generated by the corresponding preset depth generation model is better, and the applicability and accuracy when used for SAR target classification are stronger.

[0086] In an optional implementation manner, the using the hybrid recognition rate curve to evaluate the quality of the simulated SAR image generated by the preset depth generation model may include:

[0087] Determine whether, in the mixed recognition rate curve, when the proportion of the simulated SAR image data is a preset simulated SAR image proportion, the corresponding recognition rate is greater than a preset recognition rate threshold; if so, evaluate whether the quality of the simulated SAR image generated by the preset depth generation model meets the requirements.

[0088] For example, through a large number of sample experiments in advance, a preset simulated SAR image ratio can be determined, such as 50%, and a preset recognition rate threshold for the ratio can be determined, such as 0.85. Then, if in the mixed recognition rate curve, the simulated SAR image data accounts for 50% and the corresponding recognition rate is greater than 0.85, then it is evaluated that the quality of the simulated SAR image generated by the preset deep generation model meets the requirements, indicating that it can obtain a higher classification accuracy when applied to SAR target classification and meet the application requirements.

[0089] Of course, the method of using the hybrid recognition rate curve to evaluate the quality of the simulated SAR image generated by the preset depth generation model can also be to determine that when the proportion of the simulated SAR image data is within a preset interval, the corresponding recognition rate is within a preset recognition rate threshold interval. If so, it is evaluated that the quality of the simulated SAR image generated by the preset depth generation model meets the requirements, and so on.

[0090] In an embodiment of the present invention, if it is evaluated that the quality of the simulated SAR image generated by the preset depth generation model meets the requirements, the preset depth generation model can be used to generate more simulated SAR images to expand the training set. If it is evaluated that the quality of the simulated SAR image generated by the preset depth generation model does not meet the requirements, the preset depth generation model and / or the preset classification model can be adjusted accordingly to re-evaluate, etc.

[0091] In order to evaluate the performance of simulated SAR images in SAR recognition tasks, simulated SAR images are usually used as training samples to expand the real SAR training data set. In the previous prior art, real SAR images and simulated SAR images generated by different models were trained together. However, adding simulated SAR images to the training data set will cause the number of training samples to change. Therefore, it is difficult to determine whether the change in recognition rate is caused by the addition of simulated SAR images or the increase in the total number of training samples. It is therefore difficult to accurately determine whether the simulated SAR images have a positive impact on the change in recognition rate.

[0092] In the solution provided by the embodiment of the present invention, the sample data volume of the mixed sample sets of each preset ratio is the same as that of the original sample set and the simulated sample set. Therefore, the influence of the change in the sample data volume on the recognition rate can be eliminated, so as to simply obtain the influence of the change in the number of simulated SAR images in the sample set on the recognition rate. Therefore, the problem that it is difficult to accurately judge whether the change in the number of training data sets has a positive impact on the change in the recognition rate can be avoided. Moreover, the mixed recognition rate curve designed by the embodiment of the present invention can be displayed to intuitively reflect the above-mentioned degree of influence, which is convenient for personnel to observe. Moreover, the embodiment of the present invention associates the simulated SAR image with the recognition rate, which can be used to judge the application ability of the simulated SAR image in scenarios such as SAR target recognition, and realize its utility evaluation.

[0093] In an optional implementation manner, the preset depth generation model includes:

[0094] GAN (Generative Adversarial Networks) model and its variant models.

[0095] GAN is a framework for estimating generative models through an adversarial process. It trains two models: the generative model G captures the data distribution, and the discriminative model D estimates the probability that the sample comes from the training data instead of G. The training process of G is to maximize the probability that D is wrong. There are currently some variant models of GAN, including DCGAN (Deep Convolutional Generative Adversarial Networks), WGAN (Wasserstein GAN), WGAN-GP (WGAN with gradient penalty), etc. Among them, DCGAN, as the architectural topology of convolutional GAN, has made some modifications to GAN so that it can be stably trained in most settings. WGAN improves the stability of learning and eliminates problems such as mode collapse. However, WGAN sometimes still only generates low-quality samples or fails to converge. WGAN-GP proposes an alternative method to clip weights, which is to penalize the gradient norm of the discriminator and its input.

[0096] In order to illustrate the effectiveness of the method of the embodiment of the present invention, the following is an illustration based on experimental data.

[0097] 1. Experimental conditions:

[0098] The method of the embodiment of the invention is verified using the measured SAR ground stationary target data published by the public MSTAR program. Most of the data sets are SAR slice images of stationary vehicles, including target images of various vehicle targets acquired at various azimuth angles.

[0099] The embodiment of the invention uses slice data of multiple real SAR targets, including BMP2 (infantry fighting vehicle), T72 (tank) and mixed target data BTR60 (armored transport vehicle), T62 (tank), to form a data set for four types of target classification problems for verification. Among them, BMP2 (infantry fighting vehicle) and T72 (tank) have different models, including BMP2_SN9563, BMP2_SN9566, BMP2SN_c21, T72_SN132, T72_SN812, and T72_SNs7.

[0100] The training set is constructed using the images of the target obtained at a pitch angle of 17°, with a total of 1020 images, and the test set is constructed using the images of the target obtained at a pitch angle of 15°, with a total of 1246 images. For different models of BMP2 (infantry fighting vehicle) and T72 (tank), BMP_SN9563 and T72_SN132 are selected as training sets, and BMP2_SN9566, BMP2SN_c21, T72_SN812 and T72_SNs7 are selected as test sets, and finally four types of target data sets including training sets (containing 1020 images) and test sets (containing 1246 images) are obtained. Each image set is a stationary vehicle image with a pixel size of 128×128 obtained after processing the data collected by the radar.

[0101] The DCGAN, WGAN and WGAN-GP algorithms are used to build preset deep generative models respectively to verify the proposed method. The experimental operation system is Intel(R) Core(TM) i5-10400FCPU@2.90GHz and NVIDIA GeForce GTX1660SUPER, 64-bit Windows 10 operating system, and the simulation software uses Python 3.6.

[0102] 2. Experimental content and results analysis:

[0103] The embodiment of the present invention focuses on solving the quality assessment problem of SAR image simulation using a deep generative model, and compares the result with FID (Fréchet Inception Distance score, FID score) and subjective visual judgment to verify the feasibility.

[0104] The visualization results of the real image and the simulated image in the embodiment of the present invention are as follows: Figure 3 and Figure 4 shown. Figure 3 A schematic diagram of four real SAR vehicle slice data provided by an embodiment of the present invention; Figure 4 The embodiment of the present invention is directed to Figure 3 ,Simulated SAR images generated using three preset depth generation models; Figure 4 The columns in Figure 3 The columns in correspond to, Figure 4 In each column, the three images from top to bottom are obtained by the preset deep generative models constructed by DCGAN, WGAN, and WGAN-GP algorithms.

[0105] from Figure 4 The following conclusions can be drawn: The WGAN-GP and DCGAN simulation datasets are consistent with the real images in terms of shadow and scatter center locations, and have good visual similarity. The images simulated using WGAN are similar to the filtered images, and clutter and speckle noise are suppressed.

[0106] The FID results are shown in Table 1. The four-category FID scores of the images generated by WGAN-GP are lower than those generated by other models, which means that the WGAN-GP simulation dataset is closer to the real dataset.

[0107] Using the three mixed training sets obtained by DCGAN, WGAN and WGAN-GP, we get the average mixed recognition rate curves of the four types of samples, as shown in Figure 5 As shown. Among them, the horizontal axis Fake_persentage represents the proportion of simulated SAR images in the training set; the vertical axis Accuracy represents the recognition rate. The experimental results show that:

[0108] 1) The results of WGAN and DCGAN are almost the same.

[0109] 2) When the percentage of simulated images in the training set is less than 50%, the accuracy fluctuates less and the recognition rate values ​​are all greater than 90%, which proves the effectiveness of the three simulated data sets and that simulated images can be used to expand the data set.

[0110] 3) WGAN-GP has the highest fitting accuracy, indicating that the fitting performance of WGAN-GP is better than that of WGAN and DCGAN, which is consistent with the above experimental analysis results.

[0111] Table 1 FID values ​​of three simulation data sets

[0112]

[0113] In summary, in the embodiments of the present invention, for the mixed recognition rate curve, the recognition rate in the curve can be compared with the recognition rate of the total set of original samples. If the numerical fluctuation is smaller and closer, it means that the simulated image generated by the preset deep generation model is closer to the real image, which means that the application ability of the simulated image is stronger.

[0114] In a second aspect, corresponding to the above method embodiment, the embodiment of the present invention further provides a quality assessment device for SAR image simulated by a deep generative model, such as Figure 6As shown, the device comprises:

[0115] The original sample set generation module 601 is used to obtain original sample sets of each category by using a plurality of real SAR target slice data of known categories;

[0116] The simulation sample set generation module 602 is used to input all sample data in the original sample set of each category into a preset depth generation model to obtain the same number of simulated SAR image data to form a simulation sample set of the category;

[0117] A mixed sample set generation module 603 is used to mix the sample data in the original sample set and the simulated sample set of each category according to different preset ratios, and keep the number of samples in each generated mixed sample set the same as that of the original sample set of the category, so as to obtain multiple mixed sample sets of the category;

[0118] The sample set generation module 604 is used to combine the original sample sets of all categories into the original sample set, combine the simulated sample sets of all categories into the simulated sample set, and for each preset ratio, combine the mixed sample sets of all categories under the preset ratio into the mixed sample set of the preset ratio;

[0119] The quality assessment module 605 is used to obtain a mixed recognition rate curve corresponding to a preset classification model based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples, and use the mixed recognition rate curve to evaluate the quality of the simulated SAR image generated by the preset depth generation model.

[0120] Optionally, the original sample set generating module 601 is specifically configured to:

[0121] For each category, all real SAR target slice data of the category are respectively subjected to complex data modulo calculation, and all the obtained real SAR target slice data after modulo calculation are combined into the original sample set of the category.

[0122] Optionally, the preset depth generation model includes:

[0123] GAN model and its variant models.

[0124] Optionally, the mixed sample set generating module 603 is specifically used to:

[0125] For each category, the sample data in the original sample set of the category is reduced by 10% in turn, and the sample data in the simulated sample set of the category is used to supplement the missing samples each time, so as to obtain multiple mixed sample sets of the category.

[0126] Optionally, when the quality assessment module 605 is used to obtain a mixed recognition rate curve corresponding to a preset classification model based on the total set of original samples, the total set of mixed samples of each preset ratio, and the total set of simulated samples, it specifically includes:

[0127] The training and testing submodule is used to use the total set of original samples, the total set of mixed samples with preset proportions, and the total set of simulated samples as training sets of the preset classification model for model training, and after each training is completed, use the preset test set to classify the SAR target to obtain the corresponding recognition rate;

[0128] The curve generation submodule is used to obtain the mixed recognition rate curve by using the proportion of the simulated SAR image data in the training set and the obtained recognition rate.

[0129] Optionally, the curve generating submodule is specifically used for:

[0130] The mixed recognition rate curve is drawn using the proportion of the simulated SAR image data in the training set as the horizontal axis and the recognition rate as the vertical axis using the corresponding data values; wherein the proportion of the simulated SAR image data in the horizontal axis increases successively.

[0131] Optionally, when the quality assessment module 605 is used to assess the quality of the simulated SAR image generated by the preset depth generation model using the hybrid recognition rate curve, it is specifically used to:

[0132] Determine whether, in the mixed recognition rate curve, when the proportion of the simulated SAR image data is a preset simulated SAR image proportion, the corresponding recognition rate is greater than a preset recognition rate threshold; if so, evaluate whether the quality of the simulated SAR image generated by the preset depth generation model meets the requirements.

[0133] For specific details, please refer to the quality assessment method of deep generative model simulated SAR images described in the first aspect, which will not be elaborated here.

[0134] In the solution provided by the embodiment of the present invention, the sample data volume of the mixed sample sets of each preset ratio is the same as that of the original sample set and the simulated sample set. Therefore, the influence of the change in the sample data volume on the recognition rate can be eliminated, so as to simply obtain the influence of the change in the number of simulated SAR images in the sample set on the recognition rate. Therefore, the problem that it is difficult to accurately judge whether the change in the number of training data sets has a positive impact on the change in the recognition rate can be avoided. Moreover, the mixed recognition rate curve designed by the embodiment of the present invention can be displayed to intuitively reflect the above-mentioned degree of influence, which is convenient for personnel to observe. Moreover, the embodiment of the present invention associates the simulated SAR image with the recognition rate, which can be used to judge the application ability of the simulated SAR image in scenarios such as SAR target recognition, and realize its utility evaluation.

[0135] In a third aspect, an embodiment of the present invention further provides an electronic device 700, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0136] Memory 703, used for storing computer programs;

[0137] The processor 701 is configured to implement the steps of quality assessment of the deep generative model simulated SAR image as described in the first aspect when executing the program stored in the memory 703.

[0138] The electronic device may be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0139] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0140] The communication interface is used for communication between the above electronic device and other devices.

[0141] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0142] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0143] In a fourth aspect, corresponding to the quality assessment method for SAR images simulated by a deep generative model provided in the first aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the quality assessment method for SAR images simulated by a deep generative model provided in the embodiment of the present invention are implemented.

[0144] As for the device / electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] It should be noted that the device, electronic device and storage medium of the embodiments of the present invention are respectively a device, electronic device and storage medium that apply the above-mentioned quality assessment method of deep generative model simulation SAR image, then all embodiments of the above-mentioned quality assessment method of deep generative model simulation SAR image are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0146] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically limited.

[0147] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0148] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A quality assessment method for SAR images simulated by a deep generative model, characterized in that: include: Using multiple real SAR target slice data with known categories, the original sample sets of each category are obtained; For each category, all sample data in the original sample set of the category are input into the preset deep generation model to obtain multiple simulated SAR image data with the same number of input data, which together constitute the simulated sample set of the category; For each category, the sample data in the original sample set and the simulated sample set of the category are mixed according to different preset proportions, and the number of samples in each generated mixed sample set is kept the same as that of the original sample set of the category, so as to obtain multiple mixed sample sets of the category; The original sample sets of all categories are combined into a total original sample set, the simulated sample sets of all categories are combined into a total simulated sample set, and for each preset ratio, the mixed sample sets of all categories under the preset ratio are combined into a total mixed sample set of the preset ratio; Based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples, a mixed recognition rate curve corresponding to a preset classification model is obtained, and the quality of the simulated SAR image generated by the preset depth generation model is evaluated using the mixed recognition rate curve; Wherein, the mixed recognition rate curve corresponding to the preset classification model is obtained based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulation samples, including: The total set of original samples, the total set of mixed samples of each preset ratio, and the total set of simulated samples are respectively used as training sets of the preset classification model for model training, and after each training is completed, the preset test set is used to classify the SAR target to obtain the corresponding recognition rate; The mixed recognition rate curve is obtained by using the proportion of the simulated SAR image data in the training set and the obtained recognition rate.

2. The quality assessment method of the deep generative model simulation SAR image according to claim 1 is characterized in that: The method of using a plurality of real SAR target slice data of known categories to obtain original sample sets of each category includes: For each category, all real SAR target slice data of the category are respectively subjected to complex data modulo calculation, and all the obtained real SAR target slice data after modulo calculation are combined into the original sample set of the category.

3. The quality assessment method of the deep generative model simulation SAR image according to claim 1 is characterized in that: The preset depth generation model includes: GAN model and its variant models.

4. The quality assessment method of the deep generative model simulation SAR image according to claim 1 is characterized in that: For each category, the sample data in the original sample set and the simulated sample set of the category are mixed according to different preset ratios, and the number of samples in each generated mixed sample set is kept the same as that of the original sample set of the category, to obtain multiple mixed sample sets of the category, including: For each category, the sample data in the original sample set of the category is reduced by 10% in turn, and the sample data in the simulated sample set of the category is used to supplement the missing samples each time, so as to obtain multiple mixed sample sets of the category.

5. The quality assessment method of the deep generative model simulation SAR image according to claim 1 is characterized in that: The hybrid recognition rate curve is obtained by using the proportion of the simulated SAR image data in the training set and the obtained recognition rate, including: The mixed recognition rate curve is drawn using the proportion of the simulated SAR image data in the training set as the horizontal axis and the recognition rate as the vertical axis using the corresponding data values; wherein the proportion of the simulated SAR image data in the horizontal axis increases successively.

6. The quality assessment method of deep generative model simulation SAR image according to claim 5 is characterized in that: The method of evaluating the quality of the simulated SAR image generated by the preset depth generation model by using the hybrid recognition rate curve includes: Determine whether, in the mixed recognition rate curve, when the proportion of the simulated SAR image data is a preset simulated SAR image proportion, the corresponding recognition rate is greater than a preset recognition rate threshold; if so, evaluate whether the quality of the simulated SAR image generated by the preset depth generation model meets the requirements.

7. A quality assessment device for deep generative model simulation SAR images, characterized in that: include: The original sample set generation module is used to obtain the original sample sets of each category by using a plurality of real SAR target slice data of known categories; A simulation sample set generation module is used to input all sample data in the original sample set of each category into a preset depth generation model to obtain the same number of simulated SAR image data to form a simulation sample set of the category; A mixed sample set generation module is used to mix the sample data in the original sample set and the simulated sample set of each category according to different preset proportions, and keep the number of samples in each generated mixed sample set the same as that of the original sample set of the category, so as to obtain multiple mixed sample sets of the category; A sample total set generation module, used to combine original sample sets of all categories into an original sample total set, combine simulated sample sets of all categories into a simulated sample total set, and for each preset ratio, combine mixed sample sets of all categories under the preset ratio into a mixed sample total set of the preset ratio; A quality assessment module is used to obtain a mixed recognition rate curve corresponding to a preset classification model based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples, and use the mixed recognition rate curve to evaluate the quality of the simulated SAR image generated by the preset depth generation model; wherein the quality assessment module obtains the mixed recognition rate curve corresponding to the preset classification model based on the total set of original samples, the total set of mixed samples of each preset ratio and the total set of simulated samples, and is specifically used to: The total set of original samples, the total set of mixed samples of each preset ratio, and the total set of simulated samples are respectively used as training sets of the preset classification model for model training, and after each training is completed, the preset test set is used to classify the SAR target to obtain the corresponding recognition rate; The mixed recognition rate curve is obtained by using the proportion of the simulated SAR image data in the training set and the obtained recognition rate.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images