Remote sensing image target detection sample amplification method based on context background enhancement style migration network

Abstract

The invention provides a remote sensing image target detection sample amplification method based on a context background enhancement style migration network, and the method comprises the steps: carrying out the preprocessing of obtained original remote sensing data, and obtaining a to-be-processed image which comprises the position information of a to-be-amplified target; generating an annotation box according to the position information, randomly determining a target annotation box, and generating a noise box in the target annotation box; training a preset generative adversarial network model based on the to-be-processed image including the noise box and the to-be-processed image to obtain a target generative adversarial network model; inputting the to-be-processed image including the noise box into the target generative adversarial network model, and outputting a generated image including the to-be-amplified target; wherein the position information of the target to be amplified is the same as that of the noise frame. The generated image obtained by the target detection sample amplification method provided by the scheme has no sample blurring, unnatural generated background and other problems, and known information in the original remote sensing data is fully utilized to guide the background in the target noise frame to recover and obtain a more real amplification target.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and recognition, and in particular to a remote sensing image target detection sample amplification method based on a contextual background enhanced style transfer network. Background Art

[0002] Remote sensing image target detection is a technology that accurately determines the category and coordinate position of the target of interest by analyzing the appearance, shape, posture and other features of the objects in the remote sensing image. It is widely used at the national military and civilian levels.

[0003] With the rapid development of deep learning, target detection algorithms based on deep convolutional networks have become the mainstream method in the detection field. For the training of neural network models, a sufficient number of training samples is a necessary prerequisite and foundation, and the spatial distribution of sample data also plays a key role in the robustness and generalization ability of the model. However, in practical applications and remote sensing image backgrounds, due to factors such as the high cost of acquiring sample data, the presence of a large number of repetitions, or the low probability of occurrence of the required samples, the number of useful sample data is extremely limited, resulting in the so-called "big data, small sample" problem.

[0004] Data augmentation methods can make up for the problem of insufficient sample quantity, but traditional sample augmentation methods are oriented to images. By rotating, color-dithering, blurring, and scaling the entire image, the focus is more on increasing the amount of data in the entire data set and performing some data enhancement. Deep learning-based methods have great advantages, and can directly generate images of small sample targets to achieve inter-class balance. However, due to the scale effect, camouflage occlusion, complex background, weak target and other characteristics of the samples, the generated samples will eventually be blurred and the generated background will be unnatural, thus affecting the detection results of the detection algorithm. Summary of the invention

[0005] The present invention provides a remote sensing image target detection sample augmentation method based on context background enhancement style transfer network, which makes full use of known information in the original remote sensing data to guide the recovery of the background in the target noise frame and obtain a more realistic augmented target.

[0006] In a first aspect, the present invention provides a method for sample augmentation of remote sensing image target detection based on a contextual background enhanced style transfer network, comprising:

[0007] Preprocessing the acquired original remote sensing data to obtain an image to be processed including position information of the target to be amplified;

[0008] Generate a labeling frame according to the position information, randomly determine a target labeling frame, and generate a noise frame in the target labeling frame;

[0009] Training a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model;

[0010] The image to be processed including the noise frame is input into the target generative adversarial network model, and a generated image including the target to be augmented is output; wherein the position information of the target to be augmented is the same as that of the noise frame.

[0011] Optionally, generating a labeling box according to the position information, randomly determining a target labeling box, and generating a noise box in the target labeling box includes:

[0012] A labeling frame is generated based on the position information, at least one target labeling frame is randomly determined in the labeling frame, and a noise frame is generated in the target labeling frame to obtain an image to be processed including the noise frame.

[0013] Optionally, the preset generative adversarial network model includes a generator network and a discriminator network; the generator network includes a basic generator and a background enhancement module; the discriminator network includes a background restoration discriminator and a sample generation discriminator.

[0014] Optionally, the background enhancement module is used to perform the following operations:

[0015] Performing a convolution operation on the image to be processed to obtain a first image;

[0016] Cropping the first image based on the target annotation frame to obtain a sample image located within the target annotation frame and a background image located outside the target annotation frame;

[0017] Cutting and reorganizing the background image to obtain a background reconstruction convolution kernel;

[0018] The sample image is convolved using the background reconstruction convolution kernel, and then background enhancement information of the target annotation box is obtained through an activation function.

[0019] Optionally, the generator network is used to perform the following operations:

[0020] Inputting the image to be processed including the noise frame into the basic generator, and performing three downsampling processes to obtain a second image;

[0021] fusing the second image and the background enhancement information to obtain a third image;

[0022] The third image is subjected to three upsampling processes to obtain the generated image.

[0023] Optionally, it also includes:

[0024] Cropping the image to be processed and the generated image based on the target annotation frame to obtain an original target image and an augmented target image respectively;

[0025] Input the original target image and the augmented target image into the sample generation discriminator, and after passing through the convolution layer and the weighted spatial pyramid pooling layer, output a sample discrimination result; wherein the sample discrimination result includes whether the augmented target is true or the augmented target is false;

[0026] The image to be processed and the image to be processed including the noise frame are input into the background restoration discriminator, and a background discrimination result is output; the generated image and the image to be processed including the noise frame are input into the background restoration discriminator, and a background discrimination result is output; wherein the background discrimination result is that the background information is true or the background information is false.

[0027] Optionally, the weighted spatial pyramid pooling layer calculates a pooling value by the following formula:

[0028]

[0029] Among them, x i is used to represent the pixel value of the i-th pixel in the current pooling area; R is used to represent the set of all pixels in the current pooling area; w i Used to represent the weighted value corresponding to the i-th pixel in the current pooling area; Used to represent the pooling value.

[0030] Optionally, the training of a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model includes:

[0031] Inputting the image to be processed including the noise frame into the generator network to obtain the generated image;

[0032] Inputting the generated image into the discriminator network for back propagation, and obtaining the optimal network model parameters through iterative optimization;

[0033] The optimal network model parameters are loaded into the preset generative adversarial network model to obtain the target generative adversarial network model.

[0034] In a second aspect, the present invention provides a remote sensing image target detection sample amplification device based on a contextual background enhanced style transfer network, comprising:

[0035] A preprocessing module is used to preprocess the acquired original remote sensing data to obtain an image to be processed including the position information of the target to be amplified; generate a labeling frame according to the position information, randomly determine the target labeling frame, and generate a noise frame in the target labeling frame;

[0036] A training module, used for training a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model;

[0037] An augmentation module is used to input the image to be processed including the noise frame into the target generative adversarial network model, and output a generated image including the target to be augmented; wherein the position information of the target to be augmented is the same as that of the noise frame.

[0038] In a third aspect, an embodiment of the present invention further provides a computing device, comprising a memory and a target processor, wherein the memory stores a computer program, and when the target processor executes the computer program, the method described in any first aspect of this specification is implemented.

[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method described in any first aspect of this specification.

[0040] The present invention provides a method for sample augmentation of remote sensing image target detection based on context background enhancement style transfer network, the method pre-processes the acquired original remote sensing data to obtain an image to be processed with known position information of the target to be augmented, then generates a labeling frame based on the position information, and randomly selects the labeling frame as the target labeling frame, generates a noise frame in the target labeling frame, so that the image to be processed including the noise frame can be input into a preset generative adversarial network model, then generates a training set based on the image to be processed including the noise frame and the image to be processed, so as to train the preset generative adversarial network using the training set, and finally obtains a generated image including the target to be augmented through the target generative adversarial network model obtained by training. In this way, by making full use of the known information in the entire image to be processed to guide the recovery of the sample background in the noise frame, and at the same time generating a more realistic target to be augmented in the noise frame, the target detection accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of a method for sample augmentation of remote sensing image target detection based on contextual background enhanced style transfer network provided by one embodiment of the present invention;

[0043] Figure 2 is a structural diagram of a background enhancement module provided by an embodiment of the present invention;

[0044] Figure 3 is a structural diagram of a generator network provided by an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of the structure of a discriminator network provided by an embodiment of the present invention;

[0046] Figure 5 It is a detection prediction result diagram based on the yolov3 algorithm provided by an embodiment of the present invention;

[0047] Figure 6 It is a detection prediction result diagram based on the yolov3 algorithm provided by an embodiment of the present invention;

[0048] Figure 7 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention;

[0049] Figure 8 It is a structural schematic diagram of a remote sensing image target detection sample amplification device based on a contextual background enhanced style transfer network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, an embodiment of the present invention provides a method for sample augmentation of remote sensing image target detection based on a contextual background enhanced style transfer network, the method comprising:

[0052] Step 100, preprocessing the acquired original remote sensing data to obtain an image to be processed including position information of the target to be amplified;

[0053] Step 102, generating a labeling frame according to the position information, randomly determining a target labeling frame, and generating a noise frame in the target labeling frame;

[0054] Step 104, training a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model;

[0055] Step 106: input the image to be processed including the noise frame into the target generative adversarial network model, and output a generated image including the target to be augmented; wherein the position information of the target to be augmented is the same as that of the noise frame.

[0056] In the present invention, the acquired original remote sensing data is preprocessed to obtain an image to be processed with known position information of the target to be amplified, and then a labeling frame is generated based on the position information, and the labeling frame is randomly selected as the target labeling frame, and a noise frame is generated in the target labeling frame, so that the image to be processed including the noise frame can be input into a preset generative adversarial network model, and then a training set is generated based on the image to be processed including the noise frame and the image to be processed, so as to train the preset generative adversarial network using the training set, and finally a generated image including the target to be amplified can be obtained through the target generative adversarial network model obtained by training. In this way, by making full use of the known information in the entire image to be processed to guide the recovery of the sample background in the noise frame, a more realistic target to be amplified is generated in the noise frame, the number of samples is increased, and the accuracy of target detection is improved.

[0057] With respect to step 100, the acquired original remote sensing data is preprocessed to obtain an image to be processed including the position information of the target to be amplified, including: cropping the original remote sensing data to obtain a sub-remote sensing image; determining the target to be amplified and the position information of the target to be amplified in the sub-remote sensing image to obtain an image to be processed including the position information.

[0058] For step 102, a labeling box is generated according to the position information, a target labeling box is randomly determined, and a noise box is generated in the target labeling box, including: generating a labeling box based on the position information, randomly determining at least one target labeling box in the labeling box, and generating a noise box in the target labeling box, to obtain an image to be processed including the noise box.

[0059] Specifically, for example, obtain the open source remote sensing target detection source data set of the aircraft category data set in RSOD, and randomly crop it according to the size of 512*512 to obtain N original image sets, determine the location information of the target to be amplified in each original image set, generate a labeling box based on the location information bbox, randomly determine at least one target labeling box in the labeling box, and generate a Gaussian noise box (i.e., noise box) of the corresponding scale in the target labeling box, and then perform normalization preprocessing to obtain an image to be processed including a noise box, so that the image to be processed is in a format that can be input to the generator. It should be noted that the position of the target labeling box in the image to be processed of the noise box has no target information but noise information.

[0060] In step 104, the preset generative adversarial network model includes a generator network and a discriminator network; the generator network includes a basic generator and a background enhancement module; the discriminator network includes a background restoration discriminator and a sample generation discriminator.

[0061] In a preferred embodiment, the structure of the background enhancement module is as follows: Figure 2 As shown, the background enhancement module is used to perform the following operations:

[0062] Performing a convolution operation on the image to be processed to obtain a first image;

[0063] Cropping the first image based on the target annotation frame to obtain a sample image located within the target annotation frame and a background image located outside the target annotation frame;

[0064] Cut and reorganize the background image to obtain the background reconstruction convolution kernel;

[0065] The sample image is convolved using the background reconstruction convolution kernel, and then the background enhancement information of the target annotation box is obtained through the activation function.

[0066] Specifically, in the background enhancement module, two 3*3 convolutional layers are used to obtain the shallow feature information of the image to be processed, i.e., the first image. Then, according to the location information bbox (i.e., the target annotation box) where noise is added, the sample features are cropped out separately and resized to 64*64, denoted as Fo∈R B*64*64*64 , that is, the sample image; then use highlight processing outside the target annotation box to obtain the background image, recorded as Fb; then use a fixed cropping size (for example, 3*3) to crop the background image into small blocks, and then combine them together as the background reconstruction convolution kernel, recorded as bconv belonging to R B *128*64*3*3 (B is the batch size, where the background image is divided into 128 64*3*3 feature blocks.) The sample image is convolved with the background reconstruction convolution kernel, and then passed through the softmax function to obtain the background enhancement information of the target annotation box, denoted as Fe, Fe = softmax(bconv(Fb)).

[0067] In the present invention, the background enhancement module can retain as much background information of the target annotation box as possible, thereby solving the problem of unnatural generated background, further improving the authenticity of the generated image, compensating for the training balance problem of the preset generative adversarial network, and improving the training ability of the preset generative adversarial network model.

[0068] In a preferred embodiment, the structure of the generator network is as follows Figure 3 As shown, the generator network is used to perform the following operations:

[0069] Inputting the image to be processed including the noise frame into the basic generator, and performing three downsampling processes to obtain a second image;

[0070] fusing the second image with the background enhancement information to obtain a third image;

[0071] The third image is upsampled three times to obtain a generated image.

[0072] Specifically, as described in the previous example, the image to be processed including the noise frame is input into the basic generator, and the basic generator uses a U-net network, which includes three downsampling layers (downsamplei,i∈{1,2,3}) and three upsampling layers (upsamplei,i∈{1,2,3}); However, unlike the conventional U-net network, the second image F is obtained by downsampling in the last layer. min After that, the second image F min Channel splicing feature fusion (F fuse =conv(concat(F min ,Fe))), get the third image F fuse ; Then up-sample and restore the image to get the generated image; where conv represents the convolution operation.

[0073] In a preferred embodiment, in step 104, Figure 4 As shown, it also includes: cropping the to-be-processed image and the generated image based on the target annotation frame to obtain the original target image and the enlarged target image respectively;

[0074] The original target image and the augmented target image are input into the sample generation discriminator, and after passing through the convolution layer and the weighted spatial pyramid pooling layer, the sample discrimination result is output; wherein the sample discrimination result includes whether the augmented target is true or false;

[0075] The image to be processed and the image to be processed including the noise frame are input into the background restoration discriminator, and the background discrimination result is output; the generated image and the image to be processed including the noise frame are input into the background restoration discriminator, and the background discrimination result is output; wherein the background discrimination result is that the background information is true or the background information is false.

[0076] It should be noted that Figure 4 The real sample image pair in consists of the image to be processed and the image to be processed including the noise frame, the generated sample image pair consists of the generated image and the image to be processed including the noise frame, and Db is used to represent the background restoration discriminator.

[0077] Specifically, the construction of the sample generation discriminator is to determine whether the augmented target generated by the target annotation box tends to the real distribution. The sample generation discriminator consists of two parts. The first is the convolution layer for feature extraction of the original target image and the augmented target image, which consists of three 3*3 convolution layers. Figure 4 As shown in the figure, after the image features are obtained by the convolution layer, the weighted spatial pyramid pooling layer (i.e., weighted SPP) is used to weight the features and highlight the feature points with more useful information, thereby improving the authenticity of the amplified samples and solving the problem of sample ambiguity.

[0078] In the present invention, since the sizes of different target annotation frames are different, the sizes of the original target images or the amplified target images obtained by cutting out the different target annotation frames are also different. Therefore, it is unreasonable to directly use the fully connected layer to extract the image features. However, if the original target image or the amplified target image is resized during input, the features of the original target sample will be destroyed, which is not conducive to the confrontation between the discriminator and the generator. Therefore, in order to solve this problem, by using the spatial pyramid pooling layer, when facing features of different sizes, it can be adjusted to a fixed output, thereby ensuring the stability of the sample generation discriminator structure; however, for the conventional spatial pyramid pooling layer, when the generated image features are not obvious, it is not possible to obtain useful feature information well, thus resulting in poor training effect of the final basic generator. Therefore, the present invention cites the weighted spatial pyramid pooling layer to weight the features and highlight the feature points with more useful information.

[0079] Specifically, the background restoration discriminator used in the present invention is a fully convolutional structure of patchGAN similar to the discriminator of the pix2pix network.

[0080] In a more specific implementation, the weighted spatial pyramid pooling layer calculates the pooling value by the following formula:

[0081]

[0082] Among them, x i is used to represent the pixel value of the i-th pixel in the current pooling area; R is used to represent the set of all pixels in the current pooling area; w i Used to represent the weighted value corresponding to the i-th pixel in the current pooling area; Used to represent the pooling value.

[0083] In a preferred embodiment, a preset generative adversarial network model is trained based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model, including:

[0084] Input the image to be processed including the noise frame into the generator network to obtain a generated image;

[0085] The generated image is input into the discriminator network for back propagation, and the optimal network model parameters are obtained through iterative optimization;

[0086] The optimal network model parameters are loaded into the preset generative adversarial network model to obtain the target generative adversarial network model.

[0087] Specifically, a total loss function is constructed to optimize the preset generative adversarial network model. In this generative adversarial network, there are two adversarial processes, the base generator and the sample generation discriminator, and the base generator and the background restoration discriminator. In order to avoid the gradient vanishing caused by the cross entropy loss function of the original GAN ​​network, which leads to problems such as unstable training, the least squares loss is used for both adversarial processes; at the same time, in order to help the base generator learn better to generate more realistic images, the L1 norm content loss is also added to the generator loss. Specifically, the loss function of the background restoration discriminator is as follows:

[0088] L Db (G,D b )=E y～p(y) [(D b (y,z)-1) 2 ]+E x～p(x) [(D b (G(z),z)) 2 ]

[0089] Where z is the image to be processed including the noise frame; y represents the original image (i.e. the image to be processed); D b represents the background restoration discriminator; G(z) represents the generated image including the target to be augmented;

[0090] The loss function of the sample generation discriminator is as follows:

[0091] L Do (G,D o )=E y～p(y) [(D o (p)-1) 2 ]+E x～p(x) [(D o (G p (z))) 2 ]

[0092] Where p represents the original target image in the original image; G p (z) represents the amplified target image in the generated image; D ORepresents the sample generation discriminator. Although the two discriminators are different in structure, the final loss calculation uses the least squares method to overcome the training instability problem caused by the original GAN ​​loss. In addition to the adversarial loss, the generator loss function also adds the L1 norm content loss, as follows:

[0093] L Do (G, Do) = λL1(G(z), y) + E y～p(y) [(D b (y,z)-1) 2 ]+E x～p(x) [(D o (G p (z))) 2 ]

[0094] Where λ is a parameter used to balance the losses and L1 is the 1 norm.

[0095] More specifically, the image to be processed including the noise frame and the image to be processed are input into the preset generative adversarial network model as training data sets, and the fake image is generated by the generator network initialized with a Gaussian distribution value with a mean of zero, and then input into the discriminator network, and the loss is calculated for back propagation; the network weights are iteratively optimized by using the Adam optimizer, with the initial learning rate of 10 -4 . When the conditions for the end of training are met (for example, the number of training times is reached or the loss value does not decrease within the specified number of training times), the trained optimal network model parameters are saved. The optimal network model parameters are loaded into the preset generative adversarial network model, and the test set image is input to achieve sample amplification. Specifically, after the image to be processed including the noise frame is input into the generator, the amplified target image is obtained, and then according to the position information of the target standard frame, the amplified target image is replaced back into the original image to be processed including the noise frame to obtain the generated image.

[0096] In this invention, the effectiveness of the generated image is tested using the classic detection algorithm yolov3 target detection algorithm. The test results are as follows: Figure 5 and Figure 6 As shown. Figure 5 (a) Figure 6 (a) represents the test results of the detection algorithm training by simulating a small amount of normal source data. Figure 5 (b) Figure 6 (b) shows the detection results of the detection algorithm training using the method of the embodiment of the present invention under the generation and amplification of twice the amount of normal source data. By comparison, it can be seen that Figure 5 (a) shows obvious missed detection problems, and the detection accuracy of each sample is mostly less than Figure 5 (b) The detection accuracy in Figure 6In addition to missed detection, Figure 6 (a) In some areas with dense samples, two samples are detected in one frame. Figure 6 (b) can still accurately detect and identify, and the target frame can also frame the target sample more compactly. Therefore, the amplified sample (i.e., generated image) obtained by the method of the embodiment of the present invention improves the accuracy and generalization performance of the model for remote sensing target detection tasks with a small amount of sample data.

[0097] The preset generative adversarial network model designed by the present invention recomposes the background convolution kernel with all the background features outside the target annotation box, and performs convolution operation on the target sample features. Then, through the normalized activation function operation, the background similarity weight information of the target annotation box area relative to the background features is obtained; when the generator is used to generate the noise box content of the remote sensing target with a small recovery target and a relatively complex background distribution, it can restore and generate more natural and realistic background information, and avoid generating abrupt backgrounds to affect the detection effect. The original target annotation box can also be used as new annotation information to avoid the process of manual annotation. In addition, the commonly used spatial pyramid pooling layer is replaced by a weighted spatial pyramid pooling layer in the present invention, which makes up for the training balance problem of the GAN network to a certain extent.

[0098] like Figure 7 , Figure 8 As shown, the embodiment of the present invention provides a remote sensing image target detection sample amplification device based on context background enhancement style transfer network. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From the hardware level, Figure 7 As shown, it is a hardware architecture diagram of a computing device where a remote sensing image target detection sample amplification device based on a context background enhanced style transfer network provided by an embodiment of the present invention is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown in the figure, the computing device in which the device is located in the embodiment may also generally include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Figure 8 As shown, as a device in a logical sense, the CPU of the computing device in which it is located reads the corresponding computer program in the non-volatile memory into the memory and runs it. This embodiment provides a remote sensing image target detection sample amplification device based on context background enhancement style transfer network, the device comprising:

[0099] The preprocessing module 800 is used to preprocess the acquired original remote sensing data to obtain a to-be-processed image including the position information of the target to be amplified; generate a labeling frame according to the position information, randomly determine the target labeling frame, and generate a noise frame in the target labeling frame;

[0100] A training module 802 is used to train a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model;

[0101] The augmentation module 804 is used to input the image to be processed including the noise frame into the target generative adversarial network model, and output a generated image including the target to be augmented; wherein the position information of the target to be augmented is the same as that of the noise frame.

[0102] In some specific implementations, the preprocessing module 800 may be used to perform the above steps 100 and 102 , the training module 802 may be used to perform the above step 104 , and the amplification module 804 may be used to perform the above step 106 .

[0103] In one embodiment of the present invention, the preprocessing module 800 is further configured to perform the following operations:

[0104] A labeling frame is generated based on the position information, at least one target labeling frame is randomly determined in the labeling frame, and a noise frame is generated in the target labeling frame to obtain an image to be processed including the noise frame.

[0105] In one embodiment of the present invention, the preset generative adversarial network model includes a generator network and a discriminator network; the generator network includes a basic generator and a background enhancement module; the discriminator network includes a background restoration discriminator and a sample generation discriminator.

[0106] In one embodiment of the present invention, the background enhancement module is used to perform the following operations:

[0107] Performing a convolution operation on the image to be processed to obtain a first image;

[0108] Cropping the first image based on the target annotation frame to obtain a sample image located within the target annotation frame and a background image located outside the target annotation frame;

[0109] Cut and reorganize the background image to obtain the background reconstruction convolution kernel;

[0110] The sample image is convolved using the background reconstruction convolution kernel, and then the background enhancement information of the target annotation box is obtained through the activation function.

[0111] In one embodiment of the present invention, the generator network is used to perform the following operations:

[0112] The image to be processed including the noise frame is input into the basic generator, and the second image is obtained after three downsampling processes;

[0113] fusing the second image with the background enhancement information to obtain a third image;

[0114] The third image is upsampled three times to obtain a generated image.

[0115] In one embodiment of the present invention, it also includes:

[0116] The processed image and the generated image are cropped based on the target annotation box to obtain the original target image and the augmented target image respectively;

[0117] The original target image and the augmented target image are input into the sample generation discriminator, and after passing through the convolution layer and the weighted spatial pyramid pooling layer, the sample discrimination result is output; wherein the sample discrimination result includes whether the augmented target is true or false;

[0118] The weighted spatial pyramid pooling layer calculates the pooling value using the following formula:

[0119]

[0120] Among them, x i is used to represent the pixel value of the i-th pixel in the current pooling area; R is used to represent the set of all pixels in the current pooling area; w i Used to represent the weighted value corresponding to the i-th pixel in the current pooling area; Used to characterize pooling values;

[0121] The image to be processed and the image to be processed including the noise frame are input into the background restoration discriminator, and the background discrimination result is output; the generated image and the image to be processed including the noise frame are input into the background restoration discriminator, and the background discrimination result is output; wherein the background discrimination result is that the background information is true or the background information is false.

[0122] In one embodiment of the present invention, the training module 802 is further configured to perform the following operations:

[0123] Input the image to be processed including the noise frame into the generator network to obtain a generated image;

[0124] The generated image is input into the discriminator network for back propagation, and the optimal network model parameters are obtained through iterative optimization;

[0125] The optimal network model parameters are loaded into the preset generative adversarial network model to obtain the target generative adversarial network model.

[0126] It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a remote sensing image target detection sample amplification device based on a context background enhanced style transfer network. In other embodiments of the present invention, a remote sensing image target detection sample amplification device based on a context background enhanced style transfer network may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0127] The information interaction, execution process and other contents between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For the specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0128] An embodiment of the present invention also provides a computing device, including a memory and a target processor, wherein a computer program is stored in the memory, and when the target processor executes the computer program, a remote sensing image target detection sample amplification method based on a context background enhanced style transfer network in any embodiment of the present invention is implemented.

[0129] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes a remote sensing image target detection sample augmentation method based on a context background enhanced style transfer network in any embodiment of the present invention.

[0130] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0131] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0132] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0133] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling the operating system on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0134] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0135] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical factors in the process, method, article or device including the elements.

[0136] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for sample augmentation of remote sensing image target detection based on contextual background enhanced style transfer network, characterized in that: include: Preprocessing the acquired original remote sensing data to obtain an image to be processed including position information of the target to be amplified; Generate a labeling frame according to the position information, randomly determine a target labeling frame, and generate a noise frame in the target labeling frame; Training a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model; The image to be processed including the noise frame is input into the target generative adversarial network model, and a generated image including the target to be augmented is output; wherein the position information of the target to be augmented is the same as that of the noise frame.

2. The method according to claim 1, characterized in that The step of generating a label frame according to the position information, randomly determining a target label frame, and generating a noise frame in the target label frame includes: A labeling frame is generated based on the position information, at least one target labeling frame is randomly determined in the labeling frame, and a noise frame is generated in the target labeling frame to obtain an image to be processed including the noise frame.

3. The method according to claim 1, characterized in that The preset generative adversarial network model includes a generator network and a discriminator network; the generator network includes a basic generator and a background enhancement module; the discriminator network includes a background restoration discriminator and a sample generation discriminator.

4. The method according to claim 3, characterized in that The background enhancement module is used to perform the following operations: Performing a convolution operation on the image to be processed to obtain a first image; Cropping the first image based on the target annotation frame to obtain a sample image located within the target annotation frame and a background image located outside the target annotation frame; Cutting and reorganizing the background image to obtain a background reconstruction convolution kernel; Using the background reconstruction convolution kernel to perform a convolution operation on the sample image, and then obtaining the background enhancement information of the target annotation box through an activation function; and / or, The generator network is used to perform the following operations: Inputting the image to be processed including the noise frame into the basic generator, and performing three downsampling processes to obtain a second image; fusing the second image and the background enhancement information to obtain a third image; The third image is subjected to three upsampling processes to obtain the generated image.

5. The method according to claim 4, characterized in that Also includes: Cropping the image to be processed and the generated image based on the target annotation frame to obtain an original target image and an augmented target image respectively; Input the original target image and the augmented target image into the sample generation discriminator, and after passing through the convolution layer and the weighted spatial pyramid pooling layer, output a sample discrimination result; wherein the sample discrimination result includes whether the augmented target is true or the augmented target is false; The image to be processed and the image to be processed including the noise frame are input into the background restoration discriminator, and a background discrimination result is output; the generated image and the image to be processed including the noise frame are input into the background restoration discriminator, and a background discrimination result is output; wherein the background discrimination result is that the background information is true or the background information is false.

6. The method according to claim 5, characterized in that The weighted spatial pyramid pooling layer calculates the pooling value by the following formula: Among them, x i is used to represent the pixel value of the i-th pixel in the current pooling area; R is used to represent the set of all pixels in the current pooling area; w i Used to represent the weighted value corresponding to the i-th pixel in the current pooling area; Used to represent pooling values.

7. The method according to claim 5 or 6, characterized in that: The method of training a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model includes: Inputting the image to be processed including the noise frame into the generator network to obtain the generated image; Inputting the generated image into the discriminator network for back propagation, and obtaining the optimal network model parameters through iterative optimization; The optimal network model parameters are loaded into the preset generative adversarial network model to obtain the target generative adversarial network model.

8. A remote sensing image target detection sample amplification device based on context background enhanced style transfer network, characterized in that: include: A preprocessing module is used to preprocess the acquired original remote sensing data to obtain an image to be processed including the position information of the target to be amplified; And generating a labeling frame according to the position information, randomly determining a target labeling frame, and generating a noise frame in the target labeling frame; A training module, used for training a preset generative adversarial network model based on the image to be processed including the noise frame and the image to be processed to obtain a target generative adversarial network model; An augmentation module is used to input the image to be processed including the noise frame into the target generative adversarial network model, and output a generated image including the target to be augmented; wherein the position information of the target to be augmented is the same as that of the noise frame.

9. A computing device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images