A Multi-Weather Data Augmentation Method for Airport Surface Target Detection Based on Improved CycleGAN

Abstract

The present invention discloses a multi-weather data augmentation method for airport surface target detection based on an improved CycleGAN. On the basis of the original CycleGAN network, the generator network is improved by expanding the residual network blocks, cross-layer connection of the encoder-decoder, and introducing an attention mechanism. At the same time, a dynamic weighted multi-scale discriminator network is designed, and it is trained using a small sample target detection dataset of the airport surface under normal weather and an auxiliary dataset under typical bad weather to obtain a multi-weather image generation network, thereby effectively generating high-quality target detection images under various weather conditions such as rainy days, foggy days, and nights, and increasing the diversity of data samples. As an effective data augmentation method for saving annotation costs, the present invention improves the performance of the airport surface target detection network under small sample conditions.

Description

Technical Field

[0001] The present invention belongs to the field of image generation in computer vision, and relates to a multi-weather data enhancement method for airport scene target detection based on an improved CycleGAN, and particularly to a method for generating airport scene target detection data under typical bad weather through an improved cycle-consistent generative adversarial network for data enhancement. Background Art

[0002] As one of the typical tasks in the field of computer vision, object detection has achieved rapid development in recent years, and a large number of excellent network models have emerged. However, such models often require large-scale datasets to show good performance, which is difficult to meet in actual application scenarios and has extremely high costs. Therefore, data enhancement methods have become an effective means to improve network performance under small-sample data. In the field of vision, common data enhancement methods include rotation, scaling, flipping, cropping, etc. of pictures. These basic data enhancement methods have good effects on image classification tasks, but the effects are not ideal when applied to object detection tasks. Generative adversarial networks have also become a hot topic in recent years due to their excellent performance. Through the game process between the generator and the discriminator, samples that can pass for real can be generated to achieve the effect of data enhancement.

[0003] In the airport scene object detection task, various objects such as special vehicles and aircraft have strong randomness, variable scales, and image illumination changes with seasons and weather, making it difficult to collect a large number of data samples for annotation, and the object detection effect is not good under small-sample conditions. Using an improved CycleGAN network for data generation and increasing the diversity of object detection data is an effective and feasible means to improve the performance of object detection networks. Summary of the Invention

[0004] In view of the above problems, the present invention provides a multi-weather data enhancement method for airport scene target detection based on an improved CycleGAN. On the basis of the cycle-consistent generative adversarial network, the generator network and the discriminator network parts are respectively improved, and the overall loss function is adjusted at the same time, so that the model can generate high-quality picture data under various typical bad weather conditions, combine the original real data with the generated high-quality data, increase the diversity of training samples, thereby improving the generalization performance of the object detection network, so as to finally realize a more efficient small-sample object detection technology.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A multi-weather data enhancement method for airport scene target detection based on an improved CycleGAN, the method comprising the following steps:

[0007] Step S1: Image data acquisition, preprocessing, and dataset production;

[0008] The specific process of step S1 is as follows:

[0009] Collect the source image dataset through the airport surface monitoring camera, obtain the target detection data under normal weather conditions for a period of time and perform annotation, as the original small sample target detection dataset;

[0010] Collect the image data under typical bad weather conditions as the auxiliary weather dataset;

[0011] Perform basic data augmentation operations on the above two original datasets to obtain the training dataset for improving the CycleGAN network;

[0012] Step S2: Construction of an improved attention mechanism generator network;

[0013] The specific process of step S2 is as follows:

[0014] Expand the residual style conversion network in the original CycleGAN network generator, and add a two-layer convolutional residual network block after each convolutional block in the encoding and decoding parts;

[0015] In the original CycleGAN network generator, introduce the skip connection idea in U-Net, that is, perform cross-layer connection operations on the corresponding parts of the encoding and decoding;

[0016] Use the dense connection network DenseNet to replace the residual style conversion network in the original CycleGAN network generator;

[0017] Add an attention mechanism module to each of the left and right ends of the style conversion network DenseNet;

[0018] Step S3: Construction of a dynamic weighted multi-scale discriminator network;

[0019] The specific process of step S3 is as follows:

[0020] Perform downsampling operations on the image through two PatchGAN fully convolutional network branches with different scales to obtain image discrimination results of different sizes;

[0021] Step S4: Adversarial training;

[0022] The specific process of step S4 is as follows:

[0023] On the basis of the above improved generator and discriminator networks, adjust the loss function of the original CycleGAN network, that is:

[0024]

[0025] Among them , is the cyclic consistency loss, , is the identity loss, is the modified generative adversarial network loss, where the multi-scale discriminator network loss is dynamically weighted. Specifically:

[0026]

[0027] From and calculate the A-distance respectively, so as to obtain the dynamic weighting factor , finally there is

[0028]

[0029] Similarly;

[0030] In the above loss function are respectively the generator from the original small sample target detection data X to the auxiliary weather data Y, the generator from the auxiliary weather data Y to the original small sample target detection data X, the discriminator with the original small sample target detection data X as the real sample, and the discriminator with the auxiliary weather data Y as the real sample; , and , are respectively the above two discriminators with different scales, λ is the penalty coefficient, and the network parameters are obtained by gradient descent training through the following formula:

[0031] .

[0032] Furthermore, in the step S1, the normal weather is daytime and sunny.

[0033] Furthermore, in the step S1, the targets include special vehicles and aircraft.

[0034] Furthermore, in the step S1, the typical bad weather includes cloudy days, rainy days, foggy days and nights.

[0035] Furthermore, in the step S1, the data augmentation operations include rotation, scaling and flipping.

[0036] Furthermore, in the step S2, the attention mechanism module is a spatial-channel hybrid attention mechanism.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] The generator network is improved by expanding the residual network block, cross-layer connection of the encoder-decoder, and introducing the attention mechanism. At the same time, a dynamic weighted multi-scale discriminator network is designed and trained under the conditions of the airport scene small sample target detection dataset and the auxiliary weather dataset, which can effectively generate high-quality target detection images in weather conditions such as rainy days, foggy days, and nights, increase the diversity of target detection data samples, and thus effectively improve the performance of the target detection network under small sample conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the network structure diagram of the method proposed by the present invention;

[0040] Figure 2 It is the schematic diagram of the cycle-consistent generative adversarial network;

[0041] Figure 3 It is the generator network structure diagram of the method proposed by the present invention;

[0042] Figure 4 It is the discriminator network structure diagram of the method proposed by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0043] The present invention will be further described in detail below in conjunction with the drawings and the specific embodiments:

[0044] As Figure 1 shown, the original small sample target detection dataset and the auxiliary weather dataset are respectively used as the source domain data X and the target domain data Y, and the improved CycleGAN network model is trained to obtain a multi-weather image generation network, and then multi-weather target detection images are generated. The improved CycleGAN model includes a generator G and a generator F, corresponding to the mappings from data X to data Y and from data Y to data X respectively; two multi-scale discriminators Dx and Dy are used to discriminate the generations in two directions, and both Dx and Dy include two sub-discriminators. A high-quality multi-weather image generation network can be obtained through iterative training. Figure 2 It is the schematic diagram of the original cycle-consistent generative adversarial network.

[0045] Specifically, the main steps are as follows:

[0046] Step S1, Image data collection and preprocessing, dataset production;

[0047] Collect the source image dataset through the airport scene monitoring camera, obtain the target detection data under normal weather conditions within a period of time and perform annotation, as the original small sample target detection dataset; collect the image data under foggy weather conditions, as the auxiliary weather dataset; perform basic data augmentation operations on the above two original datasets to obtain the training dataset of the improved CycleGAN network;

[0048] Step S2: Construction of an improved attention mechanism generator network;

[0049] Figure 3 It is the structural diagram of the improved generator network. As shown in the figure, the residual style conversion network in the original CycleGAN network generator is expanded, and a two-layer convolutional residual network block is added after each convolutional block in the encoding and decoding parts; in the generator of the original CycleGAN network, the skip connection idea in U-Net is introduced, that is, cross-layer connection operations are performed on the corresponding parts of the encoding and decoding; the DenseNet (Dense Connection Network) is used to replace the residual style conversion network in the original CycleGAN network generator; an attention mechanism module is added to each of the left and right ends of the style conversion network DenseNet.

[0050] Step S3: Construction of a dynamic weighted multi-scale discriminator network;

[0051] Figure 4 It is the structural diagram of the improved discriminator network. As shown in the figure, the image is downsampled through two PatchGAN fully convolutional network branches with different scales to obtain image discrimination results of different sizes;

[0052] Step 4: Adversarial training;

[0053] According to the adjusted network loss function, that is:

[0054]

[0055] Optimization objective , set the initial parameters to train the model, alternately update the discriminator and the generator. Finally, after the training is completed, the generation network is obtained. Input the target detection data under normal weather, and the corresponding generated foggy weather target detection data can be obtained.

[0056] The above is only the preferred embodiment of the present invention, and it is not any other form of limitation to the present invention. Any modification or equivalent change made according to the technical essence of the present invention still belongs to the scope protected by the present invention.

Claims

1. A multi-weather data augmentation method for airport surface target detection based on improved CycleGAN, characterized in that The method includes the following steps: Step S1, image data acquisition and preprocessing, and dataset production; The specific process of the step S1 is as follows: Collect the source image dataset through the airport surface monitoring camera, obtain the target detection data under normal weather conditions for a period of time and perform annotation, as the original small sample target detection dataset; Collect the image data under typical bad weather conditions as the auxiliary weather dataset; Perform basic data augmentation operations on the above two original datasets to obtain the training dataset for improving the CycleGAN network; Step S2, construction of an improved attention mechanism generator network; The specific process of the step S2 is as follows: Expand the residual style conversion network in the original CycleGAN network generator, and add a two-layer convolutional residual network block after each convolutional block in the encoding and decoding parts; Introduce the skip connection idea in U-Net into the generator of the original CycleGAN network, that is, perform cross-layer connection operations on the corresponding parts of the encoding and decoding; Use the Dense Connection Network DenseNet to replace the residual style conversion network in the original CycleGAN network generator; Add an attention mechanism module to both the left and right ends of the style conversion network DenseNet; Step S3, construction of a dynamic weighted multi-scale discriminator network; The specific process of the step S3 is as follows: Perform downsampling operations on the image through two PatchGAN fully convolutional network branches with different scales to obtain image discrimination results of different sizes; Step S4, adversarial training; The specific process of the step S4 is as follows: Adjust the loss function of the original CycleGAN network based on the above improved generator and discriminator networks, that is: ； Among them , is the cyclic consistency loss, , is the identity loss, is the modified generative adversarial network loss, where the multi-scale discriminator network loss is dynamically weighted. Specifically: ； From and calculate the A-distance respectively, so as to obtain the dynamic weighting factor , finally there is ； Similarly; In the above loss function are respectively the generator from the original small-sample object detection data X to the auxiliary weather data Y, the generator from the auxiliary weather data Y to the original small-sample object detection data X, the discriminator with the original small-sample object detection data X as the real sample, and the discriminator with the auxiliary weather data Y as the real sample; , and , are respectively the discriminators of the above two different scales, λ is the penalty coefficient, and the network parameters are obtained by gradient descent training according to the following formula: 。 2. The multi-weather data augmentation method for airport surface target detection based on the improved CycleGAN according to claim 1, wherein: The normal weather in the step S1 is daytime and sunny.

3. The multi-weather data augmentation method for airport surface target detection based on the improved CycleGAN according to claim 1, wherein: The targets in the step S1 include special vehicles and aircraft.

4. The multi-weather data augmentation method for airport surface target detection based on the improved CycleGAN according to claim 1, wherein: The typical bad weather in the step S1 includes cloudy days, rainy days, foggy days and nights.

5. The multi-weather data augmentation method for airport surface target detection based on the improved CycleGAN according to claim 1, wherein: The data augmentation operations in the step S1 include rotation, scaling and flipping.

6. The multi-weather data augmentation method for airport surface target detection based on the improved CycleGAN according to claim 1, wherein: The attention mechanism module in the step S2 is a spatial channel hybrid attention mechanism.

Images