A sea ship target detection method based on a generative adversarial network anti-shake

Abstract

The application discloses a kind of offshore ship target detection methods based on generation confrontation network anti-shake, comprising the following steps: establishing DeblurGAN-v2 network model and training, using generator and discriminator alternately training method training, finally loss function converges and obtains the deblurred confrontation generation network model;Take out the discriminator network part in the trained network model as binary classification network continues training;Establish based on offshore ship target detection model and training, obtain YOLOv7 detection model;The generator part in the obtained trained network model, binary classification network and YOLOv7 detection model are reformed and fused, new discriminator network is used as front-end module, for fuzzy frame judgment, for selecting whether the picture sample is sent into generator network to deblur, and the target detection network of fusion generation confrontation network DeblurGAN-v2 is constructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, deep learning and image processing, in particular to a ship target detection method based on a generative adversarial network for anti-shaking. BACKGROUND

[0002] With the rapid development of social economy and the deepening of globalization, the shipping business of the near-sea channel is becoming more and more busy, which poses new challenges to the marine monitoring department and the channel management department. The development of edge computing and artificial intelligence technology provides a new solution for the channel management department. The scheme of forming a channel monitoring network by using distributed edge monitoring nodes is emerging. Using the navigation mark distributed on the near-sea channel as a carrier and carrying an embedded monitoring device to monitor the ship can effectively improve the supervision efficiency. The embedded device as an edge computing carrier can run the existing mature target detection program. However, the shaking of the offshore buoy with the sea waves poses a challenge to target detection.

[0003] In order to solve the problem of motion blur caused by camera shaking affecting the effect of target detection, the traditional methods mainly include background difference method, inter-frame difference method and optical flow method, etc. However, the effect of the algorithm improved on the basis of the traditional method is generally poor, and the algorithm is complex and has large amount of calculation. For example, a kind of motion target detection algorithm combining block gray projection, background difference and continuous inter-frame difference method (Xue Yang, Zhang Yafei, Yang Tianyu, etc. A kind of motion target detection algorithm for shaking video sequence [J]. Laser and Optoelectronics Progress, 2018, 55(9): 7.) The algorithm processes the image frames by block processing, removes the target area with low gray gradient change and local motion by combining the discretization decision mechanism, and improves the global motion vector estimation accuracy. The mutual correlation calculation is performed according to the block area gray projection curve to complete the shaking sequence correction. The fusion strategy of background difference and continuous three-frame difference method is used to process the corrected sequence frames to enhance the motion target area. The fusion difference image is smoothed and the Otsu method is used for adaptive threshold segmentation to detect the foreground motion target, so as to detect the motion target in the camera shaking scene. However, the application of these algorithms on the edge device has poor effect, and a simple and excellent target detection method against camera shaking is needed. SUMMARY

[0004] In order to solve the problems existing in the prior art, the present application provides a ship target detection method based on a generative adversarial network for anti-shaking. The target detection model of the fusion generative adversarial network is established to realize an end-to-end anti-shaking blur target detection scheme. The blurred frames in the video can be automatically detected and deblurred, and the target detection can be performed. The accuracy of the detection is improved while ensuring a certain real-time performance.

[0005] The application is implemented at least by one of the following technical solutions.

[0006] An offshore ship target detection method based on a generative adversarial network anti-shake, comprising the following steps:

[0007] S1, acquiring a video stream containing a motion blur frame caused by camera shake;

[0008] S2, establishing a target detection model of a fusion generative adversarial network;

[0009] S3, inputting the video stream into the target detection model established in step S2 for target detection.

[0010] Further, in step S2, the establishment of the target detection model of the fusion generative adversarial network comprises the following steps:

[0011] S21, establishing an offshore ship image deblurring dataset, the dataset comprising pairs of blurred images and clear images;

[0012] S22, establishing a DeblurGAN-v2 network model and training, using a method of alternating training of a generator and a discriminator, finally obtaining a deblurring adversarial generation network model through loss function convergence, and the generated model mainly comprises a trained generator and a discriminator;

[0013] S23, training a binary classification network for distinguishing whether the current input frame is a clear image or a blurred image;

[0014] S24, establishing an offshore ship target detection model based on YOLOv7, training using an offshore ship target detection dataset, obtaining a YOLOv7 target detection model, and using the model for ship target detection in a target sea area;

[0015] S25, fusing the generator part of the DeblurGAN-v2 network model trained in step S22, the binary classification network trained in step S23, and the YOLOv7 model trained in step S24 to construct a target detection network of the fusion generative adversarial network DeblurGAN-v2: the binary classification network is used as a front-end module to determine whether the current input frame is blurred, if so, the current input frame is sent to the generator network for deblurring, and after deblurring, the current input frame is sent to the YOLOV7 target detection network; if not, the current input frame is directly sent to the YOLOV7 target detection network.

[0016] Further, the DeblurGAN-v2 network model adopts a feature pyramid (FPN) structure as the core module of the generator, wherein the backbone network backbone selects a MobileNet network, a relative discriminator with a minimum square loss is adopted for the discriminator part, and a two-scale discriminant loss function is adopted.

[0017] Further, the YOLOv7 target detection model includes an input, a backbone network, a neck, and a head part, wherein the backbone network is an ELAN and MP structure.

[0018] Further, the loss function L of the DeblurGAN-v2 network model generator is: G

[0019] L G = 0.5 * L P + 0.006 * L X + 0.01 * L adv

[0020] In the formula, L P is the minimum mean square error, L X is the perception loss, and L adv is the local loss.

[0021] Further, the loss function L of the DeblurGAN-v2 network model discriminator is: D

[0022] L D = E x～Pdata(x) [(D (x) -E z～Pz(z) D(G(z))-1) 2 ]+E z～Pz(z) [(D(G(z))-E x～Pdata(x) D(x)+1) 2 ]

[0023] In the formula, E is the expectation, x ~ Pdata(x) represents that x is randomly extracted from the distribution of real data data, z ~ Pz(z) represents that noise z is extracted from a Gaussian distribution of random noise, D(x) represents the vector output by the discriminator after receiving the input in the brackets, and G(z) represents the vector output by the generator after receiving the input in the brackets.

[0024] Further, the binary classification network is obtained by continuing training the discriminator in the DeblurGAN-v2 network model.

[0025] Further, the output of the binary classification network is a probability value between 0 and 1, and when greater than 0.5, it indicates that the input picture is a clear picture, otherwise it is a blurred picture.​​

[0026] Further, the data set for the binary classification in step S231 is the sample set established in step S21 or the sample set formed by collecting the ambiguous category pictures and the clear category pictures in the marine environment.

[0027] Further, the target detection model of the fusion generative adversarial network DeblurGAN-v2 is an end-to-end model, which can automatically deblur the blurred frames and realize target detection after inputting the video stream.

[0028] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0029] (1) The binary classification network is obtained by performing secondary training on the discriminator of the trained generative adversarial network DeblurGAN-v2, part of the parameters is reused, and the discrimination accuracy of whether each frame in the video stream is blurred is improved.

[0030] (2) The target detection model of the fusion generative adversarial network DeblurGAN-v2 is established, which belongs to an end-to-end target detection model, and each frame of the input video is detected for the pre-operation of the blurred frame, only the certain frame image detected as blurred is input to the generator network for deblurring operation, and the clear frame is directly subjected to target detection, which can improve the video processing efficiency and the target detection speed of the video with shaking blur. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of an embodiment of the marine ship target detection method based on the generative adversarial network anti-shaking of the present application is shown.

[0032] Figure 2 A flowchart of the improved target detection network of the fusion generative adversarial network of the embodiment of the present application is shown.

[0033] Figure 3 A structure diagram of the target detection network of the fusion generative adversarial network of the embodiment is shown. DETAILED DESCRIPTION

[0034] This part will describe the specific embodiments of the present application in detail, and the preferred embodiments of the present application are shown in the accompanying drawings, which serve to supplement the description in the text part of the description and enable people to intuitively and visually understand the overall technical scheme of the present application, but it cannot be understood as a limitation on the protection scope of the present application.

[0035] As shown in Figure 1 An embodiment of the present application, a marine ship target detection method based on the generative adversarial network anti-shaking, includes the following steps:

[0036] S1, acquire a video stream containing motion blur frames caused by camera shake;

[0037] S2, establish a target detection model of the fusion generative adversarial network DeblurGAN-v2;

[0038] S3, input the video stream into the model established in S2 for target detection.

[0039] As a preferred embodiment, the embodiment takes a nearshore beacon as a carrier, carries a camera and an embedded edge computing platform Jetson TX2, and detects a ship target through the camera to collect a video of a target sea area. Due to the influence of sea waves, the nearshore beacon will make periodic fluctuation, so the video stream in step S1 contains motion blur frames caused by camera shake.

[0040] As shown in Figure 2 , in step S2, the establishment of the target detection model of the fusion generative adversarial network is realized through the following steps:

[0041] S21, establish a marine ship image deblurring dataset, which is composed of pairs of blurred images and clear images, and divide the training set and the test set in a ratio of 7:3. The deblurring dataset is acquired by simultaneously shooting images in the target sea area with two cameras, one of which captures blurred images through low shutter speed, and the other of which shoots clear images through high shutter speed. A pair of blurred-clear image samples is established, the total number of samples is greater than 5000, and the marine ship image deblurring dataset is composed.

[0042] S22, establish the DeblurGAN-v2 network model on a server carrying a V100 GPU and train it. The method of alternating training of the generator and the discriminator is adopted for training. Finally, the loss function converges to obtain the deblurring adversarial generation network model. The generated model mainly includes the trained generator and discriminator. At this time, the generator of the DeblurGAN-v2 network has the ability to deblur pictures. The specific training process is as follows:

[0043] S221, input the clear images in the training set into the generator network with a batch size of 8. The images are reorganized to a size of 256*256*3. The blurred images are input into the generator as input, and the generator generates clear images;

[0044] S222, input the images generated by the generator into the discriminator. The discriminator extracts feature maps through the 3*3 convolution kernel of VGG19, calculates the loss function for each patch with a size of 8*8, and takes the average value of the discrimination results of each patch as the output result of the discriminator. Then the parameters are updated through the gradient descent method.

[0045] S232, the generator updates the gradient according to the result output by the discriminator. If the average value output by the discriminator is greater than 0.5, it means that the discriminator recognizes that the picture is generated by the generator, and then the generator parameters are updated by gradient descent;

[0046] S233, alternately iterate to train both. When training the discriminator, the generator parameters are fixed. When training the generator, the discriminator parameters are fixed. The generator and the discriminator share parameters in the same batch training;

[0047] S234, because the convergence speed of the discriminator is faster than that of the generator, the discriminator parameters are no longer updated after 20 rounds of training. The generator is trained for a total of 300 rounds, and finally the DeblurGAN-v2 network model parameters are obtained;

[0048] S23, the discriminator network part in the trained DeblurGAN-v2 network model is taken out as a binary classification network for further training. The binary classification network obtained by training is used to distinguish whether the current input frame is a clear image or a blurred image. The discriminator network architecture in the DeblurGAN-v2 network model is the same as PatchGAN, and it is essentially a binary classification network, which is used to judge whether the input picture is a real picture or a picture generated by the generator. The output of the discriminator is a probability value between 0 and 1. When it is greater than 0.5, it means that the input picture is a real picture, otherwise it is a picture generated by the generator. When the DeblurGAN-v2 network model is trained, the generator and the discriminator reach Nash equilibrium, that is, any picture input has only a 0.5 chance of being considered a picture generated by the generator. At this time, the discriminator loses the ability to distinguish between real pictures and pictures generated by the generator. Therefore, the discriminator of the general adversarial network should be discarded after training, but the DeblurGAN-v2 discriminator has the function of extracting picture features and has certain classification ability. The present application uses the trained discriminator to continue the binary classification training, so that it can judge whether the current input picture is a blurred picture or a clear picture. The training steps are as follows:

[0049] S231, prepare a data set for binary classification, which contains two categories of blurred and clear: one is a labeled blurred image sample, and the other is a labeled clear image sample. After randomly shuffling, the data set is divided into training set and test set according to the ratio of 8:2;

[0050] S232, start training using the prepared data set. When the discriminator output is less than 0.5, it corresponds to the blurred image category. When the output is greater than 0.5, it corresponds to the clear image category. The loss function still uses the original loss function of the discriminator, and the parameters are updated by gradient descent method;

[0051] S233, test after training, at this time the binary classification network is trained, which can be used for binary classification of the input picture sample, to determine whether the input picture is a clear picture or a blurred picture.

[0052] The binary classification network is obtained by retraining the discriminator of the trained generative adversarial network DeblurGAN-v2, which reuses part of the parameters and improves the discrimination accuracy of whether each frame in the video stream is blurred.

[0053] S24, establish a marine ship target detection model based on YOLOv7, train it with a marine ship target detection dataset, and obtain a YOLOv7 target detection model for ship target detection in a target sea area.

[0054] As a preferred embodiment, the disclosed SeaShips ship dataset is selected. The SeaShips ship dataset is a large ship dataset that covers six common offshore ship types (ore carrier, bulk carrier, general cargo ship, container ship, fishing boat and passenger ship), a total of 7000 ship images, of which 6000 images are divided into a training set, 600 images are divided into a validation set, and 400 images are divided into a test set. The size of the input image is uniformly set to 640*640 JPG image, the batch size is set to 48, the Epoch is set to 200, the learning rate is set to 0.01, the network learning rate is adjusted by simulating the cosine annealing strategy, and the YOLOv7 model weight parameter file is obtained after training.

[0055] S25, fuse the generator part of the trained DeblurGAN-v2 network model in step S22, the trained binary classification network in step S23, and the trained YOLOv7 model in step S24 to construct a target detection network of the fused generative adversarial network DeblurGAN-v2: the binary classification network as a pre-module determines whether the current input frame is blurred, if so, the current input frame is sent to the generator network to remove blur, and after removing blur, it is sent to the YOLOV7 target detection network; if not, the current input frame is directly sent to the YOLOV7 target detection network. For each frame of the input video, the pre-operation of detecting blurred frames is performed, only the certain frame image detected as blurred is input to the generator network for deblurring operation, and the clear frame is directly subjected to target detection, which can improve the video processing efficiency and improve the target detection speed of the video with shaking blur.

[0056] In step S2, the DeblurGAN-v2 network model adopts a feature pyramid (FPN) structure as the core module of the generator, wherein the backbone network backbone selects a MobileNet network, a relativistic discriminator with a minimum square loss is adopted for the discriminator part, and a discriminant loss function combining global and local two scales is adopted.

[0057] In step S22, the loss function L G of the DeblurGAN-v2 network model generator is:

[0058] L G = 0.5*L P + 0.006*L X + 0.01*L adv

[0059] In the formula, L P is the minimum mean square error, L X is the perceptual loss, the Euclidean distance is calculated by the feature map extracted by the convolution kernel, and L adv is the local loss.

[0060] The loss function L D of the DeblurGAN-v2 network model discriminator is:

[0061] L D = E x～Pdata(x) [(D (x) -E z～Pz(z) D(G(z))-1) 2 ]+E z～Pz(z) [(D(G(z))-E x～Pdata(x) D(x)+1) 2 ]

[0062] In the formula, E is the expectation, x ~ Pdata(x) represents that x is randomly extracted from the distribution of real data data, z ~ Pz(z) represents that noise z is extracted from a Gaussian distribution of random noise, D(x) represents the output vector of the discriminator after receiving the input in the brackets, and G(z) represents the output vector of the generator after receiving the input in the brackets.

[0063] In step S24, the YOLOv7 target detection model mainly includes four parts of input (Input), backbone network (Backbone), neck (Neck), and head (Head), wherein the backbone network is mainly based on ELAN and MP structure. The marine ship target detection data set can use public or private ship data set to meet the common ships in the target sea area. The embodiment is only one of the preferred ones.

[0064] The target detection model of the fusion generative adversarial network DeblurGAN-v2 is transplanted to an edge computing platform Jetson TX2 for offshore target detection test, as shown in Figure 3 The target detection model of the fusion generative adversarial network DeblurGAN-v2 is an end-to-end model, as shown in the figure. After inputting a video stream, a binary classification network judges whether it is a blurred frame by frame. If it is a blurred frame, it is input into the generator network to generate a clear image and input into the YOLOv7 target detection model for ship target detection. If it is a clear frame, it is directly input into the YOLOv7 target detection model for ship target detection. The target detection model of the fusion generative adversarial network in this embodiment can realize end-to-end ship target detection under camera jitter, and can identify six common types of ships near the sea: ore carrier, bulk carrier, general cargo ship, container ship, fishing boat and passenger ship.

[0065] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A method for detecting maritime ship targets based on generative adversarial networks to combat jitter, characterized in that, Includes the following steps: S1. Obtain a video stream containing motion-blurred frames caused by camera shake; S2. Establish a target detection model using a fusion generative adversarial network; S3. Input the video stream into the target detection model established in step S2 for target detection; The establishment of the target detection model of the fusion generative adversarial network includes the following steps: S21. Establish a deblurred dataset of marine vessel images, which includes pairs of blurred and clear images; S22. Establish and train the DeblurGAN-v2 network model. The generator and discriminator are trained alternately. Finally, the loss function converges to obtain the deblurred adversarial generative network model. The main structure of the generated model includes the trained generator and discriminator. S23. Train a binary classification network to determine whether the current input frame is a clear image or a blurry image; S231. Prepare a dataset for binary classification, containing two categories: blurry and clear. One category consists of labeled blurry image samples, and the other consists of labeled clear image samples. After randomly shuffling the order, divide the dataset into training and test sets in an 8:2 ratio. S232. Start training using the prepared dataset. When the discriminator output is less than 0.5, it corresponds to the blurred image category. When the output is greater than 0.5, it corresponds to the clear image category. The loss function is still the original loss function of the discriminator. Update the parameters using the gradient descent method. S233. Test after training. At this point, the binary classification network is trained and can be used to perform binary classification on the input image samples to determine whether the input image is a clear image or a blurry image. S24. Establish a YOLOv7-based marine vessel target detection model, train it using a marine vessel target detection dataset, and obtain a YOLOv7 target detection model for use in target sea areas for vessel target detection. S25. The generator part of the DeblurGAN-v2 network model trained in step S22, the binary classification network trained in step S23, and the YOLOV7 model trained in step S24 are fused to construct the object detection network of the fused generative adversarial network DeblurGAN-v2: the binary classification network is used as a front-end module to determine whether the current input frame is blurred. If it is, the current input frame is sent to the generator network to deblur it, and then sent to the YOLOV7 object detection network. Otherwise, the current input frame is sent directly to the YOLOV7 object detection network.

2. The method for detecting maritime ship targets based on generative adversarial networks to combat jitter, as described in claim 1, is characterized in that: The DeblurGAN-v2 network model uses a Feature Pyramid (FPN) structure as the core module of the generator. The backbone network is MobileNet. The discriminator part uses a relative discriminator with minimum square root loss and a discriminant loss function with two scales.

3. The method for detecting maritime ship targets based on generative adversarial networks to combat jitter, as described in claim 1, is characterized in that: The YOLOv7 target detection model includes an input, a backbone network, a neck, and a head section, wherein the backbone network is an ELAN and MP structure.

4. The method for detecting maritime ship targets based on generative adversarial networks to combat jitter, as described in claim 1, is characterized in that: The loss function of the DeblurGAN-v2 network model generator for: In the formula, To minimize the mean square error, In order to perceive loss, This is a localized loss.

5. The method for detecting maritime ship targets based on generative adversarial networks to combat jitter, as described in claim 1, is characterized in that: The loss function of the discriminator in the DeblurGAN-v2 network model for: In the formula, E represents the expectation. This means randomly selecting x from the distribution of the real data data. This indicates that noise z is extracted from random noise distributed in a Gaussian pattern. G(z) represents the vector output by the discriminator after receiving the input within the brackets, and G(z) represents the vector output by the generator after receiving the input within the brackets.

6. The method for detecting maritime ship targets based on generative adversarial networks to combat jitter according to claim 1, characterized in that: The binary classification network is obtained by further training the discriminator in the DeblurGAN-v2 network model.

7. The method for detecting maritime ship targets based on generative adversarial networks to combat jitter, as described in claim 6, is characterized in that: The output of the binary classification network is a probability value between 0 and 1. When it is greater than 0.5, it indicates that the input image is a clear image; otherwise, it is a blurry image.

8. The method for detecting maritime ship targets based on generative adversarial networks to combat jitter according to claim 1, characterized in that: In step S231, the dataset used for binary classification is either the sample set established in step S21 or a sample set formed by collecting fuzzy category images and clear category images in a marine environment.

9. A method for detecting maritime ship targets based on generative adversarial networks to combat jitter, as described in any one of claims 1 to 8, characterized in that: The object detection model of the fusion generative adversarial network DeblurGAN-v2 is an end-to-end model that can automatically deblur blurred frames and achieve object detection after inputting a video stream.

Images