Water surface motion blurred image enhancement method for intelligent navigation of ship

Abstract

The invention discloses a water surface motion blurred image enhancement method for intelligent navigation of a ship. The method comprises the following steps: 1) obtaining a water surface motion blurred image training set for intelligent navigation of the ship; 2) constructing an image motion blur removing model based on the generative adversarial network, wherein the image motion blur removing model comprises a generator and a discriminator; 3) training the model; and 4) using a generator of the trained model to realize motion blur removal of the water surface image. The method can eliminate the adverse effect of motion blur on visual perception, and guarantees the intelligent navigation safety of the ship.

Description

Technical Field

[0001] The invention relates to image processing technology, and in particular to a water surface motion blur image enhancement method for ship intelligent navigation. Background Art

[0002] Ships sailing on the water are easily affected by wind, waves and currents, which can cause motion blur in the images of visual sensors, thereby interfering with visual perception and threatening the intelligent navigation safety of ships. Most existing image deblurring methods are based on deep learning theory, which enhances the feature extraction ability of the model by stacking a large number of convolutional layers. Compared with traditional methods, although the quality of restored images is improved, the computational complexity of the model is greatly increased. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a water surface motion blur image enhancement method for ship intelligent navigation in view of the defects in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: a water surface motion blur image enhancement method for ship intelligent navigation, comprising the following steps:

[0005] 1) Obtain a training set of water surface motion blurred images for ship intelligent navigation;

[0006] 2) constructing an image motion blur removal model based on a generative adversarial network, wherein the image motion blur removal model includes: a generator and a discriminator;

[0007] The generator takes the motion-blurred water surface image as input; the discriminator discriminates between the restored image generated by the generator and the real clear image. If it is judged to be the generated restored image, it will be given a low score, and if it is judged to be the real clear image, it will be given a high score. The discriminator result is passed to the generator to find the difference between the restored image and the clear image, thereby generating a higher quality restored image.

[0008] 3) Training model;

[0009] 4) Use the generator of the trained model to de-motion blur the water surface image.

[0010] According to the above scheme, the game relationship between the generator and the discriminator is expressed as:

[0011]

[0012] Among them, D is the discriminator, G is the generator, and I S For a clear image, I B is a motion blurred image, G(I B ) is the restored image output by the generator, is the gradient penalty term, which is used to ensure the stability of the training process, x is the input when the discriminator calculates the gradient, and λ GP is a hyperparameter.

[0013] According to the above scheme, the structure of the generator is as follows: three convolution layers are set at the front end of the generator for preliminarily extracting blurred image features, two deconvolution layers and one convolution layer are set at the back end of the generator for reconstructing the resolution of the blurred image, and a receptive field module is set in the middle part of the generator.

[0014] According to the above scheme, the structure of the receptive field module is as follows:

[0015] The receptive field module has a total of 5 branches. The 5 branches first use 1×1 convolution to reduce the dimension of the input data. Before one branch after dimensionality reduction is connected to the activation layer of the module, 4 branches use convolution kernels of preset sizes to further process the features after dimensionality reduction. At the end of the 4 branches, there are dilated convolutions with different numbers of holes. After the dilated convolution, the 4 branches splice the features in the channel dimension, and then undergo 1×1 convolution and fuse with the data of another branch before the activation layer of the module, and finally output through the nonlinear processing of the ReLU activation layer.

[0016] According to the above scheme, 9 receptive field modules are set in the middle part of the generator.

[0017] According to the above scheme, the discriminator is set as a cyclic combination of multiple convolutional layers, instance normalization layers, and LeakyReLU.

[0018] According to the above scheme, in step 3), when training the model, the feature loss of the restored image and the clear image is calculated as follows:

[0019]

[0020] in, W represents the feature map output after the activation of the jth convolutional layer before the i-th maximum pooling layer of the VGG19 network. i,j , H i,j are the width and height dimensions of the feature map, I S For a clear image, I B is a motion blurred image, G is the generator of the network, G(I B ) is the restored image output by the generator.

[0021] The beneficial effects produced by the present invention are:

[0022] The method of the invention can eliminate the adverse effects of motion blur on visual perception and ensure the intelligent navigation safety of ships. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0024] Figure 1 is a flow chart of motion blur removal by a generative adversarial network according to an embodiment of the present invention;

[0025] Figure 2 is a diagram of a motion blur removal model architecture according to an embodiment of the present invention;

[0026] Figure 3 is a structural diagram of a receptive field module according to an embodiment of the present invention;

[0027] Figure 4 Schematic diagram of the discrimination method of the classic discriminator in an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a discriminant method of a discriminator according to an embodiment of the present invention;

[0029] Figure 6 Schematic diagram of characteristic loss calculation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] like Figure 1 As shown, a method for enhancing a water surface motion blur image for ship intelligent navigation comprises the following steps:

[0032] 1) Obtain a training set of water surface motion blurred images for ship intelligent navigation;

[0033] The training and testing process requires pairs of clear images and blurred images to provide guidance for the network's learning process. For clear images, the original clear images can be collected by setting up visual sensors in the required navigation scene. For blurred images, considering that in the real world, after motion blur occurs in the image shooting process of a certain scene, it is often difficult to obtain a clear image that completely corresponds to the scene. Therefore, referring to the data set production method of the existing method, this embodiment adds motion blur to the collected clear images based on the physical model of motion blur, thereby constructing pairs of clear images and motion blurred images for network training and testing;

[0034] I B =K*I S +N

[0035] Among them, I Sis the original clear image, K is the blur kernel, and N is the noise. The trajectory information of the image motion blur process is included in the blur kernel K. S Convolve with blur kernel K, then add noise N to generate blurred image I B .

[0036] 2) constructing an image motion blur removal model based on a generative adversarial network, wherein the image motion blur removal model includes: a generator and a discriminator;

[0037] The generator takes the motion-blurred water surface image as input; the discriminator discriminates between the restored image generated by the generator and the real clear image. If it is judged to be the generated restored image, it will be given a low score, and if it is judged to be the real clear image, it will be given a high score. The discriminator result is passed to the generator to find the difference between the restored image and the clear image, thereby generating a higher quality restored image.

[0038] The game relationship between the generator and the discriminator is expressed as:

[0039]

[0040] Among them, D is the discriminator, G is the generator, and I S For a clear image, I B is a motion blurred image, G(I B ) is the restored image output by the generator, is the gradient penalty term, which is used to ensure the stability of the training process, x is the input when the discriminator calculates the gradient, and λ GP is a hyperparameter.

[0041] like Figure 2 ,The structure of the generator is as follows: three convolutional layers are set at the front end of the ,generator to preliminarily extract the features of the blurred image, two deconvolutional layers and one convolutional layer are set at the back end of ,generator to reconstruct the resolution of the blurred image, and 9 receptive field modules are ,set in the middle part of the generator.

[0042] like Figure 3 , the receptive field module structure is as follows:

[0043] The receptive field module has a total of 5 branches. The 5 branches first use 1×1 convolution to reduce the dimension of the input data. Before one branch after dimensionality reduction is connected to the activation layer of the module, 4 branches use convolution kernels of preset sizes to further process the features after dimensionality reduction. At the end of the 4 branches, there are dilated convolutions with different numbers of holes. After the dilated convolution, the 4 branches splice the features in the channel dimension, and then undergo 1×1 convolution and fuse with the data of another branch before the activation layer of the module, and finally output through the nonlinear processing of the ReLU activation layer.

[0044] like Figure 2 ,The discriminator is set as a cyclic combination of multiple convolutional layers, instance normalization layers, and LeakyReLU.

[0045] By cyclically combining multiple convolutional layers, instance normalization layers, and LeakyReLU, and adjusting the coverage of the discriminator's receptive field, more refined identification can be achieved compared to the classic discriminator. The discrimination process of the classic discriminator is to discriminate the entire image and output a single scalar from 0 to 1. The discriminator of this patent divides the entire image into multiple image blocks by adjusting the receptive field, and integrates the discrimination results of all image blocks into a matrix output. Each element in the matrix represents the quality of the image block within its receptive field, thereby more precisely guiding the generator's de-motion blur learning. The discrimination methods of the classic discriminator and the discriminator of this patent are as follows: Figure 4 , Figure 5 As shown in the figure, D represents the discrimination result of the entire image or image block, and a, b, c are the indexes of the image block.

[0046] 3) Training model;

[0047] When training the model, the feature loss of the restored image and the clear image is calculated as follows:

[0048]

[0049] in, W represents the feature map output after the activation of the jth convolutional layer before the i-th maximum pooling layer of the VGG19 network. i,j , H i,j are the width and height dimensions of the feature map, I S For a clear image, I B is a motion blurred image, G is the generator of the network, G(I B ) is the restored image output by the generator.

[0050] Different from the common perceptual loss that only uses a certain middle layer of the pre-trained deep convolutional neural network, the feature loss proposed in this patent uses multiple deep feature maps and shallow feature maps of the network to fully integrate image features at different levels, helping the network to further improve the feature similarity between the restored image and the clear image. Figure 6 As shown, the feature loss proposed in this application is introduced by taking the VGG19 network as an example.

[0051] First, the restored image and the clear image are respectively input into the pre-trained VGG19 network to obtain the feature maps of the two at different layers in the network. After that, the mean square error of the feature maps of the two is calculated, and the mean square error is used to represent the feature differences at different levels of the two. Unlike ordinary perceptual loss, this patent uses convolutional layer 2-2, convolutional layer 3-3, convolutional layer 4-4 and convolutional layer 5-4 to combine and construct feature loss, thereby comprehensively covering image features at different levels. Feature loss makes full use of the feature extraction capability of convolutional neural networks, improves the similarity of texture features between the restored image and the clear image by reducing the difference in the shallow feature maps between the restored image and the clear image, and ensures the consistency of semantic information between the two by reducing the difference in the deep feature maps between the two, helping the network to restore motion blurred images with higher quality. At the same time, the mean square error calculation of the feature map in the feature loss is to calculate the error of the feature points with position correspondence in the feature maps of the clear image and the restored image, so as to ensure the consistency of information at the corresponding positions of the restored image and the clear image before and after restoration.

[0052] 4) Use the generator of the trained model to de-motion blur the water surface image.

[0053] After the network training is completed, the generator has completed the learning of the de-motion blur task under the guidance of the discriminator. Therefore, when deploying the application, only the generator needs to be retained to achieve the de-motion blur of the water surface image.

[0054] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for enhancing a water surface motion blurred image for intelligent navigation of ships, characterized in that: The following steps are involved: 1) Obtaining a training set of water surface motion blurred images for ship intelligent navigation; 2) constructing an image motion blur removal model based on a generative adversarial network, wherein the image motion blur removal model includes: a generator and a discriminator; The generator takes the motion-blurred water surface image as input; the discriminator discriminates between the restored image generated by the generator and the real clear image. If it is judged to be the generated restored image, it will be given a low score, and if it is judged to be the real clear image, it will be given a high score. The discriminator result is passed to the generator to find the difference between the restored image and the clear image, thereby generating a higher quality restored image. 3) Training model; 4) Use the generator of the trained model to de-motion blur the water surface image.

2. The method for enhancing water surface motion blur image for ship intelligent navigation according to claim 1, characterized in that: The game relationship between the generator and the discriminator is expressed as: Among them, D is the discriminator, G is the generator, and I S For a clear image, I B is a motion blurred image, G(I B ) is the restored image output by the generator, is the gradient penalty term, which is used to ensure the stability of the training process, x is the input when the discriminator calculates the gradient, and λ GP is a hyperparameter.

3. The method for enhancing water surface motion blur image for ship intelligent navigation according to claim 1, characterized in that: The structure of the generator is as follows: three convolutional layers are set at the front end of the generator for preliminarily extracting blurred image features, two deconvolutional layers and one convolutional layer are set at the back end of the generator for reconstructing the resolution of the blurred image, and a receptive field module is set in the middle part of the generator.

4. The method for enhancing water surface motion blur image for ship intelligent navigation according to claim 4 is characterized in that: The receptive field module structure is as follows: The receptive field module has a total of 5 branches. The 5 branches first use 1×1 convolution to reduce the dimension of the input data. Before one branch after dimensionality reduction is connected to the activation layer of the module, 4 branches use convolution kernels of preset sizes to further process the features after dimensionality reduction. At the end of the 4 branches, there are dilated convolutions with different numbers of holes. After the dilated convolution, the 4 branches splice the features in the channel dimension, and then undergo 1×1 convolution and fuse with the data of another branch before the activation layer of the module, and finally output through the nonlinear processing of the ReLU activation layer.

5. The method for enhancing water surface motion blur image for ship intelligent navigation according to claim 3, characterized in that: Nine receptive field modules are arranged in the middle part of the generator.

6. The method for enhancing water surface motion blur image for ship intelligent navigation according to claim 1, characterized in that: The discriminator is set as a cyclic combination of multiple convolutional layers, instance normalization layers, and LeakyReLU.

7. The method for enhancing water surface motion blur image for ship intelligent navigation according to claim 1, characterized in that: In step 3), when training the model, the feature loss of the restored image and the clear image is calculated as follows: in, W represents the feature map output after the activation of the jth convolutional layer before the i-th maximum pooling layer of the VGG19 network. i,j , H i,j are the width and height dimensions of the feature map, I S For a clear image, I B is a motion blurred image, G is the generator of the network, G(I B ) is the restored image output by the generator.

8. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images