An image rectification method in deep-sea environment based on deep learning

Abstract

The application discloses a kind of based on deep learning's image correction method under deep sea environment, comprising the following steps: S10, obtain deep sea image;S20, according to sensor parameter screening needs correction image;S30, to be corrected image is input to feature extraction network and extracts feature;S40, the feature extracted and original image are input to generation network together and obtain the image after correction.The deep sea image correction method provided in the application can eliminate the error generated by the large shaking of the imaging system to improve the accuracy of deep sea three-dimensional imaging.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image correction method in deep-sea environment based on deep learning. BACKGROUND

[0002] There has been more development in deep-sea exploration using autonomous underwater vehicles (AUVs) and unmanned underwater vehicles (UUVs). However, stable shooting in underwater environment is still a major application problem. It is difficult for underwater vehicles to maintain a fixed height and angle when shooting pictures. Compared with images in nature, underwater images have the problem of low visibility. First, light is absorbed when sunlight is reflected by the water surface. In addition, absorption greatly reduces the light energy of the environment. Random attenuation and scattering reduce the contrast of the scene. In addition to the fluctuation of seawater itself, it is very difficult to shoot images at a fixed height and angle. Therefore, the images must be corrected.

[0003] In the past two decades, researchers have been focusing on improving the quality of underwater images, trying to correct the images through various mathematical transformations to obtain images at a set height and angle. However, traditional image correction methods have many limitations. On the one hand, the established mathematical correction model can only be applied to certain specific scenes and is difficult to have universality. On the other hand, the process of establishing a mathematical correction model is complex and requires a lot of time cost.

[0004] Compared with traditional methods, the image correction method based on deep learning has the characteristics of wide application range, strong correction ability, and simple model establishment. By training the network model in different scenes, a network model with wide application can be obtained. At the same time, for new application scenarios, only a small amount of data is needed to train the original model, so that the model can be applied to new scenarios. SUMMARY

[0005] Therefore, the present application proposes an image correction method in deep-sea environment based on deep learning, comprising the following steps:

[0006] S10, obtaining a deep-sea shooting image;

[0007] S20, screening images that need to be corrected according to sensor parameters;

[0008] S30, inputting the image to be corrected into a feature extraction network to extract features;

[0009] S40, inputting the extracted features and original image into a generation network to obtain a corrected image. Figure One

[0010] Preferably, the sensor comprises a height gauge and a gyroscope. ​

[0011] Preferably, the method for screening images needing correction according to sensor parameters is:

[0012] S21, each image obtained corresponds to one altimeter data and one gyroscope data;

[0013] S22, judging whether the camera is at a set height and whether rotation occurs at the time through the two data;

[0014] S23, if so, the image is judged as a to-be-corrected image.

[0015] Preferably, the to-be-corrected image is input to a feature extraction network to extract features, specifically: the to-be-corrected image is input to a feature extraction network, the network includes multiple convolution layers and pooling layers, and image features are extracted layer by layer, and the formula through the feature extraction network is:

[0016] x1=CNN(x)

[0017] x2=Attention(x1)

[0018] x3=FC(x2)

[0019] Wherein, x is the to-be-corrected image, CNN represents the convolutional neural network layer, x1 is the feature map output after the convolutional neural network layer; Attention represents the self-attention layer, x2 is the feature map output after the self-attention layer; FC represents the full connection layer, and x3 is the feature map output after the full connection layer.

[0020] Preferably, the convolutional neural network layer includes a convolution layer, a pooling layer and a nonlinear layer, wherein the formula corresponding to the nonlinear layer is,

[0021] y nl =ReLU(x3)

[0022] Wherein, y nl is the feature map output after the nonlinear layer, ReLU represents the nonlinear function, and x3 is the image output after the full connection layer.

[0023] Preferably, the convolution layer adopts a convolution kernel with a size of 3x3 and a step of 1.

[0024] Preferably, the pooling layer adopts a sliding window with a size of 2x2 and a step of 2.

[0025] Preferably, the self-attention layer adopts a multi-head attention mechanism to extract features.

[0026] Preferably, the generation network includes a deconvolution layer, a feature splicing layer, a convolution layer and an output layer.

[0027] Preferably, the feature splicing layer specifically deconvolves the feature map extracted by the feature extraction network, and then splices the low-dimensional feature map.

[0028] Beneficial effects: In most three-dimensional imaging tasks, the camera needs to maintain a certain direction and distance. However, in the deep sea environment, because of the fluctuation of seawater, the imaging device will also continue to shake, which will seriously affect the accuracy of three-dimensional imaging. The present application corrects the error caused by the movement of the shooting system when shooting underwater. For example, the shooting system rises due to the fluctuation of seawater, which is higher than the set height, resulting in the image captured being smaller than the expected size; the shooting system cannot maintain horizontal when moving, resulting in left and right shaking, resulting in the image captured being deformed. The system can repair these problems, reducing the difficulty of applying computer vision technology to the deep sea environment. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The step flow chart of the image correction method in the deep sea environment based on deep learning of the embodiment of the present application is shown in the figure.

[0030] Figure 2 The feature extraction network schematic diagram of the image correction method in the deep sea environment based on deep learning of the embodiment of the present application is shown in the figure.

[0031] Figure 3 The generation network schematic diagram of the image correction method in the deep sea environment based on deep learning of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0033] On the contrary, the present application covers any alternative, modification, equivalent method and scheme made on the essence and scope of the present application defined by the claims. Further, in order to make the public have a better understanding of the present application, some specific details are described in detail in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0034] In order to achieve the above purpose, the present application proposes an image correction method in the deep sea environment based on deep learning, which can correct the deep sea image and obtain the image with fixed height and angle. Referring to Figure 1 , comprising the following steps:

[0035] S10, acquiring a deep-sea shooting image;

[0036] S20, screening the image needing correction according to the sensor parameters;

[0037] S30, inputting the image to be corrected into a feature extraction network to extract features;

[0038] S40, inputting the extracted features and the original image into a generation network to obtain a corrected image. Figure One

[0039] The sensor includes an altimeter and a gyroscope, and the method for screening the image to be corrected according to the sensor parameters is:

[0040] S21, each image obtained corresponds to an altimeter data and a gyroscope data;

[0041] S22, judging whether the camera is at a set height and whether rotation occurs at the time through the two data;

[0042] S23, if so, the image is judged as an image to be corrected.

[0043] The image to be corrected is input into a feature extraction network to extract features, specifically: the image to be corrected is input into a feature extraction network, referring to Figure 2 , the network includes multiple convolutional layers and pooling layers, and the image features are extracted layer by layer, the convolutional layer uses a convolution kernel with a size of 3x3 and a step of 1.

[0044] Convolutional layer 1 includes 2 layers of convolution, the first layer of convolution has 3 input channels and 256 output channels. The second layer of convolution has 256 input channels and 512 output channels.

[0045] The pooling layer 1 uses a sliding window with a size of 2x2 and a step of 2.

[0046] The formula corresponding to the nonlinear layer 1 is:

[0047] y1 = ReLU(x)

[0048] Convolutional layer 2 includes 3 layers of convolution. The first layer of convolution has 256 input channels and 512 output channels. The second layer of convolution has 512 input channels and 1024 output channels. The third layer has 1024 input channels and 2048 output channels.

[0049] The pooling layer 2 and the pooling layer 1 use the same structure.

[0050] The nonlinear layer 2 and the nonlinear layer 1 use the same structure.

[0051] The self-attention layer includes 3 layers, each of which uses a 3-head self-attention structure.

[0052] The last layer of the feature extraction network is a fully connected layer. The formula through the feature extraction network is: ​

[0053] x1 = CNN(x)

[0054] x2 = Attention(x1)

[0055] x3 = FC(x2)

[0056] wherein x is the image to be corrected, CNN represents a convolutional neural network layer, x1 is a feature map output after the convolutional neural network layer, Attention represents a self-attention layer, x2 is a feature map output after the self-attention layer, and FC represents a fully connected layer, and x3 is a feature map output after the fully connected layer.

[0057] The convolutional neural network layer comprises a convolutional layer, a pooling layer and a nonlinear layer, wherein the convolutional layer corresponds to the formula: y2 = Conv_3(x), Conv_3 represents a convolutional layer with a convolution kernel size of 3x3 and a step size of 1; the pooling layer corresponds to the formula: y3 = maxpooling(x), maxpooling represents a pooling layer with a size of 2x2 and a step size of 2; and the nonlinear layer corresponds to the formula: y4 = ReLU(x3), wherein y4 is a feature map output after the nonlinear layer, ReLU represents a nonlinear function, and x3 is a feature map output after the fully connected layer. nl nl

[0058] The self-attention layer adopts a multi-head attention mechanism to extract features.

[0059] The generation network comprises a deconvolutional layer, a feature concatenation layer, a convolutional layer and an output layer. The formula of the deconvolutional layer is:

[0060] o = s*(i-1) + k-2*p

[0061] wherein i is an input size, o is an output size, k is a convolution kernel size, p is padding, and s is a step size.

[0062] The feature concatenation layer specifically performs deconvolution on the feature map extracted by the feature extraction network, and then concatenates the deconvolution result with a low-dimensional feature map.

[0063] Referring to Figure 3 , the deconvolutional layer enlarges the extracted feature map through deconvolution; the feature concatenation layer concatenates the deconvolution result with a low-dimensional feature map; the convolutional layer performs convolution on the concatenated feature map to obtain multiple output results; and the output layer averages the multiple output results of the convolutional layer to output a final corrected image.

[0064] The above completes the construction of the entire neural network, and the steps for obtaining a data set required for training include:

[0065] ​​S1, combine the camera, altimeter and gyroscope into a shooting system and put it into the pool;

[0066] S2, set a certain height and take a picture at this height as the rectification target image;

[0067] S3, move the shooting system up and down and take a series of pictures, read the altimeter and gyroscope data as the image to be rectified;

[0068] S4, rotate the shooting system at a certain angle in a certain direction at the set height and take a picture as the image to be rectified, read the altimeter and gyroscope data;

[0069] S5, calculate the intersection of the camera center and the shooting bottom at this angle, and set a height above the intersection to take a picture as the rectification target image;

[0070] S6, rotate the shooting system at a certain angle in a certain direction below the set height and take a picture as the image to be rectified, read the altimeter and gyroscope data;

[0071] S7, calculate the intersection of the camera center and the shooting bottom at this angle, and set a height above the intersection to take a picture as the rectification target image;

[0072] S8, rotate the shooting system at a certain angle in a certain direction above the set height and take a picture as the image to be rectified, read the altimeter and gyroscope data;

[0073] S9, calculate the intersection of the camera center and the shooting bottom at this angle, and set a height above the intersection to take a picture as the rectification target image.

[0074] Repeat the above steps to take enough training data sets.

[0075] Use the obtained data set to train the neural network: multiply the rectified image with the cosine of the system inclination obtained by the gyroscope data, and add the height data as the input of the entire network; calculate the mean square error between the output of the network and the corresponding rectification target image, and use the gradient descent method to train the network.

[0076] The above only describes the preferred embodiments of the present application and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based image correction method in a deep-sea environment, characterized in that: The following steps are involved: S10, acquiring deep-sea images; S20, screening images that need to be corrected according to sensor parameters; S30, inputting the image to be corrected into a feature extraction network to extract features; S40, inputting the extracted features and the original image together into a generative network to obtain a rectified image; The sensors include an altimeter and a gyroscope; The method for screening images that need to be corrected according to sensor parameters is as follows: S21, each acquired image corresponds to an altimeter acquisition data and a gyroscope acquisition data; S22, judging whether the camera is at the set height and whether it is rotating based on these two data; S23, if yes, the image is determined to be an image to be corrected; The image to be corrected is input into the feature extraction network to extract features. Specifically, the image to be corrected is input into the feature extraction network. The network includes multiple convolutional layers and pooling layers, and extracts image features layer by layer. The formula of the feature extraction network is: x1=CNN(x) x2=Attention(x1) x3=FC(x2) Where x is the image to be corrected, CNN represents the convolutional neural network layer, x1 is the feature map output after the convolutional neural network layer; Attention represents the self-attention layer, x2 is the feature map output after the self-attention layer; FC represents the fully connected layer, x3 is the feature map output after the fully connected layer; The convolutional neural network layer includes a convolution layer, a pooling layer and a nonlinear layer, wherein the formula corresponding to the nonlinear layer is: and nl =ReLU(x3) Among them, y nl is the feature map output after the nonlinear layer, ReLU represents the nonlinear function, and x3 is the feature map output after the fully connected layer; The convolution layer uses a convolution kernel with a size of 3×3 and a stride of 1; The pooling layer uses a sliding window with a size of 2×2 and a stride of 2; The self-attention layer uses a multi-head attention mechanism to extract features; The generation network includes a deconvolution layer, a feature splicing layer, a convolution layer and an output layer; The feature splicing layer specifically deconvolves the feature map extracted by the feature extraction network and then splices it with the low-dimensional feature map.

Images