A method for defogging sea surface target images

Abstract

The present invention relates to a method for defogging a sea surface target image, comprising: S1: collecting a sea surface image, and using an image fogging algorithm to fog the sea surface image to generate a sea surface foggy image, and forming a training set with the sea surface image and the sea surface foggy image; S2: constructing an image defogging model, and training the image defogging model through the training set, so that the output result of the sea surface foggy image after passing through the image defogging model is close to its corresponding sea surface image; S3: defogging the sea surface foggy image to be defogged through the trained image defogging model. The present invention proposes a multi-scale fusion network structure, which obtains deeper image information by fusing feature layers of different scales, overcomes the problems of halo phenomenon, low brightness, color distortion, etc. of traditional defogging algorithms, can retain image detail information to a greater extent, and improves the defogging effect.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a method for defogging a sea surface target image. Background Art

[0002] Water transport is a sustainable green transportation mode. Whether it is a water transport network composed of inland rivers or sea channels, it can greatly alleviate the transportation pressure of roads and railways, greatly improve transportation efficiency and efficiency, and significantly reduce transportation costs. With the continuous development of image technology, more and more ships are equipped with electronic eyes. On the one hand, they can be used for electronic cruising, effectively control the situation at sea, and reduce manpower input; on the other hand, they can identify passing ships, grasp the information of waterway facilities, and ensure navigation safety. However, the information collected by electronic monitoring is affected to varying degrees by climate factors, especially by foggy days, which is directly related to the clarity of the image collected. For the recognition of sea surface targets, it is necessary to accurately extract features through clear images, so it is particularly sensitive to foggy weather conditions. Due to the foggy weather, the sea scene is affected by the scattering effect of foggy particles, the image will be more blurred, and the quality of the collected image will be seriously attenuated, causing the relevant application systems to be affected to a greater extent. Based on this kind of situation, in order to enable the image recognition system to operate normally in foggy weather and ensure the stability and reliability of the system, it is necessary to defog the sea surface foggy image.

[0003] In recent years, with the rapid development of deep learning technology in the field of image vision, the successful application of convolutional neural network (CNN) technology in advanced image vision fields such as face recognition and image classification has made more and more researchers begin to pay attention to image dehazing algorithms based on neural networks. At present, machine learning and deep learning methods have been applied to the field of image dehazing and have achieved ideal results. However, experiments have found that these algorithms are still slightly insufficient in marine dehazing. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a method for defogging sea surface target images.

[0005] The specific plan is as follows:

[0006] A method for defogging a sea surface target image comprises the following steps:

[0007] S1: Collect sea surface images, and use image fogging algorithm to fog the sea surface images to generate sea surface foggy images, and the sea surface images and sea surface foggy images form a training set;

[0008] S2: Construct an image defogging model and train the image defogging model through the training set so that the output result of the sea foggy image after passing through the image defogging model is close to its corresponding sea surface image; the structure of the image defogging model is as follows:

[0009] The input of the image defogging model is the color image corresponding to the sea foggy image. The images of the red, green and blue color channels in the input color image are processed as follows to obtain the corresponding starting layers:

[0010] After an image of a color channel passes through two convolutional layers continuously, the output of the first convolutional layer is fused with the output of the second convolutional layer to obtain a new feature layer. After that, the new feature layer is activated by the maximum pooling and ReLU function to obtain the starting layer corresponding to the color channel.

[0011] After the starting layers corresponding to the three color channels are fused, they pass through the last convolutional layer, and the output of the last convolutional layer is the output of the image dehazing model;

[0012] S3: Defogging the foggy sea surface image to be defogged using the trained image defogging model.

[0013] Furthermore, the marine foggy image dataset is obtained by fogging the sea surface images using two different image fogging algorithms.

[0014] Furthermore, the first image fogging algorithm controls the fog density by fixing the atmospheric light value A1 while changing the atmospheric system parameter β, and the second image fogging algorithm directly changes the atmospheric light value A2 and the atmospheric transmittance t2.

[0015] Furthermore, the Relu activation function is used in the convolutional layer.

[0016] The present invention adopts the above technical solution and proposes a multi-scale fusion network structure, which obtains deeper image information by fusing feature layers of different scales, overcomes the problems of halo phenomenon, low brightness, color distortion, etc. in traditional dehazing algorithms, and can retain the detailed information of the image to a greater extent, thereby improving the dehazing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of an embodiment of the present invention.

[0018] Figure 2 Shown is a structural schematic diagram of the image defogging model in this embodiment.

[0019] Figure 3 FIG. 4 is a schematic diagram of the convolution process of the R color channel in this embodiment.

[0020] Figure 4Shown is a schematic diagram of multi-scale feature fusion in this embodiment.

[0021] Figure 5 FIG. 4 is a schematic diagram of the maximum pooling operation in this embodiment. DETAILED DESCRIPTION

[0022] To further illustrate the embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0023] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0024] The embodiment of the present invention provides a method for defogging a sea surface target image. Figure 1 As shown, the method comprises the following steps:

[0025] S1: Collect sea surface images, and use image fogging algorithm to add fog to the sea surface images to generate sea surface foggy images, and the sea surface images and sea surface foggy images together form a training set.

[0026] Since a large amount of training data samples are first required when training the network for target recognition, the image defogging network is no exception. At present, there are many different scene types in the foggy image dataset on land, but there is almost no foggy image dataset on the sea surface, so it needs to be produced in this embodiment. This dataset fogs the natural image through the fogging algorithm. In recent years, there are many algorithms that can defog the image, but there are fewer algorithms for fogging the image. At present, the foggy image dataset of the land background is also simulated by the fogging algorithm.

[0027] In order to prevent the network from overfitting, the sea surface fog image dataset in this embodiment is obtained by fogging the sea surface image using two different fogging algorithms. The first one controls the fog density by fixing the atmospheric light value A1 and changing the atmospheric system parameter β, and the second one directly changes the atmospheric light value A2 and the atmospheric transmittance t2. The principles of the two fogging algorithms are both based on the atmospheric scattering model, but the implementation methods are different.

[0028] (1) Image fogging algorithm based on atmospheric system parameters

[0029] Because the sea surface environment is different from the land, the sea surface background is relatively simple, mainly consisting of three parts: sky, water, and ships. Therefore, it does not require a lot of computing power to estimate the depth of the scene. The specific steps of the fogging algorithm are as follows:

[0030] ①Through the function f resizeThe images are uniformly compressed into a fixed size, such as (640, 480), as shown in formula (1).

[0031] J t =f resize (J)→(640,480) (1)

[0032] ② According to formula (2), the range of β is controlled between 0.05 and 0.074, and fogging effects of different concentrations are achieved by continuously increasing β.

[0033] β f =0.05+0.04*f(f=0,1,2…6) (2)

[0034] ③ According to formula (1), the image size is determined. For the convenience of expression, row is used to represent the width of the image, col is used to represent the length of the image, S is used to represent the size of the fog, and C is used to represent the size of the fog. e Indicates the atomization center:

[0035]

[0036]

[0037] ④ According to the fog size and fog center determined by formula (3) and formula (4), the scene depth d1 and transmittance t1 of the entire image are determined and substituted into the atmospheric scattering model to obtain a foggy image, as shown in formula (5) and formula (6).

[0038]

[0039]

[0040] Among them, J t (j,i,c) represents image J t The pixel value of the jth row and ith column in the c color channel (the value of each pixel after dividing by 255), A1 is set to 0.6, J f1 Indicates the image obtained after adding fog.

[0041] The change of system parameter β will lead to the change of transmittance t1 of the image, so that the image will produce a foggy effect, and the concentration of fog will gradually increase with the increase of β value.

[0042] (2) Image fogging algorithm based on atmospheric model

[0043] The image fogging algorithm based on the atmospheric model uses a double random method to randomly generate the atmospheric light value A2 and the transmission rate t2. The specific implementation steps are as follows:

[0044] ① The same as the fogging algorithm based on atmospheric system parameters, first through the function fresize The images are uniformly compressed into (640, 480), as shown in formula (7).

[0045] J t =f resize (J)→(640,480) (7)

[0046] ② Randomly generate atmospheric illumination values. In order to distinguish it from the fogging algorithm based on atmospheric system parameters, in this embodiment, the range of A2 is set to be randomly generated between 0.6-0.95, that is:

[0047] A2=f rand (0.6, 0.95) (8)

[0048] ③ After determining the atmospheric light value A2 according to formula (8), the atmospheric transmittance t2 is randomly generated and the range is set between 0.3-0.58, that is:

[0049] t f =f rand (0.3+f*0.4, 0.3+(f+1)*0.4) f=0, 1, 2...(9)

[0050] ④ By substituting the atmospheric light value and transmission rate determined by formula (8) and formula (9) into the atmospheric scattering model, a foggy image can be obtained, as shown in formula (10).

[0051] J f2 (c) = J t (c)*t f +A2*(1-t f ) c∈r,g,b (10)

[0052] Among them, J t (c) represents image J t In the c color channel pixel value (the value of each pixel after dividing by 255), J f2 Indicates the image obtained after adding fog.

[0053] As the atmospheric transmittance t f The natural image will produce a foggy effect. In this embodiment, the foggy levels range from mild to severe, and the ranges of t are 0.3-0.34, 0.34-0.38, 0.38-0.42, 0.42-0.46, and 0.46-0.5, respectively. The atmospheric light value is randomly generated between (0.6, 0.95) each time.

[0054] Through the above two different fogging algorithms, the overfitting phenomenon in the network model training process is avoided. In this embodiment, three different sea surface scenes are selected to convert the natural image into 14 foggy scene images.

[0055] S2: Construct an image defogging model and train the image defogging model through the training set so that the output result of the sea foggy image after passing through the image defogging model is close to its corresponding sea surface image.

[0056] The network structure used by the image defogging model in this embodiment is composed of seven convolutional layers and three maximum pooling layers. Figure 2 As shown in the figure, each color channel of the color image in the sea fog image dataset passes through convolution layers of different scales to extract features and fuse feature layers of different scales to obtain deeper image information, thereby achieving the purpose of defogging.

[0057] The convolutional neural network has a convolutional layer, a pooling layer and a fully connected layer. The network model used in this embodiment is composed of a convolutional layer and a pooling layer. The convolutional layer is the core layer of the convolutional neural network. It is composed of many filters, each of which has only a small part and is only connected to a small part of the original image at a time, thereby improving the calculation efficiency. Since there are many results after the convolutional layer, and due to the existence of the sliding window, a lot of information overlaps, so there is a pooling layer. It divides the results obtained by the convolutional layer into several parts without overlap, and then selects the maximum value, average value or two-norm of each part. The maximum value is selected in the model in this embodiment. The specific implementation process of the model is as follows:

[0058] (1) By Figure 2 It can be seen that the input image passes through two convolutional layers and one maximum pooling layer, and the first starting layer (Dense Layer) can be obtained through the DepthConcat operation. Figure 3 Shows the convolution process of the R color channel and the foggy sea surface image J f After processing, three color channels, R, G, and B, are generated respectively, and each color channel is downsampled by the convolution kernel. The size of the first convolution kernel is 3*3. According to formula (11), the image size after the downsampling operation can be obtained.

[0059]

[0060] Wherein, W represents the size of the sea fog image, S represents the step size, C represents the convolution kernel size, and P represents the padding size. In this embodiment, the step size S is set to 1. If the size of the image remains unchanged after each convolution operation, it is necessary to set N = W and substitute it into formula (11) to obtain the value of the padding size P, which is:

[0061]

[0062] Therefore, the result of the first layer convolution operation J can be obtained f_1 , as shown in formula (13).

[0063] J f_1 =Relu.Conv1(J f ,C1,P1,N1) C1=3; P1=1; N1=3 (13)

[0064] Among them, Conv1 represents the first layer of convolution function, using Relu activation function, C1 and P1 represent the first layer of convolution kernel and padding size respectively, and N1 represents the number of convolution kernels (number of output channels).

[0065] (2) Similar to the first stage process, such as Figure 4 The image J obtained by the first layer of convolution operation is shown f_1 Image J is obtained through the second layer of convolution operation f_2 The two feature layers are fused through DepthConcat to obtain a new feature layer image. As shown in equation (14) and equation (15).

[0066]

[0067] J f_2 =Relu.Conv2(J f ,C2,P2,N2) C2=5; P2=2; N2=3 (15)

[0068] Among them, Conv2 represents the second-layer convolution function, using the Relu activation function, C2 and P2 represent the second-layer convolution kernel and padding size respectively, N2 represents the number of second-layer convolution kernels, and N d1 Indicates the number of DepthConcat1 output channels.

[0069] (3) New feature layer image After activation by the maximum pooling and Relu function, the first starting layer (Dense Layer) is obtained. The specific implementation process is as follows Figure 5As shown. The difference from step (1) here is that there is an additional maximum pooling layer. The addition of the pooling layer is because the convolution layer in a typical convolutional neural network convolves a neighborhood of an image to obtain the neighborhood features of the image, and the subsampling layer uses pooling technology to integrate the feature points in the neighborhood to obtain new features. The role of pooling is to reduce features and parameters, but the purpose of pooling in this embodiment is not this, its purpose is to maintain certain invariance (rotation, translation, scaling, etc.). The error in feature extraction mainly comes from two aspects: the increase in valuation variance caused by the limited neighborhood size and the estimated mean shift caused by the convolution layer parameter error. Generally speaking, average pooling can reduce the first error and retain more background information of the image; maximum pooling can reduce the second error and retain more texture information. Therefore, the maximum pooling layer is used in this embodiment. Random pooling is between the two. By assigning probabilities to pixels according to the numerical size and then performing subsampling according to the probability, it also obeys the maximum pooling criterion in a local sense.

[0070] The maximum pooling operation must also satisfy the principle of constant sum of gradients. Its forward propagation is to pass the largest value in the neighborhood to the next layer, while the values ​​of other pixels are directly discarded. Then the back propagation is to pass the gradient directly to a certain pixel in the previous layer, while other pixels do not receive the gradient, that is, 0. Therefore, the difference between the maximum pooling operation and the average pooling operation is that it is necessary to record the location of the maximum value pixel during the pooling operation, because the position correspondence is required in the back propagation. According to formula (16), the value of the first starting layer can be obtained.

[0071]

[0072] in, Represents the value of the first starting layer after being activated by the Relu function. Indicates the number of output channels of the first starting layer. Maxp1 indicates the maximum pooling function of the first layer.

[0073] (4) The specific calculation process of the model is similar to the above steps, as shown in equations (17), (18) and (19).

[0074]

[0075]

[0076] Among them, Conv3, Conv4, and Conv5 represent the third, fourth, and fifth convolution functions, respectively, using the Relu activation function. C3 and P3 represent the third-layer convolution kernel and padding size, respectively. N3 represents the number of second-layer convolution kernels. C5 and P5 represent the fifth-layer convolution kernel and padding size, respectively. N5 represents the number of fifth-layer convolution kernels. d2Indicates the number of output channels of DepthConcat2. Dense2 Represents the value of the second starting layer after being activated by the Relu function, N Dense2 Indicates the number of output channels of the second starting layer. Maxp2 indicates the maximum pooling function of the second layer.

[0077]

[0078] Among them, Conv6 represents the sixth layer convolution function, using Relu activation function, C6 and P6 represent the sixth layer convolution kernel and padding size respectively, N6 represents the number of sixth layer convolution kernels, N d3 Indicates the number of DepthConcat3 output channels. Dense3 Represents the value of the third starting layer after being activated by the Relu function, N Dense3 Indicates the number of output channels of the third starting layer. Maxp3 indicates the maximum pooling function of the third layer.

[0079] Therefore, the final model output Y is shown in Equations (20) and (21).

[0080] J D =Concat(J fD1 ,J fD2 ,J fD3 ,N D ) N D =3 (20)

[0081] Y = Relu.Conv7(J D ,C7,P7,N7) C7=3; P7=1; N7=3 (21)

[0082] Among them, Conv7 represents the seventh-layer convolution function, which adopts Relu activation function, C7 and P7 represent the seventh-layer convolution kernel and padding size respectively, and N7 represents the number of seventh-layer convolution kernels.

[0083] The error function in this embodiment adopts the mean square error (MSE). In mathematical statistics, the mean square error refers to the expected value of the square of the difference between the parameter estimate and the true value of the parameter. The true value here is a natural fog-free image. As shown in formula (21), MSE can evaluate the degree of change of the data. The smaller the MSE, the higher the accuracy of the prediction model in describing the experimental data.

[0084]

[0085] Among them, n represents the number of training samples, m represents the number of image pixels, and J t represents the foggy sea surface image, Y iRepresents the i-th pixel value of the output image. The output result of the sea foggy image after passing through the image defogging model is close to its corresponding sea surface image, that is, the mean square error is less than the preset error threshold.

[0086] S3: Defogging the foggy sea surface image to be defogged using the trained image defogging model.

[0087] The embodiment of the present invention proposes a multi-scale fusion network structure, which obtains deeper image information by fusing feature layers of different scales, thereby achieving the purpose of defogging. By comparing with other different defogging algorithms, the multi-scale fusion-based convolutional neural network single image defogging method adopted by the present invention may retain more image detail information and also has better performance in objective image evaluation standards.

[0088] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A method for defogging a sea surface target image, characterized in that: The following steps are involved: S1: Collect sea surface images, and use image fogging algorithm to fog the sea surface images to generate sea surface foggy images, and the sea surface images and sea surface foggy images form a training set; S2: Construct an image defogging model and train the image defogging model through the training set so that the output result of the sea foggy image after passing through the image defogging model is close to its corresponding sea surface image; the structure of the image defogging model is as follows: The input of the image defogging model is the color image corresponding to the sea foggy image. The images of the red, green and blue color channels in the input color image are processed as follows to obtain the corresponding starting layers: After an image of a color channel passes through two convolutional layers continuously, the output of the first convolutional layer is fused with the output of the second convolutional layer to obtain a new feature layer. After that, the new feature layer is activated by the maximum pooling and ReLU function to obtain the starting layer corresponding to the color channel. The scale of the convolutional layers that each color channel passes through is different; After the starting layers corresponding to the three color channels are fused, they pass through the last convolutional layer, and the output of the last convolutional layer is the output of the image dehazing model; S3: Defogging the foggy sea surface image to be defogged using the trained image defogging model.

2. The method for defogging a sea surface target image according to claim 1, characterized in that: The marine foggy image dataset is obtained by fogging the sea surface images using two different image fogging algorithms.

3. The method for defogging a sea surface target image according to claim 2, characterized in that: The first image fogging algorithm controls the fog density by fixing the atmospheric light value A1 while changing the atmospheric system parameter β. The second image fogging algorithm directly changes the atmospheric light value A2 and the atmospheric transmittance t2.

4. The method for defogging a sea surface target image according to claim 1, characterized in that: The Relu activation function is used in the convolutional layer.

Images