A high-precision underwater imaging method and device

Abstract

The present invention discloses a high-precision underwater imaging method and device, wherein the method comprises: constructing a scene radiation sub-network, the input of which is a collected underwater distorted image, and the output of which is a scene radiation map; estimating a transmittance map and a background light map from the underwater distorted image, and synthesizing the scene radiation map, the transmittance map and the background light map according to the underwater imaging formula to simulate the underwater distorted image; constructing a priori discriminant sub-network based on the underwater distorted image, the input of which is the scene radiation map, and the output of the underwater image degradation degree; optimizing the scene radiation sub-network parameters based on the underwater image degradation degree to output the optimized high-precision scene radiation map. The present invention aims at the problem of underwater images having no true value when solving underwater imaging problems using deep learning, avoids the dependence on the true value map, combines the underwater imaging model, and uses an underwater distorted image to restore a clear, high-contrast and color-corrected underwater image.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater imaging, and in particular to a high-precision underwater imaging method and device. Background Art

[0002] In recent years, marine environment detection, especially underwater detection technology, has attracted more and more attention. The urgent demand for underwater resource development has put forward requirements for high-quality underwater imaging technology. Underwater imaging technology is mainly restricted by two factors: one is the absorption of water and particles in the water, and the other is the scattering of water and particles in the water, which causes underwater image color shift, low contrast, and image blur. Therefore, restoring the normal color of underwater scenes, improving contrast, and enhancing details have become the focus of many scholars.

[0003] Traditional underwater imaging methods are divided into two parts. One part is the image restoration method based on the physical imaging model. For example, Schechner et al. proposed the classic underwater polarization imaging model, which estimates the transmittance by calculating the polarization degree of the backscattered light, and then restores the contrast-enhanced image based on the underwater imaging model. The other part is to improve the visual quality of underwater images by image enhancement. By using two degraded images that have been color corrected and filtered in advance as the input for fusion, the expected results can be obtained. However, traditional underwater imaging methods face the problems of complex algorithms and poor universality.

[0004] With the rapid development of deep learning in recent years, there are more and more methods for restoring underwater images using neural networks. The CycleGAN network is used to generate an underwater image dataset, and then another network UGAN is proposed to restore underwater images. The WaterGAN network generates a large training dataset containing relative depth, ground color images, and realistic underwater images, which are used as input to a two-stage network for color correction of monocular images. A dataset of underwater images UIEBD is constructed and a fusion-based underwater image enhancement model DUIENet is proposed. The network fuses the input and predicted confidence maps to achieve enhanced results. However, due to the particularity of the underwater environment, there are few datasets of real underwater scenes, and it is difficult to learn prior information from underwater images using a small number of datasets. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, in response to the above problems, the present invention proposes a high-precision underwater imaging method, which avoids the problem that underwater images have no true value. By combining an underwater imaging model and a visual enhancement method, a clear, high-contrast and color-corrected underwater image is restored using a distorted underwater image.

[0007] Another object of the present invention is to provide a high-precision underwater imaging device.

[0008] To achieve the above object, the present invention provides a high-precision underwater imaging method, comprising the following steps:

[0009] S1, builds a scene radiation sub-network, the input is the collected underwater distorted image, and the output is the scene radiation map;

[0010] S2, estimating a transmittance map and a background light map from the underwater distorted image, and synthesizing the scene radiation map, the transmittance map and the background light map to simulate an underwater distorted image according to an underwater imaging formula;

[0011] S3, based on the underwater distorted image, constructing a priori discriminant subnetwork, inputting the scene radiation map, and outputting the degree of underwater image degradation;

[0012] S4, based on the degree of underwater image degradation, optimizing the scene radiation sub-network parameters to output an optimized high-precision scene radiation map.

[0013] The high-precision underwater imaging method of the embodiment of the present invention constructs a scene radiation sub-network, the input is the collected underwater distorted image, and the output is the scene radiation map; the transmittance map and the background light map are estimated from the underwater distorted image, and the scene radiation map, the transmittance map and the background light map are synthesized according to the underwater imaging formula to simulate the underwater distorted image; based on the underwater distorted image, a priori discriminant sub-network is constructed, the input is the scene radiation map, and the underwater image degradation degree is output; based on the underwater image degradation degree, the scene radiation sub-network parameters are optimized to output the optimized high-precision scene radiation map. The present invention aims at the problem that the underwater image has no true value when solving the underwater imaging problem by deep learning, avoids the dependence on the true value map, combines the underwater imaging model and the visual enhancement method, and uses an underwater distorted image to restore a clear, high-contrast and color-corrected underwater image.

[0014] In addition, the high-precision underwater imaging method according to the above embodiment of the present invention may also have the following additional technical features:

[0015] Furthermore, in one embodiment of the present invention, S1 includes: introducing a self-attention mechanism and adopting a residual structure, and the output is an image with three channels of the same size as the input; wherein the self-attention mechanism includes a BN layer, a self-attention layer and a ReLU layer.

[0016] Furthermore, in one embodiment of the present invention, estimating the transmittance map and the background light map from the underwater distorted image includes: utilizing Non-local prior information of the color image and prior information of red channel attenuation of the underwater image to estimate the transmittance map and the background light map of the underwater image.

[0017] Furthermore, in one embodiment of the present invention, estimating the transmittance map and the background light map from the underwater distorted image includes: utilizing Non-local prior information of the color image and prior information of red channel attenuation of the underwater image to estimate the transmittance map and the background light map of the underwater image.

[0018] Furthermore, in one embodiment of the present invention, the S3 includes: a priori discriminant subnetwork, which uses a sigmoid function in the last layer to normalize the output to the interval [0,1], and the output is the degree of underwater image degradation; wherein the priori discriminant subnetwork includes a variety of backbone structures in the form of T2T module and Deep-Narrow using a T2T-ViT network.

[0019] Furthermore, in one embodiment of the present invention, S4 includes: establishing a loss function, the loss function includes two parts, the first part is the loss function of the scene radiation sub-network, and the second part is the degree of underwater image degradation. Based on the loss function, the Adam gradient descent algorithm is used to optimize the network parameters so that the simulated underwater distorted image approaches the collected underwater distorted image, and at the same time minimizes the degree of underwater image degradation.

[0020] Furthermore, in one embodiment of the present invention, the self-attention mechanism is introduced and the residual structure is adopted, and the output is an image with three channels of the same size as the input, including: using Non-local to estimate the transmittance map and background light map The three-channel image The underwater physical imaging formula is combined to obtain

[0021]

[0022] Furthermore, in one embodiment of the present invention, the loss function, wherein,

[0023] The first one is:

[0024] E rec =||I(x)-I'(x)|| 2

[0025] The second one is:

[0026]

[0027] in, for The brightness of the image, for The saturation of the image.

[0028] Furthermore, in one embodiment of the present invention, the step S3 further includes: converting the scene radiation map The trained priori discriminant subnetwork is input, and the output is the degradation degree P of the underwater image. The degradation degree P is added to the loss function, and the loss function is minimized. The scene radiation subnetwork is updated, and the optimized scene radiation map J(x) is output as an underwater image of preset clarity.

[0029] To achieve the above object, the present invention provides a high-precision underwater imaging device, comprising:

[0030] The first construction module is used to construct a scene radiation sub-network, the input is the collected underwater distorted image, and the output is the scene radiation map;

[0031] A synthesis simulation module, used for estimating a transmittance map and a background light map from the underwater distorted image, and synthesizing the scene radiation map, the transmittance map and the background light map to simulate an underwater distorted image according to an underwater imaging formula;

[0032] A second construction module is used to construct a priori discriminant subnetwork based on the underwater distorted image, with the scene radiation map as input and the underwater image degradation degree as output;

[0033] The image output module is used to optimize the scene radiation sub-network parameters based on the degree of underwater image degradation to output an optimized high-precision scene radiation map.

[0034] The high-precision underwater imaging device of the embodiment of the present invention comprises a first construction module for constructing a scene radiation sub-network, the input of which is a collected underwater distorted image, and the output of which is a scene radiation map; a synthesis simulation module for estimating a transmittance map and a background light map from the underwater distorted image, and synthesizing the scene radiation map, the transmittance map and the background light map to simulate the underwater distorted image according to the underwater imaging formula; a second construction module for constructing a priori discriminant sub-network based on the underwater distorted image, the input of which is a scene radiation map, and the output of the degree of underwater image degradation; and an image output module for optimizing the parameters of the scene radiation sub-network based on the degree of underwater image degradation to output an optimized high-precision scene radiation map. The present invention aims at the problem of underwater images having no true value when solving underwater imaging problems using deep learning, avoids the reliance on the true value map, combines the underwater imaging model and the visual enhancement method, and uses an underwater distorted image to restore a clear, high-contrast and color-corrected underwater image.

[0035] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 is a flow chart of a high-precision underwater imaging method according to an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a network structure according to an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of the structure of a high-precision underwater imaging device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] The following describes a high-precision underwater imaging method and device according to an embodiment of the present invention with reference to the accompanying drawings.

[0043] Figure 1 It is a flow chart of a high-precision underwater imaging method according to an embodiment of the present invention.

[0044] like Figure 1 As shown, the high-precision underwater imaging method includes the following steps:

[0045] S1, builds a scene radiation sub-network, the input is the collected underwater distorted image, and the output is the scene radiation map.

[0046] S2, estimating the transmittance map and the background light map from the underwater distorted image, and synthesizing the scene radiation map, the transmittance map and the background light map to simulate the underwater distorted image according to the underwater imaging formula;

[0047] S3, based on the underwater distorted image, builds a priori discriminant sub-network, takes the scene radiation map as input, and outputs the degree of underwater image degradation;

[0048] S4, based on the degree of underwater image degradation, optimizes the scene radiation sub-network parameters to output an optimized high-precision scene radiation map.

[0049] As an example, the present invention includes but is not limited to introducing a self-attention mechanism and adopting a residual structure, and outputting an image with three channels of the same size as the input; wherein the self-attention mechanism includes a BN layer, a self-attention layer, and a ReLU layer.

[0050] As an example, the present invention estimates the transmittance map and background light map of the underwater image by using the non-local prior information of the color image and the prior information that the red channel of the underwater image has a large attenuation.

[0051] As an example, the prior discriminant subnetwork of the present invention includes but is not limited to the T2Tmodule of the T2T-ViT network and the backbone structure in the form of Deep-Narrow. The sigmoid function is used in the last layer to normalize the output to the interval [0,1], and the output is the degree of underwater image degradation.

[0052] As an example, the present invention establishes a loss function, which includes two parts. The first part is the loss function of the scene radiation sub-network, and the second part is the degree of underwater image degradation. The Adam gradient descent algorithm is used to optimize the network parameters so that the synthesized simulated underwater distorted image is close to the collected underwater distorted image, and at the same time minimizes the degree of underwater image degradation.

[0053] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0054] Figure 2 is a schematic diagram of a network structure according to an embodiment of the present invention, as shown in Figure 2 As shown:

[0055] First, a distorted underwater image I(x) is collected by a camera underwater. The image is input into the network, and the self-attention mechanism is introduced to divide the input image into blocks, and self-attention is performed in each image block. The network contains a BN layer, a self-attention layer, and a ReLU layer, and adopts a residual structure. The output is an image with three channels of the same size as the input. Using Non-local to estimate the transmittance map and background light map The three images are combined using the underwater physical imaging formula to construct

[0056]

[0057] There are two loss functions used by the network. The first one is:

[0058] E rec =||I(x)-I'(x)|| 2

[0059] E rec The purpose of the design is to optimize the network parameters so that the output scene radiation map can better synthesize underwater distorted images.

[0060] The second one is:

[0061]

[0062] in for The brightness of the image, for The saturation of the image. The tone of the underwater image after white balance will be white. At this time, the brightness of the image will increase, the saturation will decrease, and the difference between the two will increase. In a clear image, the difference between the two will decrease, so this loss function should be minimized.

[0063] The prior network uses the T2T module of the T2T-ViT network and the Deep-Narrow backbone structure. The sigmoid function is used in the last layer to normalize the output to the interval [0,1]. No image pairs corresponding to underwater images and ground truth images are required for training. Only one part of the training set is underwater images and the other part is ground images. The input is fed into the trained prior network, and the output is the degradation degree P of the underwater image. P is added to the loss function, and the loss function is minimized, the scene radiation sub-network is updated, and the optimized scene radiation map J(x) is output as a clear and high-precision underwater image.

[0064] In summary, the present invention constructs a scene radiation sub-network, whose input is a collected underwater distorted image and whose output is a scene radiation map; estimates a transmittance map and a background light map from the collected underwater distorted image; synthesizes the scene radiation map, the transmittance map and the background light map to simulate an underwater distorted image; constructs a priori discriminant sub-network, whose input is a scene radiation map and outputs the degree of underwater image degradation; optimizes the parameters of the scene radiation sub-network to make the synthesized simulated underwater distorted image approach the collected underwater distorted image and minimize the degree of underwater image degradation at the same time; and outputs an optimized high-precision scene radiation map.

[0065] The high-precision underwater imaging method of the embodiment of the present invention addresses the problem that underwater images have no true values ​​when using deep learning to solve underwater imaging problems, avoids dependence on true value maps, combines underwater imaging models and visual enhancement methods, and uses a distorted underwater image to restore a clear, high-contrast, and color-corrected underwater image.

[0066] In order to implement the above embodiment, Figure 3 As shown, this embodiment also provides a high-precision underwater imaging device 10, which includes: a first construction module 100, a synthesis simulation module 200, a second construction module 300 and an image output module 400.

[0067] The first construction module 100 is used to construct a scene radiation sub-network, the input of which is a collected underwater distorted image, and the output of which is a scene radiation map;

[0068] The synthesis simulation module 200 is used to estimate the transmittance map and the background light map from the underwater distorted image, and synthesize the scene radiation map, the transmittance map and the background light map to simulate the underwater distorted image according to the underwater imaging formula;

[0069] The second construction module 300 is used to construct a priori discriminant sub-network based on the underwater distorted image, with the input being the scene radiation map and the output being the degree of underwater image degradation;

[0070] The image output module 400 is used to optimize the scene radiation sub-network parameters based on the degree of underwater image degradation to output an optimized high-precision scene radiation map.

[0071] Furthermore, the above-mentioned first building module 100 is also used to: introduce a self-attention mechanism and adopt a residual structure to output an image with three channels of the same size as the input; wherein the self-attention mechanism includes a BN layer, a self-attention layer and a ReLU layer.

[0072] According to the high-precision underwater imaging device of the embodiment of the present invention, the first construction module is used to construct a scene radiation sub-network, the input is the collected underwater distorted image, and the output is the scene radiation map; the synthesis simulation module is used to estimate the transmittance map and the background light map from the underwater distorted image, and the scene radiation map, the transmittance map and the background light map are synthesized to simulate the underwater distorted image according to the underwater imaging formula; the second construction module is used to construct a priori discriminant sub-network based on the underwater distorted image, the input is the scene radiation map, and the underwater image degradation degree is output; the image output module is used to optimize the scene radiation sub-network parameters based on the underwater image degradation degree to output the optimized high-precision scene radiation map. The present invention aims at the problem that the underwater image has no true value when solving the underwater imaging problem by deep learning, avoids the dependence on the true value map, combines the underwater imaging model and the visual enhancement method, and uses an underwater distorted image to restore a clear, high-contrast and color-corrected underwater image.

[0073] It should be noted that the above explanation of the embodiment of the high-precision underwater imaging method is also applicable to the high-precision underwater imaging device of this embodiment, and will not be repeated here.

[0074] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0076] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A high-precision underwater imaging method, characterized in that: The following steps are involved: S1, builds a scene radiation sub-network, the input is the collected underwater distorted image, and the output is the scene radiation map; S2, estimating a transmittance map and a background light map from the underwater distorted image, and synthesizing the scene radiation map, the transmittance map and the background light map to simulate an underwater distorted image according to an underwater imaging formula; S3, based on the underwater distorted image, constructing a priori discriminant subnetwork, inputting the scene radiation map, and outputting the degree of underwater image degradation; S4, optimizing the scene radiation sub-network parameters based on the underwater image degradation degree to output an optimized high-precision scene radiation map; Said S1 comprises: A self-attention mechanism is introduced and a residual structure is adopted, wherein the output is an image with three channels of the same size as the input; wherein the self-attention mechanism includes a BN layer, a self-attention layer and a ReLU layer; The S4 comprises: Establish a loss function, which includes two parts: the first part is the loss function of the scene radiation sub-network, and the second part is the degree of underwater image degradation. Based on the loss function, the Adam gradient descent algorithm is used to optimize the network parameters so that the simulated underwater distorted image approaches the collected underwater distorted image, and at the same time minimizes the degree of underwater image degradation; The self-attention mechanism is introduced and the residual structure is adopted, and the output is an image with three channels of the same size as the input, including: Using Non-local to estimate the transmittance map and background light map The three-channel image It is constructed by combining the underwater physical imaging formula The loss function, where The first one is: The second one is: in, for The brightness of the image, for The saturation of the image.

2. The high-precision underwater imaging method according to claim 1, characterized in that: The estimating a transmittance map and a background light map from the underwater distorted image comprises: The transmittance map and background light map of the underwater image are estimated by using the non-local prior information of the color image and the prior information of the red channel attenuation of the underwater image.

3. The high-precision underwater imaging method according to claim 1, characterized in that: The S3 includes: The prior discriminant subnetwork uses a sigmoid function in the last layer to normalize the output to the interval [0,1], and the output is the degree of underwater image degradation; wherein the prior discriminant subnetwork includes a variety of backbone structures in the form of T2T module and Deep-Narrow using a T2T-ViT network.

4. The high-precision underwater imaging method according to claim 3, characterized in that: The S3 further includes: The scene radiance map The trained prior discriminant subnetwork is input, and the output is the degradation degree P of the underwater image. The degradation degree P is added to the loss function, and the loss function is minimized. The scene radiation subnetwork is updated, and the optimized scene radiation map J(x) is output as an underwater image of preset clarity.

5. A high-precision underwater imaging device using the method as claimed in claim 1, characterized in that: include: The first construction module is used to construct a scene radiation sub-network, the input is the collected underwater distorted image, and the output is the scene radiation map; A synthesis simulation module, used for estimating a transmittance map and a background light map from the underwater distorted image, and synthesizing the scene radiation map, the transmittance map and the background light map to simulate an underwater distorted image according to an underwater imaging formula; A second construction module is used to construct a priori discriminant subnetwork based on the underwater distorted image, with the scene radiation map as input and the underwater image degradation degree as output; The image output module is used to optimize the scene radiation sub-network parameters based on the degree of underwater image degradation to output an optimized high-precision scene radiation map.

6. The high-precision underwater imaging device according to claim 5, characterized in that: The first building block is further used for: A self-attention mechanism is introduced and a residual structure is adopted, and the output is an image with three channels of the same size as the input; wherein the self-attention mechanism includes a BN layer, a self-attention layer and a ReLU layer.

Images