Underwater optical hybrid imaging method and device for complex ocean scene

Abstract

The invention relates to an underwater optical hybrid imaging method and device for a complex ocean scene, and the method comprises the steps: presetting any clean image as an undistorted image, simulating the light absorption and scattering effects of a plurality of water bodies at a preset water depth for the undistorted image based on a pre-constructed multi-stage attenuation underwater physical imaging model, and carrying out the simulation of the light absorption and scattering effects of the undistorted image; obtaining an initial underwater optical image; performing style adjustment considering color deviation on the initial underwater optical image by using a pre-trained underwater image detail migration model to obtain a final underwater optical image; wherein the underwater image detail migration model comprises a style generator and a style discriminator, and the style generator and the style discriminator perform cooperative training based on an adversarial learning strategy. Compared with the prior art, high-quality, self-adaptive and robust underwater optical imaging of a complex ocean scene is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ocean engineering and computer vision, and particularly relates to an underwater optical hybrid imaging method and device for complex marine scenes. BACKGROUND

[0002] The underwater optical imaging model is an important theoretical basis for underwater visual processing and application research. The commonly used underwater optical imaging modeling can be divided into two categories: underwater imaging modeling based on Jaffe-McGlamery simplified model and underwater imaging modeling based on deep learning theory.

[0003] The Jaffe-McGlamery model considers that the light radiation received by the camera in the underwater optical imaging process is derived from three parts: light absorption, forward scattering and backscattering. Jaffe found that the forward scattering effect has little effect on the quality of underwater imaging by studying the influence of the inherent optical properties and external optical properties of the water body on imaging. Therefore, most of the existing methods are to use the simplified Jaffe-McGlamery model to model the underwater imaging process. However, this type of modeling method has the following limitations: 1) ignoring the influence of water depth on light absorption: only focusing on the light absorption process of the depth of field, and ignoring the light absorption process along the water surface to the camera path with the increase of water depth; 2) ignoring the influence of background light: in the light scattering process, the background light is determined by random sampling, and the underwater image appears significant color error; 3) limited modeling ability: only considering two factors of underwater light absorption and light scattering, which cannot be applied to modeling complex and variable underwater environment.

[0004] Due to the complex and changeable underwater environment, it is very difficult to establish a formulaic and quantitative description of the underwater optical image degradation model. Therefore, some researchers propose a new idea of establishing an underwater optical image degradation model with the help of deep learning theory. The modeling method based on deep learning mainly uses style transfer technology to convert atmospheric images into images with underwater style. For example, a research team from the University of Michigan uses a single generative adversarial network to model underwater imaging in three stages: 1) considering the attenuation of different lights, simulating the light absorption process; 2) considering light scattering, using a convolutional neural network to simulate the fog effect; 3) considering the characteristics of the camera lens, simulating vignetting to produce shadow effects on the edges of the image. However, this imaging model must be trained on paired images. In order to reduce the demand for image paired training, the U.S. Air Force Research Laboratory proposes to use a bidirectional recurrent consistency adversarial network to construct an underwater optical imaging model, learning the transfer between any style of atmospheric images and underwater optical images. The underwater optical imaging modeling based on deep learning theory usually uses unsupervised learning to learn the statistical characteristics of atmospheric images and underwater optical images, and directly establishes a mapping relationship to reduce the dependence on labeled data. However, this type of modeling method has the following limitations: 1) ignores the complexity and diversity of underwater optical image degradation; 2) does not fully consider the physical principles of underwater optical imaging, resulting in monotonous colors of generated underwater images and easy model collapse problems; 3) the ability to simulate the fog effect of underwater optical images is very limited.

[0005] In summary, although the research on underwater imaging model has attracted widespread attention, the existing model still has shortcomings in describing the complex optical imaging process and basic laws in the real marine environment, which limits the further development of underwater vision research. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the prior art and provide a mixed underwater optical imaging method and device for complex marine scenes, which realizes high-quality, adaptive and robust underwater optical imaging of complex marine scenes.

[0007] The purpose of the present application can be achieved by the following technical solutions:

[0008] A mixed underwater optical imaging method for complex marine scenes, the method comprising:

[0009] Pre-set any clean image as a distortion-free image, based on a pre-constructed multi-stage attenuation underwater physical imaging model, simulate the light absorption and scattering effects of multiple water bodies at a pre-set water depth on the distortion-free image to obtain an initial underwater optical image; use a pre-trained underwater image detail transfer model to adjust the style of the initial underwater optical image considering color deviation to obtain a final underwater optical image;

[0010] The underwater image detail migration model comprises a style generator and a style discriminator, and the style generator and the style discriminator are collaboratively trained based on an adversarial learning strategy.

[0011] Further, the multi-stage attenuation underwater physical imaging model comprises a light absorption branch and a light scattering branch, the light absorption branch is used to simulate a light absorption process and generate an absorption image, and the light scattering branch is used to simulate a light scattering process and calculate a global background light of an underwater image, and based on the absorption image and the global background light of the underwater image, an initial underwater optical image is obtained.

[0012] Further, the light absorption branch simulating the light absorption process comprises:

[0013] A wavelength of a blue channel, a green channel and a red channel in the distortion-free image is selected according to a preset water depth and a plurality of water body types;

[0014] According to the water body type, absorption coefficients and attenuation coefficients of the three wavelengths of light in the current water body type are determined;

[0015] According to the preset water depth, the absorption coefficients and the wavelengths of light, a water depth direction light absorption component is calculated;

[0016] A distance between an object in the image and a camera is determined according to the distortion-free image, and a scene depth is defined;

[0017] According to the attenuation coefficients, the scene depth and the absorption coefficients, a scene depth direction light absorption component is calculated;

[0018] According to the water depth direction light absorption component and the scene depth direction light absorption component, the absorption image is generated.

[0019] Further, the calculation expression of the water depth direction light absorption component is:

[0020] c λ,x =e -α(λ)×D(x) , λ ∈ {r, g, b}

[0021] Wherein, x is a pixel coordinate, λ is a wavelength, α(λ) represents an absorption coefficient of the current medium to the λ wavelength of light, T λ,x represents a residual light intensity of pixel x in the water depth direction, r, g and b respectively represent a red channel, a green channel and a blue channel, and D(x) represents a water depth path of pixel x;

[0022] The calculation expression of the scene depth direction light absorption component is:

[0023] T λ,x =e -β(λ)×d(x) , λ ∈ {r, g, b}

[0024] Wherein, Tλ,x β(λ) represents the residual light intensity in the scene depth direction of pixel x, β(λ) represents the attenuation coefficient of the current medium for light of wavelength λ, and d(x) represents the scene depth of pixel x.

[0025] The calculation expression for generating the absorption image is:

[0026]

[0027] in, I represents the intensity of pixel x in the absorbed image. λ (x) represents the light intensity value at pixel x color channel λ in a distortion-free image.

[0028] Furthermore, the light scattering branch simulates the process of light scattering, including:

[0029] Obtain preset water depths and various water body types, and select the wavelengths of the blue, green, and red channels in the distortion-free image;

[0030] Based on the water body type, determine the scattering coefficients of the three wavelengths of light in the current water body type;

[0031] Based on the preset background light intensity value of any one channel, the background light intensity values ​​of the remaining channels are calculated to obtain the global background light of the underwater image.

[0032] Furthermore, the expression for calculating the background light intensity value is as follows:

[0033]

[0034] Where B(λ) is the background light intensity of the underwater scene in the λ channel, B r B represents the background light intensity value of the red channel. g B represents the background light intensity value of the green channel. b λ represents the background light intensity value of the blue channel, λ is the wavelength, and r, g and b represent the red channel, green channel and blue channel, respectively.

[0035] Furthermore, the style generator is based on the U-Net architecture, and the style discriminator adopts the PatchGAN structure.

[0036] Furthermore, the loss function of the underwater image detail transfer model is:

[0037] L = L adv +λ1·L sty +λ2·L con

[0038] Where λ1 and λ2 represent weight parameters, L adv Indicates WGAN-GP adversarial loss, Lsty denotes style loss, L con denotes content loss;

[0039]

[0040] where D s denotes style discriminator, denotes the sampling distribution obtained by uniformly sampling from the converted optical image x st and the real underwater optical image x r l denotes the l-th layer of the pre-trained VGG-19 model, and denote the i-th and j-th vectorized feature map in the l-th layer respectively, k denotes the number of channels, G l denotes Gram matrix, L s denotes the set of layers in the VGG-19 model participating in the calculation of style loss, ω s l denotes the weight of the l-th layer, φ l denotes the l-th layer of the pre-trained VGG-19 model, L c denotes the set of layers in the VGG-19 model participating in the calculation of content loss, ω c l denotes the weight of the l-th layer, φ l (x st ) and φ l (x s ) denote the converted optical image x st feature extraction result in the l-th layer, the physically modeled image x s feature extraction result in the l-th layer of the VGG-19.

[0041] Further, in the collaborative training based on the adversarial learning strategy, the style generator is used to convert the synthesized optical image into an optical image similar in style to the real optical image, and the style discriminator is used to accurately distinguish the synthesized optical image and the real optical image.

[0042] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the underwater optical hybrid imaging method for complex marine scenarios as described above.

[0043] Compared with the prior art, the beneficial effects of the present application include:

[0044] ​1.The application realizes high-quality, adaptive and robust underwater optical imaging of complex marine scenes. In view of the scene limitation problem of the existing underwater optical imaging modeling, a hybrid modeling method is used to construct an underwater optical imaging model. A pre-constructed multi-stage attenuation underwater physical imaging model is used to simulate the light absorption and scattering effects of various water bodies at a preset water depth. A pre-trained underwater image detail transfer model is used to adjust the style of the initial underwater optical image considering color deviation. The dynamic relationship between the background light of the RGB channel and the water depth and the water type is determined, so that the light attenuation characteristics and color deviation rules of the synthesized image are more consistent with the physical laws of the real marine environment, solving the problem of physical distortion. The application not only retains the physical authenticity, but also has the detail richness of real images, avoiding pattern collapse and single style.

[0045] 2.The application considers the influence of various factors on image quality degradation in the real marine environment, and combines hierarchical modeling of various complex factors at the physical and data levels to overcome the limitations of the original model and build a more complete underwater optical imaging model, providing a theoretical basis for subsequent underwater vision research.

[0046] 3.The light absorption branch of the multi-stage attenuation underwater physical imaging model of the application introduces water depth (vertical distance) as an independent variable into the light absorption process, superimposes the absorption effect of scene depth (horizontal distance), and completely describes the attenuation process of light in the vertical and horizontal double paths underwater, solving the brightness deviation problem caused by the traditional model which only considers the horizontal distance and ignores the influence of water depth.

[0047] 4.The light scattering branch of the multi-stage attenuation underwater physical imaging model of the application replaces random sampling by presetting any color channel and calculating the remaining color channels, so that the background light is strongly associated with the water depth and the water type, and the core defect of the traditional model that the background light randomly causes color deviation is completely solved. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The method flowchart of the application;

[0049] Figure 2 The network overall structure diagram of the application;

[0050] Figure 3 The construction flowchart of the underwater optical image synthesis data set in the application;

[0051] Figure 4 The underwater image example synthesized by the multi-stage attenuation underwater physical imaging model in the application;

[0052] Figure 5Examples of underwater images synthesized by using the present application;

[0053] Figure 6 Visual comparison of results synthesized by different underwater optical imaging modeling methods in embodiments of the present application;

[0054] Figure 7 Comparison of results of enhanced models trained on different synthetic data sets in embodiments of the present application on the UIEB data set. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0056] Embodiment 1

[0057] This embodiment discloses an underwater optical hybrid imaging method for complex marine scenes, and the method process is as shown in Figure 1 The method considers that underwater imaging in a real marine environment is affected by water body light absorption, scattering, and complex environment and lighting conditions, and the steps are specifically as follows:

[0058] Step S1, preset any clean image as a distortion-free image;

[0059] Step S2, based on a pre-constructed multi-stage attenuation underwater physical imaging model, simulate the light absorption and scattering effects of multiple water bodies at a preset water depth on the distortion-free image to obtain an initial underwater optical image;

[0060] Step S3, use a pre-trained underwater image detail transfer model to perform color deviation considering style adjustment on the initial underwater optical image to obtain a final underwater optical image.

[0061] In step S1, the clean image specifically means an image that does not contain underwater distortion. This image can be an image on land.

[0062] Underwater optical imaging modeling is an important theoretical basis for underwater visual processing and application research. The commonly used underwater optical imaging modeling can be divided into two categories: underwater imaging modeling based on the Jaffe-McGlamery simplified model and underwater imaging modeling based on deep learning theory.

[0063] Based on the propagation characteristics of light in water, the Jaffe-McGlamery model considers that the light radiation received by the camera in the underwater optical imaging process is derived from three parts: light absorption, forward scattering and backscattering. Jaffe found that the forward scattering effect has little effect on the quality of underwater imaging and can usually be ignored by studying the effects of inherent optical properties and external optical properties of water bodies on imaging. Therefore, most existing methods use the simplified Jaffe-McGlamery model to model the underwater imaging process, that is:

[0064] I w = I ab + I bs

[0065] wherein I w , I ab and I bs represent the underwater optical imaging image, the light absorption image and the light backscattering image respectively. The light absorption component I ab is the light component directly reflected by the target object into the camera, which can be represented as:

[0066]

[0067] wherein I a represents a clean image without underwater distortion, i.e. a distortion-free image, d represents the horizontal distance from the camera to the scene, i.e. the depth of field, λ represents the red, green and blue color channels, T represents the transmission map, and represents the degree of light preservation after absorption. In addition, the light will be affected by scattering effect when propagating in water, thereby producing a haze effect, which is defined as:

[0068]

[0069] wherein I bs represents the light scattering image, and B λ represents the ambient light. This simplified physical model can describe the selective attenuation characteristics of light in water, model the characteristics and basic rules of underwater imaging.

[0070] However, this type of modeling method has the following limitations:

[0071] 1) Ignoring the effect of water depth on light absorption: only focusing on the light absorption process of the depth of field, while ignoring the light absorption process along the path from the water surface to the camera with increasing water depth;

[0072] 2) Ignoring the effect of background light: in the light scattering process, the background light is determined by random sampling, which models the significant color error in the underwater image;

[0073] 3) Limited modeling capability: Only considering two factors of underwater light absorption and light scattering, it is not suitable for modeling complex and variable underwater environment.

[0074] Based on the above analysis, the application proposes a hybrid underwater optical imaging method based on physical principles and data driving, referred to as HUIS. The method first constructs a multi-stage attenuation underwater physical imaging model at the physical level, and further fuses data-driven clues to model more realistic underwater optical images. Figure 2 The overall architecture of HUIS is shown.

[0075] In step S2, the multi-stage attenuation underwater physical imaging model includes a light absorption branch and a light scattering branch. The light absorption branch is used to simulate the light absorption process to generate an absorption image, and the light scattering branch is used to simulate the light scattering process to calculate the global background light of the underwater image and generate a scattering image. Based on the absorption image and the global background light of the underwater image, an initial underwater optical image is obtained.

[0076] The quality degradation of the underwater optical image is mainly caused by the absorption and scattering of light by water. Therefore, the underwater physical imaging model is usually expressed as the linear superposition of two optical processes, defined as follows:

[0077] I λ (x)=I ab,λ (x)+I sc,λ (x)

[0078] =J λ (x)·T λ (x)+B(λ)·(1-T λ (x))

[0079] Where I ab,λ (x) represents the underwater light absorption process, I sc,λ (x) represents the underwater light scattering process, I λ (x) represents the light intensity value of pixel x in the optical synthetic image, J λ (x) represents the light intensity value of pixel x in the optical clear image, λ∈{r,g,b} represents three color channels, B(λ) represents background light, and transmission map T λ (x) represents the remaining energy ratio of scene radiant energy reaching the camera, defined as a composite function of wavelength λ and object distance from the camera, i.e. scene depth d(x), i.e. T λ (x)=e -β(λ)×d(x) , where β(λ) represents the attenuation coefficient, which varies with water body.

[0080] In fact, there are two light absorption mechanisms in the process of underwater light propagation: light absorption caused by scene depth and light attenuation caused by water depth, so the embodiment uses a light absorption branch and a light scattering branch to simulate respectively.

[0081] The process of simulating light absorption by the light absorption branch includes:

[0082] Obtaining a preset water depth and a plurality of preset water body types, and selecting wavelengths of a blue channel, a green channel and a red channel in the undistorted image;

[0083] According to the water body type, determining absorption coefficients and attenuation coefficients of light of the three wavelengths in the current water body type;

[0084] According to the preset water depth, the absorption coefficients and the wavelengths of the light, calculating a water depth direction light absorption component that increases with an increase of a water depth path D(x);

[0085] According to the undistorted image, determining a distance between an object in the image and a camera, and defining a scene depth;

[0086] According to the attenuation coefficients, the scene depth and the absorption coefficients, calculating a scene depth direction light absorption component;

[0087] According to the water depth direction light absorption component and the scene depth direction light absorption component, generating an absorption image.

[0088] The calculation expression of the water depth direction light absorption component is:

[0089] c λ,x =e -α(λ)×D(x) , λ∈{r, g, b}

[0090] wherein x is a pixel coordinate, λ is a wavelength, α(λ) represents an absorption coefficient of the current medium to light of the wavelength λ, T λ,x represents a residual light intensity of pixel x in the water depth direction, i.e., a light intensity remaining after the incident light energy is attenuated, r, g and b respectively represent a red channel, a green channel and a blue channel, and D(x) represents a water depth path of pixel x;

[0091] The calculation expression of the scene depth direction light absorption component is:

[0092] T λ,x =e -β(λ)×d(x) , λ∈{r, g, b}

[0093] wherein T λ,x represents a residual light intensity of pixel x in the scene depth direction, β(λ) represents an attenuation coefficient of the current medium to light of the wavelength λ, and d(x) represents a scene depth of pixel x.

[0094] The calculation expression of the generated final absorption image is:

[0095]

[0096] wherein, represents the intensity of pixel x in the absorption image, I λ (x) represents the light intensity value of the undistorted image at pixel x in color channel λ.

[0097] The light scattering branch simulates the process of light scattering, which is similar to I sc,λ (x) in the existing underwater physical imaging model described above, except that a background light generation balance factor is introduced in this method, which takes into account various dependencies such as water depth, water type, etc., and clearly defines the complex mathematical relationship between the background light intensity values in the RGB channels at a point in the underwater scene. Specifically, in the underwater scene, it is assumed that there is a small water disc with a thickness of dz. The amount of radiation scattered by the water disc can be represented as:

[0098] dL(z,λ)=b(λ)·E(D,λ)dz

[0099] where E(D,λ) represents the background light value at water depth D, L(z,λ) represents the scene object, and b(λ) represents the scattering coefficient.

[0100] According to the Lambert-Beer theory, the received radiation intensity at distance z is:

[0101] dB(z,λ)=dL(z,λ)·e -β(λ)·z

[0102] Substitute the expression of the amount of radiation scattered by the water disc into the expression of the radiation intensity, and integrate z from z1=0 to z2=z to obtain the background light illumination:

[0103]

[0104] where E(0,λ) represents the incident light original component, i.e. the sea surface irradiance. Based on the above background light illumination formula, it can be found that the background light B(λ) of the underwater optical image is affected by multiple factors, including the scattering coefficient a(λ) of seawater, the attenuation coefficient b(λ), the distance D between the object and the sea surface, and the sea surface irradiance E(0,λ).

[0105] In the terrestrial environment, the light rays have the same color channel attenuation rate by default. Therefore, at the sea surface, the energy values of the RGB three channels are equal, i.e. E(0,r)=E(0,g)=E(0,b). Combined with the above background light illumination formula, the RGB channel value intensity ratio of the background light at a certain point underwater can be represented as:

[0106]

[0107] The above formula shows that there is a complex dynamic relationship between the RGB channel values of the background light at a certain point underwater, and therefore cannot be randomly assigned. To solve this problem, the present application modifies the background light B(λ) synthesis formula to:

[0108]

[0109] The intensity ratio expression of the RGB channel values of the background light at a certain point underwater and the modified expression of the background light B(λ) synthesis formula are defined as the balance factor for background light generation. In the data synthesis process, first, the background light value of one channel (for example, the green channel) is specified, and then the background light values of the remaining two channels are directly calculated using the balance factor formula. Compared with randomly generating background light, the balance factor establishes a more explicit mapping relationship, dynamically considers the dependency between channels, more accurately simulates the background light, and thus improves the quality of the underwater optical synthesized image.

[0110] In summary, the process of simulating light scattering by the light scattering branch includes:

[0111] Obtaining a preset water depth and a plurality of preset water body types, and selecting the wavelengths of the blue channel, the green channel and the red channel in the undistorted image;

[0112] According to the water body type, the scattering coefficients of the three wavelengths of light in the current water body type are determined;

[0113] According to the preset intensity value of any channel background light, the intensity values of the background light of the remaining channels are calculated, and the global background light of the underwater image is obtained, that is, Figure 2 the scattering image.

[0114] The calculation expression of the background light intensity value is:

[0115]

[0116] Wherein, B(λ) is the underwater scene background light intensity of the λ channel, B r is the background light intensity value of the red channel, B g is the background light intensity value of the green channel, B b is the background light intensity value of the blue channel, λ is the wavelength, and r, g and b represent the red channel, the green channel and the blue channel respectively.

[0117] The multi-stage attenuation underwater physical imaging model proposed by the method effectively simulates the absorption and scattering effects of water on light, and the generated optical image presents attenuation characteristics close to the real underwater scene, and can simulate underwater optical images of different degradation types and water areas. However, this model only considers the optical properties of water and cannot effectively simulate complex underwater optical imaging environments. In fact, the underwater optical imaging process is affected by many complex factors. For example, the presence of artificial light sources will cause uneven lighting changes in the image; the movement of dynamic water flow will bring noise to the underwater optical image. These factors are difficult to fully consider in the physical model, and there is still a significant difference between the synthesized underwater optical image and the real underwater optical image. To solve this problem, the method further introduces an underwater detail transfer model as a supplement to the physical model as shown in step S3, to simulate more degradation factors of underwater optical images in a data-driven manner, and further reduce the domain gap between the synthesized underwater optical image and the real underwater optical image.

[0118] Specifically, the underwater optical image detail transfer model aims to further adjust the style of the underwater optical image synthesized by the physical model to the style of the real underwater optical image through style level transformation.

[0119] In step S3, the underwater image detail transfer model includes a style generator and a style discriminator, and the style generator and the style discriminator are collaboratively trained based on an adversarial learning strategy.

[0120] The style generator is based on the U-Net architecture and can efficiently extract and fuse multi-scale features to achieve accurate detail transfer. The style discriminator adopts the PatchGAN structure, which can distinguish at the patch level of the image compared with the traditional discriminator, pays more attention to image details, and improves the calculation efficiency.

[0121] In the collaborative training based on the adversarial learning strategy, the style generator is used to convert the synthesized optical image x st to an optical image similar in style to the real optical image x r .

[0122] In order to stabilize the training process, the model adopts WGAN-GP adversarial loss as the objective function. For the converted optical image x st , its style performance should be consistent with the real underwater optical image x r , while retaining the same scene content as the synthesized optical image x s . Therefore, in addition to the distribution-based adversarial loss, two perception losses are further introduced, including style loss and content loss, to achieve style transfer and content preservation. For the style loss, the method adopts the Gram matrix G lTo characterize the style correlation between feature responses, the invention defines the content loss as the sum of distances between output features on specified layers of a pre-trained VGG-19 model.

[0123] Specifically, the total loss function used in the training process of the underwater image detail transfer model is:

[0124] L=L adv +λ1·L sty +λ2·L con

[0125] Wherein, λ1 and λ2 represent weight parameters, L adv represents the WGAN-GP adversarial loss, L sty represents the style loss, L con represents the content loss.

[0126]

[0127] Wherein, D s represents the style discriminator, represents the sampling distribution obtained by uniformly sampling from the converted optical image x st and the real underwater optical image x r , φ l represents the l-th layer of the pre-trained VGG-19 model, and respectively represent the i-th and j-th vectorized feature maps in the l-th layer, k represents the number of channels, G l represents the Gram matrix, L s represents the set of layers in the VGG-19 model participating in the calculation of the style loss, ω s l represents the weight of the l-th layer, φ l represents the l-th layer of the pre-trained VGG-19 model, L c represents the set of layers in the VGG-19 model participating in the calculation of the content loss, ω c l represents the weight of the l-th layer, φ l (x st ) and φ l (x s ) respectively represent the feature extraction results of the converted optical image x st in the l-th layer, the physically modeled synthetic image x s in the l-th layer of the VGG-19 feature extraction results.

[0128] Embodiment 2

[0129] Based on the above embodiment 1, this embodiment discloses a specific application example of an underwater optical hybrid imaging method for complex marine scenes.

[0130] To facilitate the research of data-driven underwater optical image enhancement methods, based on the underwater optical hybrid imaging method for complex marine scenes, a more realistic and more comprehensive general underwater optical image synthesis dataset is established.

[0131] As shown in Figure 3 The synthesis process mainly consists of two steps:

[0132] ① Based on the multi-stage attenuation underwater optical imaging model in Example 1, combined with the absorption and scattering characteristics of 10 types of real water bodies, the light absorption and scattering effects of different water bodies at different water depths are simulated more systematically, improving the realism of the synthesized underwater optical images.

[0133] Specifically, 1000 land images are randomly selected from the existing RTTS RGB-D depth image dataset as undistorted images, which are used for underwater images in different water areas, and the degradation of different degrees is simulated by adjusting the parameters of the underwater imaging model.

[0134] First, 450nm, 525nm and 650nm are selected as the wavelengths of blue, green and red channels in the RGB image, and based on the measurement results of the existing Jerlov water type, the absorption coefficient a(λ) and scattering coefficient b(λ) of the three wavelengths of light in different water types are determined, as shown in Tables 1 and 2.

[0135] Table 1 Absorption coefficient α(λ) values of Jerlov water type

[0136]

[0137] Table 2 Absorption coefficient b(λ) values of Jerlov water type

[0138]

[0139] Subsequently, a random parameter between [0.5, 1] is assigned as the background light intensity value of the green channel, and the background light intensity values of the red and blue channels are calculated according to the formula, thereby obtaining the global background light B(λ) of the underwater image. Next, adjust the value range of the scene depth d(x) (i.e. the distance between the object and the camera) in the RTTS dataset to be randomly changed between [0.25, 20] meters, and introduce the water depth parameter D(x) to represent the distance between the object and the water surface, which is randomly set to be between 0 meters and 5 meters, i.e. 0m≤D(x)≤5m.

[0140] Finally, the absorption coefficient a(λ) and scattering coefficient b(λ) corresponding to the water area are selected, combined with the background light B(λ), scene depth d(x) and water depth D(x), and the underwater image corresponding to the water area is synthesized according to the formula. Through this process, 1000×10 underwater optical images are generated. Figure 4 Examples of 10 synthesized underwater optical images of water areas are shown and compared with optical images synthesized using traditional underwater imaging models. It is observed that the proposed physical imaging model synthesizes optical images that are highly similar to real underwater optical images, accurately simulating the attenuation characteristics and color tones of real underwater scenes. In contrast, underwater optical images synthesized using traditional imaging models differ significantly from actual underwater scenes. This difference is attributed to the limitations of traditional models: they do not consider the exponential attenuation of incident light with water depth, and they cannot provide accurate background light simulation, resulting in significant color differences. These experimental results further demonstrate that the multi-stage attenuation model can effectively simulate the absorption and scattering effects of water on light, presenting attenuation characteristics close to real underwater scenes. Figure 4

[0141] 2. Based on the underwater optical image detail transfer model in embodiment 1, the style of the underwater optical image synthesized by the physical model is adjusted to make it closer to the style of the real underwater optical image.

[0142] Specifically, first, considering that one of the key features of underwater optical images is color deviation, mainly manifested as six types: light blue, dark blue, blue-green, light green, dark green and yellow, this embodiment divides the underwater optical image synthesized by the physical model and the existing underwater dataset UID in the real world into six color subsets according to the average value of the blue channel in the CIELAB color space.

[0143] Subsequently, for each color subset, the same number of images as in the real dataset are randomly selected from the synthesized dataset to construct 4434 pairs of unpaired image pairs, further narrowing the distribution gap caused by their feature differences. Finally, adversarial learning training is performed to generate underwater synthetic images with more detailed details, higher authenticity and diversity.

[0144] The underwater optical image detail transfer model is built based on the PyTorch platform and undergoes 200 cycles of end-to-end training. In the first 100 training cycles, the learning rates of the generator G s and discriminator D s are set to 1×10 -4 and 2×10 -4 respectively. After 100 cycles of training, the learning rate is reduced by 0.1 every 10 cycles. G s and D s ​All of them are optimized by Adam optimizer; the batch size is set to 6. The default values of β1 and β2 are set to 0.5 and 0.999, respectively, and the weight decay is set to 0.00005. The size of all input images is adjusted to 256x256, and the pixel value is normalized to the range of [-1, 1]. In addition, various data augmentation techniques are used to expand the data volume during the training stage, including random rotation by 90°, 180°, 270°, and horizontal flip.

[0145] In summary, the method effectively combines physical prior knowledge and data-driven clues to generate 4434 pairs of training samples. Figure 5 Examples of the finally synthesized underwater optical images are given. It can be observed that, in addition to the image content (the data set is outdoor images), the synthesized underwater optical images are highly similar to the real underwater optical images, exhibiting diverse underwater degradation characteristics, such as different color tones, fog effects, non-uniform illumination, and low contrast, etc. Figure 5

[0146] Example 3

[0147] This example is based on the above-mentioned example 1, to verify the superiority of the underwater optical hybrid imaging method (HUIS) of the present application facing complex marine scenes, the specific process of comparing the underwater optical hybrid imaging method facing complex marine scenes with the existing underwater optical image modeling method is disclosed.

[0148] The existing underwater optical image modeling methods include WaterGAN, CycleGAN, Physical, and UWGAN. To ensure the fairness of the comparison, the codes of all the compared methods are adopted from the codes provided by the original authors, and are trained and tested on the same data set as the present method. In this experiment, the Frechet Inception Distance (FID) is used as the evaluation index to quantitatively compare the differences in feature distribution between the synthesized data set and the real underwater image data set. The smaller the FID score, the closer the two distributions are in the feature space. To enhance the accuracy of the evaluation of the realism of the generated optical images, the present application also conducts a subjective evaluation comparison experiment. 100 images are randomly selected from the synthesized images, and 15 image processing experts are invited to participate in the selection of the closest real results from the image realism. In addition, this experiment also investigates the influence of the quality of the synthesized optical images on the enhancement effect of the underwater optical images. The trained enhancement model is tested on the SQUID and UIEB two real underwater benchmark data sets, and the UIQM, CCF, and Edge intensity (Edge) three non-reference indexes are used to evaluate the results. In addition, 15 users are invited to subjectively evaluate the quality of 100 randomly selected enhancement results to obtain more accurate evaluation results. The evaluation results are shown in Table 3 below.

[0149] ​Table 3. Comparison of objective metrics of different underwater optical image modeling methods

[0150]

[0151] Table 3 shows the performance of different underwater optical image modeling methods in FID evaluation and subjective evaluation. As shown in Table 3, the method proposed in Example 1 achieved the lowest score in FID index. Compared with the suboptimal method UWGAN, the improvement of this method reached 61.3%. This result shows that the data synthesized by this method has stronger correlation with the real underwater images compared with other synthesized data. The subjective evaluation results further confirm the advantages of this method. In the subjective reality evaluation, the results synthesized by this method are most similar to the original underwater images in physical appearance, significantly better than other methods. The experimental results show that HUIS can generate images that are more similar to real underwater images, which benefits from the effective combination of physical prior knowledge and data-driven clues.

[0152] Figure 6 Visual comparisons of the results of different underwater optical image modeling methods are shown. It can be observed that the underwater optical images synthesized by this method are very close to the real underwater optical images in terms of diversity, color deviation, and fog effect. In contrast, the results of other comparative methods are not satisfactory. Due to the insufficient modeling of light absorption and light scattering processes, the WaterGAN method has limited performance in simulating real underwater color distortion and fog effect. The CycleGAN method learns more realistic color distortion by incorporating additional constraints (such as pixel-level cycle consistency). However, due to the lack of guidance from the underwater optical imaging process, this method is difficult to simulate the fog effect, and there are obvious artifacts in its results. The UWGAN method can simulate a certain degree of fog effect by enhancing the light scattering process, but due to the use of only unpaired image adversarial loss for training, this method cannot effectively simulate color distortion. It is worth noting that existing deep learning-based underwater optical imaging modeling methods do not fully consider the diversity of underwater degradation, especially color distortion. Therefore, they are prone to pattern collapse problems, resulting in monotonous underwater images. Although physical methods can synthesize underwater optical images with various color distortions and fog effects, they usually rely on randomly determined background light values and ignore other influencing factors, thereby limiting their generalization ability to generate diverse underwater images. In contrast, this method can generate underwater optical images with diverse features by combining physical priors and data-driven clues, making it more superior. This method considers the diversity of underwater degradation, including color distortion and fog effect, and uses physical priors and data-driven clues to guide the image synthesis process. Therefore, it can generate underwater optical images with rich details and realistic effects, effectively avoiding the pattern collapse problem and the defect of monotonous style.

[0153] Next, the performance of the enhanced models trained on different synthetic underwater optical image datasets on real underwater optical images is compared to verify the practicability of the synthetic datasets.

[0154] The synthetic datasets involved in the comparison include: underwater optical images synthesized by WaterGAN (denoted as Synthetic A); underwater optical images synthesized by CycleGAN (denoted as Synthetic B); underwater optical images synthesized by UWGAN (denoted as Synthetic C); underwater images synthesized by a traditional underwater optical imaging model (denoted as Synthetic D); and underwater images synthesized by the HUIS method proposed in the present application (denoted as Synthetic E). In the experiment, the number and content of the training samples remain consistent, and only the synthesis method is changed. Table 4 shows the quantitative comparison results of the enhanced models trained on different synthetic data on the SQUID and UIEB datasets. It can be seen that in most cases, the enhanced model trained based on the HUIS synthetic data (Synthetic E) achieves the best results on both datasets, and only suboptimal results are achieved in the CCF and Edge indicators on the UIEB dataset.

[0155] This result shows that, compared with other synthesis methods, HUIS has a significant advantage and can help the network achieve more excellent enhancement effect. Figure 7 Further, the visual comparison results of different enhanced models on the UIEB dataset are shown. It can be observed that the enhanced model trained based on the HUIS synthetic data (Synthetic E) performs well in correcting color, enhancing contrast, and improving details, and produces results with good visual effects. In contrast, the results of other enhanced models still have obvious artifacts and fog effects, and even introduce local over-enhancement. The experimental results show that the HUIS method can generate more realistic synthetic underwater optical images. The models trained on these synthetic data can perform better generalization on real underwater optical images.

[0156] Table 4 Comparison of objective indicators of enhanced models trained on different synthetic datasets on SQUID and UIEB datasets

[0157]

[0158] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be encompassed within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A hybrid underwater optical imaging method for complex marine scenarios, characterized in that, The method includes: An arbitrary clean image is preset as a distortion-free image. Based on a pre-constructed multi-stage attenuation underwater physical imaging model, the light absorption and scattering effects of various water bodies at a preset water depth are simulated on the distortion-free image to obtain an initial underwater optical image. Using a pre-trained underwater image detail transfer model, the initial underwater optical image is styled to take into account color deviation to obtain a final underwater optical image. The underwater image detail transfer model includes a style generator and a style discriminator, which are trained collaboratively based on an adversarial learning strategy.

2. The underwater optical hybrid imaging method for complex marine scenarios according to claim 1, characterized in that, The multi-stage attenuation underwater physical imaging model includes a light absorption branch and a light scattering branch. The light absorption branch is used to simulate the light absorption process and generate an absorption image. The light scattering branch is used to simulate the light scattering process and calculate the global background light of the underwater image. Based on the absorption image and the global background light of the underwater image, an initial underwater optical image is obtained.

3. The underwater optical hybrid imaging method for complex marine scenarios according to claim 2, characterized in that, The light absorption branch simulates the light absorption process, which includes: Obtain preset water depths and various water body types, and select the wavelengths of the blue, green, and red channels in the distortion-free image; Based on the water body type, determine the absorption coefficient and attenuation coefficient of the three wavelengths of light in the current water body type; Calculate the light absorption component in the water depth direction based on the preset water depth, absorption coefficient, and wavelength of light. Determine the distance between objects in the image and the camera based on the distortion-free image, and define the scene depth; Calculate the light absorption component in the scene depth direction based on the attenuation coefficient, scene depth, and absorption coefficient; An absorption image is generated based on the light absorption components in the water depth direction and the light absorption components in the scene depth direction.

4. The underwater optical hybrid imaging method for complex marine scenarios according to claim 3, characterized in that, The expression for calculating the light absorption component in the water depth direction is: C λ,x =e -α(λ)×D(x) ,λ∈{r,g,b} Where x is the pixel coordinate, λ is the wavelength, α(λ) represents the absorption coefficient of the current medium for light of wavelength λ, and C λ,x The remaining light intensity in the water depth direction of pixel x is represented by r, g, and b, which represent the red, green, and blue channels, respectively. D(x) represents the water depth path of pixel x. The expression for calculating the light absorption component in the depth direction of the scene is: T λ,x =e -β(λ)×d(x) ,λ∈{r,g,b} Among them, T λ,x β(λ) represents the residual light intensity in the scene depth direction of pixel x, β(λ) represents the attenuation coefficient of the current medium for light of wavelength λ, and d(x) represents the scene depth of pixel x. The calculation expression for generating the absorption image is: in, I represents the intensity of pixel x in the absorbed image. λ (x) represents the light intensity value at pixel x color channel λ in a distortion-free image.

5. The underwater optical hybrid imaging method for complex marine scenarios according to claim 2, characterized in that, The light scattering branch simulates the process of light scattering, including: Obtain preset water depths and various water body types, and select the wavelengths of the blue, green, and red channels in the distortion-free image; Based on the water body type, determine the scattering coefficients of the three wavelengths of light in the current water body type; Based on the preset background light intensity value of any one channel, the background light intensity values ​​of the remaining channels are calculated to obtain the global background light of the underwater image.

6. The underwater optical hybrid imaging method for complex marine scenarios according to claim 5, characterized in that, The formula for calculating the background light intensity value is: Where B(λ) is the background light intensity of the underwater scene in the λ channel, B r B represents the background light intensity value of the red channel. g B represents the background light intensity value of the green channel. b λ represents the background light intensity value of the blue channel, λ is the wavelength, and r, g and b represent the red channel, green channel and blue channel, respectively.

7. The underwater optical hybrid imaging method for complex marine scenarios according to claim 1, characterized in that, The style generator is based on the U-Net architecture, and the style discriminator adopts the PatchGAN structure.

8. The underwater optical hybrid imaging method for complex marine scenarios according to claim 1, characterized in that, The loss function of the underwater image detail transfer model is: L=L adv +λ1·L sty +λ2·L con Where λ1 and λ2 represent weight parameters, L adv Indicates WGAN-GP adversarial loss, L sty L represents style loss. con This indicates content loss; Among them, D s Indicates a style discriminator, Indicates the transformation from the optical image x st and real underwater optical images x r The sampling distribution obtained by uniform sampling, φ l This represents the l-th layer of the pre-trained VGG-19 model. and Let G represent the i-th and j-th vectorized feature maps in the l-th layer, respectively, where k represents the number of channels. l Let L represent the Gram matrix. s ω represents the set of layers involved in style loss calculation in the VGG-19 model. s l φ represents the weight of the l-th layer. l L represents the l-th layer of the pre-trained VGG-19 model. c ω represents the set of layers involved in the content loss calculation in the VGG-19 model. c l φ represents the weight of the l-th layer. l (x st ) and φ l (x s (x) represent the converted optical image. st The feature extraction results at layer l, and the physical model synthesizes the image x. s Feature extraction results from layer l of VGG-19.

9. The underwater optical hybrid imaging method for complex marine scenarios according to claim 1, characterized in that, In collaborative training based on adversarial learning strategies, the style generator is used to convert synthetic optical images into optical images with styles similar to real optical images, and the style discriminator is used to accurately distinguish between synthetic optical images and real optical images.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the underwater optical hybrid imaging method for complex marine scenarios as described in any one of claims 1-9.

Images