Deep sea image color restoration method based on binocular camera

Abstract

The invention belongs to the technical field of underwater image processing, and particularly relates to a deep sea image color restoration method based on a binocular camera, which comprises the following steps: firstly, calibrating the binocular camera, determining an internal parameter matrix, an external parameter matrix and a distortion coefficient, and carrying out distortion correction and stereo correction through the distortion coefficient and the external parameter matrix; determining a parallax matrix between the left camera and the right camera through parallax calculation, and converting the parallax matrix into a depth matrix in combination with the internal reference matrix; and after the depth matrix is obtained, performing pixel-by-pixel correction on the red channel in the RGB format image in combination with the proportion of the multi-attenuation part of the red channel to the existing part of the green channel compared with that of the green channel, thereby realizing red channel compensation. According to the scheme, pixel-level dynamic compensation is carried out on the red channel of the camera image, so that the method is suitable for color calibration of any underwater equipment capable of obtaining a depth matrix and an image at the same time, including an RGB-D camera, a range gating camera and the like, and has relatively high practical application and popularization values.

Description

Technical Field

[0001] The invention belongs to the technical field of underwater image processing, and in particular relates to a deep-sea image color restoration method based on a binocular camera. Background Art

[0002] At present, deep-sea optical detection is already a widely used detection method, and direct imaging through cameras is one of the important ways for humans to directly understand the deep-sea environment. However, deep-sea camera imaging faces a significant problem, that is, due to the large difference in the attenuation of light of different bands in water, there is a very serious color distortion problem in deep-sea optical imaging.

[0003] As the distance from the light source increases, the propagation of light in the water will be significantly attenuated, and the attenuation degree of seawater to light in different bands is completely different. Blue and green light are less attenuated, while red light is more attenuated. This difference causes the red channel intensity to be low in the images taken by common RGB three-channel cameras, which visually presents a blue or green hue. This hue deviation not only affects the aesthetics of the image, but more importantly, it can lead to the loss of image information and misjudgment.

[0004] At present, most of the existing underwater image restoration technologies are based on prior information and empirical model parameters, with poor universality. Most of them are aimed at processing shallow sea images and cannot solve the problem of color distortion of deep sea images. Summary of the invention

[0005] In view of the problems of poor universality and color distortion in current underwater image processing methods, the present invention proposes a deep-sea image color restoration method based on a binocular camera, which restores the color of deep-sea images by combining the image and depth information of the binocular camera.

[0006] The present invention is implemented by adopting the following technical solution: a deep sea image color restoration method based on a binocular camera, comprising the following steps:

[0007] Step A: calibrate the binocular camera, determine the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient, perform distortion correction and stereo correction through the distortion coefficient and extrinsic parameter matrix, determine the disparity matrix between the left and right cameras through disparity calculation, and convert it into a depth matrix in combination with the intrinsic parameter matrix;

[0008] Step B: After obtaining the depth matrix, the red channel in the RGB format image after distortion correction and stereo correction is corrected pixel by pixel, specifically including:

[0009] Step B1, determining the ratio R(D) of the portion of the red channel that is more attenuated than the green channel to the existing portion of the green channel;

[0010] First, determine the attenuation of the red channel and the green channel:

[0011]

[0012]

[0013] The red channel is attenuated more than the green channel as shown below:

[0014]

[0015] The ratio of the red channel's attenuation to the green channel's attenuation is:

[0016]

[0017] Among them, ΔI red (D) represents the attenuation of the red channel, ΔI green (D) represents the attenuation of the green channel, represents the current light intensity of the green channel received by the camera, I0 represents the light intensity of the incident light, α is the proportion of the red light component in the light source, β is the proportion of the green light component in the light source, the proportions of red and green in the light source α and β can be measured by a spectrometer, k red With k green is the attenuation coefficient of water to red light and green light, and D is the distance that light propagates in water;

[0018] Step B2: Determine the distance D that light propagates in water, and perform red channel compensation in combination with R(D) obtained in step B1; the distance D that light propagates in water is determined by the following method:

[0019]

[0020] Assume that the coordinates of the lighting fixture in the camera coordinate system are (X1, Y1, Z1), then |C| is:

[0021]

[0022] Among them, A represents the vector from the camera to the light source, B represents the vector from the target point to the camera calculated by the binocular camera, C represents the vector from the light source to the target point, and X, Y, and Z are the spatial position coordinates of the target point stored in the corresponding pixel in the depth matrix; when combining R (D) to compensate the red channel, the following method is used:

[0023] I′ red (D)=I red (D)+R(D)·I green (D)

[0024] According to the method of calculating the distance of light propagation, the D calculated by the target point using the coordinates stored in the corresponding pixel point in the depth matrix is ​​certain, so R(D) can be expressed in the pixel coordinate system using R(i,j) as follows:

[0025] I′ red (i,j)=I red (i,j)+R(i,j)·I green (i,j)

[0026]

[0027] Among them, I′ red (D) represents the red channel value after compensation, I green (D) represents the current light intensity of the green channel received by the camera, I red (D) represents the current light intensity of the red channel received by the camera, I red (i,j) and I green (i, j) is the value of the red channel and green channel of each pixel in the RGB format image matrix before restoration, and (i, j) is the position index of the pixel in the image; I′ red (i, j) is the value of the red channel of the pixel at the restored position (i, j), D(i, j) represents the propagation distance of the light collected by the camera pixel in the water, and R(i, j) represents the proportion of light collected by the pixel that attenuates the red component more than the green component in the green channel.

[0028] Furthermore, in step A, the conversion relationship between the intrinsic parameter matrix and the depth matrix is ​​as follows:

[0029]

[0030] Where i and j are the coordinates of the image pixel coordinate system, and the depth matrix (X, Y, Z) is the coordinate of the camera coordinate system solved by (i, j); c x , c y Respectively represent the pixel coordinates of the center point of the image in the horizontal and vertical directions; f x , f y is the focal length expressed in pixels, f x is the focal length in the x direction, f y is the focal length in the y direction; b represents the horizontal distance between the two cameras, and d is the disparity matrix.

[0031] Compared with the prior art, the advantages and positive effects of the present invention are:

[0032] This scheme uses the depth map to calculate the transmission distance of illumination light and reflected light in target imaging, and corrects the red channel in the RGB format image pixel by pixel based on the ratio of the more attenuated part of the red channel to the existing part of the green channel. The green channel is used to compensate the red channel intensity based on the illumination light transmission distance and the water attenuation coefficient. It is suitable for color calibration of any underwater equipment that can simultaneously obtain depth matrix and image, including RGB-D cameras, range-gated cameras, etc., and has high practical application and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the color restoration method according to an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the propagation route of light in water according to an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of image comparison before and after restoration according to an embodiment of the present invention, (a) is before restoration, (b) is after restoration;

[0036] Figure 4 Schematic diagram of point cloud comparison before and after restoration according to an embodiment of the present invention, (a) is before restoration, and (b) is after restoration. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the specific embodiments disclosed below.

[0038] Embodiment: According to the attenuation coefficient of deep sea water to light of different wavelengths, it can be known that the green channel of the three-channel camera is the channel that is better preserved in deep sea pictures, and the red channel is the channel with the worst shooting effect. This solution creatively proposes to use the information of the green channel of the image to compensate the red channel of the image in combination with the distance data collected by the binocular camera, thereby achieving the purpose of underwater image color correction. Specifically:

[0039] A deep sea image color restoration method based on binocular camera, combined with Figure 1 As shown, the following steps are included:

[0040] Step 1: calibrate the binocular camera, determine the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient, perform distortion correction and stereo correction through the distortion coefficient and extrinsic parameter matrix, determine the disparity matrix between the left and right cameras through disparity calculation, and convert it into a depth matrix in combination with the intrinsic parameter matrix;

[0041] Camera calibration is a relatively mature technical means. This process can obtain the camera parameters of the binocular camera in the seawater medium to ensure the accuracy and quality of imaging. The calibration process will generate parameters such as the camera's intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient. The distortion coefficient is used to perform distortion correction on the left and right views of the binocular camera to obtain an undistorted image. The extrinsic parameter matrix is ​​used to perform stereo correction on the undistorted images of the left and right views of the binocular camera to align the polar lines of the left and right views of the camera. The intrinsic parameter matrix is ​​used to calculate the depth matrix to ensure accurate depth information in the seawater environment.

[0042] This solution uses a mature disparity calculation method to determine the disparity matrix d between the left and right cameras, and converts it into a depth matrix through the camera intrinsic parameter matrix obtained by calibration. The specific conversion relationship is as follows:

[0043]

[0044] Where i and j are the coordinates of the image pixel coordinate system, and the depth matrix (X, Y, Z) is the coordinate of the camera coordinate system solved by (i, j); c x , c y are the principal point coordinates, indicating the pixel coordinates of the center point of the image in the horizontal and vertical directions; f x , f y is the focal length expressed in pixels, f x is the focal length in the x direction, and f y is the focal length in the y direction; b represents the baseline distance, which refers to the horizontal distance between the two cameras, and d is the disparity matrix.

[0045] Step 2: After obtaining the depth matrix, perform pixel-by-pixel correction on the red channel in the RGB format image. Specifically:

[0046] (1) Determine the ratio of the red channel’s attenuation to the green channel’s existing attenuation:

[0047] Due to the absorption of sunlight by seawater, there is almost no natural light in the deep sea, and the light source for deep sea photography is basically artificial light. In this case of a relatively single light source, according to Lambert-Beer's law, the attenuation of light in water can be approximately expressed by the following formula:

[0048] I(D)=I0·e -kD

[0049] Among them, I0 is the intensity of the incident light; I(D) is the intensity of the light received by the camera; D is the distance the light propagates in the water; k is the attenuation coefficient of seawater (k is a function related to the wavelength). Since different water qualities have different absorption and scattering characteristics of light, the attenuation coefficient will also be different (it can be determined based on experiments, and the deep-sea environment usually uses the attenuation coefficient of ocean water as an empirical value for calculation).

[0050] The core of this solution is to use the green channel of the image to compensate the red channel, so it is necessary to solve the ratio of the red light attenuated more than the green light to the existing green light. In this solution, assuming that the total intensity of the light source is I0, the attenuation of the red and green light in the illumination light source can be obtained as:

[0051]

[0052] The part where red light is attenuated more than green light is expressed as follows:

[0053]

[0054] The proportion of the red light attenuated more than the green light to the existing green light is:

[0055]

[0056] R(D) is the ratio of the red light attenuated more than the green light to the existing green light; ΔI red (D) represents the attenuation of the red channel, ΔI green (D) represents the attenuation of the green channel, It represents the existing light intensity of the green channel received by the camera, I0 represents the light intensity of the incident light, α is the proportion of the red light component in the light source, β is the proportion of the green light component in the light source, the proportions of red and green in the light source α and β can be measured by a spectrometer, and D is the distance that light propagates in water, which can be estimated by the depth information obtained by the binocular camera.

[0057] (2) Determine the distance D that light travels in water:

[0058] The path of light propagation in water is as follows Figure 2 As shown, 1 is the target object in the scene, 2 is the underwater light source, 3 is the underwater binocular camera, A represents the vector from the camera to the light source, B represents the vector from the target point to the camera calculated by the binocular camera, and C represents the vector from the light source to the target point. The positional relationship between the camera and the light source depends on the fixing method, so the vector A is fixed and known, and the vector B from the target point to the camera can be obtained in real time by the binocular camera. According to the vector calculation rule, C = B + A can be obtained, and the propagation distance D of light in water can be expressed as follows:

[0059] D=|C|+|B|

[0060]

[0061] C=B+A

[0062] Assuming that the coordinates of the lighting fixture in the camera coordinate system are (X1, Y1, Z1), |C| is:

[0063]

[0064] In this method, the propagation of light in water is simplified, and multiple reflections of light are ignored to simplify the calculation model. When the illumination direction of the light source is consistent with the shooting direction of the camera, and the camera position and the light source position should be as close as possible, the propagation distance of light can be approximated to twice the depth value of the pixel corresponding to the target point in the depth map.

[0065] (3) After determining the distance D that light travels in water, compensation for the red channel is performed:

[0066] I′ red (D)=I red (D)+R(D)·I green (D)

[0067] Using the calculation method provided in (2), we can calculate the light transmission distance D(i, j) corresponding to the pixel (i, j) in the depth matrix, and convert R(D) to the pixel coordinate system and use R(i, j) to represent it:

[0068] I′ red (i,j)=I red (i,j)+R(i,j)·I green (i,j)

[0069]

[0070] Among them, I red (i,j) and I green (i, j) is the value of the red channel and green channel of each pixel in the RGB format image matrix before restoration; I′ red (i, j) is the value of the restored red channel pixel, k red With k green is the attenuation coefficient of water to red and green light.

[0071] In specific operations, if the camera performs a white balance operation on the light source in the air, its internal balance method will automatically balance α and β. At this time, α = β, and R(i,i) can be degenerated into the following formula:

[0072]

[0073] This compensation process aims to correct the attenuation of red light in underwater environments and improve the imaging quality, so that the final image can more accurately reflect the actual deep seabed scene. The typical calibration results of this calibration method for underwater images are shown below: Figure 3 As shown in Figure 2, the calibration results for the point cloud are as follows: Figure 4 As shown, although this model ignores the influence of the object itself on the absorption and scattering of light of different wavelengths to a certain extent, experiments have verified that the color restoration of deep-sea objects still has a very high accuracy.

[0074] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A deep-sea image color restoration method based on a binocular camera, characterized in that: The following steps are involved: Step A: calibrate the binocular camera, determine the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient, perform distortion correction and stereo correction through the distortion coefficient and extrinsic parameter matrix, determine the disparity matrix between the left and right cameras through disparity calculation, and convert it into a depth matrix in combination with the intrinsic parameter matrix; Step B: After obtaining the depth matrix, the red channel in the RGB format image after distortion correction and stereo correction is corrected pixel by pixel, specifically including: Step B1, determining the ratio R(D) of the portion of the red channel that is more attenuated than the green channel to the existing portion of the green channel; Step B2: determine the distance D that the light propagates in water, and perform red channel compensation based on R(D) obtained in step B1.

2. The deep sea image color restoration method based on a binocular camera according to claim 1, characterized in that: In step B1, the attenuation of the red channel and the green channel is first determined as: The red channel is attenuated more than the green channel as shown below: The ratio of the red channel's attenuation to the green channel's attenuation is: Among them, ΔI red (D) represents the attenuation of the red channel, ΔI green (D) represents the attenuation of the green channel, represents the current light intensity of the green channel received by the camera, I0 represents the light intensity of the incident light, α is the proportion of the red light component in the light source, β is the proportion of the green light component in the light source, and k red With k green is the attenuation coefficient of water to red light and green light, and D is the distance light travels in water.

3. The deep sea image color restoration method based on a binocular camera according to claim 1, characterized in that: In step B2, the propagation distance D of light in water is determined by: D=|C|+|B| C=B+A Assume that the coordinates of the lighting fixture in the camera coordinate system are (X1, Y1, Z1), then |C| is: Among them, A represents the vector from the camera to the light source, B represents the vector from the target point to the camera calculated by the binocular camera, C represents the vector from the light source to the target point, and X, Y, and Z are the spatial position coordinates of the target point stored in the corresponding pixel in the depth matrix.

4. The deep sea image color restoration method based on a binocular camera according to claim 1, characterized in that: In step B2, when red channel compensation is performed in combination with R(D), the following method is adopted: I′ red (D)=I red (D)+R(D)·I green (D) Corresponding to the pixel coordinate system, R(D) is expressed as R(i,j): I′ red (i,j)=I red (i,j)+R(i,j)·I green (i,j) Among them, I′ red (D) represents the red channel value after compensation, I green (D) represents the current light intensity of the green channel received by the camera, I red (D) represents the current light intensity of the red channel received by the camera, I red (i,j) and I green (i, j) is the value of the red channel and green channel of each pixel in the RGB format image matrix before restoration, and (i, j) is the position index of the pixel in the image; I′ red (i, j) is the value of the red channel of the pixel at the restored position (i, j), D(i, j) represents the propagation distance of the light collected by the camera pixel in the water, and R(i, j) represents the proportion of light collected by the pixel that attenuates the red component more than the green component in the green channel.

5. The deep sea image color restoration method based on binocular camera according to claim 1, characterized in that: In step A, the conversion relationship between the intrinsic parameter matrix and the depth matrix is ​​as follows: Where i and j are the coordinates of the image pixel coordinate system, and the depth matrix (X, Y, Z) is the coordinate of the camera coordinate system solved by (i, j); c x , c y Respectively represent the pixel coordinates of the center point of the image in the horizontal and vertical directions; f x , f y is the focal length expressed in pixels, f x is the focal length in the x direction, f y is the focal length in the y direction; b represents the horizontal distance between the two cameras, and d is the disparity matrix.

Images