A method for correcting water-air imaging distortion based on dynamic window

Abstract

The application discloses a kind of based on dynamic window's water-penetrating air imaging distortion correction method, comprising the following steps: reconstructing water surface range, take the intersection of the water surface range with camera natural collection window range, obtain dynamic window;Based on the intersection of the dynamic window and camera natural collection window, obtain correction area;Pixel reset is carried out to the image in the correction area, and rearranged image is obtained, and distortion correction is completed.The application calculates dynamic window using the spatial relationship between water surface and camera, then calculates the correctable range of the method using dynamic window and camera natural collection window, and finally completes distortion correction in the correctable range using the proposed secondary water-penetrating dynamic window spatial position rearrangement method, to realize a kind of water-penetrating air imaging distortion correction method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image imaging distortion correction, and particularly relates to a water-air imaging distortion correction method based on a dynamic window. BACKGROUND

[0002] The water medium in the marine environment can cause light refraction and scattering, resulting in image distortion obtained by an underwater camera. The goal of water image distortion correction is to restore the real color, contrast and geometric shape in the image to provide more accurate and clear visual information. However, distortion correction is relatively complex, and it is a big challenge for the underwater camera to be immersed in water without the assistance of water equipment. Relatively speaking, it is relatively easy to shoot underwater targets in the air because more prior information is obtained by using equipment in the air, however, the prior information obtained by the system in the water is less and has instability under the influence of the medium. The water surface fluctuation causes corresponding distortion and distortion of the air shooting, and the camera natural collection window (snell) also limits the camera shooting field of view in the refraction process. Exceeding the window will cause the camera sampling light to not be able to exit the water surface, and cannot form effective pixel points. Therefore, it is necessary to develop and design an image imaging distortion correction method for water-air scene by using computer vision technology.

[0003] In the wave surface estimation field, Marina Alterman et al. proposed an article of designing a constant star ocean refraction imaging system, and gradient data of the water surface at a specific position can be obtained by using the hole array method, but the article does not mention a more effective water surface recovery algorithm, so the water surface data cannot be accurately obtained. Iosif M. Levin designed a set of cross-medium data acquisition experimental equipment based on the refraction and reflection principle of the water surface, which is composed of an air light source and a water surface full scattering light source. The water surface is irradiated by a red light source at a certain inclination angle, and the camera can obtain the reflection point to solve the water surface data; a calibration plate is arranged underwater, and a green light source with no cross wavelength with the red light source is used to irradiate the bottom of the calibration plate, and the camera obtains the pixel points carrying green as the target pixel information. However, the information points of the two colors cannot be stripped out, resulting in information loss.

[0004] In the image correction field, the current correction method applies more image sequences. There are two kinds of algorithms based on image sequence recovery: mean image and lucky image. The mean image is used as the reference image, and the image is recovered to a certain extent in the framework structure, but because the details of each frame image are blurred by the mean value, using the mean image as a template will introduce unstructured sparse noise, causing the loss of image details.

[0005] The above method has the following problems in constructing a water-penetrating air imaging distortion correction method: 1. The existing wave surface estimation system cannot accurately obtain water surface data. 2. The image sequence restoration algorithm will cause image detail loss. Therefore, it is urgent to propose a water-penetrating air imaging distortion correction method based on a dynamic window to solve the above technical problems. SUMMARY

[0006] To solve the above technical problems, the present application proposes a water-penetrating air imaging distortion correction method based on a dynamic window, which regards the distortion of the sea surface and the image as a joint change, calculates a dynamic window using the spatial relationship between the water surface and the camera, obtains the correctable range of the method by the intersection of the dynamic window and the natural acquisition window of the camera, and then uses the proposed secondary water-penetrating dynamic window spatial position rearrangement method to complete the distortion correction. Finally, a plurality of groups of data including Chinese, English, cartoon images and real object images are tiled on the water surface to verify the method of the present application from different depths of the camera in the water and different focal lengths of the camera at the same depth.

[0007] To achieve the above purpose, the present application provides a water-penetrating air imaging distortion correction method based on a dynamic window, comprising the following steps:

[0008] reconstructing a water surface range, and obtaining an intersection of the water surface range and a natural acquisition window range of the camera to obtain a dynamic window;

[0009] obtaining a correction region based on the intersection of the dynamic window and the natural acquisition window of the camera;

[0010] performing pixel resetting on the image in the correction region to obtain a rearranged image and complete the distortion correction.

[0011] Optionally, the process of reconstructing the water surface range comprises: taking the horizontal plane of the water surface as a base surface, obtaining the light length exceeding or below the horizontal plane, and determining the corresponding horizontal and vertical coordinates and the gradient vector of the corresponding two points of the light length of different lengths on the horizontal plane; based on the obtained different light lengths and the corresponding horizontal and vertical coordinates and the gradient vector, the water surface range is reconstructed by using the light array algorithm.

[0012] Optionally, the process of obtaining the dynamic window comprises: judging whether there is an intersection point between the light received by the natural acquisition window of the camera and the reconstructed water surface range; if there is an intersection point, obtaining the refraction vector coordinates of the light in the air based on the snell rule, and judging whether the refraction vector coordinates are in complex form; wherein the range formed by the pixel points corresponding to the light which is not in complex form is regarded as the dynamic window.

[0013] Optionally, the intersection of the dynamic view window and the camera natural acquisition window includes three position relationships: the dynamic view window is contained in the camera natural acquisition window, the dynamic view window intersects with the camera natural acquisition window, and the camera natural acquisition window is contained in the dynamic view window.

[0014] Optionally, the process of obtaining the spatial pixels includes: performing boundary scanning on the image acquired by the camera in the correction region, solving the maximum pixel distribution region, obtaining the arrangement range of the rearranged image pixels in space, and then obtaining the spatial pixels.

[0015] Optionally, the process after obtaining the spatial pixels further includes: obtaining the position relationship between the spatial pixels and the image pixels acquired by the camera based on the adjacent principle assignment method, performing grid assignment and rearrangement on the spatial pixels based on the position relationship, obtaining the corrected image, and then performing irregular interpolation on the corrected image to obtain the final corrected image.

[0016] Compared with the prior art, the present application has the following technical effects:

[0017] The present application calculates the dynamic view window by using the spatial relationship between the water surface and the camera, then calculates the correctable range by using the dynamic view window and the camera natural acquisition window, and finally completes distortion correction in the correctable range by using the proposed secondary water-permeable dynamic view window spatial position rearrangement method, so that the distortion of the water-permeable air-to-space imaging is corrected. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the present application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and the explanations provided herein are to facilitate understanding of the present application and are not intended to impose unnecessary limitations upon the present application. In the drawings:

[0019] Figure 1 The figure is a flowchart of the water-permeable air-to-space imaging distortion correction method based on the dynamic view window of the embodiment of the present application;

[0020] Figure 2 The figure is a schematic diagram of the experimental simulation experimental device of the embodiment of the present application;

[0021] Figure 3 The figure is a schematic diagram of the three position relationships between the snell and the dynamic window of the embodiment of the present application;

[0022] Figure 4 The figure is a comparison diagram of water surface restoration of the embodiment of the present application, wherein (a) is a schematic diagram of the real state of the water surface, and (b) is a restored water surface diagram;

[0023] Figure 5 The figure is a comparison diagram of the correction result of the embodiment of the present application, wherein (a) is a distortion diagram, (b) is a schematic diagram of the correctable range of the dynamic view window, and (c) is a schematic diagram of the correction result.

[0024] Figure 6 Imaging diagrams of different heights of the camera of an embodiment of the present application, wherein (a) is an imaging diagram of the camera at -50 cm, (b) is an imaging diagram of the camera at -30 cm, and (c) is an imaging diagram of the camera at -10 cm;

[0025] Figure 7 Imaging diagrams of different focal lengths of the camera of an embodiment of the present application, wherein (a) is an imaging diagram of the camera with a focal length of 4 mm, (b) is an imaging diagram of the camera with a focal length of 8 mm, and (c) is an imaging diagram of the camera with a focal length of 12 mm;

[0026] Figure 8 Light array diagram of a water surface hole plate of an embodiment of the present application. DETAILED DESCRIPTION

[0027] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0028] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0029] Embodiment One

[0030] As shown in Figure 1 , the present embodiment provides a water-penetrating air-to-space imaging distortion correction method based on a dynamic window, comprising the following steps:

[0031] Step 1: Reconstruct the water surface shape using a light array algorithm.

[0032] 1) As shown in Figure 8 , each light ray corresponds to a high and low undulating water surface, the present embodiment proposes to take the horizontal plane as the base surface, and the length exceeding or below the horizontal plane is set as SL1, SL MN (M = 10 rows, N = 13 columns). p x , p y is the horizontal and vertical coordinates of the SL light ray in space, (T1_x, T1_y, T1_z), (T2_x, T2_y, T2_z) are the gradient vectors of the two points, respectively, wherein the two points in the present embodiment are the upper endpoints of adjacent light columns in the calculation process.

[0033] In the x direction:

[0034] First column:

[0035]

[0036] 2~(M-1) column:

[0037] Δ x(i,j) = SL (j + (i * N) - SL (j + (i - 2) * N)

[0038] M column:

[0039]

[0040] 1st column and M column:

[0041] Δ_x_i(1) = (p x (m + N) - p x (m)) * (-(T1_x(m + N) + T2_x(m)) / 2) / (T1_z(m + N) + T2_z(m))

[0042] 2~(M-1) column:

[0043]

[0044] The comprehensive formula is: Δ_x_i = Δ x

[0045] According to the above equation, it can be written as:

[0046]

[0047] G1 * SL = Δ x

[0048] In the y direction:

[0049] 1st row:

[0050]

[0051] 2~(N-1) row:

[0052] Δ x(i,j) = SL (j + ((i - 1) * N - 1) - SL (j + ((i - 1) * N + 1)

[0053] N row:

[0054]

[0055] 1st row and N row:

[0056] Δ_y_j(1) = (p y (m) - p y(m-1))*(-(T1_y(m)+T2_y(m-1)) / 2) / (T1_z(m)+T2_z(m-1)) / 2

[0057] 2~(M-1) columns:

[0058] Δ_y_j(2) = (p y (m+1)-p y (m))*(-(T1_y(m+1)+T2_y(m)) / 2) / (T1_z(m+1)+T2_z(m)) / 2

[0059] The comprehensive formula is: Δ_y_i = Δ y

[0060] According to the above difference equation, it can be written as:

[0061]

[0062] Simplified as G2*SL = Δ y

[0063] The formula is arranged and combined into a (2MN)*(2MN) linear system:

[0064]

[0065] Step 2: Determine the dynamic window range

[0066] The dynamic window is the intersection of the reconstructed water surface range and the natural camera collection window (snell) range. The range of the dynamic window can be converted into two conditions. First, determine whether the light received by the camera has an intersection with the reconstructed water surface through the relationship between the camera and the dynamic window. If there is an intersection, proceed to the second step. The second step calculates the refraction vector in the air based on the refraction law axiom (snell rule) and determines whether the refraction vector coordinates are not complex numbers. If there is no complex number form, the light received by the pixel penetrates the dynamic window.

[0067] Calculate the dynamic window

[0068] A good dynamic window can improve the effect of water-penetrating sky imaging distortion correction. The present embodiment is based on a designed wave surface collection simulation experiment system, as shown in Figure 2 , and a water surface restoration algorithm, which calculates the dynamic window using the spatial relationship between the water surface and the camera. The simulation experiment system includes four parts: a digital camera imaging system, a water surface simulation system, a parallel light simulation system, and a water surface restoration reconstruction system.

[0069] Calculate the range of the dynamic window that can be corrected

[0070] The camera shooting range is limited to the Snell window, because the light in the window can pass through, beyond the window light can not penetrate the water surface imaging. Therefore, the secondary pixel reverse tracking process is limited to the Snell window and the dynamic window area of the known water surface. The intersection of the two areas is the correctable area of the pixel reverse index. The known water surface area obtained by calculation is a dynamic window. When the camera is placed underwater, the dynamic window will have three different positional relationships as shown in Figure 3 The known sea surface is the dynamic window proposed in this embodiment.

[0071] Pixel rearrangement distortion correction algorithm

[0072] The pixels in the dynamic window correctable range are correctable, because the window is three-dimensional, and the boundary is undulating, the corresponding light needs to be judged again. On the one hand, due to the high and low state of the boundary, the correctable pixel area is irregularly shaped, which increases the maximum space range of the periphery, and subsequently causes the vacancy of the spatial point coordinates. Because the spatial range is large, the number of pixels is reduced, and the image needs to be pixel reset. The image of the secondary pixel tracking is scanned for the boundary, and the maximum pixel distribution area is solved, which is the number of pixels of the rearranged image.

[0073] {p x,y ∈((xmin,ymin),(xmax,ymax))}

[0074] The formula is the range of the spatial rearranged pixel p, which represents the real manifestation of the camera collected pixels in space. The rearranged image can make the pixels in space more evenly distributed. The relationship between the obtained spatial pixels and the image pixel position needs to be determined, and because the range is relatively large, the adjacent principle assignment method is used.

[0075] x = round((x-xmin) / det_x) + 1

[0076] y = round((y-ymin) / det_y) + 1

[0077] Where:

[0078] det_x = (xmax-xmin) / L pixels_x

[0079] det_y = (ymax-ymin) / L pixels_y

[0080] Based on the relationship between the spatial pixels and the image pixel position, the spatial pixel grid is valued, the spatial coordinates are converted into two-dimensional image coordinates, and then the pixels are rearranged to obtain a new image, which presents the best pixel distribution effect. Due to the deviation in the reverse tracking process, part of the pixels will be lost. Therefore, the non-regularized interpolation of the corrected image is needed to make the image more detailed.

[0081] In this embodiment, the sea surface and the image distortion are regarded as a joint change. Based on the designed wave surface acquisition simulation experiment system and the water surface restoration algorithm, the dynamic window is calculated using the spatial relationship between the water surface and the camera. The intersection of the calculated dynamic window and the natural camera acquisition window obtains the correctable range of the method. Then, the secondary water-permeable dynamic window spatial position rearrangement method is used to complete the distortion correction. The advantages are as follows: (1) The problem that the existing wave surface estimation system cannot accurately obtain water surface data is solved. (2) The existing restoration algorithm can cause image detail loss.

[0082] Next, this embodiment will use multiple sets of data including Chinese, English, cartoon images and real object images tiled on the water surface to verify the method of this embodiment from different depths of the camera in the water and different focal lengths of the camera at the same depth.

[0083] Experimental analysis of water-permeable distortion correction method for air imaging based on dynamic window

[0084] Water surface restoration experiment results:

[0085] During the restoration of the water surface, the spots of sunlight passing through the water surface and mapping on the receptor are collected as the information input for the water surface restoration process. Due to the real-time changes of the water surface, the obtained spot map also changes with the water surface. Using the algorithm proposed in this embodiment, the error of the water surface restoration can be controlled within a very low range. Whether in shape or spatial position, it can be highly restored. The effect of restoring the water surface using the technical means of this embodiment is very good. This embodiment verifies the accuracy and limitation points of the algorithm through comparison experiments of various water surface conditions. The experimental results are shown in Figure 4 . Figure 4 Fig. (a) is the real state of the water surface, and Fig. (b) is the water surface restored by the algorithm of this embodiment, which has very small data difference in shape from the real data. Because gradient interpolation calculation and certain systematic errors are used in the restoration process, the roughness of the water surface is relatively large. This error can be calculated by an error function as follows:

[0086]

[0087] Wherein, H represents the true height of the water surface, h is the height recovered by the algorithm of the embodiment, and Error is the unit pixel error value. The formula can be used as a comprehensive index for evaluating the shape difference and spatial position similarity of two water surfaces.

[0088] Image correction results:

[0089] As Figure 5 shown, to verify the effectiveness of the improved target detection algorithm, the embodiment uses several data for correction. The image is tiled on the water surface, and the multiple data includes Chinese, English, cartoon images, and real object images. The diverse images logically achieve the diversity of the correction experiment. It is found in the result image that the algorithm of the embodiment has a correction blind area, which explains why there are incomplete contents in the correction image of the embodiment. The image is only a part of the image obtained by the camera. The reason is that in the process of reverse secondary pixel indexing of the camera, most of the light of the camera is outside the dynamic window. The algorithm proposed in the embodiment can only correct the pixel area through which the light passes the window.

[0090] The method regards the distortion of the sea surface and the image as a joint change, calculates the dynamic window using the spatial relationship between the water surface and the camera, obtains the correctable range of the method by calculating the intersection of the dynamic window and the natural acquisition window of the camera, and then completes the distortion correction by using the rearrangement method of the secondary water-penetrating dynamic window spatial position.

[0091] Image correction results of underwater camera at different depths:

[0092] According to the depth of the camera under the water, the relationship between the original view angle and the dynamic window is analyzed. The length of the water surface area covered by the light of the camera is:

[0093] L = 2h * tan(θ c / 2)

[0094] When the depth of the camera reaches a certain level, most of the vector light exceeds the snell imaging interval, as shown in the formula:

[0095] L s = 2h * tan(θ s / 2)

[0096] Wherein, θ c and θ s are the camera view angle and the snell area angle, respectively. Since the height of the camera from the water surface is large, θ c > θ s , so L >> L s . From the spatial relationship, the range of the view angle covers the snell area. Since the view in the air is large, the images obtained by penetrating the visible range at different heights. The experimental result image is as followsFigure 6 As shown.

[0097] The images obtained from the three sets of height data (A, B, and C) in the figure show significant differences in distortion. At greater depths, the camera's light travels a longer distance, increasing the vector angle difference between adjacent pixels, thus widening the angle between the light ray and the normal at the water surface intersection. Refraction is pronounced at the sampled pixels, resulting in almost overall image distortion. As the depth from the water surface decreases, i.e., the pixel vector angles become too close together, the sampled pixels deviate less. Furthermore, from C to A... Figure 5 (a) The field of view changes little when shooting in the air, but in (b) and (c) the field of view increases, and the correctable image range expands, as shown in the formula:

[0098] x d =2h*tan(θ) d )

[0099] Where θ d It is the angle between the dynamic viewport and the camera pixel rays. As the height changes, h decreases, resulting in θ. d Increase, given θ d <θ s This leads to an increase in the range of secondary pixel inverse indexing, ultimately increasing the number of calibrable pixels within the dynamic viewport.

[0100] The relationship between the camera's height, depth, and dynamic viewport can be intuitively shown by comparing pixel points. Relevant registration formulas can be used to obtain the relationship between the image within the dynamic viewport and the distortion-free image.

[0101]

[0102] Where xmax, xmin, ymax, and ymin are the spatial pixel positions corresponding to the secondary pixel indices of the dynamic viewport, respectively. The ratio of the corrected image portion to the overall image is obtained based on the index position. This method yields high accuracy in determining the correctable proportion. It is known that the range of the dynamic viewport is related to the water surface morphology; Table 1 presents the conclusions drawn from combining various water surface morphology data.

[0103] Table 1

[0104]

[0105] Correction results for underwater camera images at the same depth but different focal lengths:

[0106] Through observation Figure 7It can be seen that the imaging of different focal lengths of the camera (a), (b), (c) is the camera at 4mm, 8mm, 12mm, and the focal length of the camera has little effect on the imaging. For example, the focal length of 4mm and 12mm, the difference between the focal lengths is the range of the imaging. The imaging angle of 4mm is larger, and 12mm is more suitable for shooting distant targets. Similarly, the selection of the focal length is also related to the distance of the camera from the water surface. The selection of the device and the determination of the parameters are not single, but a complete system, which is complementary to each other.

[0107] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for correcting the distortion of a water-penetrating air-to-space image based on a dynamic window, characterized in that, The method comprises the following steps: reconstructing a water surface range, taking an intersection of the water surface range and a camera natural collection window range to obtain a dynamic window; obtaining a correction region based on an intersection of the dynamic window and the camera natural collection window; performing pixel resetting on an image in the correction region to obtain a spatial pixel; performing pixel rearrangement on the spatial pixel again to complete distortion correction; the process of reconstructing the water surface range comprises: taking a horizontal plane of the water surface as a base surface, obtaining light lengths exceeding or being lower than the horizontal plane, and determining horizontal and vertical coordinates of an upper end of the horizontal plane of light beams of different lengths and a gradient vector of a corresponding light column; based on the obtained different light lengths and the corresponding horizontal and vertical coordinates and the gradient vector, the water surface range is reconstructed by using a light array algorithm; the process of obtaining the dynamic window comprises: judging whether there is an intersection point between a light beam received by the camera natural collection window and the reconstructed water surface range; if there is an intersection point, obtaining a refraction vector coordinate of the light beam in the air based on a refraction law axiom, and judging whether the refraction vector coordinate is in a complex form; wherein a range formed by pixel points corresponding to the light beam which is not in the complex form is regarded as the dynamic window; the process of obtaining the spatial pixel comprises: performing boundary scanning on an image collected by the camera in the correction region, solving a maximum pixel distribution region, obtaining an arrangement range of rearranged image pixels in space, and then obtaining the spatial pixel; the process after obtaining the spatial pixel further comprises: obtaining a position relationship between the spatial pixel and image pixels collected by the camera based on an adjacent principle assignment method, performing grid assignment and rearrangement on the spatial pixel based on the position relationship, obtaining a corrected image, and then performing irregular interpolation on the corrected image to obtain a final corrected image.

2. The water-penetrating sky imaging distortion correction method based on a dynamic window according to claim 1, characterized in that the intersection of the dynamic window and the camera natural collection window comprises three position relationships: the dynamic window is contained in the camera natural collection window, the dynamic window intersects with the camera natural collection window, and the camera natural collection window is contained in the dynamic window.

Images