Underwater binocular camera image matching enhancement device and method based on spectrum feature projection

Abstract

The invention relates to the technical field of underwater imaging, in particular to an underwater binocular camera image matching enhancement device and method based on spectrum feature projection. According to the method, the image matching effect of the underwater binocular camera is enhanced by projecting the pattern with the specific frequency characteristic in the underwater environment; meanwhile, the matched image is processed by using a frequency domain filtering technology so as to eliminate the interference of the projection pattern. According to the invention, the imaging effect of the underwater binocular camera can be effectively improved, and the normal imaging of the binocular camera is prevented from being interfered. The method is particularly suitable for imaging optimization of low-feature scenes such as seabed sand slopes, and the three-dimensional imaging effect of the underwater binocular camera can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of underwater imaging technology, and in particular to an underwater binocular camera image matching enhancement device and method based on spectrum feature projection. Background Art

[0002] Underwater binocular imaging technology plays a vital role in many fields, including underwater robot navigation, marine science research, environmental monitoring, military reconnaissance, and marine resource exploration. Underwater binocular cameras use two cameras to capture two-view images of the same scene and obtain three-dimensional information through parallax estimation. However, in practical applications, underwater binocular camera image acquisition and processing face many challenges, especially in underwater low-feature scenes such as seabed sand slopes and sediment areas. These scenes lack significant texture and structural features due to single color and sparse features, and the image matching and depth estimation of binocular cameras often have poor results. This lack of features directly affects the accuracy and reliability of three-dimensional reconstruction.

[0003] In order to improve these problems, researchers have proposed various methods, including optimizing image processing algorithms (training neural networks specifically for underwater binocular camera image matching), improving light source configuration, and using more advanced imaging sensors. However, these methods have failed to fundamentally solve the problem of sparse image features, have limitations, and have limited improvement in image matching accuracy.

[0004] In recent years, projection technology has been introduced into the field of underwater binocular image processing as a possible solution. Projection technology enhances the feature information of an image by projecting speckles or specific patterns in an underwater environment. However, existing projection technology still faces challenges in practical applications. In order to avoid the absorption of projection light by seawater, projection light usually uses a blue-green band light source in the visible light region. The projection pattern in this band will seriously interfere with the image quality, destroy the color information of the image, cover up the real image features, and interfere with the extraction of underwater original image features. Therefore, increasing scene features and avoiding the negative impact of projection patterns on scene images are key issues that need to be urgently solved in current underwater binocular imaging. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned defects of the prior art, and proposes an underwater binocular camera image matching enhancement device and method based on spectral feature projection, which utilizes a narrow-band spectrum pattern projection method to enhance scene features to optimize the image matching of the binocular camera and improve the accuracy and reliability of underwater binocular camera image processing; at the same time, through the image frequency domain filtering method, the projection pattern is filtered out and the original color of the scene is restored.

[0006] The technical solution of the present invention is: an underwater binocular camera image matching enhancement device based on spectrum feature projection, comprising a projection system and an imaging system, wherein the imaging system comprises a left camera and a right camera;

[0007] The projection system and the left camera have the same field of view angle, and the field of view of the projection system coincides with the field of view of the left camera.

[0008] In the present invention, a flat beam splitter is provided on the side of the left camera facing the scene to be observed;

[0009] When the projection system and the camera are arranged in parallel, a reflector is provided on the side of the projection system facing the scene to be observed, and the projection light beam emitted by the projection system is reflected by the reflector and the flat beam splitter in sequence, and then overlaps with the field of view of the left camera;

[0010] When the projection system and the camera are arranged vertically, the projection light beam emitted by the projection system overlaps with the field of view of the left camera after being reflected by the flat beam splitter.

[0011] The present invention also includes an underwater binocular camera image matching enhancement method based on spectrum feature projection, the method comprising the following steps:

[0012] S1, generating a projection pattern;

[0013] S2. After the generated projection pattern is transmitted to the scene to be observed, the binocular camera is used to collect and calibrate the image of the scene to be observed. The calibrated image is used to generate point cloud data on the one hand, and frequency filtering of the image is performed on the other hand to filter out the projection fringes of the projected composite image obtained by the left camera and restore the image color;

[0014] S3. Assign the color and texture information of the image obtained in step S2 to the point cloud data to obtain color point cloud data that is not affected by the projection fringes.

[0015] The specific implementation process of step S1 is as follows:

[0016] S1.1, forming a uniform fringe projection pattern in the spatial domain and providing a fringe function f(x);

[0017] S1.2, performing Fourier transform on the fringe function provided in step S1.1 to obtain a spectrum function F(k);

[0018] S1.3. Crop the spectrum of the spectrum function to synthesize the combined spectrum features to be projected:

[0019] The combined spectrum feature contains low-frequency components, and the spectrum feature is selected near the fundamental frequency of the fringe function f(x). The selected spectrum components are synthesized into the combined spectrum feature P(k) to be projected, and the remaining spectrum components are set to zero.

[0020] S1.4. Through Fourier transform, the combined spectrum characteristics obtained in step S1.3 are converted to the spatial domain again, and a new fringe function p(x) is formed. The new fringe function p(x) is amplified in the vertical direction to obtain a projection pattern.

[0021] In step S1.2, the fringe function f(x) is transformed by Fourier transform to obtain the spectrum function F(k):

[0022]

[0023] Where N represents the length of the fringe function f(x); j is the imaginary unit; k represents the discrete frequency component;

[0024] In step S1.4, the Fourier inverse transform equation for converting the combined spectrum feature P(k) to form a new fringe function p(x) is:

[0025]

[0026] Where x represents the spatial domain sampling point index of the fringe function.

[0027] In step S2, the binocular camera performs feature matching, disparity map estimation, and depth map calculation on the captured and calibrated images in sequence, and finally generates point cloud data.

[0028] In step S2, filtering out the projection fringes of the projected composite image acquired by the left camera includes the following specific steps:

[0029] S2.1. Perform discrete Fourier transform on the projected composite image acquired by the left camera row by row along the horizontal direction according to the pixel row, and convert the projected composite image from the spatial domain to the frequency domain. After the conversion, the frequency domain function F m (k);

[0030] S2.2, by binarizing and inverting the combined spectrum feature P(k) obtained in S1.3, a frequency domain filter Filter(k) is generated;

[0031] S2.3, using the frequency filter Filter(k) generated in step S2.2, the transformed spectrum function F m (k) After performing the filtering operation, the image function f′ after the stripes are filtered out can be obtained by performing the inverse discrete Fourier transform. m (x).

[0032] In step S2.1, the frequency domain function F obtained after conversion m (k) is:

[0033]

[0034] Among them, m represents the pixel row index of the image along the vertical direction, M is the height of the image, which represents the total number of pixels in the vertical direction, that is, the total number of rows, and f m (x) is the function of the pixels in the mth row of the projected composite image;

[0035] In step S2.3, the image function f′ m The calculation equation for (x) is:

[0036]

[0037] In step S2.2, on the basis of generating the frequency domain filter, an adjustment coefficient α for the DC component of k=0 is added to suppress the stray light background caused by water scattering and to correct the color deviation caused by seawater absorption to a certain extent;

[0038] The value of α depends on the optical parameters of seawater, the brightness of the illumination source, and the projection brightness.

[0039] The beneficial effects of the present invention are:

[0040] (1) This application proposes a method and device for enhancing the image feature matching of an underwater binocular camera by projecting an image with specific narrowband spectrum characteristics, and suppressing the projection color interference by using a frequency domain filtering method. By projecting an image with specific narrowband spectrum characteristics onto an underwater low-feature scene through projection technology, the feature information in the image can be effectively enhanced, thereby improving the left and right field of view image matching effect of an underwater binocular camera;

[0041] (2) The spatial domain image is transformed into the frequency domain, and the specific frequency band where the projection pattern is located is filtered using filtering technology in the frequency domain to eliminate the frequency domain features corresponding to the projection pattern in the image. Finally, the frequency domain image is restored to the spatial domain to obtain an underwater pattern with the projection features eliminated. This method can process the projection image without interfering with the normal imaging of the binocular camera. While improving the matching accuracy of the binocular image, the original color image without projection interference is obtained. This solves the shortcomings of the existing technology in low-feature scenes and retains the image color information of the original scene, providing new ideas and solutions for the development of underwater imaging technology.

[0042] In summary, the present application can not only enhance the image matching success rate of underwater low-feature scenes, but also avoid the interference of spatial color in underwater speckle projection, effectively solving the challenges in the current field of underwater three-dimensional imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a first structural schematic diagram of the device described in this application;

[0044] Figure 2It is a second structural schematic diagram of the device described in this application;

[0045] Figure 3 It is the fringe projection pattern formed in the spatial domain in the second step;

[0046] Figure 4 It is the projection pattern transformed back to the spatial domain after spectrum clipping;

[0047] Figure 5 is the flow chart of the generation of the projection pattern in the second step;

[0048] Figure 6(a) is the original deep-sea scene;

[0049] Fig. 6(b) is the projected composite image obtained by the left camera after projection;

[0050] Figure 7 is a schematic diagram of a filtering process of projected fringes of a projected composite image acquired by a left camera;

[0051] Figure 8 is a flow chart of the method described in the present application.

[0052] In the figure: 1. Projection system; 2. Left camera; 3. Right camera; 3. Projector; 4. Flat beam splitter; 5. Reflector. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0054] In the following description, specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in a variety of other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0055] The present application includes an underwater binocular camera image matching enhancement device based on spectrum feature projection. The enhancement device includes a projection system and an imaging system. In order to achieve the goal of projecting enhanced image features, the projection system and the imaging system need to be assembled according to certain rules. The specific assembly structure is as follows: Figure 1 shown.

[0056] The imaging system includes a left camera 2 and a right camera 3, and the left camera 2, the right camera 3, and the projection system 1 are placed in parallel. The projection system 1 and the left camera 2 have the same field of view, and in the assembly relationship, the field of view of the projection system coincides with the field of view of the left camera 2 after passing through the reflector 5 and the flat beam splitter 4, so as to ensure that the projection pattern has scale invariance in the image of the left camera 2, that is, no matter how the imaging distance is, the pixel position of the projection pattern in the image of the camera 2 is unchanged.

[0057] In the device, the function of the reflector 5 is to adjust the direction of the projection field of view, reflect the projection light beam emitted in the horizontal direction to the flat beam splitter 4, and overlap with the field of view of the left camera 2 after further reflection by the flat beam splitter 4.

[0058] In this application, the location of the projection system 1 is not limited to Figure 1 When the projection system 1 is placed vertically to the camera, the reflector 5 can be omitted, and the field of view of the projection system 1 can be directly projected into the flat beam splitter 4, as shown in FIG. Figure 2 shown.

[0059] The present application also discloses a method for implementing image matching enhancement using the above device, and the method includes the following steps.

[0060] The first step is to assemble the underwater binocular camera image matching enhancement device so that the field of view of the projection system in the device coincides with the field of view of the left camera.

[0061] The second step is the generation of projection patterns.

[0062] In order to facilitate subsequent filtering and avoid serious loss of real image information caused by filtering operations, the projection pattern should have a limited narrow frequency band range in the frequency domain space and have obvious texture features in the spatial domain. It is difficult to design a pattern with a limited frequency range in the spatial domain or frequency domain alone. The frequency characteristics are reflected in the overall image, and the design in the spatial domain is not intuitive; the expansion of the spatial domain image in the frequency domain is continuous, and often cannot meet the limited narrow frequency band design requirements. In this application, the texture features of the projection pattern are designed in the spatial domain to meet the texture requirements; after the transformation operation, it is cropped in the frequency domain to meet the limited narrow frequency band requirements; and finally transformed to the spatial domain to obtain the projection pattern. A projection pattern with obvious change characteristics is more helpful in enhancing image features and improving the matching effect of left and right cameras. In order to achieve this design goal, this scheme adopts a combination of multiple frequency features, that is, in the frequency domain space, multiple single-frequency functions are synthesized according to the function synthesis law to produce a periodic spatial domain image with significant intensity changes. The specific implementation process is described as follows.

[0063] (a) forming a uniform fringe projection pattern in the spatial domain and providing a fringe function f(x).

[0064] In a binocular imaging system, in order to speed up the matching of left and right image features, the left and right camera images are aligned in rows during the camera stereo calibration process, and search and match are performed on the corresponding row pixels on the left and right. Therefore, this application uses periodic projection features distributed along pixel rows to utilize the row alignment characteristics of the left and right images to reduce the amount of calculation. This is reflected in the projection pattern as periodic vertical stripes distributed along the horizontal direction.

[0065] In this embodiment, a Figure 3 The fringe function of the uniform fringe projection pattern with a period of 2π and a number of 5 fringes in the horizontal direction can be expressed as follows:

[0066]

[0067] (ii) Performing Fourier transformation on the fringe function provided in the above steps to obtain a spectrum function.

[0068] In the present application, the uniform fringe projection image obtained in the previous step is subjected to a fast Fourier transform method as shown in the following formula, and the fringe function is Fourier transformed to obtain a spectrum function F(k).

[0069]

[0070] Where N represents the length of the fringe function f(x); j is a complex unit; and k represents a discrete frequency component.

[0071] (iii) cropping the spectrum of the spectrum function to synthesize the combined spectrum features to be projected.

[0072] In the present application, the combined spectrum feature includes a low-frequency component and the spectrum feature is selected near the fundamental frequency of the fringe function, which can ensure that the generated projection pattern is similar to the designed pattern.

[0073] In this embodiment, the first 5-8 spectral components of the F(k) function are selected to synthesize the combined spectral feature P(k) to be projected, and the remaining spectral components are set to zero.

[0074]

[0075] (iv) Through inverse Fourier transform, the combined spectrum features are converted back to the spatial domain to form a new fringe function p(x), and the fringe function p(x) is amplified in the vertical direction.

[0076] In this application, the combined spectrum feature P(k) is converted to the spatial domain by inverse Fourier transform to form a new fringe function p(x). The specific frequency domain inverse transform equation is as follows:

[0077]

[0078] Where x represents the spatial domain sampling point index of the fringe function.

[0079] By amplifying the fringe function p(x) in the vertical direction, we can obtain Figure 4 The projection pattern shown.

[0080] The projection pattern generated through the above steps is projected to the field of view of the left camera 2 through the projection system, and completely overlaps with the field of view, which is convenient for subsequent frequency domain filtering.

[0081] Taking the original deep sea scene shown in Figure 6(a) as an example, after projection, the scene in the figure can obtain the projected composite image shown in Figure 6(b). The projected stripes can help the binocular camera improve the feature matching success rate of the left and right views in low-feature scenes.

[0082] In the third step, the generated projection pattern is projected into the scene through the image matching enhancement device, and the scene is imaged using a binocular camera. The captured binocular image is calibrated using internal and external parameters to remove the effects of changes in the refractive index of the imaging medium and system aberrations on the image. The calibrated image is sent to two processing branches for processing: one is a standard binocular camera processing process to generate point cloud data; the other is an image frequency domain filtering process to filter out the effects of the projection pattern and restore the image color.

[0083] The calibration and processing of the binocular image adopt conventional standard procedures, which mainly include binocular camera image feature matching, disparity map estimation, depth map calculation, and finally generate point cloud data, which will not be described in detail here. This application will mainly describe the filtering process of the projected stripes of the projected composite image obtained by the left camera. The left camera view after filtering will be used as the main field of view image of the binocular camera for point cloud color rendering to obtain a colored point cloud model without projection interference. The specific implementation process is described in detail below.

[0084] (i) The projected composite image acquired by the left camera is subjected to discrete Fourier transform row by row in the horizontal direction according to pixel rows, so as to convert the projected composite image from the spatial domain to the frequency domain.

[0085] The frequency domain function F obtained after conversion m (k) is:

[0086]

[0087] Where m represents the pixel row index of the image along the vertical direction; M is the height of the image, which represents the total number of pixels in the vertical direction, that is, the total number of rows; f m (x) is the function of the pixels in the mth row of the projected composite image.

[0088] (ii) Generate a frequency domain filter by binarizing and inverting the combined spectrum feature P(k) obtained in S1.3.

[0089] On the basis of generating the frequency domain filter, an adjustment coefficient α for the DC component of k=0 is added to suppress the stray light background caused by water scattering and to correct the color deviation caused by seawater absorption to a certain extent. The value of α depends on the optical parameters of seawater, the brightness of the illumination source and the projection brightness, and needs to be adjusted according to the specific situation during use.

[0090] In this embodiment, the form of the frequency domain filter is shown in the following formula:

[0091]

[0092] (III) Use the frequency filter generated in the previous step to transform the spectrum function F m (k) After performing the filtering operation, the image function f′ after the stripes are filtered out can be obtained by performing the inverse discrete Fourier transform. m (x). As shown below:

[0093]

[0094] Arranging the image functions according to the corresponding row positions can obtain the original image after filtering out the projection pattern. The specific image filtering process is as follows: Figure 7 shown.

[0095] In the above implementation process, filters with sharp edges may introduce ringing effects in the image space, so a smoothing strategy can be used to smooth the filter; further, incorporating the harmonic spectrum of the fringe function into the filter can also improve the filtering effect. The smoothing method of the frequency domain filter includes but is not limited to Gaussian smoothing, and other smoothing methods can also achieve similar effects.

[0096] By assigning the color and texture information of the processed image to the point cloud data obtained by the binocular camera processing flow, color point cloud data that is not affected by projection fringes can be obtained.

[0097] Underwater projection usually uses blue or green lasers, so the filtering of underwater images can be performed only in the corresponding blue or green channels. The method described in this application can minimize the impact of frequency domain filtering on the inherent frequency characteristics of the image and retain the original color information of the scene. The overall workflow of this system and the steps of image filtering are as follows: Figure 8 shown.

[0098] The above is a detailed introduction to an underwater binocular camera image matching enhancement method based on spectral feature projection provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined in this article can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this article, but will conform to the widest scope consistent with the principles and novel features disclosed in this article.

Claims

1. An underwater binocular camera image matching enhancement device based on spectrum feature projection, comprising a projection system and an imaging system, characterized in that: The imaging system includes a left camera and a right camera; The projection system and the left camera have the same field of view angle, and the field of view of the projection system coincides with the field of view of the left camera.

2. The underwater binocular camera image matching enhancement device based on spectrum feature projection according to claim 1, characterized in that: A flat beam splitter is provided on the side of the left camera facing the scene to be observed; When the projection system and the camera are arranged in parallel, a reflector is provided on the side of the projection system facing the scene to be observed, and the projection light beam emitted by the projection system is reflected by the reflector and the flat beam splitter in sequence, and then overlaps with the field of view of the left camera; When the projection system and the camera are arranged vertically, the projection light beam emitted by the projection system overlaps with the field of view of the left camera after being reflected by the flat beam splitter.

3. An underwater binocular camera image matching enhancement method based on spectral feature projection, characterized in that: The following steps are involved: S1, generating a projection pattern; S2. After the generated projection pattern is transmitted to the scene to be observed, the binocular camera is used to collect and calibrate the image of the scene to be observed. The calibrated image is used to generate point cloud data on the one hand, and frequency filtering of the image is performed on the other hand to filter out the projection fringes of the projected composite image obtained by the left camera and restore the image color; S3. Assign the color and texture information of the image obtained in step S2 to the point cloud data to obtain color point cloud data that is not affected by the projection fringes.

4. The underwater binocular camera image matching enhancement method based on spectral feature projection according to claim 3 is characterized in that: The specific implementation process of step S1 is as follows: S1.1, forming a uniform fringe projection pattern in the spatial domain and providing a fringe function f(x); S1.2, performing Fourier transform on the fringe function provided in step S1.1 to obtain a spectrum function F(k); S1.

3. Crop the spectrum of the spectrum function to synthesize the combined spectrum features to be projected: The combined spectrum feature contains low-frequency components, and the spectrum feature is selected near the fundamental frequency of the fringe function f(x). The selected spectrum components are synthesized into the combined spectrum feature P(k) to be projected, and the remaining spectrum components are set to zero. S1.

4. Through Fourier transform, the combined spectrum characteristics obtained in step S1.3 are converted to the spatial domain again, and a new fringe function p(x) is formed. The new fringe function p(x) is amplified in the vertical direction to obtain a projection pattern.

5. The underwater binocular camera image matching enhancement method based on spectral feature projection according to claim 4 is characterized in that: In step S1.2, the fringe function f(x) is transformed by Fourier transform to obtain the spectrum function F(k): Where N represents the length of the fringe function f(x); j is the imaginary unit; k represents the discrete frequency component; In step S1.4, the Fourier inverse transform equation for converting the combined spectrum feature P(k) to form a new fringe function p(x) is: Where x represents the spatial domain sampling point index of the fringe function.

6. The underwater binocular camera image matching enhancement method based on spectral feature projection according to claim 3, characterized in that: In step S2, the binocular camera performs feature matching, disparity map estimation, and depth map calculation on the captured and calibrated images in sequence, and finally generates point cloud data.

7. The underwater binocular camera image matching enhancement method based on spectral feature projection according to claim 3 is characterized in that: In step S2, filtering out the projection fringes of the projected composite image acquired by the left camera includes the following specific steps: S2.

1. Perform discrete Fourier transform on the projected composite image acquired by the left camera row by row along the horizontal direction according to the pixel row, and convert the projected composite image from the spatial domain to the frequency domain. After the conversion, the frequency domain function F m (k); S2.2, by binarizing and inverting the combined spectrum feature P(k) obtained in S1.3, a frequency domain filter Filter(k) is generated; S2.3, using the frequency filter Filter(k) generated in step S2.2, the transformed spectrum function F m (k) After performing the filtering operation, the image function f′ after the stripes are filtered out can be obtained by performing the inverse discrete Fourier transform. m (x).

8. The underwater binocular camera image matching enhancement method based on spectral feature projection according to claim 7, characterized in that: In step S2.1, the frequency domain function F obtained after conversion m (k) is: Among them, m represents the pixel row index of the image along the vertical direction, M is the height of the image, which means the total number of pixels in the vertical direction, that is, the total number of rows, and f m (x) is the function of the pixels in the mth row of the projected composite image; In step S2.3, the image function f′ m The calculation equation for (x) is:

9. The underwater binocular camera image matching enhancement method based on spectral feature projection according to claim 7, characterized in that: In step S2.2, on the basis of generating the frequency domain filter, an adjustment coefficient α for the DC component of k=0 is added to suppress the stray light background caused by water scattering and to correct the color deviation caused by seawater absorption to a certain extent; The value of α depends on the optical parameters of seawater, the brightness of the illumination source, and the projection brightness.

Images