A method for geometric correction of underwater hyperspectral push-broom images
By combining USBL and laser ranging data in an underwater hyperspectral imager for image stitching and correction, the finite element discrete allocation method is used to solve the image distortion problem in the underwater environment, and efficient geometric correction and visual effects are achieved Improvement.
Patent Information
- Application Number
- CN202111654296.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-12-30
AI Technical Summary
Underwater hyperspectral imager is severely distorted due to waves and robot posture instability in underwater environments, and traditional geometric correction methods are difficult to achieve effective image correction, especially in underwater environments, which cannot accurately obtain the image of each frame. Directional element.
The underwater visible light camera and hyperspectral camera are used to collect data in push-scan mode, combine USBL positioning and laser ranging data for image stitching and correction, and use the finite element discrete discrete allocation method to perform geometric correction on hyperspectral images, avoiding the use of the IMU data dependence and precise correction of images is achieved through polynomial model fitting and chunking matching.
The efficiency and visual effect of image correction are improved, so that the image is closer to the real morphology of the target, avoiding the influence of IMU data error and underwater environment limitations in traditional methods, and is suitable for geometric correction of underwater hyperspectral continuous images.
Smart Images
Figure CN114494039B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of underwater hyperspectral imaging, and in particular to a method for geometric correction of underwater hyperspectral push-scan images. Background Art
[0002] Underwater hyperspectral imaging technology has been applied to underwater environmental research in recent years due to its high spatial resolution and spectral resolution, and its ability to identify and classify targets at a fine scale. With the development of marine engineering technology, the detection capability of hyperspectral imaging technology in underwater and deep-sea areas has gradually increased. By being mounted on an underwater robot system, a hyperspectral imager can achieve continuous, large-scale, low-risk and efficient investigation of the underwater environment, and can be applied to fields such as deep-sea mineral exploration, seabed habitat mapping, underwater coral resource investigation, and underwater archaeology.
[0003] Usually, underwater robot platforms are also equipped with lighting systems, high-definition cameras, USBL and other sensors to provide seabed lighting and underwater positioning and navigation functions. Due to the uncertainty of the underwater environment and the influence of water flow fluctuations, underwater robots cannot move in an ideal situation (constant speed, height and no shaking), so the continuous images collected by the hyperspectral imager will produce serious distortion, which is not conducive to the classification and identification of underwater targets. At present, the data processing for underwater hyperspectral imagers is mainly the calibration of radiation and reflectivity and simple geometric correction. There is a lack of correction means for image distortion. Affected by underwater waves, the posture of underwater robots has great uncertainty and changes rapidly over time. The platform has poor stability. The geometric correction method of traditional satellite-borne and airborne push-broom hyperspectral images requires accurate input of the exterior orientation elements of each frame of the image. For underwater robot platforms, it is difficult to accurately obtain the exterior orientation elements of each frame of the image. Therefore, traditional remote sensing methods are not suitable for underwater hyperspectral continuous images. Summary of the invention
[0004] The purpose of the present invention is to provide a method for geometric correction of underwater hyperspectral push-broom images. Compared with traditional overall image correction, this method has higher correction efficiency, better image correction visual effect, and is closer to the actual morphology of the target.
[0005] The objective of the present invention is achieved through the following technical solutions:
[0006] A method for geometric correction of underwater hyperspectral push-broom images, the method comprising:
[0007] Step 1: First, use an underwater visible light camera and an underwater hyperspectral camera to collect data in an underwater push-scan mode to obtain a visible light image and a continuous push-scan hyperspectral image; wherein the underwater visible light camera and the underwater hyperspectral camera are mounted in a photoelectric ball of an underwater robot platform, and the optical paths of the two cameras are coaxial;
[0008] Step 2: During the underwater robot detection process, the real-time positioning data of the underwater robot is obtained by using an ultra-short baseline positioning system USBL, and the time when each frame of the visible light image is acquired and the USBL positioning data are recorded;
[0009] Step 3: After data collection is completed, the collected visible light images are spliced and corrected in combination with the laser ranging data and USBL positioning data to obtain orthophotos, and the spliced and corrected orthophotos are used as reference images;
[0010] Step 4: Use the reference image in step 3 to perform geometric correction on the continuous push-scan hyperspectral image obtained in step 1 using the finite element discrete allocation calibration method.
[0011] It can be seen from the technical solution provided by the present invention that the above method is more efficient than the traditional whole image correction, has a better image correction visual effect, and is closer to the real shape of the target. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0013] Figure 1 A schematic flow chart of a method for geometric correction of underwater hyperspectral push-broom images provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0015] like Figure 1 The figure is a schematic flow chart of a method for geometric correction of underwater hyperspectral push-broom images provided by an embodiment of the present invention, the method comprising:
[0016] Step 1: First, use an underwater visible light camera and an underwater hyperspectral camera to collect data in an underwater push-scan mode to obtain a visible light image and a continuous push-scan hyperspectral image;
[0017] The underwater visible light camera and the underwater hyperspectral camera are mounted in the photoelectric ball of the underwater robot platform, and the optical paths of the two cameras are coaxial.
[0018] In this step, the underwater hyperspectral camera is a linear array push-broom hyperspectral imager, the working wavelength of the data acquisition part covers the visible-near infrared band of 380-800nm, the spectral resolution is 4nm, and the number of spectral bands is 120; the frame rate of the underwater visible light camera is 30FPS, and the pixel resolution is 4096×2160.
[0019] In the specific implementation process, the photoelectric ball is installed in the photoelectric pod of the underwater robot platform. The photoelectric pod adopts a two-axis two-frame structure design, which can realize movement in azimuth and pitch. The rotation range is 360°×n continuous in azimuth, and pitch is -150°~+20°. The horizontal optical axis is 0°, and the vertical downward viewing is -90°.
[0020] When the underwater robot performs underwater push-broom imaging tasks, the optoelectronic pod will rotate to a specific angle so that the underwater hyperspectral camera and the underwater visible light camera form a fixed angle with the underwater ground. The underwater robot moves along a predetermined route, thereby achieving push-broom imaging of underwater targets.
[0021] Step 2: During the underwater robot detection process, the real-time positioning data of the underwater robot is obtained by using an ultra-short baseline positioning system USBL (Ultra-Short Baseline), and the time and USBL positioning data of each frame of the visible light image are recorded;
[0022] Step 3: After data collection is completed, the collected visible light images are spliced and corrected in combination with the laser ranging data and USBL positioning data to obtain orthophotos, and the spliced and corrected orthophotos are used as reference images;
[0023] In this step, the process of stitching and correcting the collected visible light image by combining the laser ranging data and the USBL positioning data is as follows:
[0024] Since underwater imaging is affected by water absorption and backscattering, the image brightness is dark and the details are blurred. In order to extract more feature points in the registration, the collected visible light images are first preprocessed, including gamma correction and Laplace edge enhancement.
[0025] Since the visible light camera acquires video images, the content overlap rate of adjacent frame images is very high. Therefore, in order to improve the stitching speed, a stitching experiment is required before formal stitching to determine the appropriate number of stitching images and image selection interval. The SIFT algorithm is used to extract the features of the preprocessed visible light image, and the RANSC algorithm is used to screen the matching points. The transformation matrix is calculated based on the screened matching point pairs and the image is stitched. Among them, the feature detection of the SIFT algorithm is mainly divided into four steps:
[0026] (1) Construct a scale space and generate multi-scale images; (2) Detect feature points and eliminate mismatched points; (3) Assign directions to feature points to make them rotation invariant; (4) Generate descriptors for feature points.
[0027] The above-mentioned SIFT algorithm feature extraction is invariant to image rotation, scaling, and brightness changes, and has good stability to perspective, affine transformation, and noise. Since the SIFT algorithm may have one-to-many mismatches during feature point matching, the RANSC algorithm can be used to eliminate mismatched points and improve the accuracy of registration.
[0028] After the stitched images are obtained, the stitched images are further geometrically corrected. Specifically, the least squares polynomial model fitting correction method is adopted. The polynomial model expresses the coordinate relationship between the corresponding points of the images before and after correction with a general polynomial. This model avoids the spatial geometric process during imaging and directly performs mathematical simulation on the image deformation itself. All the geometric error sources and overall deformation of the remote sensing images are regarded as the result of the comprehensive effects of translation, scaling, rotation, affine, torsion, bending and higher-order basic deformations. The deformation of the original image is regarded as a certain surface, and the output image is a regular plane, which meets the requirements of geometric correction.
[0029] Among them, the polynomial model introducing the elevation Z is expressed as:
[0030]
[0031] In the formula, the image plane coordinates of the image point are (x, y); the geodetic coordinates of the object point corresponding to the image point are (X, Y, Z); the coefficients of the polynomial are (a i ,b i )(i=1,2,…9), this coefficient is also the unknown coefficient to be solved;
[0032] In order to solve the undetermined coefficients in the above polynomial model, multiple sets of known image plane-object plane corresponding coordinate point pairs are required. Although the USBL positioning point is not the center point of the underwater visible light image, since the positioning module of the underwater robot and the visible light camera maintain the same relative position, the positioning point can be used as a pseudo center point, that is, the image center point of each spliced image is used as the image point in the control point pair, and the USBL positioning data is used as the seabed coordinate point in the control point pair. The USBL positioning data is used to provide longitude and latitude information (X, Y), and the laser ranging data is used to provide the height information (Z) from the seabed when measuring each frame of the image;
[0033] In geometric correction, the coordinate positions (X, Y, Z) of multiple control point pairs are substituted, and the undetermined coefficients (ai, bi) in the polynomial model (1) are calculated using the least squares method. It can be seen from the polynomial model (1) that when the subscript i is 6, only 7 center points are needed to calculate the size of the parameters. However, in practical applications, more points with the same name can be substituted to obtain better fitting results.
[0034] Substitute the image plane coordinates (x, y) of all image points into the polynomial model (1) to obtain the corresponding object space coordinates (X, Y). ref ,Y ref ) into the polynomial model (1) and calculate the transformed (x ref ,y ref ) pixel value, since (x ref ,y ref ) is not an integer, so it is necessary to obtain new pixels by interpolation on the original pixels, that is, to obtain the corrected visible light image after image grayscale resampling.
[0035] The interpolation method uses cubic polynomial interpolation. The corrected visible light image can be mapped to obtain object coordinate information, which can be used to splice into a larger image in subsequent processing.
[0036] In a specific implementation, in addition to the least squares polynomial model fitting and correction method described in the embodiment of the present invention, other algorithms that use USBL data as control point coordinates for solution are also feasible.
[0037] Step 4: Use the reference image in step 3 to perform geometric correction on the continuous push-scan hyperspectral image obtained in step 1 using the finite element discrete allocation calibration method.
[0038] In this step, due to the instability factors such as pitch, roll and non-uniform forward movement of the underwater robot platform, the underwater hyperspectral image obtained by push-scanning is distorted, so it is necessary to correct the distortion of the underwater hyperspectral image. Specifically, the visible light image after splicing correction is used as the reference image, and the underwater hyperspectral image is used as the image to be registered for matching. Since the reference image is large in size and has a large number of pixels, the traditional registration method for the hyperspectral image is not only slow, but also has poor results after matching correction. In order to solve the above problems, this embodiment adopts the idea of finite element discrete unit allocation registration to perform block matching on the hyperspectral image. The specific geometric correction process is:
[0039] 1) According to the idea of finite element discrete demarcation calibration, the reference image in step 3 is first gridded and divided into blocks. The block size is determined by the image size and the algorithm operation speed. The acquired hyperspectral image is extracted. Since the hyperspectral image has not been spliced, the continuous push-scan hyperspectral image obtained in step 1 can be directly extracted in blocks according to the acquisition time. The block size is similar to the gridded reference image, and is generally slightly smaller than the reference image.
[0040] 2) After obtaining the reference image and the hyperspectral image after the blocks are obtained, the similarity detection is performed on the two, and for each block of the hyperspectral image, the reference image with the highest similarity is found; in the specific implementation, this process can be detected manually or automatically by algorithm. The manual method is for the human eye to judge whether the reference image after the blocks and the hyperspectral image match, and the reference block image with the highest similarity is found for each block of the hyperspectral image; the automatic detection by algorithm is to use the algorithm to calculate and find the reference block image with the highest similarity for each block of the hyperspectral image, in which the algorithm part can use the ORB algorithm to detect the similarity of the image blocks, and the specific method is to calculate the feature point descriptors one by one and match them two by two. The image with the most matching points is considered to have the highest similarity between the two blocks, that is, the largest overlapping area;
[0041] 3) Then, each hyperspectral image with the highest similarity is accurately registered and spliced with the reference image to achieve geometric correction of the continuous push-scan hyperspectral image.
[0042] In a specific implementation, the means for accurately registering and splicing each hyperspectral image with the highest similarity with the reference image specifically includes:
[0043] The SIFT algorithm is used to calculate the feature descriptors, and after matching and screening, the homography transformation matrix of the hyperspectral image block relative to the reference image block is obtained;
[0044] The homography transformation matrix is then applied to the hyperspectral image to be corrected and the image stitching based on the reference image area, wherein the stitching edge uses an image fusion method to make the edge transition more natural.
[0045] In addition, in addition to the ORB and SIFT algorithms used in this embodiment, other achievable similarity detection and registration algorithms are also feasible.
[0046] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.
[0047] In summary, the method described in the embodiment of the present invention has the following advantages:
[0048] 1. Use USBL data to stitch and correct visible light images first, and then correct the hyperspectral image based on the visible light image. This method does not require the system's IMU data, that is, it does not require the exterior orientation elements of each frame. This can avoid the error caused by interpolation and filtering of IMU data, so that the system has no accuracy requirements for IMU in image correction;
[0049] 2. In the visible light image stitching correction method, multiple USBL positioning data are used as the pseudo image center, and the least squares polynomial fitting method does not require ground control points, which is very important for underwater imaging systems. Because DEM data cannot be obtained in underwater environments, it is difficult to set control points to orthorectify images;
[0050] 3. Using visible light images as reference image correction can reduce the impact of image distortion caused by changes in underwater robot speed and water wave fluctuations. The image correction result has a better visual effect and is closer to the true appearance of the target.
[0051] 4. The finite element matching method is more efficient than the traditional overall image correction method. In addition, when there are many distortions in the image, the registration and correction effect of the homography matrix calculated in blocks is better.
[0052] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.
Claims
1. A method for geometric correction of underwater hyperspectral push-broom images, characterized in that: The method comprises: Step 1: First, use an underwater visible light camera and an underwater hyperspectral camera to collect data in an underwater push-scan mode to obtain a visible light image and a continuous push-scan hyperspectral image; wherein the underwater visible light camera and the underwater hyperspectral camera are mounted in a photoelectric ball of an underwater robot platform, and the optical paths of the two cameras are coaxial; Step 2: During the underwater robot detection process, the real-time positioning data of the underwater robot is obtained by using an ultra-short baseline positioning system USBL, and the time when each frame of the visible light image is acquired and the USBL positioning data are recorded; Step 3: After data collection is completed, the collected visible light images are spliced and corrected in combination with the laser ranging data and USBL positioning data to obtain orthophotos, and the spliced and corrected orthophotos are used as reference images; In step 3, the process of stitching and correcting the collected visible light image by combining the laser ranging data and the USBL positioning data is as follows: First, the acquired visible light images are preprocessed, including gamma correction and Laplace edge enhancement; Then, the SIFT algorithm is used to extract the features of the preprocessed visible light image, and the RANSC algorithm is used to screen the matching points. The transformation matrix is calculated based on the screened matching point pairs and the images are spliced. After the stitched image is obtained, the stitched image is geometrically corrected. Specifically, the least squares polynomial model fitting correction method is adopted, and the polynomial model of the height Z is introduced and expressed as: In the formula, the image plane coordinates of the image point are (x, y); the geodetic coordinates of the object point corresponding to the image point are (X, Y, Z); the coefficients of the polynomial are (a i ,b i )(i=1,2,…9), this coefficient is also the unknown coefficient to be solved; Then, the center point of the image plane of each stitched image is used as the image point in the control point pair, and the USBL positioning data is used as the seabed coordinate point in the control point pair. The USBL positioning data is used to provide longitude and latitude information (X, Y), and the laser ranging data is used to provide the height information (Z) from the seabed when each frame of image is measured; In geometric correction, the coordinate positions (X, Y, Z) of multiple control point pairs are substituted and the undetermined coefficients (ai, bi) in the polynomial model (1) are calculated using the least squares method; Substitute the image plane coordinates (x, y) of all image points into the polynomial model (1) to obtain the corresponding object space coordinates (X, Y). ref ,Y ref )Calculate the transformed (x ref ,y ref ), and then resample the grayscale of the image to obtain a corrected visible light image; Step 4: Using the reference image in step 3, a finite element discrete partitioning calibration method is used to perform geometric correction on the continuous push-scan hyperspectral image obtained in step 1; The photoelectric ball is installed in the photoelectric pod of the underwater robot platform. The photoelectric pod adopts a two-axis two-frame structure design, which can realize movement in azimuth and pitch. The rotation range is 360°×n continuous in azimuth, and pitch is -150°~+20°. The horizontal optical axis is 0°, and the vertical downward viewing is -90°. When the underwater robot performs underwater push-broom imaging tasks, the optoelectronic pod will rotate to a specific angle so that the underwater hyperspectral camera and the underwater visible light camera form a fixed angle with the underwater ground. The underwater robot moves along a predetermined route, thereby achieving push-broom imaging of underwater targets.
2. The method for geometric correction of underwater hyperspectral push-broom images according to claim 1, characterized in that: In step 1, the underwater hyperspectral camera is a linear array push-broom hyperspectral imager, the working wavelength of the data acquisition part covers the visible-near infrared band of 380-800nm, the spectral resolution is 4nm, and the number of spectral bands is 120; The frame rate of the underwater visible light camera is 30FPS and the pixel resolution is 4096×2160.
3. The method for geometric correction of underwater hyperspectral push-broom images according to claim 1, characterized in that: The process of step 4 is specifically as follows: 1) First, the reference image in step 3 is gridded and divided into blocks. The block size is determined according to the image size and the algorithm operation speed. Then, for the continuous push-scan hyperspectral image obtained in step 1, block extraction is performed according to the acquisition time, and the block size is similar to that of the gridded reference image. 2) After obtaining the reference image and the hyperspectral image after the blocks are divided, the similarity between the two is detected, and the reference image with the highest similarity is found for each hyperspectral image; 3) Then, each hyperspectral image with the highest similarity is accurately registered and spliced with the reference image to achieve geometric correction of the continuous push-scan hyperspectral image.
4. The method for geometric correction of underwater hyperspectral push-broom images according to claim 3, characterized in that: The means for accurately registering and splicing each hyperspectral image with the highest similarity with the reference image specifically include: The SIFT algorithm is used to calculate the feature descriptors, and after matching and screening, the homography transformation matrix of the hyperspectral image block relative to the reference image block is obtained; The homography transformation matrix is then applied to the hyperspectral image to be corrected and the image stitching based on the reference image area.