Underwater depth camera refraction correction and online self-calibration method, device and system

Abstract

The application discloses a method, device and system for underwater depth camera refraction correction and online self-calibration, and relates to the technical field of underwater robot environment perception and three-dimensional measurement. A composite medium refraction model of air-observation window-water is established to correct the refraction ray of the pixel line of sight, effectively reducing the systematic geometric error caused by the refraction of the observation window. Further, the IMU attitude and the fitting result of the environment plane are combined to construct a constraint, and the refraction model parameters are estimated and updated online to realize adaptive calibration under the condition of assembly deviation or running drift. Meanwhile, the confidence is calculated based on multiple point cloud quality indicators, and the reliable point cloud is output, so that the downstream close-range ranging, wall sticking and docking tasks can use the point cloud data according to the reliability, thereby improving the stability and reliability of the underwater close-range perception, and reducing the manual calibration and maintenance cost.

Description

Technical Field

[0001] This invention relates to the field of underwater robot environmental perception and three-dimensional measurement technology, specifically to underwater depth camera refraction correction and online self-calibration methods, devices and systems. Background Technology

[0002] In recent years, underwater robots (such as ROVs and AUVs) have been widely used in scenarios such as pool inspection, ship and bridge pier inspection, aquaculture operations, and close-range observation of pipelines / structures. Underwater operations typically require robots to complete tasks such as wall-hugging navigation, fixed-distance hovering, close-range docking, target ranging, and 3D reconstruction within a limited space. These tasks place higher demands on the accuracy and stability of close-range 3D perception. Compared to traditional monocular vision, binocular depth cameras or structured light / infrared depth cameras can output dense depth images and point cloud information at close range, thereby supporting fine ranging and target geometry extraction. Therefore, they are often integrated into underwater robots as close-range point cloud sensors.

[0003] However, depth cameras used underwater typically need to be installed inside a waterproof hull and imaged through observation windows (such as flat or curved windows). Light passing through the observation window from the air inside the hull and then into the water undergoes multiple refractions, rendering the imaging geometry model calibrated in air inapplicable. This introduces systematic geometric errors, such as depth scale deviation, increased distortion at the edge of the field of view, shift in the normal direction of the point cloud, and nonlinear errors that change with attitude. Particularly at close range, refraction errors significantly affect the reliability of wall-hull distance control, docking positioning, and precise ranging.

[0004] Furthermore, underwater environments commonly suffer from issues such as light attenuation, scattering by suspended particles, and unstable echoes caused by reflective or dark materials. These problems easily lead to holes in depth images, increased noise points, and depth jumps, resulting in localized and time-varying degradation of point cloud quality. Existing technologies commonly employ methods such as fixed-ratio correction of depth results, obtaining compensation parameters through single offline calibration, or smoothing through simple filtering in the image / point cloud backend. These methods typically rely on manual calibration and fixed parameters, making it difficult to adapt to factors such as observation window assembly tolerances, minor camera attitude deviations within the cabin, relative displacement changes caused by vibration / collision during operation, and refractive index fluctuations due to temperature changes. Simultaneously, existing solutions often lack a quantitative evaluation mechanism for point cloud reliability, making downstream functions such as wall-hugging control, close-range ranging, or docking susceptible to misleading data from low-quality point cloud data, thus reducing system stability and safety.

[0005] Therefore, there is an urgent need for methods and devices that can effectively compensate for the refraction effect caused by the observation window of the waterproof hull for underwater near-range point clouds, and can achieve online self-calibration by combining inertial measurement and environmental structural constraints during operation, while outputting point cloud reliability information, so as to improve the accuracy, robustness and engineering usability of underwater near-range point cloud sensing, and reduce the cost of manual recalibration and maintenance. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method, apparatus, and system for underwater depth camera refraction correction and online self-calibration, thereby solving the problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for refraction correction and online self-calibration of an underwater depth camera, comprising:

[0008] Acquire raw depth images and / or infrared intensity images output by a depth camera installed in a waterproof enclosure, as well as attitude data output by an inertial measurement unit (IMU).

[0009] A composite medium refraction model of air-observation window-water is established based on the parameters of the observation window of the waterproof cabin.

[0010] Based on the refraction model, the pixel line of sight corresponding to the original depth image is corrected by refraction ray to generate an underwater compensated depth image and / or compensated point cloud.

[0011] Within a preset time window, environmental plane detection and fitting are performed on the compensated point cloud to obtain plane parameters;

[0012] By using the IMU attitude data and the plane parameters to construct constraints, the parameters to be estimated in the refraction model are estimated and updated online to achieve online self-calibration of the depth camera;

[0013] The confidence level of the point cloud is calculated based on the depth void ratio, local geometric consistency, infrared intensity statistics and / or distance interval indicators, and the confidence level of the point cloud is output.

[0014] Preferably, establishing the air-observation window-water composite medium refraction model based on the parameters of the waterproof cabin observation window includes:

[0015] Acquire or set the thickness, refractive index, and normal direction parameters of the observation window;

[0016] Based on the thickness, the refractive index, and the window normal direction parameters, the refraction and propagation relationship of the pixel line of sight in the three media of air-viewing window-water is constructed.

[0017] The direction of the refracted ray and / or equivalent imaging geometry parameters of the pixel line of sight are determined based on the aforementioned refraction propagation relationship for subsequent refracted ray correction.

[0018] Preferably, after establishing the composite medium refraction model of air-observation window-water, the method further includes:

[0019] Intrinsic parameter matrix based on depth camera pixel coordinates Converted to incident direction vector in camera coordinate system and the incident direction vector The incident direction vector is normalized. The expression is as follows:

[0020]

[0021] The angle of refraction was calculated using Snell's law at both the air-observation window interface and the observation window-water interface. The expression for Snell's law is as follows:

[0022]

[0023] in, Let be the refractive index of the incident medium. Let be the refractive index of the refracting medium. Angle of incidence The angle of refraction;

[0024] The underwater ray direction vector is obtained from the two interface refractions. and based on the The propagation direction of the pixel's line of sight in the underwater coordinate system is determined for the purpose of the refracted ray correction.

[0025] Preferably, the step of performing environmental plane detection and fitting on the compensated point cloud within a preset time window to obtain plane parameters includes:

[0026] The candidate planes in the compensated point cloud are fitted and the plane fitting quality index of the candidate planes is calculated. The plane fitting quality index includes the proportion of inliers and / or the fitting residual from the points to the plane.

[0027] In response to determining that the proportion of inliers is greater than a preset threshold and / or the fitting residual is less than a preset threshold, the corresponding candidate plane point set and its plane parameters are determined as the target plane parameters for the online self-calibration.

[0028] Preferably, the fitting residual in the plane fitting quality index is calculated using the following formula:

[0029]

[0030] in, Represents the fitting residual. This represents the number of points used for plane fitting. This represents the three-dimensional coordinate vector of the i-th point. Denotes the unit normal vector of the fitted plane. The plane bias parameters represent the fitted plane, such that the plane equation is... .

[0031] This invention also discloses an underwater depth camera refraction correction and online self-calibration device, used to implement the aforementioned underwater depth camera refraction correction and online self-calibration method, comprising:

[0032] The data acquisition module is configured to acquire raw depth images and / or infrared intensity images output by a depth camera installed in the waterproof chamber, as well as attitude data output by an inertial measurement unit (IMU).

[0033] The refraction modeling module is configured to establish a composite medium refraction model of air-observation window-water based on the parameters of the observation window of the waterproof cabin;

[0034] The ray correction module is configured to perform refracted ray correction on the pixel line of sight corresponding to the original depth image based on the refraction model, and generate an underwater compensated depth image and / or compensated point cloud.

[0035] The plane estimation module is configured to perform environmental plane detection and fitting on the compensated point cloud within a preset time window to obtain plane parameters;

[0036] The online self-calibration module is configured to construct constraints using the IMU attitude data and the plane parameters, and to estimate and update the parameters to be estimated in the refraction model online, so as to realize the online self-calibration of the depth camera.

[0037] The confidence calculation module is configured to calculate the confidence of the point cloud based on the depth hole rate, local geometric consistency, infrared intensity statistics and / or distance interval indicators, and output the reliable point cloud and its confidence information.

[0038] The output module is configured to output the compensated depth image and / or the compensated point cloud and the corresponding point cloud confidence information.

[0039] Preferably, the confidence calculation module is based on the depth hole rate, the local geometric consistency, the infrared intensity statistics, and the distance interval index, which are calculated according to relevant data corresponding to the original depth image and / or infrared intensity image; the relevant data includes at least the following:

[0040] Depth void ratio, depth jump rate, local geometric consistency index, infrared intensity mean, infrared intensity variance, image contrast index and / or distance interval information.

[0041] This invention also discloses an underwater depth camera refraction correction and online self-calibration system, used to implement the aforementioned underwater depth camera refraction correction and online self-calibration method, comprising:

[0042] A waterproof cabin, wherein the waterproof cabin is equipped with an observation window;

[0043] A depth camera, installed inside the waterproof chamber, is used to output raw depth images and / or infrared intensity images;

[0044] An inertial measurement unit (IMU) is used to output attitude data.

[0045] The processing unit is communicatively connected to the depth camera and the IMU, and is configured to execute the underwater depth camera refraction correction and online self-calibration method to output compensated depth images and / or compensated point clouds and their point cloud confidence information.

[0046] Preferably, the processing unit includes at least one of an edge computing board, an embedded processor, and / or an industrial computer.

[0047] The technical effects and advantages of this invention are as follows:

[0048] The underwater depth camera refraction correction and online self-calibration method, device, and system provided in this application correct the refracted rays of the pixel line of sight by establishing a composite medium refraction model of air-observation window-water, effectively reducing the systematic geometric error caused by observation window refraction. Furthermore, by combining the IMU attitude and environmental plane fitting results to construct constraints, the refraction model parameters are estimated and updated online to achieve adaptive calibration under assembly deviation or operational drift conditions. At the same time, confidence is calculated based on multiple point cloud quality indicators and a reliable point cloud is output, enabling downstream close-range ranging, wall-hugging, and docking tasks to use point cloud data according to confidence, thereby improving the stability and reliability of underwater close-range sensing and reducing manual calibration and maintenance costs. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of the method of the present invention;

[0051] Figure 2 This is a schematic diagram of the air-viewing window-water composite medium refraction model and pixel line-of-sight refraction propagation in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the environmental plane detection and fitting process based on compensated point cloud according to an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the online self-calibration process based on IMU attitude and planar parameter constraints according to an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the point cloud confidence calculation and trusted point cloud output process according to an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of a depth camera refraction compensation and online self-calibration device for underwater near-field point cloud perception, according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This embodiment discloses a method for refraction correction and online self-calibration of an underwater depth camera. This method corrects the refracted rays of the pixel line of sight by establishing an air-viewing window-water composite medium refraction model, and further constructs constraints by combining the IMU attitude and environmental plane fitting results to estimate and update the parameters to be estimated in the refraction model online. At the same time, it calculates the point cloud confidence based on multiple point cloud quality indicators and outputs the reliable point cloud and its confidence information, thereby improving the accuracy and robustness of underwater near-range point cloud perception and reducing manual calibration and maintenance costs.

[0058] The technical solution of this application will be described in detail below through specific embodiments.

[0059] refer to Figure 1 One embodiment of this application provides a depth camera refraction compensation and online self-calibration method for underwater near-range point cloud sensing, which specifically includes the following steps:

[0060] Step S101: Acquire the raw depth image and / or infrared intensity image output by the depth camera installed in the waterproof chamber, as well as the attitude data output by the inertial measurement unit (IMU).

[0061] In this step, the depth camera can be a binocular depth camera or an infrared structured light depth camera, and its output can include the raw depth image. Infrared intensity image (or grayscale / texture image), the IMU output may include attitude quaternions. Euler angles, angular velocity and / or acceleration Data such as these. To improve the stability of subsequent refraction compensation and self-calibration, depth camera data and IMU data can be synchronized in time or aligned to approximately the same time, for example, by interpolation alignment based on timestamps. No specific limitations are made here.

[0062] Step S102: Establish a composite medium refraction model of air-observation window-water based on the parameters of the observation window of the waterproof cabin.

[0063] In this step, the observation window can be a flat window or a curved window; taking a flat window as an example, the observation window parameters may include: observation window thickness. Observation window material refractive index Window normal direction (Unit vector in camera coordinate system or cabin coordinate system), and air refractive index With the refractive index of water A composite medium refraction model is established based on parameters to describe the propagation relationship of two interface refractions during the process of the pixel's line of sight entering the observation window from air and then entering the water body. As an implementation method, Snell's law can be used to model the two interface refractions, and its expression is:

[0064] ;

[0065] in, The refractive index of the incident medium is... The refractive index of the refractive medium, Angle of incidence It is the angle of refraction.

[0066] It should be noted that if the observation window is a curved window, curvature parameters, local normal calculation methods, etc. can be further introduced to construct a refraction model, which is not limited here.

[0067] Step S103: Based on the refraction model, the pixel line of sight corresponding to the original depth image is corrected by refraction ray to generate an underwater compensated depth image and / or compensated point cloud.

[0068] In this step, each pixel in the original depth image can be processed. The corresponding line of sight is corrected at the ray level. As a implementation method, it is first based on the camera intrinsic parameter matrix... pixel coordinates Converted to incident direction vector in camera coordinate system and to Normalization, example as follows:

[0069]

[0070] Subsequently, based on the composite medium refraction model established in step S102, refraction was calculated at the air-observation window interface and the observation window-water interface to obtain the ray direction vector after underwater refraction. (Unit vector). Based on this, the original depth value can be mapped to a compensated distance along the underwater refracted ray, thus obtaining a compensated depth image. Or directly generate a compensated point cloud. Its point coordinates can be represented as:

[0071]

[0072] in, The distance parameter along the underwater refracted ray. It can be determined by the original depth value, the thickness of the viewing window, and the refractive geometry. It should be noted that... The solution can be obtained analytically or numerically iteratively; no specific method is required here.

[0073] By correcting the refracted rays as described above, the systematic geometric errors caused by refraction in the observation window can be reduced, especially improving point cloud distortion and scale deviation in the edge region of the field of view.

[0074] Step S104: Perform environmental plane detection and fitting on the compensated point cloud within a preset time window to obtain plane parameters.

[0075] In this step, the preset time window can be a fixed frame number window or a fixed duration window, such as a time window of 0.5s to 2s (this is just an example and does not constitute a limitation). The compensated point cloud is then collected within the window. Then, plane detection and fitting can be performed on the point cloud to obtain the plane parameters, where the plane equation can be expressed as:

[0076]

[0077] in, The unit normal vector of the plane. For plane offset parameters, is a three-dimensional point coordinate vector on a plane.

[0078] As an implementation method, plane detection can employ methods such as RANSAC, least squares fitting, or region growing. After fitting, the plane fitting quality index is calculated, which may include the inlier ratio ρ and / or the fitting residual. Response to determination Greater than a preset threshold and / or If the values ​​are less than a preset threshold, the corresponding candidate plane point set and its plane parameters are determined as the target plane parameters for online self-calibration. The fitting residual e can be calculated using the following formula:

[0079]

[0080] in, The number of points used for plane fitting. Let be the three-dimensional coordinate vector of the i-th point.

[0081] Step S105: Using IMU attitude data and planar parameters to construct constraints, online estimation and updating of the parameters to be estimated in the refraction model are performed to achieve online self-calibration of the depth camera.

[0082] In this step, the parameters to be estimated for the refraction model may include at least the following: depth scale factor. Depth bias term Observation window normal micro-deviation (or equivalent small pitch / yaw deviation) , ), observation window thickness correction amount Observation window refractive index correction amount wait.

[0083] As a method of implementation, the attitude information provided by the IMU can be used to determine the direction of gravity. The direction in the camera coordinate system or the cabin coordinate system, combined with the target plane normal obtained in step S104. Construct consistency constraints. For example, for the pool bottom plane, the plane normal should be approximately collinear with the gravity direction (either opposite or the same direction, depending on the coordinate system definition), thus introducing a normal consistency term into the optimization objective; simultaneously, a point-to-plane residual minimization term is introduced to estimate and update the parameters to be estimated in the refraction model online. An exemplary optimization objective function can be expressed as:

[0084]

[0085] in, This represents the set of points corresponding to the target plane. It is the unit vector of gravity direction (or its opposite direction) obtained from the IMU attitude calculation. These are the weighting coefficients.

[0086] The optimization solution can employ iterative least squares, gradient descent, extended Kalman filtering, or other online estimation methods to update the refraction model parameters, enabling subsequent refraction ray corrections to adapt to changes caused by assembly deviations or operational drift.

[0087] Step S106: Calculate the point cloud confidence level based on indicators such as depth void ratio, local geometric consistency, infrared intensity statistics and / or distance interval, and output the reliable point cloud and its confidence level information.

[0088] In this step, to quantify the local reliability of the underwater point cloud, the confidence level of the compensated depth image and / or compensated point cloud can be calculated. The confidence level can be pixel-level or point-level. As an implementation method, the depth hole rate... It can be calculated based on the percentage of invalid depth pixels:

[0089]

[0090] in, Invalid depth pixels This represents the total number of pixels within the statistical area.

[0091] Local geometric consistency can be obtained by measures such as neighborhood depth variance, neighborhood normal consistency, or local plane residuals of the point cloud; infrared intensity statistics can be obtained by... The mean, variance, or contrast index is obtained; distance interval information can be used to weight the credibility of different distance segments (e.g., higher weight for closer distance segments or reduced weight for refraction-sensitive areas).

[0092] Furthermore, the above indicators can be combined according to their weights to obtain the overall confidence level, as exemplified below:

[0093]

[0094] in, Indicates the local geometric consistency score. Indicates infrared intensity score, Indicates the score for the distance interval. These are the weighting coefficients.

[0095] When the confidence level of the point cloud is lower than the preset threshold, the corresponding depth value or point cloud point can be removed, downweighted and / or interpolated for repair. When the confidence level of the point cloud is higher than the preset threshold, the reliable point cloud and its confidence level information are output for downstream applications such as wall-mounted distance measurement, close-range distance measurement, docking positioning or fine target geometry extraction.

[0096] As can be seen, the depth camera refraction compensation and online self-calibration method for underwater near-range point cloud sensing provided in this embodiment effectively reduces the systematic geometric error caused by observation window refraction by establishing an air-observation window-water composite medium refraction model and correcting the refracted rays. Furthermore, by combining the IMU attitude and environmental plane fitting results to construct constraints, the parameters to be estimated in the refraction model are estimated and updated online, realizing adaptive self-calibration during operation. At the same time, the point cloud confidence level is output and a reliable point cloud is generated, enabling the system to use point cloud data according to the confidence level, thereby improving the stability and reliability of underwater near-range point cloud sensing and reducing manual calibration and maintenance costs.

[0097] refer to Figure 2 , Figure 2 This illustration shows a composite medium refraction model and a schematic diagram of pixel-line-of-view refraction propagation when a depth camera is mounted inside a waterproof enclosure and images underwater through an observation window. This embodiment is used to implement the key step of "establishing an air-observation window-water composite medium refraction model and correcting pixel-line-of-view refraction rays" in the method of this application, and can correspond to... Figure 2 S201 to S206 are shown.

[0098] In this embodiment, a camera coordinate system is established. Its origin is the optical center of the depth camera. , The axis is the positive direction of the camera's optical axis. shaft and The axes form a right-handed coordinate system. The pixel coordinate system has its origin at the top left corner of the image, and the pixel is denoted as (u, v). The viewing window can be a planar window (…). Figure 2 Taking a flat window as an example, its two side interfaces are approximately two parallel planes: the air-viewing window interface. With the observation window—water interface The thickness of the observation window is... The refractive index of the observation window material is The refractive index of the air inside the cabin is The refractive index of water is The unit normal of the observation window (the direction pointing towards the air or water side can be standardized by convention) is denoted as... , exist The Chinese character is used to represent this. For ease of description, let's assume... relatively along Directional translation The result is (i.e., the two interfaces are parallel and the distance between them is ). When the observation window has an assembly tilt angle, It can be a unit vector in any direction, and still satisfy the above description.

[0099] Step S201: Obtain pixel coordinates Camera intrinsic parameter matrix and observation window parameters , Window normal .

[0100] Among them, the camera intrinsic parameter matrix The thickness of the observation window can be obtained from camera calibration. With the refractive index of the material It can be given by structural design parameters or obtained by measurement; window normal It can be given by assembly parameters or estimated and updated by the subsequent online self-calibration module (105). Air refractive index With water refractive index Commonly used approximate values ​​can be used, or compensation can be made based on temperature / salinity (this embodiment is not limited).

[0101] Step S202: From Will Convert to the incident direction vector in the camera coordinate system and normalize to obtain .

[0102] For any pixel Construct homogeneous pixel vectors Then the incident direction vector in the camera coordinate system can be expressed as:

[0103]

[0104] in, This represents the 2-norm of a vector. The unit direction vector corresponds to Figure 2 The viewing direction of pixels within the air medium.

[0105] Step S203: Calculate the direction of the ray inside the window by applying Snell's law to the first refraction at the air-observation window interface. .

[0106] First, calculate the line of sight and interface. intersection .set up The previous known point is (For example, the center point of the window can be taken at) (The coordinates in the coordinates) can be used to find the intersection point using the ray-plane intersection method:

[0107]

[0108]

[0109] when When the value is close to 0, it indicates that the ray is nearly parallel to the interface, and the pixel can be skipped or processed according to a preset strategy (no limitation).

[0110] Then the first refraction is calculated. According to Snell's law:

[0111]

[0112] in, Angle of incidence Let be the angle of refraction. For ease of engineering implementation, the refraction direction can be calculated using a vector form. Let the refractive index of the incident medium be... The refractive index of the refractive medium is refractive index ratio ;make (Agreement) When the direction points outward from the incident medium, a negative sign is used (the specific sign can be standardized according to implementation), thus obtaining the refraction direction:

[0113]

[0114] in,

[0115]

[0116] When the value inside the square root is negative, it corresponds to total internal reflection. The pixel can be marked as invalid or protected (no limitation). This is a unit direction vector, or can be further normalized, representing the ray direction within the medium of the observation window, corresponding to... Figure 2 In .

[0117] Step S204: Calculate the second refraction at the observation window-water interface according to Snell's law to obtain the direction of the underwater ray. .

[0118] because relatively along Translation It is acceptable A little bit Calculate the distance the ray reaches within the observation window. intersection :

[0119]

[0120]

[0121] Then A second refraction occurs at the point, according to Snell's law:

[0122]

[0123] in, To observe the angle of incidence from the side of the window, Let be the angle of refraction on the water side. The direction of refraction is calculated using vector form, let . , , ,

[0124]

[0125] in,

[0126]

[0127] The refracted ray direction vector within the water medium corresponds to Figure 2 In .

[0128] Step S205: Calculate the ray offset and distance parameter s based on the observation window thickness and the geometric relationship between the two refractions.

[0129] After underwater refraction, the pixel's line of sight travels along the water... Directional propagation. To generate underwater compensation points, it is necessary to determine the direction along... Distance parameters in direction As an implementation method, it can be determined according to any of the following methods. (This application is not limited to):

[0130] (1) Geometric mapping method: The depth values ​​in the original depth image are mapped to the depth values ​​in the original depth image. Convert to original 3D points in camera coordinate system and combined and Establish constraints and solve for the point after refraction. Satisfying geometric consistency ;

[0131] (2) Analytical / Approximate Method: Under the assumption of parallel plane windows, based on the incident / refraction angle and window thickness Calculate the lateral offset of the optical path and the equivalent scale change caused by refraction, and then from Converted ;

[0132] (3) Numerical iteration method: Establish an error function for the unknowns (e.g., reprojection error or residual from a point to a constraint structure), and solve it using a one-dimensional search or iterative method. .

[0133] Furthermore, a scaling factor can be introduced when solving for s. With bias (For subsequent online self-calibration updates), for example:

[0134]

[0135] in , It can be estimated and updated in real time by the online self-calibration module (105) based on the environmental plane constraints.

[0136] Step S206: Based on Generate underwater compensation points / Compensation point cloud, used for subsequent refracted ray correction.

[0137] In obtaining With distance parameters Afterwards, underwater compensation points can be generated. When with light as the center Starting point and When it is a unit vector, it can be represented as:

[0138]

[0139] Or equivalently, if the interface intersection is taken into account As the starting point of underwater rays, it can be represented as:

[0140]

[0141] in From Starting edge The distance parameter in the direction. By repeating the above calculation for all valid pixels in the image, a compensated point cloud can be generated. With / or compensated depth image Compensation point cloud It will serve as the input for subsequent environmental plane detection and fitting, as well as online self-calibration and confidence calculation.

[0142] pass Figure 2 The composite medium refraction model shown in this embodiment, along with the calculation of refraction and propagation twice along the pixel line of sight, explicitly considers the influence of multi-medium refraction (air-viewing window-water) on the pixel line of sight, achieving ray-level geometric correction. This reduces the systematic point cloud error caused by viewing window refraction and provides a foundation for subsequent online self-calibration and reliable point cloud output.

[0143] refer to Figure 3 , Figure 3 This illustration shows a schematic diagram of the environmental plane detection and fitting process based on compensated point clouds in an embodiment of this application. This embodiment is used to implement the key step in the method of this application: "performing environmental plane detection and fitting of the compensated point cloud within a preset time window to obtain plane parameters," and can correspond to... Figure 3 S301 to S307 are shown.

[0144] In this embodiment, the compensation point cloud can be Figure 1 Step S103 or Figure 2 Step S206 obtains the compensated point cloud, which is underwater point cloud data corrected by refracted rays from the air-viewing window-water composite medium. The environmental plane can be the bottom plane or wall plane of a pool / tank scene, or the dock wall or local plane of a ship's hull in other scenes, etc., and this application does not limit it. To improve the robustness of the plane parameters, this embodiment uses a point cloud accumulation within a preset time window or a sliding window strategy for plane detection and fitting.

[0145] Step S301: Obtain the compensated point cloud within the preset time window. .

[0146] In this step, the preset time window can be a fixed duration window (e.g., 0.5s to 2s) or a fixed frame number window (e.g., 10 frames to 60 frames) to improve the number of points and stability of the plane fitting while ensuring real-time performance. For sliding window scenarios, the latest frame point cloud can be added to the window and the oldest frame point cloud can be removed each time an update is performed; for cumulative window scenarios, plane fitting can be performed all at once at the end of the window. This application does not impose any limitations on this.

[0147] Step S302: Preprocess the compensated point cloud, including downsampling, denoising, and / or region of interest cropping.

[0148] In this step, to reduce computational load and suppress the impact of noise on plane fitting, the following steps can be taken: Perform at least the following preprocessing operations:

[0149] (1) Downsampling: The point cloud is downsampled to a preset resolution using methods such as voxel grid filtering or uniform sampling;

[0150] (2) Noise removal: Outlier noise points are removed by methods such as Statistical Outlier Removal and Radius Outlier Removal;

[0151] (3) Region of Interest (ROI) cropping: Crop the point cloud according to a preset spatial range (e.g., limit the distance in the Z direction, limit the X / Y range) or according to the central area of ​​the camera's field of view, so as to retain the point set that is more likely to belong to the bottom / wall of the pool.

[0152] It should be noted that the preprocessing parameters (such as voxel size, outlier threshold, and clipping range) can be set according to the actual underwater environment, camera range, and noise level, and this application does not limit them.

[0153] Step S303: Extract candidate planar point sets (pool bottom plane and / or pool wall plane) from the compensated point cloud.

[0154] In this step, candidate planar point sets can be extracted using at least the following methods:

[0155] (1) Candidate extraction based on normal: Estimate the normal vector of the preprocessed point cloud and select the point set whose normal vector is close to the preset direction as the candidate pool bottom / pool wall point set;

[0156] (2) Candidate extraction based on height / distance: In a pool scene, the bottom of the pool is usually located within a certain distance range of the camera coordinate system, and candidate points can be filtered by height threshold or distance threshold;

[0157] (3) Candidate extraction based on region growing: Using flat regions as seed points, region growing is performed to obtain a large set of planar candidate points;

[0158] (4) Candidate extraction based on RANSAC: Random sampling consensus (RANSAC) is directly performed on the point cloud to obtain the largest set of points in the plane, and this set is used as the candidate set of points in the plane.

[0159] In a pool scenario, if both a bottom plane and a wall plane exist simultaneously, the candidate point sets for the bottom and wall can be distinguished based on the relationship between the plane normal and the direction of gravity (provided by the IMU); or the plane extraction can be performed twice to obtain multiple candidate planes. This application does not limit this.

[0160] Step S304: Perform plane fitting on the candidate plane point set to obtain the candidate plane parameters. .

[0161] In this step, least squares fitting, RANSAC fitting, or weighted least squares fitting can be used to obtain the candidate plane parameters, where the plane equation can be expressed as:

[0162]

[0163] in, For unit normal vector, For plane offset parameters, Let be the coordinate vector of a three-dimensional point on a plane. If the fitted result... Since it's not a unit vector, it can be normalized. The vector obtained in this step... Used for subsequent quality evaluation and threshold determination.

[0164] Step S305: Calculate the plane fitting quality index, including the proportion of interior points. and / or fitting residuals .

[0165] In this step, the interior point ratio... It can be represented as:

[0166]

[0167] in, To satisfy the condition that the distance from a point to a plane is less than a threshold The number of interior points, The number of points in the candidate plane point set.

[0168] Fitting residuals This can be expressed as the root mean square (RMS) of the distance from a point to a plane:

[0169]

[0170] Where N is the number of points involved in the residual calculation. For the first The coordinates of a point. (Through) With / or It can quantify the quality of plane fitting and suppress the influence of noise, floating objects and depth holes on the fitting results.

[0171] Step S306, determine Is it greater than a preset threshold and / or Is it less than the preset threshold?

[0172] In this step, the preset threshold may include: interior point ratio threshold. With residual threshold .when and / or If the candidate plane fitting quality meets the requirements, it can be considered that the quality meets the requirements. If it does not meet the requirements, strategies such as re-extracting the candidate point set, adjusting the preprocessing parameters, expanding the time window, or outputting the "invalid plane" flag can be triggered. This application does not limit these strategies.

[0173] It should be noted that under different water qualities (clear / turbid), different reflectivity of materials, and different camera ranges, and It can be adaptively adjusted or set based on experience.

[0174] Step S307: In response to meeting the threshold condition, the candidate plane parameters are determined as the target plane parameters. And output it.

[0175] In this step, when the threshold condition is met, the candidate plane parameters are... Determined as target plane parameters And output it to subsequent online self-calibration steps (e.g. Figure 4 (S401~S408) is used. The output Π may include: unit normal. Planar offset In-plane point set index and fitting quality index , If multiple planes are fitted simultaneously (e.g., the pool bottom and pool wall), they can be output separately. and Alternatively, the optimal plane parameters can be output according to the priority of fitting quality; this application does not limit this to any particular method.

[0176] It can be seen that, through the above Figure 3 As shown in the flowchart, this embodiment can stably extract the environmental plane and output the plane parameters based on the compensated point cloud, providing reliable geometric constraints for subsequent online self-calibration based on IMU attitude and plane parameter constraints, thereby improving the accuracy and robustness of underwater near-range point cloud perception.

[0177] refer to Figure 4 , Figure 4 This illustration shows an online self-calibration process based on IMU attitude and plane parameter constraints in an embodiment of this application. This embodiment is used to implement the key step in the method of this application: "constructing constraints using IMU attitude data and environmental plane parameters to estimate and update the parameters to be estimated in the refraction model online," and can be used to... Figure 4 S401 to S408 are shown.

[0178] In this embodiment, the target plane parameters can be Figure 3 Step S307 outputs the IMU attitude data. can be Figure 1 The parameters of the refraction model are obtained in step S101. Figure 2 The composite medium refraction ray correction process shown (e.g., determining the window normal, scale factor, bias term, etc.) continuously updates the refraction model parameters online, thereby continuously improving the geometric consistency of the compensation point cloud on the environmental plane, thus offsetting the error drift caused by assembly deviations, small displacements / vibrations during operation, and changes in the medium.

[0179] Step S401: Obtain the current IMU attitude data U and target plane parameters. .

[0180] In this step, IMU attitude data It can include quaternions Euler angles, angular velocity acceleration Target plane parameters, etc. Including unit normal Offset from the plane (The equation of the plane is) ), and optionally include in-plane point set index and fit quality index. , These are used for calibration status determination and weight setting. This application does not limit this.

[0181] Step S402: Calculate the unit vector of gravity direction based on IMU attitude data. .

[0182] In this step, the gravity direction can be transformed from the world coordinate system to the camera coordinate system or the cabin coordinate system based on the IMU attitude. As an implementation method, let the gravity direction in the world coordinate system be... (Unit vector, for example only), derived from attitude quaternions Construct rotation matrix Then the direction of gravity in the camera coordinate system can be expressed as:

[0183]

[0184] And by normalizing it, we get:

[0185]

[0186] in, This represents the L2 norm. It should be noted that if the coordinate systems are defined differently (e.g., ...), the L2 norm may vary. (Axial direction upwards), the sign of the gravity direction can be adjusted accordingly; in addition, low-pass filtering can be used to smooth the IMU output to suppress high-frequency noise, but this application does not limit this.

[0187] Step S403: Construct constraint terms, including plane normal consistency constraints. and / or point-to-plane residual constraints ( .

[0188] In this step, to ensure that the point cloud after refraction compensation matches the structure of the real environment, two types of constraints can be introduced:

[0189] (1) Plane normal consistency constraint: When the target plane is the bottom plane of the pool, its normal direction should be approximately parallel (in the same direction or opposite to the direction of gravity). Therefore, a normal consistency constraint term can be constructed, for example:

[0190]

[0191] in, A sign factor (+1 or -1) is used to handle cases where the normal and gravity directions may be opposite; a dot product constraint can also be used, such as... This application does not impose any limitations on this.

[0192] (2) Point-to-plane residual constraint: For the set of points belonging to the target plane (Source: Figure 3 The cost of constructing a point-to-plane connection from the extracted in-plane points:

[0193]

[0194] in, The number of points in the plane. This is used to make the compensated point cloud fit the plane more geometrically.

[0195] Step S404: Determine the set of parameters to be estimated. Including scale factor Bias and / or observation window normal micro-deviation .

[0196] In this step, the set of parameters to be estimated Used to characterize the parts of a refractive model that require online correction as assembly, environment, or operating conditions change. It may include at least the following:

[0197] (1) Scale factor Used to correct depth scale deviation;

[0198] (2) Bias Used to correct the overall depth offset;

[0199] (3) Observation window normal slight deviation One-dimensional or two-dimensional angles can be used. , This indicates, or is equivalent to, a normal perturbation. This characterizes the tilt angle of the observation window assembly or the slight deviation in the camera's attitude relative to the window surface.

[0200] (4) Other parameters: such as the thickness correction of the observation window Refractive index correction etc. (can be included if necessary) ).

[0201] To facilitate real-time implementation, priority can be given to selecting and Composed of a small number of angular parameters This reduces the dimensionality of the online solution.

[0202] Step S405: Establish the optimization objective function The point-to-plane residual and the normal consistency term are combined.

[0203] In this step, a joint optimization objective function can be constructed to achieve online self-calibration, for example:

[0204]

[0205] in, Indicates the use of parameters The in-plane point-to-plane residual obtained after refracted ray correction. Represents the normal consistency constraint. The parameters are from the previous time step. , , These are the weighting coefficients. A regularization term is introduced. This can suppress drastic parameter jumps and improve the stability of online estimation. Weighting coefficients can be based on plane fitting quality indices (such as...). Figure 3 of , Adaptive adjustment, for example, reducing when the plane fitting quality is poor. or This application does not impose any limitations on this.

[0206] Step S406: Solve using iterative least squares and / or filtering methods. Update volume .

[0207] In this step, the following online estimation method can be used to solve the problem. :

[0208] (1) Iterative least squares: in the current Within the neighborhood of Linearization, establishing normal equations and solving them. .

[0209] (2) Extended Kalman Filter / Unscented Kalman Filter: As a state variable, and Relevant quantities are used as observations for prediction and updating;

[0210] (3) Gradient descent or quasi-Newton method: Calculation Gradient and update .

[0211] In practical engineering, to ensure real-time performance, the maximum step size for each update can be limited, and... This application does not impose any restrictions on the setting of a limiting strategy.

[0212] Step S407, update the refraction model parameters And generate an updated compensated point cloud / depth.

[0213] In this step, we obtain Then update the refraction model parameters and use the updated parameters. Figure 2 The refracted ray correction process shown regenerates the compensated point cloud. and / or compensated depth images To reduce computational overhead, fast recalculation can be performed only on the relevant planar regions or sparsely sampled pixels; alternatively, the updated parameters can be directly used for compensation in the next frame of data processing. This application does not limit this approach.

[0214] Step S408: Output the refractive model parameters and calibration status flags after online self-calibration.

[0215] In this step, the updated version can be output. (For example , , (etc.) and a calibration status flag (Flag) to indicate whether the current self-calibration is valid / converged. As an implementation, when the plane fitting quality meets a threshold and When the amplitude is less than a preset threshold, the Flag can be set to "calibration valid"; when the plane fitting quality is insufficient or the IMU attitude is unstable, the Flag can be set to "calibration paused / invalid", and the parameters from the previous moment will not be updated or will be updated with a smaller weight. This application does not impose any limitations on this.

[0216] It can be seen that, through the above Figure 4 As shown in the flowchart, this embodiment can utilize the prior gravity direction and environmental plane geometric constraints provided by the IMU to estimate and update the key parameters of the refraction model online during operation, realize the adaptive self-calibration of the depth camera under the observation window conditions of the waterproof hull, thereby continuously reducing the systematic error of underwater near-field point clouds and improving long-term stability.

[0217] refer to Figure 5 , Figure 5 This illustration shows a schematic diagram of the point cloud confidence calculation and reliable point cloud output process in an embodiment of this application. This embodiment is used to implement the key steps of the method in this application: "calculating point cloud confidence based on indicators such as depth hole rate, local geometric consistency, infrared intensity statistics, and / or distance interval, and outputting reliable point clouds and their confidence information." Figure 5 S501 to S508 are shown.

[0218] In this embodiment, the depth image is compensated. and / or compensated point cloud can be Figure 1 Step S103 or Figure 4 Step S407 obtains the infrared intensity image. can be Figure 1 The point cloud confidence level is obtained in step S101. It can provide pixel-level confidence (giving confidence for each pixel / depth point) and / or point-level confidence (giving confidence for each 3D point in the point cloud). Through confidence output, downstream tasks such as wall adhesion, docking, and close-range ranging can be guided to selectively use point cloud data, thereby improving system stability and security.

[0219] Step S501: Obtain the compensated depth image and / or compensated point cloud and infrared intensity images .

[0220] In this step, This is the depth image after refraction compensation. For the reason Point clouds generated or directly corrected from refracted rays, This is an infrared intensity image (or grayscale intensity image) synchronously output from a depth camera. In practical implementation, it can be... , and Timestamp alignment is performed to ensure that confidence calculations correspond to observation data at the same time, but this application does not impose any limitations on this.

[0221] Step S502, calculate the depth void ratio and / or depth jump rate .

[0222] In this step, the depth void ratio Used to measure the degree of missing depth data. As an implementation method, it can be implemented in the entire map or a local window. Internal calculation:

[0223]

[0224] in, Number of invalid depth pixels (e.g., depth is 1) , (or pixels beyond the range). This represents the total number of pixels within the statistical area.

[0225] The depth hop rate *r* measures the degree of depth abrupt change. One implementation method is to statistically analyze the differences between pixel neighborhoods, for example, calculating the depth gradient for 4-neighborhoods or 8-neighborhoods, and counting gradients exceeding a threshold. Pixel ratio:

[0226]

[0227] in, To meet The number of neighboring pixel pairs or the number of corresponding pixels. and It can be used to reflect depth instability caused by underwater scattering, reflection, or insufficient texture.

[0228] Step S503: Calculate the local geometric consistency index. This includes neighborhood variance, normal consistency, and / or local plane residuals.

[0229] In this step, local geometric consistency is used to measure the interpretability and continuity of the point cloud in its local space. As an implementation, at least the following metrics can be calculated and formed. :

[0230] (1) Neighborhood variance: variance of pixels or point cloud points Calculate the variance of the nearest neighbor depth / distance value The smaller the variance, the better the geometric consistency.

[0231] (2) Normal consistency: Estimating the normal vector of the point cloud Calculate the distribution of the included angle of the neighborhood normals or the average dot product value. The more consistent the normals are, the better the geometric consistency.

[0232] (3) Local plane residual: Fit a small plane in the local neighborhood and calculate the residual from the point to the plane. The smaller the residual, the more stable the local geometry.

[0233] The above metrics can be achieved at the pixel level (calculated for the neighborhood of the depth map) or at the point cloud level (calculated for the neighborhood of three-dimensional points), and this application does not limit them.

[0234] Step S504: Calculate infrared intensity statistical indicators This includes the mean, variance, and / or contrast.

[0235] In this step, the infrared intensity image It can reflect texture intensity, reflection, and attenuation, thus indirectly indicating the reliability of depth matching / ranging. As an implementation method, it can be done in a local window. Internal calculation:

[0236] (1) Mean With variance ;

[0237] (2) Contrast indicators, such as the maximum and minimum difference ( or local gradient mean;

[0238] (3) Saturation ratio or dark ratio is used to reflect strong reflection or insufficient signal.

[0239] Generally speaking, areas that are too dark, oversaturated, or have extremely low contrast are more prone to producing deep holes and noise points. It can serve as an important reference for confidence level.

[0240] Step S505: Obtain distance interval information And set weights or penalties for different distance segments.

[0241] In this step, distance interval information can be Alternatively, the depth / distance values ​​of the point cloud can be obtained. Since underwater depth cameras exhibit different noise characteristics at different distances (e.g., closer distances are more sensitive to refraction geometry errors, while farther distances are more susceptible to scattering attenuation), weights can be assigned to different distance segments. Or penalty items As a possible implementation, the distance can be divided into multiple intervals. Different weights can be set for points within the same interval; points that exceed the recommended working distance can also be assigned a lower confidence level, but this application does not limit this.

[0242] Step S506: Calculate the point cloud confidence level by integrating the above indicators. This yields pixel-level confidence and / or point-level confidence.

[0243] In this step, you can , , , , The confidence level is obtained by fusion using a weighted method. As an example implementation, the overall confidence level can be expressed as:

[0244]

[0245] in, These are the weighting coefficients; , , For the normalized geometric consistency score, intensity score, and distance score, the normalization range can be [0,1]. Under pixel-level confidence conditions, for each pixel... Calculate the corresponding ; In the case of point-level confidence, for each point Calculate the corresponding .

[0246] It should be noted that the weighting coefficients can be configured based on water quality, scenario, and downstream task requirements. For example, for tasks involving adhering to the wall, the weighting of geometric consistency can be increased. In reflective scenes, the intensity constraint weight can be increased. This application does not impose any limitations on this.

[0247] Step S507: Determine the confidence level of the point cloud. Is it below the preset threshold?

[0248] In this step, a preset threshold is set. It can be a global threshold or a threshold segmented by distance. For pixel-level confidence, it can be determined... For point-level confidence, it can be determined that... The threshold can be set according to the fault tolerance requirements of the task: for example, a stricter threshold can be set for docking and positioning tasks to avoid misleading; a relatively lenient threshold can be set for coarse ranging tasks to retain more points. This application does not limit this.

[0249] In step S508, in response to the confidence level being lower than the threshold, perform elimination / downweighting / interpolation repair processing, and output the reliable point cloud and its confidence level information.

[0250] In this step, when the confidence level is below the threshold, the following minimum processing strategy can be implemented:

[0251] (1) Removal: Mark low-confidence pixels / points as invalid and remove them from the point cloud;

[0252] (2) Weight reduction: assigning lower weights to low-confidence points in subsequent mapping, registration or control;

[0253] (3) Interpolation repair: Neighborhood interpolation, guided filtering, or local plane / surface fitting-based filling are performed on deep hole regions to generate more continuous depth results;

[0254] (4) Morphological or connected component filtering: Remove isolated noise point clusters and improve the availability of point clouds.

[0255] Then output a reliable point cloud. and its confidence information (For example, outputting a confidence array that corresponds one-to-one with the point cloud, or outputting a confidence map / heatmap). Confidential point clouds can be used for tasks such as downstream wall-mounted distance determination, docking positioning, close-range 3D measurement, or fine target geometry extraction.

[0256] Through the above Figure 5 As shown in the process, this embodiment can quantitatively evaluate the point cloud quality from multiple dimensions such as depth data integrity, geometric consistency, infrared intensity reliability and range characteristics, and perform adaptive processing under low confidence conditions, and finally output a reliable point cloud and its confidence information, thereby improving the robustness and engineering usability of underwater near-range point cloud perception in complex underwater environments.

[0257] refer to Figure 6 , Figure 6 A schematic diagram of a depth camera refraction compensation and online self-calibration device for underwater near-field point cloud sensing, according to an embodiment of this application, is shown. This device is used to perform the method embodiments of this application (e.g., Figure 1 As shown in S101~S106, Figure 2 As shown in S201~S206, Figure 3 As shown in S301 to S307, Figure 4 As shown in S401~S408 and Figure 5 The functional modules corresponding to S501 to S508 shown are implemented. In this embodiment, the device can be deployed on an embedded computing board, industrial computer, or edge computing unit of an underwater ROV / AUV, and communicate with the depth camera and IMU.

[0258] like Figure 6 As shown, the device includes at least: a data acquisition module 101, a refraction modeling module 102, a ray correction module 103, a plane estimation module 104, an online self-calibration module 105, a confidence calculation module 106, and an output module 107. The input to the device may include raw depth images. (201) Infrared intensity image (202, optional) and IMU attitude data U (203); the output of the device may include a compensated depth image. (301, optional) Compensated point cloud (302) and point cloud confidence and / or trusted point clouds (303).

[0259] This invention also discloses an underwater depth camera refraction correction and online self-calibration system, used to implement an underwater depth camera refraction correction and online self-calibration method, including:

[0260] A waterproof cabin, wherein the waterproof cabin is equipped with an observation window;

[0261] A depth camera, installed inside the waterproof chamber, is used to output raw depth images and / or infrared intensity images;

[0262] An inertial measurement unit (IMU) is used to output attitude data.

[0263] The processing unit is communicatively connected to the depth camera and the IMU, and is configured to execute the underwater depth camera refraction correction and online self-calibration method to output compensated depth images and / or compensated point clouds and their point cloud confidence information.

[0264] The processing unit includes at least one of an edge computing board, an embedded processor, and / or an industrial computer.

[0265] The following provides a further explanation of the functions of each module and their interconnections.

[0266] (1) Data acquisition module 101

[0267] The data acquisition module 101 is configured to acquire raw depth images from the depth camera. (201) and / or infrared intensity images (202), and attitude data U (203) from the IMU. In some embodiments, the data acquisition module 101 may also perform time synchronization or near-time alignment operations, such as timestamp-based interpolation alignment, to ensure that depth, infrared and IMU data correspond at the same time or within the same time window, thereby improving the accuracy and stability of subsequent refraction compensation, self-calibration and confidence calculation.

[0268] (2) Refraction Modeling Module 102

[0269] The refraction modeling module 102 is connected to the data acquisition module 101 and is configured to establish a refraction model of the air-observation window-water composite medium based on the observation window parameters of the waterproof cabin. The observation window parameters may include thickness. Material refractive index Window normal and air refractive index and water refractive index The refraction modeling module 102 outputs refraction model parameters for the ray correction module 103 to use. Furthermore, a parameter feedback connection is established between the refraction modeling module 102 and the online self-calibration module 105, so that after the online self-calibration module 105 updates the parameters to be estimated in the refraction model, the refraction modeling module 102 can refresh the refraction model in real time.

[0270] (3) Ray correction module 103

[0271] Ray correction module 103 is connected to refraction modeling module 102 and is configured to apply a refraction model to the original depth image. (201) The corresponding pixel line of sight is corrected by refraction ray to generate a compensated depth image. (301, optional) and / or compensated point cloud (302). In the implementation, the ray correction module 103 corrects the pixel... via camera intrinsic parameter matrix Convert to incident direction vector The direction of the ray inside the window was calculated based on Snell's law for two interfaces. With underwater ray direction This maps the original depth to a distance parameter along the underwater refracted ray direction and generates a compensated point cloud. The output of the ray correction module 103 can be used as input to the plane estimation module 104 and the confidence calculation module 106.

[0272] (4) Plane estimation module 104

[0273] Plane estimation module 104 is connected to ray correction module 103 and is configured to perform compensation point cloud calculations within a preset time window. (302) Perform environmental plane detection and fitting to obtain the target plane parameters. The environmental plane can be the pool bottom plane and / or pool wall plane, etc. The plane estimation module 104 can perform downsampling, denoising, region of interest clipping, candidate plane point set extraction, plane fitting, and fitting quality evaluation (such as interior point ratio). and / or residuals Operations such as ) will be performed, and the target plane parameters that meet the threshold conditions will be... The output is sent to the online self-calibration module 105.

[0274] (5) Online self-calibration module 105

[0275] The online self-calibration module 105 is connected to the plane estimation module 104, the data acquisition module 101, and the refraction modeling module 102, respectively, and is configured to utilize IMU attitude data U (203) and target plane parameters. Construct constraints for the parameters to be estimated in the refraction model. Perform online estimation and updates. Parameters to be estimated. May include scale factor Bias and / or observation window normal micro-deviation The online self-calibration module 105 can calculate the unit vector of gravity direction. Furthermore, plane normal consistency constraints and point-to-plane residual constraints are constructed, and an optimization objective function is established. The parameter update amount is solved by iterative least squares and / or filtering methods. The updated θ to refraction modeling module 102 is then output, forming a closed-loop update. Furthermore, the online self-calibration module 105 can also provide updated parameters to the ray correction module 103, enabling subsequent refraction ray corrections to take effect in real time.

[0276] (6) Confidence calculation module 106

[0277] The confidence calculation module 106 is connected to the ray correction module 103 and the data acquisition module 101, and is configured to calculate based on the compensated depth image. (301, optional) and / or compensated point cloud (302), and infrared intensity images (202, optional) Calculate the confidence level of the point cloud And generate a reliable point cloud. Confidence level calculation can comprehensively consider depth and void ratio. Depth jump rate Local geometric consistency index (Neighborhood variance, normal consistency, local plane residuals, etc.), infrared intensity statistical indicators (Mean, variance, contrast, etc.) and distance interval information The data is then fused to obtain pixel-level and / or point-level confidence scores. When the confidence score is lower than a preset threshold, the corresponding point can be removed, downweighted, or interpolated for repair, in order to output more reliable point cloud data.

[0278] (7) Output module 107

[0279] Output module 107 is connected to ray correction module 103 and confidence calculation module 106 respectively, and is configured to output compensated depth image. (301, optional) Compensated point cloud (302) and point cloud confidence and / or trusted point clouds (303). In some embodiments, the output module 107 may also output calibration status flags (e.g., calibration valid / paused / invalid) and key parameters (e.g., ... , , It is used for display and recording by the host computer or for downstream control and planning modules.

[0280] In conclusion, Figure 6 The device shown achieves refraction error compensation and online self-calibration of the underwater depth camera under observation window conditions through a closed-loop structure of "refractive modeling - ray correction - plane estimation - online self-calibration". At the same time, the confidence calculation module (106) quantifies the point cloud quality and outputs a reliable point cloud, enabling the system to use point cloud data according to confidence, thereby improving the accuracy, robustness and engineering usability of underwater near-range point cloud perception.

[0281] In the above embodiments, the implementation of the air-viewing window-water composite medium refraction model and pixel line-of-sight refraction propagation (see...) Figure 2 ), Implementation of environmental plane detection and fitting process based on compensated point cloud (see Figure 3 ), Implementation of an online self-calibration process based on IMU attitude and plane parameter constraints (see Figure 4 ), Implementation of point cloud confidence calculation and trusted point cloud output process (see Figure 5 ) and the implementation of the device structure (see Figure 6 These components are used together to implement the underwater depth camera refraction correction and online self-calibration method, apparatus and system of this application, and have the beneficial effects of the corresponding method and apparatus embodiments, including but not limited to: reducing the refraction system error caused by the observation window of the waterproof hull, realizing online self-calibration of parameters during operation, outputting point cloud confidence and generating reliable point clouds to improve the stability and reliability of downstream tasks, which will not be elaborated here.

[0282] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for refraction correction and online self-calibration of an underwater depth camera, characterized in that, include: Acquire raw depth images and / or infrared intensity images output by a depth camera installed in a waterproof enclosure, as well as attitude data output by an inertial measurement unit (IMU). A composite medium refraction model of air-observation window-water is established based on the parameters of the observation window of the waterproof cabin. Based on the refraction model, the pixel line of sight corresponding to the original depth image is corrected by refraction ray to generate an underwater compensated depth image and / or compensated point cloud. Within a preset time window, environmental plane detection and fitting are performed on the compensated point cloud to obtain plane parameters; By using the IMU attitude data and the plane parameters to construct constraints, the parameters to be estimated in the refraction model are estimated and updated online to achieve online self-calibration of the depth camera; The confidence level of the point cloud is calculated based on the depth void ratio, local geometric consistency, infrared intensity statistics and / or distance interval indicators, and the confidence level of the point cloud is output.

2. The underwater depth camera refraction correction and online self-calibration method according to claim 1, characterized in that, The step of establishing a composite medium refraction model of air-observation window-water based on the parameters of the waterproof cabin observation window includes: Acquire or set the thickness, refractive index, and normal direction parameters of the observation window; Based on the thickness, the refractive index, and the window normal direction parameters, the refraction and propagation relationship of the pixel line of sight in the three media of air-viewing window-water is constructed. The direction of the refracted ray and / or equivalent imaging geometry parameters of the pixel line of sight are determined based on the aforementioned refraction propagation relationship for subsequent refracted ray correction.

3. The underwater depth camera refraction correction and online self-calibration method according to claim 2, characterized in that, Following the establishment of the composite medium refraction model of air-observation window-water, the following is also included: Intrinsic parameter matrix based on depth camera pixel coordinates Converted to incident direction vector in camera coordinate system and the incident direction vector The incident direction vector is normalized. The expression is as follows: The angle of refraction was calculated using Snell's law at both the air-observation window interface and the observation window-water interface. The expression for Snell's law is as follows: in, Let be the refractive index of the incident medium. Let be the refractive index of the refracting medium. Angle of incidence The angle of refraction; The underwater ray direction vector is obtained from the two interface refractions. and based on the The propagation direction of the pixel's line of sight in the underwater coordinate system is determined for the purpose of the refracted ray correction.

4. The underwater depth camera refraction correction and online self-calibration method according to claim 1, characterized in that, The step of performing environmental plane detection and fitting on the compensated point cloud within a preset time window to obtain plane parameters includes: The candidate planes in the compensated point cloud are fitted and the plane fitting quality index of the candidate planes is calculated. The plane fitting quality index includes the proportion of inliers and / or the fitting residual from the points to the plane. In response to determining that the proportion of inliers is greater than a preset threshold and / or the fitting residual is less than a preset threshold, the corresponding candidate plane point set and its plane parameters are determined as the target plane parameters for the online self-calibration.

5. The underwater depth camera refraction correction and online self-calibration method according to claim 1, characterized in that, The fitting residual in the plane fitting quality index is calculated using the following formula: in, Represents the fitting residual. This represents the number of points used for plane fitting. This represents the three-dimensional coordinate vector of the i-th point. Denotes the unit normal vector of the fitted plane. The plane bias parameters represent the fitted plane, such that the plane equation is... .

6. An underwater depth camera refraction correction and online self-calibration device, used to implement the underwater depth camera refraction correction and online self-calibration method as described in any one of claims 1-5, characterized in that, include: The data acquisition module (101) is configured to acquire raw depth images and / or infrared intensity images output by a depth camera installed in the waterproof chamber, as well as attitude data output by an inertial measurement unit (IMU). The refraction modeling module (102) is configured to establish a composite medium refraction model of air-observation window-water based on the parameters of the observation window of the waterproof cabin; The ray correction module (103) is configured to perform refracted ray correction on the pixel line of sight corresponding to the original depth image based on the refraction model, and generate an underwater compensated depth image and / or compensated point cloud. The plane estimation module (104) is configured to perform environmental plane detection and fitting on the compensated point cloud within a preset time window to obtain plane parameters; The online self-calibration module (105) is configured to construct constraints using the IMU attitude data and the plane parameters, and to estimate and update the parameters to be estimated in the refraction model online, so as to realize the online self-calibration of the depth camera. The confidence calculation module (106) is configured to calculate the confidence of the point cloud based on the depth void ratio, local geometric consistency, infrared intensity statistics and / or distance interval indicators, and output the confidence point cloud and its confidence information. The output module (107) is configured to output the compensated depth image and / or the compensated point cloud and the corresponding point cloud confidence information.

7. The underwater depth camera refraction correction and online self-calibration device according to claim 6, characterized in that, The confidence calculation module (106) is based on the depth void ratio, the local geometric consistency, the infrared intensity statistics, and the distance interval index, which are calculated based on relevant data corresponding to the original depth image and / or infrared intensity image; the relevant data includes at least the following: Depth void ratio, depth jump rate, local geometric consistency index, infrared intensity mean, infrared intensity variance, image contrast index and / or distance interval information.

8. An underwater depth camera refraction correction and online self-calibration system, used to implement the underwater depth camera refraction correction and online self-calibration method as described in any one of claims 1-5, characterized in that, include: A waterproof cabin, wherein the waterproof cabin is equipped with an observation window; A depth camera, installed inside the waterproof chamber, is used to output raw depth images and / or infrared intensity images; An inertial measurement unit (IMU) is used to output attitude data. The processing unit is communicatively connected to the depth camera and the IMU, and is configured to execute the underwater depth camera refraction correction and online self-calibration method to output compensated depth images and / or compensated point clouds and their point cloud confidence information.

9. The underwater depth camera refraction correction and online self-calibration system according to claim 8, characterized in that, The processing unit includes at least one of an edge computing board, an embedded processor, and / or an industrial computer.

Images