A method, device and storage medium for underwater three-dimensional reconstruction

Abstract

Embodiments of the present application provide a method and device for underwater three-dimensional reconstruction and a storage medium, and relate to the field of computer vision and image processing. The method comprises: performing optical quality evaluation on optical images to obtain optical quality scores and degradation flags; performing preset rule calculation on the optical quality scores and processing modes to obtain frame confidence of an input frame; performing filtering on the input frame according to a pose matrix to obtain a first reserved frame, determining a gating score according to the union of the sonar coverage index of the first reserved frame in an index window and the window; performing filtering on the first reserved frame according to the gating score and the conflict degree to obtain a second reserved frame; obtaining an observation increment of a voxel corresponding to a sonar image of the second reserved frame, obtaining an observation count according to the observation increment and the frame confidence, and determining a voxel occupancy state according to the observation count and an occupancy count, which is used to determine a three-dimensional reconstruction result of a target scene. The method improves the accuracy and stability of underwater three-dimensional reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing, and more specifically to a method, apparatus, and storage medium for underwater 3D reconstruction. Background Technology

[0002] Underwater robots are widely used in underwater intervention operations such as pipeline inspection, hydraulic infrastructure maintenance, and underwater scientific exploration. Real-time 3D environmental perception is the core technological foundation for robots to achieve autonomous obstacle avoidance, operational surveying, and environmental modeling. Due to the interference of complex underwater environments such as turbidity, uneven lighting, and water scattering, a single sensor is difficult to balance 3D reconstruction accuracy and environmental adaptability. Therefore, a multimodal 3D reconstruction architecture that integrates sonar and optics has become the mainstream research approach for underwater perception technology.

[0003] Currently, existing underwater multimodal 3D reconstruction technologies often combine the sensing advantages of imaging sonar and optical cameras. These solutions typically rely on 2D imaging sonar combined with voxel sculpting algorithms to construct a 3D geometric skeleton of the underwater environment, while simultaneously using optical cameras to acquire texture information to compensate for the lack of texture in sonar imaging. To adapt to dynamic underwater environments, this system sets up three processing modes: pure sonar, hybrid, and optical enhancement. Mode switching is completed based on optical quality scores, and effective acquisition frames are selected through pose thresholding to balance reconstruction accuracy and computational cost.

[0004] However, this type of sonar-optical fusion reconstruction architecture still has many technical shortcomings, which limit the reconstruction effect of complex underwater scenes. First, the voxel update adopts an equal-weight accumulation mechanism, which does not distinguish the differences in the quality of the acquired frames. Regardless of the image sharpness and contrast of the frame, the existing system uses a fixed increment to update the voxel observation count and occupancy count. Low-quality frames such as blurry and low-contrast frames have the same weight as high-quality frames, which can easily generate false occupancy clusters and geometric glitch in the reconstruction model, reducing the accuracy of the 3D model.

[0005] Secondly, the mode switching logic has poor robustness and is prone to frequent switching issues. Current technology relies on fixed thresholds to directly determine the operating mode. When the optical quality score fluctuates within the threshold range, the system repeatedly switches between adjacent operating modes, leading to frequent changes in the voxel update strategy and additional computational overhead. Furthermore, when water visibility continues to decrease, the system lacks an environmental degradation adaptation mechanism and may still rely on low-quality optical data to perform enhanced updates, resulting in insufficient adaptability.

[0006] Third, the frame selection mechanism is simplistic and does not incorporate voxel space historical consistency. Existing technologies rely solely on pose translation and rotation to determine whether to process the current frame, failing to distinguish between novel and conflicting geometric information within a frame. Conflicting frames with structural overlap and observational contradictions will update voxels with equal weight, interfering with the already constructed geometric structure and causing model distortion.

[0007] In summary, current underwater sonar-optical fusion 3D reconstruction technology suffers from problems such as unreasonable frame weight allocation, unstable mode switching, and one-sided frame selection logic, making it difficult to meet the high-precision and high-stability underwater operation perception requirements. Summary of the Invention

[0008] The purpose of this invention is to provide a method, apparatus, and storage medium for underwater three-dimensional reconstruction, which improves the accuracy and stability of underwater three-dimensional reconstruction.

[0009] To achieve the above objectives, embodiments of the present invention provide a method for underwater 3D reconstruction. The method includes: acquiring an input frame of a target scene, including a sonar image, a pose matrix, and an optical image; performing optical quality assessment on the optical image to obtain an optical quality score and a degradation flag; setting a state machine to determine the processing mode of the input frame based on the optical quality score and the degradation flag; calculating the frame confidence of the input frame using preset rules based on the optical quality score and the processing mode; filtering the input frame according to the pose matrix to obtain a first retained frame that meets a pose increment threshold; setting an index window based on the input frame; determining the conflict degree and novelty of the first retained frame based on the union of the sonar coverage index of the first retained frame and the window, for determining a gating score; filtering the first retained frame based on the gating score and the conflict degree to obtain a second retained frame; acquiring the observation increment of the voxel corresponding to the sonar image of the second retained frame; obtaining an observation count based on the observation increment and the frame confidence; determining the voxel occupancy state based on the observation count and the occupancy count, for determining the 3D reconstruction result of the target scene.

[0010] Optionally, the step of performing optical quality assessment on the optical image to obtain an optical quality score and a degradation flag includes: converting the optical image into a grayscale image, the grayscale image including brightness, contrast, and sharpness; performing normalized weighted summation on the three statistical measures of brightness, contrast, and sharpness to obtain an optical quality score; setting the degradation flag to true when at least two of the three statistical measures of brightness, contrast, and sharpness are below a threshold, otherwise setting it to false; when the optical image is missing, the optical quality score is set to zero, and the degradation flag is set to true.

[0011] Optionally, the state machine setting, used to determine the processing mode of the input frame based on the optical quality score and degradation flag, includes: the state machine having an asymmetric hysteresis threshold; the states of the state machine including pure sonar mode, hybrid mode, and optical mode; the asymmetric hysteresis threshold including a hybrid mode entry threshold, a hybrid mode exit threshold, an optical mode entry threshold, and an optical mode exit threshold; when the state machine is in pure sonar mode, if the optical quality score is not less than the optical mode entry threshold, the next state is optical mode; if the optical quality score is not less than the hybrid mode entry threshold, the next state is hybrid mode; otherwise, the next state remains pure sonar mode. When the state machine is in hybrid mode, if the optical quality score is less than the hybrid mode exit threshold, the next state is pure sonar mode; if the optical quality score is not less than the optical mode entry threshold, the next state is optical mode; otherwise, the next state remains hybrid mode. When the state machine is in optical mode, if the optical quality score is less than the hybrid mode exit threshold, the next state is pure sonar mode; if the optical quality score is less than the optical mode exit threshold, the next state is hybrid mode; otherwise, the next state remains optical mode. When the degradation sliding window is full and the degradation ratio is not less than the degradation trigger ratio threshold, the next state is set to pure sonar mode.

[0012] Optionally, the state machine also includes a minimum dwell time constraint: when the candidate next state calculated based on the asymmetric hysteresis threshold is different from the current state, but the difference between the current frame index and the previous mode switching frame index is less than the minimum dwell time frame, the current state remains unchanged and no switching is performed; otherwise, a switching is performed and the switching event is recorded; if the switching direction is opposite to the previous switching direction and the interval between the two switching frames does not exceed the preset number of frames, then the current switching is marked as a jitter event and the jitter count is accumulated.

[0013] Optionally, the step of calculating the frame confidence of the input frame by applying preset rules to the optical quality score and processing mode includes: cropping the optical quality score by upper and lower bounds, and then weighting it according to the processing mode to obtain the frame confidence of the input frame.

[0014] Optionally, the step of setting an index window based on the input frames, and determining the conflict degree and novelty of the first retained frame based on the union of the sonar coverage index of the first retained frame and the window, for determining the gating score, includes: taking the union of the coverage index sets of all input frames in the index window to obtain the historical coverage set; and calculating the conflict degree by the ratio of the overlap between the coverage index of the first retained frame and the historical coverage set to the total coverage of the current frame. Novelty for: The gating score is determined based on the degree of conflict and novelty. ,in, For novel gain coefficients, This represents the conflict penalty coefficient.

[0015] Optionally, the step of filtering the first retained frame to obtain the second retained frame based on the gating score and conflict degree includes: determining the processing action for the first retained frame based on the gating score and conflict degree; the processing action includes full-weight processing, deweighting processing, and skipping, wherein the deweighting processing is to conditionally reduce the confidence of the frame; if the number of consecutively skipped frames reaches a preset skipping limit, the current processing action is changed to deweighting processing; when the processing action for the first retained frame is full-weight processing or deweighting processing, the first retained frame is the second retained frame.

[0016] Optionally, the observation count is the cumulative number of voxel observation increments corresponding to the sonar image multiplied by the frame confidence; the occupancy count is the cumulative number of the same voxel observation increment multiplied by the frame confidence when the effective projection of the voxel on the sonar image is a strong echo.

[0017] Optionally, the step of determining the voxel occupancy state based on the observation count and occupancy count to determine the 3D reconstruction result of the target scene includes: obtaining a first ratio of the occupancy count to the observation count; comparing the first ratio with an occupancy rate threshold to obtain the voxel occupancy state; and determining the 3D reconstruction result of the target scene based on the occupancy state.

[0018] On the other hand, this application also proposes an underwater 3D reconstruction device, which includes: an acquisition module for acquiring input frames of a target scene, including sonar images, pose matrices, and optical images; a first processing module for performing optical quality assessments on the optical images to obtain optical quality scores and degradation flags; a second processing module for setting a state machine to determine the processing mode of the input frames based on the optical quality scores and degradation flags; a third processing module for calculating the frame confidence of the input frames using preset rules based on the optical quality scores and processing modes; and a fourth processing module for filtering the input frames according to the pose matrix. The first reserved frame that meets the pose increment threshold is selected. An index window is set according to the input frame. The conflict degree and novelty of the first reserved frame are determined according to the union of the sonar coverage index of the first reserved frame and the window, which is used to determine the gating score. The fifth processing module is used to filter the first reserved frame according to the gating score and conflict degree to obtain the second reserved frame. The sixth processing module is used to obtain the observation increment of the voxel corresponding to the sonar image of the second reserved frame. The observation count is obtained according to the observation increment and the frame confidence. The voxel occupancy state is determined according to the observation count and the occupancy count, which is used to determine the three-dimensional reconstruction result of the target scene.

[0019] On the other hand, this application also proposes a machine-readable storage medium storing instructions for causing a machine to perform the underwater three-dimensional reconstruction method described above.

[0020] Through the above technical solution, this application uses frame confidence as a unified mediator for judging optical quality, processing mode, and spatial consistency, and converges the outputs of the three modules—sonar image, pose matrix, and optical image—at the same weighted voxel update output. This achieves three levels of improvement within the same frame processing flow: first, it enables the 3D voxel reconstruction to suppress the cumulative contribution of low-quality frames when there is inter-frame quality heterogeneity, alleviating the problem of false occupancy introduced by low-quality observations; second, it enables the processing mode output to have temporal stability near the critical quality, avoiding mode jitter from being transmitted as fluctuations in the voxel update strategy; and third, it enables voxel accumulation in repeated scanning scenarios to have spatial redundancy awareness, avoiding repeated weighting of already stable regions.

[0021] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a method for underwater three-dimensional reconstruction according to this application; Figure 2 This is a schematic diagram of one embodiment of this application; Figure 3 This is the state transition diagram of the degenerate state machine in this application; Figure 4 This is the keyframe consistency gating logic diagram of this application; Figure 5 This is a schematic diagram of an underwater three-dimensional reconstruction device according to this application.

[0023] Explanation of reference numerals in the attached figures 100 - Underwater 3D reconstruction device; 200 - Acquisition module; 300 - First processing module; 400 - Second processing module; 500 - Third processing module; 600 - Fourth processing module; 700 - Fifth processing module; 800 - Sixth processing module. Detailed Implementation

[0024] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0025] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0026] This invention provides a method for underwater three-dimensional reconstruction, such as... Figure 1 and Figure 2 As shown, the method includes: Step S101: Obtain the input frame of the target scene, including sonar image, pose matrix, and optical image; Step S102: Perform optical quality assessment on the optical image to obtain an optical quality score and a degradation flag bit; Step S103: Set up a state machine to determine the processing mode of the input frame based on the optical quality score and degradation flag. Step S104: Calculate the frame confidence of the input frame using preset rules for the optical quality scoring and processing mode; Step S105: Filter the input frames according to the pose matrix to obtain the first retained frame that meets the pose increment threshold. Set an index window according to the input frame. Determine the conflict degree and novelty of the first retained frame according to the union of the sonar coverage index of the first retained frame and the window, which is used to determine the gating score. Step S106: Filter the first retained frame according to the gating score and conflict degree to obtain the second retained frame; Step S107: Obtain the observation increment of the voxel corresponding to the sonar image of the second retained frame, obtain the observation count based on the observation increment and the frame confidence, and determine the voxel occupancy state based on the observation count and the occupancy count, which is used to determine the three-dimensional reconstruction result of the target scene.

[0027] The target scenario described above is an underwater scene. The input frames include sonar images, pose matrices, and optical images acquired from the underwater scene. Specifically, an underwater vehicle or shipborne, while moving and scanning, scans the water topography and object contours to generate underwater acoustic imaging, i.e., sonar images. The pose matrix is ​​obtained by directly calculating the user's own position (x, y, z) and attitude using an IMU (Inertial Measurement Unit), GPS, LiDAR, or visual SLAM. Optical images are acquired through real-time capture or video frame extraction using visible light cameras, industrial cameras, binocular cameras, or webcams.

[0028] The aforementioned optical quality assessment is performed by normalizing and weighting the sum of three statistical measures of the optical image: brightness, contrast, and sharpness. Brightness is calculated by dividing the grayscale mean by 255; contrast is calculated by dividing the grayscale standard deviation by 64 (normalized to the 0–1 range); and sharpness is calculated by dividing the variance of the Laplacian operator response by 1000. The optical quality score ranges from 0 to 1, with higher values ​​indicating higher image quality. The aforementioned degradation flag is a degradation flag set for the input frame based on the brightness, contrast, and sharpness of the optical image. For example, after converting the optical image to grayscale, the three normalized statistical measures (brightness, contrast, and sharpness) are calculated, cropped to [0, 1], and then weighted and summed with weights of 0.35 (brightness), 0.30 (contrast), and 0.35 (sharpness) to obtain a comprehensive optical quality score (hereinafter referred to as s), ranging from 0 to 1, with higher values ​​indicating better image quality. If two or more of the following three criteria are met simultaneously: brightness below 0.20, contrast below 0.15, and sharpness below 0.08, the current frame is determined to be a globally degraded frame (global_degraded=True); otherwise, global_degraded=False. The degradation flag for this frame is written into a sliding window of fixed length W for subsequent degradation trend statistics.

[0029] The aforementioned state machine has an asymmetric hysteresis threshold, and its states include pure sonar mode, hybrid mode, and optical mode. Pure sonar mode uses only 2D imaging sonar for 3D voxel reconstruction, without utilizing the texture and detail enhancement from the optical camera; this mode is used when underwater turbidity makes camera visibility poor. Hybrid mode involves constructing a geometric skeleton while the optical camera moderately enhances details; both sonar and optics participate in voxel updates, serving as an intermediate transitional mode. Optical mode is used when optical image quality is good and water visibility is high; based on the sonar skeleton, the optical camera is primarily used to enhance texture and fine geometric details.

[0030] The above processing mode determines the next state of the input frame based on the optical quality score and degradation flag. The above preset rule calculation includes applying upper and lower bound pruning to the optical quality score, for example, limiting the score to the range of [0.45, 0.90]. The lower bound of 0.45 is used to ensure that the sonar-dominated frame still has the lowest update weight when the optical quality is extremely low; the upper bound of 0.90 is used to prevent the influence of the optical frame from being over-amplified when the visual score is abnormally high. The physical meaning of the above frame confidence can be: after upper and lower bound pruning and mode weighting, the frame confidence of all input frames falls within the range of approximately [0.38, 0.90] and is not greater than 1.0; the higher the value, the higher the observation quality of the frame and the greater the cumulative contribution to voxel statistics; the lower the value, the more suppressed the low-quality frame is relative to the high-quality frame. The global range [0.35, 1.25] is a protective pruning boundary for the allowed values ​​of the frame confidence, used to prevent the confidence from exceeding the boundary under abnormal parameter configurations, and does not require every frame to reach this upper bound.

[0031] The above-described filtering of the input frames based on the pose matrix constitutes setting a basic pose frame selection gate for the input frames, which is the first layer of frame filtering. For example, if there is no previous retained frame, the first frame passes unconditionally and proceeds to subsequent steps. When there is a previous retained frame (the previous first retained frame), a three-dimensional translation vector is extracted from the pose matrix, and the Euclidean translation distance between the current input frame and the previous retained frame is calculated. When this Euclidean translation distance is not lower than the minimum pose spacing threshold, the current input frame passes directly. If the Euclidean translation distance is lower than the minimum pose spacing threshold, further judgment can be made based on the extended selection strategy. The extended selection strategy includes a rotation strategy and a coverage novelty strategy. For the rotation strategy, the relative rotation angle is extracted based on the product of the rotation matrices of the two frame poses, and the current input frame is allowed when the rotation angle is not lower than the minimum rotation threshold; for the coverage novelty strategy, the non-overlap ratio between the voxel set covered by the sonar field of view of the current input frame and the voxel set covered by the previous retained frame is calculated, and the current input frame is allowed when the ratio is not lower than the overlap threshold. If none of the above conditions are met, skip this input frame.

[0032] The above conditions are met when the pose increment threshold is not less than the Euclidean translation distance and the minimum pose spacing threshold, or when the non-overlap ratio between the voxel set covered by the sonar field of view of the current input frame and the voxel set covered by the previous retained frame is not less than the overlap threshold.

[0033] The index window mentioned above includes the input frame sequence, such as (F=[f0, f1, f2, ..., f{N-1}]), where N is the total number of frames, window size is W (the number of frames contained in a single window), sliding step size is S (the number of frames the window moves each time), and the current input frame index is idx (the number of the current processed frame).

[0034] The method for obtaining the sonar coverage index is as follows: During system initialization, the voxel template is pre-calculated only once. That is, in the sonar coordinate system, according to the voxel grid resolution, all three-dimensional grid points are enumerated within the horizontal field of view, vertical field of view, and effective range of the sonar to obtain a fixed set of voxels that can be theoretically covered by a single sonar observation (i.e., the voxel template). When processing each frame (the first retained frame), the sonar pose matrix of that frame is applied to the voxel template to transform the template grid points from the sonar coordinate system to the world coordinate system. Then, each point is discretized (i.e., rounded down) using the grid origin and voxel size. After filtering out points that exceed the grid boundary, the remaining effective three-dimensional grid coordinates are converted into a one-dimensional flat index and deduplicated to obtain the sonar coverage index set for that frame (denoted as the coverage index).

[0035] The index window described above is a first-in-first-out keyframe window with a capacity of K, where each keyframe... Store its sonar coverage index set The union of the sonar coverage index of the first retained frame and the window is used to obtain the total historical coverage set by taking the union of the coverage index sets of all historical keyframes within the window. .

[0036] For example, suppose the window size K=3, and the current window already contains three historical keyframes, whose sonar coverage index sets are respectively... ={101, 102, 103, 104}, ={103, 104, 105, 106} If the sum of the given values ​​is {105, 106, 107, 108}, then the union of the windows (the total set of history overlays) is the sum of the given values. ={101, 102, 103, 104, 105, 106, 107, 108}, a total of 8 voxel indices. Let the sonar coverage index set of the current first reserved frame also be defined. ={106, 107, 108, 109, 110}, a total of 5 voxel indices. This current frame coverage index set is used to find the intersection with the historical coverage set U to calculate the conflict degree and novelty as described below.

[0037] The method for obtaining the aforementioned conflict level is as follows: calculate the coverage index set of the current first reserved frame. The number of overlaps (intersections) with the historical coverage set U, as a percentage of the total number of coverage indices in the current frame, i.e., the conflict level. ,in This indicates the number of elements in the set. The conflict level characterizes the degree of spatial overlap between the current frame's observation area and the historically observed areas within the window. The value ranges from [0, 1]. A larger value indicates more overlap between the current frame and historical observations, and more severe redundancy and conflict.

[0038] The method for obtaining the above novelty is as follows: Novelty represents the proportion of newly added (uncovered) areas in the current frame relative to historical observations, with a value ranging from [0, 1]. A larger value indicates more new information observed in the current frame. Specifically, if the window is empty (system cold start, no historical keyframes yet), the current frame is unconditionally considered completely novel and directly returns to full-weight processing, with a gating score of 1.0; if the current frame coverage index set is empty (the sonar field of view is completely outside the grid), the current frame is skipped directly.

[0039] For example, the current frame ={106, 107, 108, 109, 110} =5, the total historical coverage set U={101, ...,108}, and the intersection of the two. ={106, 107, 108}, with an overlap of 3, then the conflict degree is... Novelty Further substituting into the entry-level scoring formula (taking the recommended value) =1.0, =1.4) Since g = -0.44 is less than the weighting threshold, =0.05, so the gating action for this frame is skipped, meaning that the high-conflict frame does not participate in voxel updates, thus avoiding its interference with the established stable geometry.

[0040] Set the current frame If the set {106, 107, 108, 109, 110} remains unchanged, but the total historical coverage set is {101, 102, 103, 104}, then the intersection of the two sets is empty, the overlap is 0, and the conflict degree is zero. Novelty Gating score Since g=1.0 is not less than the full-weight threshold =0.25, and the conflict degree 0 does not exceed the maximum permissible conflict ratio. =0.35, therefore the gating action is processed with full weight, and the current frame participates in voxel update with full confidence.

[0041] The above gating score: ,in, For novelty, For the degree of conflict, This is the novelty gain factor, with a recommended value of 1.0; The collision penalty coefficient is recommended to be 1.4. Since the collision penalty coefficient is greater than the novelty gain coefficient, high-collision frames, even if they bring a small amount of novel information, will be preferentially downweighted or skipped.

[0042] The observation count of the voxels corresponding to the sonar image is the cumulative number obtained by multiplying the voxel observation increment of the sonar image by the frame confidence. The occupancy count of the voxels corresponding to the sonar image is the number of voxels whose effective projection on the sonar image is a strong echo.

[0043] The voxel occupancy status mentioned above is obtained by comparing the first ratio of the occupancy count to the observation count with the occupancy rate threshold. A voxel is the smallest cube in 3D space, with length × width × height, and inherent volume and position. The observation count is the cumulative number of voxel observation increments corresponding to the sonar image multiplied by the frame confidence level; the occupancy count is the cumulative number of observation increments of the same voxel multiplied by the frame confidence level when the effective projection of the voxel on the sonar image is a strong echo. Specifically, in each frame of the sonar image, the observation increment value obtained by a single voxel is multiplied by the confidence level of that frame, and the results of all frames are continuously added together, and the final sum is the observation count. Whenever a voxel projects a high-intensity echo signal in the sonar image, it is counted once, and the accumulated total is the occupancy count. Observation count = voxel single-frame observation increment × frame confidence level, accumulated frame by frame; Occupancy count = the number of times a voxel appears with a strong echo projection.

[0044] The specific method for determining the 3D reconstruction result of the target scene based on the voxel occupancy status is as follows: For each voxel, calculate its occupancy rate (i.e., the first ratio of the occupancy count occ_count to the observation count obs_count). The occupancy rate is compared with an occupancy threshold, occ_ratio_thresh. When the occupancy rate is greater than the threshold, the voxel is considered occupied and added to the 3D voxel mesh. Its color depth is encoded according to the occupancy rate (e.g., color values ​​change linearly with the occupancy rate, with higher occupancy rates resulting in darker colors). When the occupancy rate is not greater than the threshold, the voxel is considered unoccupied and removed from the mesh. Two independent configurations are used for the occupancy threshold: when weighted updates are disabled, the threshold is indirectly determined by the sonar false negative rate parameter false_negative_rate (occ_ratio_thresh = 1 - false_negative_rate); when weighted updates are enabled, an independent calibration parameter is used, with a recommended value of 0.42.

[0045] After traversing the entire voxel mesh, an occupation grid is formed by all occupied voxels, which is a Boolean 3D array where each element being true indicates that the corresponding voxel is occupied. Based on this occupation grid, isosurface extraction (e.g., the Marching Cubes algorithm) is performed to convert the discrete occupied voxels into a continuous triangular mesh surface: first, mesh vertices and triangular faces are generated from the occupation grid; then, the vertex coordinates are multiplied by the voxel size and the mesh origin is added, thus mapping the mesh index back to the true physical scale of the world coordinate system, exporting a 3D mesh model (e.g., a .obj format file). Further smoothing processing can be performed on the occupation grid or the resulting mesh to generate a smoother mesh model with a more refined surface. The resulting 3D mesh model is the 3D reconstruction result of the target scene, which can be used for autonomous obstacle avoidance, operational surveying, and environmental modeling of underwater robots.

[0046] Through the above technical solution, this application uses frame confidence as a unified mediator for judging optical quality, processing mode, and spatial consistency, and converges the outputs of the three modules—sonar image, pose matrix, and optical image—at the same weighted voxel update output. This achieves three levels of improvement within the same frame processing flow: first, it enables the 3D voxel reconstruction to suppress the cumulative contribution of low-quality frames when there is inter-frame quality heterogeneity, alleviating the problem of false occupancy introduced by low-quality observations; second, it enables the processing mode output to have temporal stability near the critical quality, avoiding mode jitter from being transmitted as fluctuations in the voxel update strategy; and third, it enables voxel accumulation in repeated scanning scenarios to have spatial redundancy awareness, avoiding repeated weighting of already stable regions.

[0047] In one embodiment, the step of performing optical quality assessment on the optical image to obtain an optical quality score and a degradation flag includes: converting the optical image into a grayscale image, the grayscale image including brightness, contrast, and sharpness; performing normalized weighted summation on the three statistical measures of brightness, contrast, and sharpness to obtain an optical quality score; setting the degradation flag to true when at least two of the three statistical measures of brightness, contrast, and sharpness are below a threshold, otherwise setting it to false; when the optical image is missing, the optical quality score is set to zero, and the degradation flag is set to true.

[0048] For example, the optical image is converted to grayscale, and three normalized statistics—brightness, contrast, and sharpness—are calculated. Brightness is defined as the grayscale mean divided by 255; contrast is defined as the grayscale standard deviation divided by 64, normalized to the 0–1 interval; and sharpness is defined as the Laplacian operator response variance divided by 1000. After cropping the three statistics to the interval [0, 1], they are weighted and summed with weights of 0.35, 0.30, and 0.35 respectively to obtain the overall optical quality score, which ranges from 0 to 1, with higher values ​​indicating higher image quality. Degradation status is determined as follows: if two or more of the following conditions are met simultaneously—brightness below 0.20, contrast below 0.15, and sharpness below 0.08—the current frame is considered a globally degraded frame (global_degraded=True); otherwise, it is marked as False. The degradation flag for this frame is written into a sliding window of fixed length W for degradation trend statistics. If the optical image is missing, all three statistics mentioned above are set to zero, and the global degraded frame is set to global_degraded=True, score=0, and the optical usability flag is set to usable=False.

[0049] In one embodiment, the state machine is configured to determine the processing mode of the input frame based on the optical quality score and degradation flag, such as... Figure 3As shown, the method includes: the state machine has an asymmetric hysteresis threshold, and the states of the state machine include pure sonar mode, hybrid mode, and optical mode; the asymmetric hysteresis threshold includes a hybrid mode entry threshold, a hybrid mode exit threshold, an optical mode entry threshold, and an optical mode exit threshold; when the state machine is in pure sonar mode, if the optical quality score is not less than the optical mode entry threshold, the next state is optical mode; if the optical quality score is not less than the hybrid mode entry threshold, the next state is hybrid mode; otherwise, the next state remains pure sonar mode; when the state machine is in hybrid mode, if the... If the optical quality score is less than the hybrid mode exit threshold, the next state is pure sonar mode; if the optical quality score is not less than the optical mode entry threshold, the next state is optical mode; otherwise, the next state remains hybrid mode. When the state machine is in optical mode, if the optical quality score is less than the hybrid mode exit threshold, the next state is pure sonar mode; if the optical quality score is less than the optical mode exit threshold, the next state is hybrid mode; otherwise, the next state remains optical mode. When the degradation sliding window is full and the degradation ratio is not less than the degradation trigger ratio threshold, the next state is set to pure sonar mode. The state machine also has a minimum dwell time constraint: when the candidate next state calculated based on the asymmetric hysteresis threshold is different from the current state, but the difference between the current frame index and the previous mode switching frame index is less than the minimum dwell time frame number, the current state remains unchanged and no switching is performed; otherwise, a switching is performed and the switching event is recorded; if the current switching direction is opposite to the previous switching direction and the interval between the two switching frames does not exceed a preset number of frames, the current switching is marked as a jitter event and the jitter count is accumulated.

[0050] Specifically, this state machine is a discrete state machine, and its state set is as follows: The current state is denoted as .in, This is the pure sonar mode (system default safe state), which uses only 2D imaging sonar to perform 3D voxel reconstruction and does not enable texture and detail supplementation from the optical camera; use this mode when the underwater environment is turbid and the camera cannot see clearly. For hybrid fusion mode sonar (sonar and optics are processed in a hybrid manner), a geometric skeleton is constructed and an optical camera is used to supplement details. Sonar and optics participate in voxel updates together, which is an intermediate transition mode. This is an optical enhancement mode, providing high-quality optical images and excellent water visibility. Based on the sonar skeleton, it focuses on enhancing texture and fine geometric details using an optical camera. This method performs rapid degradation in optically unavailable scenarios. As shown in Table 1, when the optical availability flag is False (optical image missing), the system directly returns... This rule skips all subsequent state machine logic and takes precedence over the hysteresis threshold and minimum dwell time constraint to ensure the system can immediately revert to safe mode when camera signals are missing or completely unavailable. The percentage of globally degraded frames (global_degraded=True) within a degraded sliding window of length W is denoted as the degradation ratio. When forced downgrade configuration is enabled, and the following conditions are met... At that time, the system directly sets the candidate mode to And mark this event as a forced degradation, parameter The threshold is configurable, ranging from (0, 1], with a recommended value of 0.7. Candidate next states are calculated based on an asymmetric hysteresis threshold. The rules for calculating candidate states under different current states are shown in the table. Among them, the entry threshold... Greater than the exit threshold ,For example ,in, for The entry threshold below, for The exit threshold below, for The exit threshold below.

[0051] This creates a hysteresis range near the threshold, suppressing pattern oscillations. When And the current frame index Compared to the last switched frame index The difference satisfies hour( (Number of dwell periods), the system maintains its current state. If the current mode remains unchanged, no switch is performed; otherwise, a mode switch event is recorded. This switch event includes at least the state before the switch, the state after the switch, the current optical quality score, and whether it is a forced degradation. Furthermore, if the current switch direction is opposite to the previous switch direction, and the frame interval between the two switches does not exceed two frames, then this switch is marked as a jitter event and the jitter counter is accumulated. After recording is complete, update... .

[0052] Table 1:

[0053] This method can form a stable buffer when the optical quality score is in the critical range of the threshold, suppress the back-and-forth oscillation of the mode between adjacent frames, and reliably fall back to the pure sonar mode in the continuous degradation environment, thereby reducing the number of mode switching times per unit time.

[0054] In one embodiment, the step of calculating the frame confidence of the input frame by applying preset rules to the optical quality score and processing mode includes: cropping the optical quality score by upper and lower bounds, and then weighting it according to the processing mode to obtain the frame confidence of the input frame. The asymmetric hysteresis threshold includes a hybrid mode entry threshold, a hybrid mode exit threshold, an optical mode entry threshold, and an optical mode exit threshold, wherein the hybrid mode entry threshold is greater than the hybrid mode exit threshold, and the optical mode entry threshold is greater than the optical mode exit threshold; the difference between the entry threshold and the corresponding exit threshold constitutes the hysteresis interval.

[0055] Specifically, the optical quality score is subject to upper and lower bound pruning, limiting the score to the range of [0.45, 0.90]. The lower bound of 0.45 ensures that sonar-dominated frames still have the lowest update weight even when optical quality is extremely low; the upper bound of 0.90 prevents excessive amplification of the optical frame's influence when the visual score is abnormally high. The current mode is multiplied by a mode weight, and the result is then globally pruned. The weights for the three modes are: SONAR_ONLY = 1.0, HYBRID = 0.9, and OPTICAL_ENHANCED = 0.85. The OPTICAL_ENHANCED mode weight is intentionally set below 1.0 to maintain sonar robustness as the primary geometric backbone and prevent excessive voxel updates caused by visual misjudgments. The mode-weighted result is then pruned to the global range of [0.35, 1.25] to obtain the final frame confidence score.

[0056] For example, the frame confidence scores mentioned above are as follows: after upper and lower bound pruning and pattern weighting, the frame confidence scores of all input frames fall within the range of approximately [0.38, 0.90] and are all no greater than 1.0; the higher the value, the higher the observation quality of the frame and the greater its cumulative contribution to voxel statistics, while the lower the value, the more suppressed the low-quality frames are relative to the high-quality frames. The global range [0.35, 1.25] is a protective pruning boundary for the allowed values ​​of frame confidence scores, used to prevent confidence scores from exceeding the boundary under abnormal parameter configurations, and it is not required that every frame reaches this upper bound.

[0057] In one embodiment, setting an index window based on the input frames and determining the conflict degree and novelty of the first retained frame based on the union of the sonar coverage index of the first retained frame and the window for determining the gating score includes: taking the union of the coverage index sets of all input frames in the index window to obtain the historical coverage set; and calculating the conflict degree by the proportion of the overlap between the coverage index of the first retained frame and the historical coverage set to the total coverage of the current frame. Novelty for: The gating score is determined based on the degree of conflict and novelty. ,in, For novel gain coefficients, This represents the conflict penalty coefficient.

[0058] Specifically, set a capacity of First-in-first-out keyframe window, where each keyframe Store its sonar coverage index set During the cold start phase, if the keyframe window... If the value is empty (i.e., the system has no historical keyframes), the current frame will directly return process_full (full weight processing), the gating score will be recorded as 1.0, and subsequent calculations will be skipped.

[0059] During the normal phase, the voxel template and the current frame coverage index are calculated first. During initialization, the voxel template is pre-calculated only once. This involves enumerating all 3D grid points within the sonar horizontal field of view, vertical field of view, and effective range in the sonar coordinate system according to the voxel grid resolution, to obtain a fixed set of voxels that can theoretically be covered by a single sonar observation. When processing each frame, the current sonar pose matrix is ​​applied to the voxel template, transforming the template points from the sonar coordinate system to the world coordinate system. Then, each point is discretized using the grid origin and voxel size. After filtering out points exceeding the grid boundaries, the remaining effective grid coordinates are converted into a one-dimensional flat index and deduplicated to obtain the voxel coverage index set for the current frame. If this coverage set is empty, meaning the sonar field of view is completely outside the grid range, the system directly returns and skips the step. The system calculates novelty and conflict based on the historical information of the keyframe window. Specifically, the union of the coverage index sets of all keyframes within the window is taken to obtain the historical coverage set; then, the proportion of overlap between the current frame coverage index and this historical coverage set is calculated relative to the total coverage of the current frame. Since the collision penalty coefficient is greater than the novelty gain coefficient, high-collision frames, even if they bring a small amount of novel information, will be preferentially downweighted or skipped.

[0060] This method can compress the contamination weights of low-quality frames on voxel statistics and provide appropriate enhancements to high-quality frames, thereby reducing the pseudo-occupancy ratio and improving geometric accuracy, increasing the surface voxel ratio, reducing the number of connected components, and effectively suppressing pseudo-occupancy clusters.

[0061] In one embodiment, the second retained frame is obtained by filtering the first retained frame based on the gating score and conflict degree, such as... Figure 4As shown, the method includes: determining the processing action for the first retained frame based on the gating score and conflict degree; the processing action includes full-weight processing, deweighting processing, and skipping, wherein the deweighting processing is to conditionally reduce the confidence of the frame; if the number of consecutively skipped frames reaches a preset skipping limit, the current processing action is changed to deweighting processing; when the processing action for the first retained frame is full-weight processing or deweighting processing, the first retained frame is the second retained frame.

[0062] As shown in Table 2, based on gated scoring With conflict level Further implement the three-level gating action decision. Specifically, when the gating score... When the gated score is greater than or equal to the full-weight threshold and the conflict degree does not exceed the maximum allowable conflict ratio, full-weight processing is performed; when the gated score is greater than or equal to the full-weight threshold and the conflict degree exceeds the maximum allowable conflict ratio, deweighting processing is performed; when the gated score is between the deweighting threshold and the full-weight threshold, deweighting processing is performed; when the gated score is less than the deweighting threshold, it is skipped without processing. The recommended value for the full-weighted threshold is 0.25. The recommended value for the weighting reduction threshold is 0.05. The recommended value for the maximum permissible conflict ratio is 0.35. The corresponding decision relationships are shown in Table 2.

[0063] Table 2:

[0064] In addition, the system maintains a continuous skip counter. When the judgment result is a skip and the skip count is greater than or equal to the maximum value of the skip counter, the action is forcibly upgraded to a deweighted processing to avoid dead zones caused by prolonged periods without updates in stable scenarios. When the gating action is deweighted processing, the portion of the confidence score above the global lower bound of 0.35 is multiplied by a conflict deweighting coefficient, with a recommended value of 0.55, so that the final confidence score approaches the lower bound but does not fall below it. For example, when the original confidence score is 0.70, the compressed score is approximately... .

[0065] This method can identify frames that highly overlap with and conflict with existing stable structures, perform weight reduction processing on such frames or skip them directly, thereby reducing the damage to stable geometric structures, reducing the overall element update runtime compared to the baseline, and shortening the overall element update runtime.

[0066] In one embodiment, determining the voxel occupancy state based on the observation count and occupancy count to determine the 3D reconstruction result of the target scene includes: obtaining a first ratio of the occupancy count to the observation count; comparing the first ratio with an occupancy rate threshold to obtain the voxel occupancy state; and determining the 3D reconstruction result of the target scene based on the occupancy state.

[0067] Specifically, adaptive weighted updates of the voxel grid are performed using frame confidence and gating actions, including: first, sonar image preprocessing. The input sonar polar coordinate image is converted to a Cartesian grayscale image through a polar-Cartesian coordinate mapper, and range-window filtering and empty echo region processing are performed in the polar coordinate space. Then, it is mapped back to Cartesian coordinates, and maximum downsampling is performed according to the configured downsampling factor to obtain the processed sonar image. In this image, a pixel value of 0 represents no echo, and a pixel value of 255 represents strong echo.

[0068] Calculate the voxel observation increment and sonar pose The method is applied to a voxel template to obtain the set of voxel positions in the world coordinate system, and the number of observations at each grid coordinate is counted to obtain the voxel observation increment vector. .

[0069] Perform a projection of the voxel into the sonar image coordinate system. For each valid voxel, the voxel's world coordinates are transformed by subtracting the sonar translation vector from the sonar pose rotation matrix, thus obtaining the voxel's 3D position in the sonar coordinate system. This 3D position is then converted to polar coordinates to obtain the voxel's range coordinates and horizontal azimuth angle, and mapped to row and column pixel coordinates in the sonar image based on the voxel's size. If the rounded pixel coordinates fall within the valid area of ​​the image, the projection is considered valid; otherwise, the voxel is skipped in this update.

[0070] During the weighted count update phase, for each valid voxel, its observation increment is multiplied by the frame confidence score and then accumulated into the observation count `obs_count` for that voxel. If the pixel value projected onto the sonar image by that voxel is 255, the same weighted increment is synchronously accumulated into the occupancy count `occ_count`; if the pixel value is 0, the occupancy count is not updated. When weighted updates are enabled, both the observation count array and the occupancy count array are of data type `float32` to support decimal accumulation; when weighted updates are disabled, both can still be implemented using integer paths, maintaining backward compatibility. Voxel occupancy rate is calculated. When the occupancy rate exceeds the threshold, the voxel is marked as occupied and added to the 3D voxel grid, and its display color depth can be encoded according to the occupancy rate; when the occupancy rate is below the threshold, the corresponding voxel is removed from the grid.

[0071] The occupancy threshold is set to two independent configurations: when weighted updates are disabled, the threshold is indirectly determined by the sonar false negative rate parameter; when weighted updates are enabled, an independent parameter is used, with a recommended value of 0.42. The dual-threshold design is adopted because weighted updates change the count from integer accumulation to floating-point accumulation, resulting in a corresponding change in the occupancy value distribution for the same number of physical observations. Therefore, the threshold needs to be recalibrated to maintain occupancy determination sensitivity comparable to the original system. Finally, keyframe window and system state updates are performed. If the gating action is non-skip, the current frame coverage index set is pushed into the keyframe window, and the window capacity is reached. The oldest entry pops up automatically; updates are also performed simultaneously. And reset the consecutive skip counter.

[0072] In one specific embodiment, which is an underwater oil and gas pipeline detection scenario (high turbidity, pure sonar reconstruction scenario), the system is deployed in a high turbidity underwater oil and gas pipeline detection environment, and the configuration parameters are as follows: global lower bound of frame confidence 0.35, global upper bound of frame confidence 1.25, lower limit of optical score pruning value in confidence calculation 0.45, occupancy threshold when weighted update is enabled 0.42, lower limit of optical score when entering HYBRID mode 0.40, upper limit of optical score when exiting HYBRID mode 0.28, lower limit of optical score when entering OPTICAL_ENHANCED mode 0.62, upper limit of optical score when exiting OPTICAL_ENHANCED mode 0.48, minimum number of resident frames after mode switching 8, degradation sliding window size (number of frames) 5, and forced degradation triggered when the proportion of degradation frames in the window exceeds this value 0.7. In this scenario, the water turbidity is high (greater than 12 NTU), and the optical camera image quality score consistently satisfies s < 0.28, which is below the HYBRID entry threshold. Therefore, the system remains in SONAR_ONLY mode, while module gating filtering is disabled to reduce additional computational overhead, resulting in a frame confidence level of 0.45. This demonstrates that even when the optical score is close to zero, the sonar-dominated update weight is maintained at a minimum, ensuring the continuous accumulation of geometric information.

[0073] In another specific embodiment, this embodiment is an underwater experimental tank (a scenario with dynamically changing optical conditions). In this embodiment, the visibility in the experimental tank varies randomly, and the optical quality score fluctuates within the range of 0.30–0.65. If the current score is s=0.41, then when the system is in the SONAR_ONLY state, since 0.41≥0.40, the system switches to HYBRID; however, if the system is already in the HYBRID state, since 0.41>0.28, the system continues to maintain HYBRID and will not immediately exit due to critical fluctuations, thus demonstrating the stabilizing effect of the hysteresis mechanism. In HYBRID mode, the frame confidence level is 0.405. When the robotic arm returns to the scanned area, if there is... =0.72> =0.35, then the gating score is:

[0074] At this point, the gating action is skipped, thus preventing high-collision frames from continuing to disrupt the established stable geometry.

[0075] In another specific embodiment, which is a fine underwater structure inspection (high-definition scene), the ambient water quality is clear with turbidity less than 0.5 NTU and the optical score consistently meets s>0.62. Therefore, the system runs stably in OPTICAL_ENHANCED mode and will not trigger forced degradation.

[0076] In this scenario, the frame confidence level is:

[0077] When the robotic arm scans a new area, if Then we have:

[0078] At this point, the gate control action is process_full. When the robotic arm returns to the scanned area, if... Then the gating action is reduced to process_downweight, and the corresponding confidence scaling is:

[0079] In this type of high-definition scene, the overall pixel update runtime is reduced by about 37% compared to the ungated version, while the geometric noise introduced by high-collision frames is suppressed, and the voxel ratio of the 3D reconstructed surface is improved by about 7%.

[0080] On the other hand, this application also proposes an underwater three-dimensional reconstruction device, such as... Figure 5As shown, the underwater 3D reconstruction device 100 includes: an acquisition module 200 for acquiring input frames of a target scene, including sonar images, pose matrices, and optical images; a first processing module 300 for performing optical quality assessments on the optical images to obtain optical quality scores and degradation flags; a second processing module 400 for setting a state machine to determine the processing mode of the input frames based on the optical quality scores and degradation flags; a third processing module 500 for calculating the frame confidence of the input frames using preset rules based on the optical quality scores and processing modes; and a fourth processing module 600 for filtering the input frames according to the pose matrix. The first reserved frame that meets the pose increment threshold is obtained. An index window is set according to the input frame. The conflict degree and novelty of the first reserved frame are determined according to the union of the sonar coverage index of the first reserved frame and the window, which is used to determine the gating score. The fifth processing module 700 is used to filter the first reserved frame according to the gating score and conflict degree to obtain the second reserved frame. The sixth processing module 800 is used to obtain the observation increment of the voxel corresponding to the sonar image of the second reserved frame, obtain the observation count according to the observation increment and the frame confidence, and determine the voxel occupancy state according to the observation count and the occupancy count, which is used to determine the three-dimensional reconstruction result of the target scene.

[0081] In one embodiment, the first processing module is further configured to convert the optical image into a grayscale image, the grayscale image including brightness, contrast, and sharpness; perform normalized weighted summation on the three statistical measures of brightness, contrast, and sharpness to obtain an optical quality score; when at least two of the three statistical measures of brightness, contrast, and sharpness are below a threshold, the degradation flag is set to true, otherwise it is set to false; when the optical image is missing, the optical quality score is set to zero, and the degradation flag is set to true.

[0082] Through the above technical solution, this application uses frame confidence as a unified mediator for judging optical quality, processing mode, and spatial consistency, and converges the outputs of the three modules—sonar image, pose matrix, and optical image—at the same weighted voxel update output. This achieves three levels of improvement within the same frame processing flow: first, it enables the 3D voxel reconstruction to suppress the cumulative contribution of low-quality frames when there is inter-frame quality heterogeneity, alleviating the problem of false occupancy introduced by low-quality observations; second, it enables the processing mode output to have temporal stability near the critical quality, avoiding mode jitter from being transmitted as fluctuations in the voxel update strategy; and third, it enables voxel accumulation in repeated scanning scenarios to have spatial redundancy awareness, avoiding repeated weighting of already stable regions.

[0083] The underwater 3D reconstruction device 100 includes a processor and a memory. The aforementioned acquisition module 200, first processing module 300, second processing module 400, third processing module 500, fourth processing module 600, fifth processing module 700 and sixth processing module 800 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0084] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the accuracy and stability of underwater 3D reconstruction.

[0085] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0086] This invention provides a storage medium storing a program that, when executed by a processor, implements the underwater three-dimensional reconstruction method.

[0087] This invention provides a processor for running a program, wherein the program executes the underwater three-dimensional reconstruction method during runtime.

[0088] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring an input frame of a target scene, including a sonar image, a pose matrix, and an optical image; performing optical quality assessment on the optical image to obtain an optical quality score and a degradation flag; setting a state machine to determine the processing mode of the input frame based on the optical quality score and the degradation flag; calculating the frame confidence of the input frame using preset rules based on the optical quality score and the processing mode; and processing the input frame according to the pose matrix. The process involves filtering to obtain a first retained frame that meets the pose increment threshold. An index window is set based on the input frame. The conflict degree and novelty of the first retained frame are determined by the union of the sonar coverage index of the first retained frame and the window, used to determine the gating score. The first retained frame is then filtered based on the gating score and conflict degree to obtain a second retained frame. The observation increment of the voxel corresponding to the sonar image of the second retained frame is obtained. An observation count is obtained based on the observation increment and the frame confidence. The voxel occupancy state is determined based on the observation count and the occupancy count, used to determine the 3D reconstruction result of the target scene. The devices mentioned in this paper can be servers, PCs, tablets, mobile phones, etc.

[0089] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: acquiring an input frame of a target scene, including a sonar image, a pose matrix, and an optical image; performing optical quality assessment on the optical image to obtain an optical quality score and a degradation flag; setting a state machine to determine the processing mode of the input frame based on the optical quality score and the degradation flag; calculating the frame confidence of the input frame using preset rules based on the optical quality score and the processing mode; filtering the input frame according to the pose matrix to obtain a first retained frame that meets the pose increment threshold; setting an index window according to the input frame; determining the conflict degree and novelty of the first retained frame based on the union of the sonar coverage index of the first retained frame and the window, for determining a gating score; filtering the first retained frame according to the gating score and the conflict degree to obtain a second retained frame; acquiring the observation increment of the voxel corresponding to the sonar image of the second retained frame; obtaining the observation count according to the observation increment and the frame confidence; determining the voxel occupancy state according to the observation count and the occupancy count, for determining the 3D reconstruction result of the target scene.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0095] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0096] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0097] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0098] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for underwater three-dimensional reconstruction, characterized in that, The method includes: Acquire the input frames of the target scene, including sonar images, pose matrix, and optical images; The optical images are subjected to optical quality assessment to obtain optical quality scores and degradation flags. A state machine is set up to determine the processing mode of the input frame based on the optical quality score and degradation flag. The frame confidence of the input frame is calculated by applying preset rules to the optical quality scoring and processing mode. The input frames are filtered according to the pose matrix to obtain the first retained frames that meet the pose increment threshold. An index window is set according to the input frames. The conflict degree and novelty of the first retained frames are determined according to the union of the sonar coverage index of the first retained frames in the index window and the window, which are used to determine the gating score. The first reserved frame is filtered based on the gating score and the conflict degree to obtain the second reserved frame; The observation increment of the voxel corresponding to the sonar image of the second retained frame is obtained, the observation count is obtained based on the observation increment and the frame confidence, and the voxel occupancy state is determined based on the observation count and the occupancy count, which is used to determine the three-dimensional reconstruction result of the target scene.

2. The method according to claim 1, characterized in that, The process of performing optical quality assessments on the optical images to obtain optical quality scores and degradation flags includes: The optical image is converted into a grayscale image, which includes brightness, contrast, and sharpness. The optical quality score is obtained by normalizing and weighting the three statistical measures of brightness, contrast and sharpness; When at least two of the three statistical measures of brightness, contrast and sharpness are below the threshold, the degradation flag is set to true; otherwise, it is set to false. When the optical image is missing, the optical quality score is set to zero and the degradation flag is set to true.

3. The method according to claim 1, characterized in that, The state machine is configured to determine the processing mode of the input frame based on the optical quality score and degradation flag, including: The state machine has an asymmetric hysteresis threshold, and the states of the state machine include pure sonar mode, hybrid mode, and optical mode. The asymmetric hysteresis thresholds include the hybrid mode entry threshold, the hybrid mode exit threshold, the optical mode entry threshold, and the optical mode exit threshold. When the state machine is in pure sonar mode, if the optical quality score is not less than the optical mode entry threshold, the next state is optical mode; if the optical quality score is not less than the hybrid mode entry threshold, the next state is hybrid mode; otherwise, the next state remains pure sonar mode. When the state machine is in the hybrid mode, if the optical quality score is less than the hybrid mode exit threshold, the next state is the pure sonar mode; if the optical quality score is not less than the optical mode entry threshold, the next state is the optical mode; otherwise, the next state remains the hybrid mode. When the state machine is in optical mode, if the optical quality score is less than the hybrid mode exit threshold, the next state is pure sonar mode; if the optical quality score is less than the optical mode exit threshold, the next state is hybrid mode; otherwise, the next state remains optical mode. When the degradation sliding window is full and the degradation ratio is not less than the degradation trigger ratio threshold, the next state is set to pure sonar mode.

4. The method according to claim 3, characterized in that, The state machine also has a minimum dwell time constraint: when the candidate next state calculated based on the asymmetric hysteresis threshold is different from the current state, but the difference between the current frame index and the previous mode switching frame index is less than the minimum dwell time, the current state is maintained and no switching is performed; otherwise, a switching is performed and the switching event is recorded. If the current switching direction is opposite to the previous switching direction and the interval between the two switching frames does not exceed the preset number of frames, then the current switching is marked as a jitter event and the jitter count is accumulated.

5. The method according to claim 1, characterized in that, The step of calculating the frame confidence of the input frame using preset rules for the optical quality scoring and processing mode includes: After cropping the optical quality score by upper and lower bounds, the frame confidence of the input frame is obtained by weighted calculation according to the processing mode.

6. The method according to claim 1, characterized in that, The step of setting an index window based on the input frame, and determining the conflict degree and novelty of the first retained frame based on the union of the sonar coverage index of the first retained frame and the window, for determining the gating score, includes: The total historical coverage set is obtained by taking the union of the coverage index sets of all input frames in the index window. The conflict degree is obtained by calculating the proportion of overlap between the first reserved frame coverage index and the total historical coverage set to the total coverage of the current frame. Novelty for: ; The gating score is determined based on the degree of conflict and novelty. ,in, For novel gain coefficients, This represents the conflict penalty coefficient.

7. The method according to claim 1, characterized in that, The step of filtering the first retained frames based on the gating score and conflict degree to obtain the second retained frames includes: The processing action for the first retained frame is determined based on the gating score and the degree of conflict. The processing actions include full-weight processing, deweighting processing, and skipping. The deweighting processing is to conditionally reduce the confidence of the frame. If the number of consecutive frames skipped reaches the preset skip limit, the current processing action will be changed to downweighting. When the processing action of the first retained frame is full weight processing or deweighting processing, the first retained frame becomes the second retained frame.

8. The method according to claim 1, characterized in that, The observation count is the cumulative number obtained by multiplying the voxel observation increment corresponding to the sonar image by the frame confidence. The occupancy count is the cumulative number of observation increments of the same voxel multiplied by the frame confidence when the effective projection of the voxel on the sonar image is a strong echo.

9. The method according to claim 1, characterized in that, The step of determining the voxel occupancy state based on the observation count and occupancy count, used to determine the 3D reconstruction result of the target scene, includes: Obtain the first ratio of the occupancy count to the observation count; The occupancy status of the voxel is obtained by comparing the first ratio with the occupancy threshold. The 3D reconstruction result of the target scene is determined based on the occupancy state.

10. An underwater three-dimensional reconstruction device, characterized in that, The device includes: The acquisition module is used to acquire input frames of the target scene, including sonar images, pose matrices, and optical images; The first processing module is used to perform optical quality assessment on the optical image to obtain an optical quality score and a degradation flag bit; The second processing module is used to set up a state machine and to determine the processing mode of the input frame based on the optical quality score and degradation flag. The third processing module is used to calculate the frame confidence of the input frame by performing preset rules on the optical quality score and processing mode. The fourth processing module is used to filter the input frames according to the pose matrix to obtain a first retained frame that meets the pose increment threshold, set an index window according to the input frame, and determine the conflict degree and novelty of the first retained frame according to the union of the sonar coverage index of the first retained frame and the window, which is used to determine the gating score. The fifth processing module is used to filter the first retained frame based on the gating score and the conflict degree to obtain the second retained frame; The sixth processing module is used to obtain the observation increment of the voxel corresponding to the sonar image of the second retained frame, obtain the observation count based on the observation increment and the frame confidence, determine the voxel occupancy state based on the observation count and the occupancy count, and determine the three-dimensional reconstruction result of the target scene.

11. A machine-readable storage medium storing instructions thereon, characterized in that, This instruction is used to cause the machine to perform the underwater three-dimensional reconstruction method according to any one of claims 1-9 of this application.

Images