An underwater multi-modal intelligent perception method for ocean ranching

Abstract

The application provides an underwater multi-modal intelligent sensing method for a mariculture, comprising the following steps: an underwater multi-modal detection module based on YOLO, introducing a sonar sensing mode, and constructing a dynamic self-adaptive multi-modal sensing system; performing semantic SLAM and dense coarse mapping; the process of performing semantic SLAM and dense coarse mapping comprises: preprocessing, initialization and joint optimization of visual-inertial data, dense mapping based on a surface element, and semantic fusion mapping; performing physically driven 3DGS fine reconstruction, comprising: initial 3DGS modeling and underwater medium estimation and compensation; and outputting a multi-scale semantic map. The application embeds a physical imaging model into a 3DGS rendering process, iteratively estimates attenuation coefficients and scattering parameters, and improves the color fidelity of the reconstructed scene. The reconstruction loss of the compensated image and the original image is significantly reduced compared with the traditional method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal perception methods, and in particular, especially relates to an underwater multi-modal intelligent perception method for ocean ranching. BACKGROUND

[0002] With the rapid development of ocean ranching, the demand for intelligent perception of underwater complex environment has significantly increased, and the key technical requirements cover accurate detection of marine products and marine garbage, simultaneous localization and mapping (SLAM) of underwater robots, and high-precision three-dimensional environment modeling. However, existing underwater perception technologies still face many challenges. Mainly including:

[0003] Limitations of underwater target detection. Existing underwater target detection mainly relies on optical cameras, but in turbid water or low light conditions, visual information is severely degraded, resulting in a significant decrease in detection accuracy. Although sonar provides supplementary information in low-visibility environments, existing methods lack adaptive multi-modal complementary mechanisms.

[0004] Insufficient environmental perception capability of SLAM system. Traditional SLAM systems only construct geometric maps and cannot simultaneously identify and label marine product distribution, making it difficult to support marine resource assessment tasks. In addition, feature point-based SLAM systems have difficulty generating dense point clouds rich in scene structure information, while RGB-D or lidar-based solutions are susceptible to scattering interference underwater, and water absorption and scattering cause image blurring and color distortion, affecting feature extraction and matching accuracy, and further increasing the pose estimation error of visual SLAM.

[0005] Precision and robustness issues of three-dimensional reconstruction. Existing underwater three-dimensional reconstruction usually uses multi-view stereo vision or structured light scanning, but due to the influence of water medium, the reconstruction results often have color distortion, strong geometric noise, and insufficient real-time performance.

[0006] To address the above challenges, the present application implements multi-granularity and multi-modal collaborative perception through a three-level intelligent perception architecture of target detection-semantic SLAM-medium compensation 3DGS, improving detection accuracy, mapping efficiency, and three-dimensional reconstruction quality, and solving the environmental perception problem in the ocean ranching scene. SUMMARY

[0007] According to the technical problems mentioned in the above background technology, an underwater multi-modal intelligent perception method for ocean ranching is provided. The present application constructs a multi-modal collaborative and multi-granularity progressive underwater intelligent perception framework, organically integrates target detection, semantic SLAM, and medium compensation 3D rendering, and forms full-dimensional perception capability from macro marine product distribution to micro scene structure.

[0008] The technical means adopted by the present application are as follows:

[0009] An underwater multi-modal intelligent sensing method for ocean ranching, comprising the following steps:

[0010] Step 1, underwater multi-modal detection module based on YOLO, introducing sonar sensing mode, constructing dynamic self-adaptive multi-modal sensing system;

[0011] Step 2, semantic SLAM and dense coarse mapping; the process of performing semantic SLAM and dense coarse mapping comprises: preprocessing of visual-inertial data, initialization and joint optimization, dense mapping based on surface element, and semantic fusion mapping;

[0012] Step 3, physical driven 3DGS fine reconstruction, including: initial 3DGS modeling and underwater medium estimation and compensation;

[0013] Step 4, output multi-scale semantic map.

[0014] Further, the step 1 comprises the following steps:

[0015] Step 11, the quality of visual data is evaluated, and a visual quality evaluation system based on the average value of brightness gradient amplitude is established; the brightness gradient amplitude calculation formula is:

[0016] ;

[0017] Wherein, represents the pixel intensity of the image at the coordinate , and respectively represent the horizontal and vertical direction gradients calculated by the Sobel operator;

[0018] Step 12, the quantitative evaluation index is:

[0019] ;

[0020] Wherein, represents the image resolution, represents the visual image quality score;

[0021] Step 13, whether the visual image quality score is low is judged; when the visual score is low, the weight of the sonar detection result is increased, otherwise the weight of the visual detection result is increased, and the final detection result is corrected by the visual data quality score.

[0022] Further, the visual and sonar detection weight calculation formula is:

[0023] ;

[0024] Wherein, visual clarity score, and modal sensitivity coefficient.

[0025] Further, in step 2, the pre-processing of visual-inertial data is:

[0026] ;

[0027] wherein, is the rotation pre-integration, i.e., the relative rotation change from time i to j , which is obtained by integrating the continuous IMU angular velocity measurements; is the angular velocity measured by the k th IMU; is the velocity pre-integration, i.e., the velocity change obtained by integrating in the coordinate system at time i ; is the linear acceleration measured by the k th IMU; is the position pre-integration, i.e., the displacement change in the coordinate system at time i .

[0028] Further, the initialization and joint optimization are in a loosely coupled manner, aligning the pre-integration of the IMU and the visual trajectory:

[0029] ;

[0030] wherein, s is the scale factor, and g represents the gravity direction;

[0031] Using a sliding window to do local optimization for the visual-inertial system, the state variable is:

[0032] ;

[0033] wherein, is the pose, velocity, and IMU bias; and the optimized objective function is:

[0034] ;

[0035] wherein, is the IMU pre-integration residual, is the visual re-projection error, is the Huber robust kernel coefficient;

[0036] Detecting whether there is a loop in the semantic SLAM and the dense coarse mapping process using the visual bag-of-words model; matching the feature points of the current frame with the historical key frames and calculating the similarity score, if it is judged as a similar frame, it is marked as a loop; if there is a loop, global optimization is performed, that is, the loop constraint is added to the pose graph for optimization:

[0037] ;

[0038] wherein represents the optimal pose estimation of the i-th key frame obtained by minimizing the objective function, otherwise the 6-DOF pose is directly output. i

[0039] Further, each face element of the face element-based dense mapping is represented by the following parameters:

[0040] ;

[0041] wherein, represents the center position of the face element, represents the normal vector of the face element, represents the radius of the face element, represents the color information, is the last updated timestamp of the face element;

[0042] First, pixels with similar color, texture and spatial position are extracted from visual images to form superpixels and are associated with face elements; the method is to obtain a set of 3D points by depth map back projection for each superpixel, and fit a local plane; whether it can be successfully associated with the existing face element is judged by the center position distance threshold and the normal vector angle threshold, if the association is successful, the face element information is updated by weighted fusion:

[0043] ;

[0044] wherein, and represent the updated face element position and normal vector, represents the center position of the superpixel, represents the normal vector of the superpixel, and both represent the confidence;

[0045] The color and radius of the face element are updated:

[0046] ;

[0047] wherein, represents the expansion coefficient;

[0048] ​If the association with the existing face element is unsuccessful, the center point, normal vector, color information and other parameters of the superpixel fitting are stored in the new face element;

[0049] The timestamp is modified to the most recently updated timestamp; when the SLAM system detects a loop, all face elements need to be rigidly transformed according to the pose graph optimization result, and no loop is needed to be processed:

[0050] ;

[0051] wherein, represents a loop optimization pose transformation matrix from the key frame i to j ;

[0052] Global consistency is ensured through pose graph optimization:

[0053] ;

[0054] wherein, represents an edge set of loop constraints; the final map is a set of all face elements, and if there is a need for dense point cloud map, such as navigation requirements, the center point position parameter can be retained to convert the face element map into a dense point cloud map.

[0055] Further, the method for semantic fusion mapping in step 2 is:

[0056] In the mapping process, the semantic information is fused, and the picture frames collected by the camera in the semantic SLAM and the dense rough mapping process are sent into the underwater multi-modal detection module based on YOLO in real time to obtain the semantic label and position information of the target object.

[0057] Since the detection result is based on the camera coordinate system, it is converted to the world coordinate system through pose transformation, and the conversion formula is:

[0058] ;

[0059] wherein, represents a 3D point in the camera coordinate system, represents a corresponding point in the world coordinate system, represents a 3x3 rotation matrix, i.e., the pose conversion from the camera coordinate system to the world coordinate system, represents a 3x1 translation vector, i.e., the position of the camera optical center in the world coordinate system;

[0060] is directly obtained from the output of the SLAM module This allows us to calculate the target's position in the world coordinate system. After time and spatial filtering, we can determine whether the target is stable and whether it has been recorded. If it has not been recorded, we record the target's label and its position in the world coordinate system. If it has been recorded, we skip this recording step.

[0061] Temporal filtering is used to filter transient noise to ensure temporal consistency across multiple detection frames; sliding window detection is employed, with temporal stability based on the following criteria:

[0062] ;

[0063] in, Indicates the length of the sliding window. Indicates the first Frame detection confidence, and These represent the confidence threshold and the minimum pass rate, respectively.

[0064] Finally, the recorded target locations and label information are represented in the map construction described above, thus completing the semantic fusion map construction.

[0065] Furthermore, in step 3, the physical imaging model is integrated into the 3DGS mapping process. By analyzing the depth map generated by 3DGS, the water attenuation coefficient and scattering parameters are estimated in real time, and an underwater optical compensation model is constructed accordingly.

[0066] Given a viewpoint, an image can be rendered efficiently by sorting Gaussians from front to back, projecting them onto the camera plane, and performing alpha synthesis; for a pixel x, the color of pixel x is calculated as:

[0067] ;

[0068] in, and Indicates the first i Given a 3D Gaussian color and density, the underwater image formation process can be represented as follows:

[0069] ;

[0070] in, Indicates the actual captured image. This represents a realistic underwater scene image after removing the influence of the medium. Indicates the attenuation coefficient. Represents the backscattering coefficient. Representing the backscattered color at infinity, it is assumed that... Used as the background color.

[0071] Furthermore, the method for embedding the underwater image physical formation model into 3DGS is as follows:

[0072] First, the depth map of the scene is obtained by the initial several rounds of 3DGS rendering , by estimating the attenuation coefficient , scattering coefficient and background color , the attenuation and scattering effects can be obtained:

[0073]

[0074]

[0075] Then superimposed on the estimated underwater real image by the following formula , the first round can directly set to the original image;

[0076]

[0077] Then the resulting image is constructed to narrow the distance between the original image and the reconstruction loss, and after several rounds of iteration, the image close to the underwater real image J .

[0078] Compared with the prior art, the present application has the following advantages:

[0079] 1. The dynamic weight fusion mechanism provided by the present application realizes adaptive weighted fusion of visual-sonar detection results in underwater complex environment through the synergistic effect of the visual quality evaluation system (VQA) and the sonar compensation coefficient (K). Svis When the visual score is β ≤15, the system automatically increases the sonar weight to 0.27-0.88 (see weight table in Example 1), solving the problem of failure of traditional single visual detection in turbid water. Svis

[0080] 2. Through the real-time linkage of the facet parameterization representation and the YOLO detection module, a dense map containing the semantic labels of marine product distribution is constructed. The superpixel association mechanism improves the map updating efficiency, and the pose graph optimization ensures the global consistency in the closed loop

[0081] 3. The physical imaging model is embedded into the 3DGS rendering process, and the attenuation coefficient and scattering parameter are estimated through iteration, so that the color fidelity of the reconstructed scene is improved. The reconstruction loss of the compensated image and the original image is significantly reduced compared with the traditional method.

[0082] Based on the above reasons, the present application can be widely popularized in the fields of marine ranching resource survey and ecological monitoring, underwater robot autonomous navigation and operation, shipwreck archaeology and coral reef three-dimensional digitization.​ Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0085] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0086] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0087] like Figure 1 As shown, this invention provides an underwater multimodal intelligent sensing method for marine ranches, comprising the following steps:

[0088] Step 1: To address the dual needs of marine ranching yield assessment and pollution monitoring, this study developed an intelligent detection system integrating visual and sonar data. The core of the system employs an underwater target detection module based on the YOLOv5 architecture, enabling simultaneous marine product yield statistics and marine debris monitoring. Furthermore, to overcome the visual limitations commonly encountered in underwater environments, such as low illumination and turbid water, this application incorporates sonar sensing modalities based on the YOLO underwater multimodal detection module to construct a dynamically adaptive multimodal perception system.

[0089] As a preferred embodiment, in the present application, the step 1 comprises the following steps:

[0090] Step 11, the quality of visual data is evaluated, and a visual quality evaluation system based on the average value of the brightness gradient amplitude is established; the brightness gradient amplitude calculation formula is:

[0091]

[0092] wherein, represents the pixel intensity of the image at the coordinate ; and respectively represent the horizontal and vertical direction gradients calculated by the Sobel operator;

[0093] Step 12, the quantitative evaluation index is:

[0094]

[0095] wherein, represents the image resolution, represents the visual image quality score;

[0096] Step 13, it is judged whether the visual image quality score is low; when the visual score is low, the weight of the sonar detection result is increased, otherwise the weight of the visual detection result is increased, and the final detection result is corrected by the visual data quality score. As a preferred, the visual and sonar detection weight calculation formula is:

[0097]

[0098] wherein, represents the visual clarity score, and represent the modal sensitivity coefficients.

[0099] Further, in the process of underwater operation, it is crucial to obtain real-time information of the own position and the surrounding environment. Therefore, step 2, semantic SLAM and dense coarse mapping are performed; considering the characteristics that the visual key frame frequency is low and the inertial measurement unit (IMU) sampling frequency is high, in order to optimize the calculation efficiency, the system first performs pre-integration processing on the IMU data. This method converts the acceleration and angular velocity information provided by the IMU into the relative displacement, velocity and rotation amount between adjacent key frames, effectively reducing the calculation burden, while ensuring the accuracy and real-time performance of positioning and mapping. The process of performing semantic SLAM and dense coarse mapping comprises: preprocessing of visual-inertial data, initialization and joint optimization, face-based dense mapping, and semantic fusion mapping;

[0100] In the step 2, the preprocessing of visual-inertial data is:​​​

[0101] ;

[0102] in, Represents the rotational pre-integral quantity, i.e., from time t. i arrive j The relative rotational change is obtained by integrating the continuous IMU angular velocity measurements; Indicates the first k Angular velocity measured by each IMU; This represents the velocity pre-integral quantity, that is, in i The velocity change obtained by integration in the time-time coordinate system; Indicates the first k Linear acceleration measured by an IMU; Indicates the position pre-integral quantity, that is, in i The change in displacement in the time coordinate system.

[0103] In a preferred embodiment, FAST corner detection and KLT optical flow tracing are performed on the visual image for feature extraction and tracking. Then, joint initialization of inertial and visual components is performed. This invention employs a loosely coupled approach, utilizing pure visual structure of motion (SfM) technology to estimate the poses of 3D map points and keyframes, and then aligning the IMU pre-integral values ​​with the visual trajectory. The initialization and joint optimization employ a loosely coupled approach, aligning the IMU pre-integral values ​​with the visual trajectory:

[0104] ;

[0105] in, s represents the scale factor, and g represents the direction of gravity;

[0106] When using a sliding window to perform local optimization on the visual-inertial navigation system, the state variables are:

[0107] ;

[0108] in, Indicate pose, velocity, and IMU bias; then the optimized objective function is:

[0109] ;

[0110] in, This represents the IMU pre-integration residual. This indicates visual reprojection error. Represents Huber robust kernel coefficients;

[0111] The visual bag-of-words model is used to detect loop closures during semantic SLAM and dense coarse-scale graph construction. Feature points in the current frame are matched with historical keyframes, and similarity scores are calculated. If similar frames are identified, they are marked as loop closures. If loop closures exist, global optimization is performed, i.e., loop closure constraints are added to the pose graph for optimization.

[0112] ;

[0113] in This represents the solution obtained by minimizing the objective function. i The optimal pose estimation for each keyframe is calculated; otherwise, the pose with 6 degrees of freedom is directly output.

[0114] Preferably, each element of the element-based dense mapping is represented by the following parameters:

[0115] ;

[0116] in, Indicates the center position of the face element. The normal vector of a surface element. Indicates the radius of the element. Indicates color information, The timestamp of the last update of the face element;

[0117] First, pixels with similar colors, textures, and spatial locations are extracted from the visual image to form superpixels, which are then associated with surface elements. The method involves backprojecting each superpixel through a depth map to obtain a set of 3D points and fitting a local plane. A threshold for the distance between the center position and a threshold for the angle between the normal vectors are used to determine whether the superpixel can be successfully associated with existing surface elements. If the association is successful, the surface element information is updated through weighted fusion.

[0118] ;

[0119] in, and This represents the updated face element position and normal vector. Indicates the center position of the superpixel. The normal vector of a superpixel. and All represent confidence levels;

[0120] Update the color and radius of the opposite element:

[0121] ;

[0122] in, Indicates the expansion coefficient;

[0123] If the association with the existing face element is unsuccessful, the center point, normal vector, color information and other parameters of the superpixel fitting are stored in the new face element;

[0124] The timestamp is modified to the most recently updated timestamp; when the SLAM system detects a loop, all face elements need to be rigidly transformed according to the pose graph optimization result, and no loop is needed to be processed:

[0125] ;

[0126] wherein, represents a loop optimization pose transformation matrix from the key frame i to j ;

[0127] Global consistency is ensured by pose graph optimization:

[0128] ;

[0129] wherein, represents an edge set of loop constraints; the final map is a set of all face elements, and if there is a need for dense point cloud map, such as navigation requirements, the face element map can be converted into a dense point cloud map by only retaining the center point position parameter.

[0130] Preferably, the method for semantic fusion mapping is:

[0131] In the mapping process, semantic information is fused, and the picture frames collected by the camera in the semantic SLAM and dense rough mapping process are sent into the underwater multi-modal detection module based on YOLO in real time to obtain the semantic label and position information of the target object.

[0132] To realize intelligent perception of the environment, the system fuses semantic information in the mapping process. When the SLAM module is running, the key frames are sent into the YOLO detection module in real time to obtain the semantic label and position information of the target object. Since the detection result is based on the camera coordinate system, it needs to be converted to the world coordinate system through pose transformation. Since the detection result is based on the camera coordinate system, it needs to be converted to the world coordinate system through pose transformation, and the conversion formula is:

[0133] ;

[0134] wherein, represents a 3D point in the camera coordinate system, represents the corresponding point in the world coordinate system, represents a 3x3 rotation matrix, i.e., the pose conversion from the camera coordinate system to the world coordinate system, represents a 3x1 translation vector, i.e., the position of the camera optical center in the world coordinate system;

[0135] The output of the SLAM module is directly obtained The position of the target in the world coordinate system can thus be calculated, and after time domain and space domain filtering, it can be determined whether the target is stably present and whether the target has been recorded. If not, the label of the target and the position information in the world coordinate system are recorded. If yes, the current recording is skipped.

[0136] Temporal filtering is used to filter transient noise for the time consistency of multiple frame detection results. A sliding window is used for detection, and the temporal stability is based on:

[0137]

[0138] wherein, represents the length of the sliding window, represents the detection confidence of the i-th frame, and and respectively represent the confidence threshold and the minimum passing rate.

[0139] Finally, the recorded target position and label information are expressed in the mapping described above, and semantic fusion mapping is completed. Spatial domain filtering is used to realize target deduplication. When a target object is detected, the system calculates the three-dimensional position of the target in the world coordinate system and compares it with the target recorded in the target storage table. If the spatial distance between the two is less than a predetermined threshold and they have the same semantic label, the system determines that they are the same target and will not be recorded again. If the difference is large, the target is stored as a new entry in the target storage table.

[0140] In this way, the system can track the position changes of target objects in real time during the SLAM process and determine which targets are stably present. Based on this constantly updated target storage table, the system can accurately count the number of targets to estimate the yield. At the same time, the system also integrates these target object position information with semantic labels into the environment map, and finally constructs a navigation map containing rich semantic information.

[0141] ​Furthermore, to address the need for detailed underwater mapping in edge computing scenarios, this invention proposes a 3DGS mapping strategy based on an underwater physical imaging model. This scheme enables detailed modeling of specific areas. Considering the unique light attenuation and backscattering issues in the underwater environment, this method integrates the physical imaging model into the 3DGS mapping process: by analyzing the depth map generated by 3DGS, the water attenuation coefficient and scattering parameters are estimated in real time, and an underwater optical compensation model is constructed accordingly. This model uses an iterative optimization algorithm to superimpose the simulated attenuation and scattering effects onto the original image and compares it with the actual observed image, gradually correcting the reconstruction error, and finally obtaining a high-fidelity 3D reconstruction result that closely approximates the real underwater scene. This technology effectively solves the problem of insufficient modeling accuracy of traditional pixel methods in complex underwater environments. Traditional 3DGS uses a set of 3D Gaussians to parameterize the scene. Each Gaussian has attributes such as mean µ, covariance Σ derived from scale S and rotation R, opacity o derived from spherical harmonic coefficients, and color c. Given a viewpoint, an image can be rendered efficiently by sorting Gaussians from front to back, projecting them onto the camera plane, and performing alpha synthesis.

[0142] Therefore, step 3, performing physical-driven 3DGS fine reconstruction, includes: initial 3DGS modeling and underwater medium estimation and compensation; integrating the physical imaging model into the 3DGS mapping process, estimating the water attenuation coefficient and scattering parameters in real time by analyzing the depth map generated by 3DGS, and constructing an underwater optical compensation model accordingly.

[0143] Given a viewpoint, an image can be rendered efficiently by sorting Gaussians from front to back, projecting them onto the camera plane, and performing alpha synthesis; for a pixel x, the color of pixel x is calculated as:

[0144] ;

[0145] in, and Indicates the first i Given a 3D Gaussian color and density, the underwater image formation process can be represented as follows:

[0146] ;

[0147] in, Indicates the actual captured image. This represents a realistic underwater scene image after removing the influence of the medium. Indicates the attenuation coefficient. Represents the backscattering coefficient. Representing the backscattered color at infinity, it is assumed that... Used as the background color.

[0148] The method for embedding underwater image physical modeling models into 3DGS is as follows:

[0149] First, the depth map of the scene is obtained from the initial few rounds of 3DGS rendering. ,pass Estimate attenuation coefficient scattering coefficient and background color The effects of attenuation and scattering can then be obtained:

[0150]

[0151]

[0152] Then, the estimated underwater real image is superimposed using the following formula. Up, the first round can Set it directly to the original image;

[0153]

[0154] The resulting image will then be generated. By constructing a reconstruction loss model that closely approximates the original image, and through multiple iterations, an image close to the actual underwater image can be obtained. J .

[0155] Finally, step 4: output a multi-scale semantic map.

[0156] Example 1

[0157] This invention mounts the system on an underwater robot to perform real-time underwater sensing tasks:

[0158] 1. Generate semantic information:

[0159] Semantic Information Generation Method: This invention employs a multimodal data collaborative detection strategy. First, a YOLOv5 detection model is trained based on underwater visual and sonar image datasets, respectively. In actual deployment, visual-sonar data is simultaneously acquired using a ZED2i binocular camera and an M1200d sonar device. A dual-thread parallel processing mechanism is used to input the two modalities of data into the YOLOv5 model for real-time inference. To improve detection reliability, the system dynamically calculates image quality scores through a visual quality assessment module and adaptively weights and fuses the detection results from the visual and sonar modalities accordingly, ultimately outputting a semantic detection result with a confidence score.

[0160] ;

[0161] Visual sensitivity coefficient Set to 5, sonar compensation intensity coefficient Set it to 3 and increase the visual score.

[0162] Normalized to [0, 1], the weight formula output characteristics are as follows:

[0163]

[0164] Thus, the inference results of the two modalities are weighted, and then time domain and space domain filtering are performed to determine whether the target object is stably present:

[0165]

[0166] The sliding window length N in the time domain stability basis is set to 10, the confidence threshold γ is set to 0.5, and the minimum passing ratio α is set to 0.7. The basis for whether the two inferences are close to the target is:

[0167]

[0168] is the distance threshold, and then combined with the label, it can be determined whether it is the same target and the target storage table is updated.

[0169] 2. Real-time construction of coarse-grained map:

[0170] The visual data and inertial data collected by the ZED2i device are packaged through the ROS platform and sent to the SLAM module in the form of topics. The SLAM module first performs data preprocessing, including:

[0171] The received image is histogram equalized, and mirror and tangential distortion correction is performed according to the pre-calibrated camera intrinsic parameters and distortion coefficients.

[0172] Feature points are extracted using FAST corner detection, and the image grid is divided into blocks. Here, 20x20 is set as a block, and the top 5 points with the highest response value are retained in each block. KLT optical flow method is used to track the feature points of adjacent frames.

[0173] The polar line correction is performed on the adjacent frame images, and the fundamental matrix F is calculated through the polar geometric constraint. Then, the essential matrix E can be obtained by combining the camera intrinsic parameter K.

[0174]

[0175] Then, the essential matrix E is decomposed by SVD to solve the relative pose between the two frames .

[0176] According to the relative pose, the matching feature points are calculated by linear triangulation to obtain 3D coordinates, and an initial map point set is constructed.

[0177] The raw data of angular velocity and acceleration of the IMU in the time interval​​​ Pre-integration is performed to obtain attitude and position information.

[0178] ;

[0179] ;

[0180] The camera pose of the visual SfM estimation is aligned with the IMU pre-integration result, and then the visual re-projection error, the IMU pre-integration error, and the scale / gravity error are jointly optimized and updated in real time using the Gauss-Newton method.

[0181] The current frame features are matched with the historical key frames using the visual bag-of-words model (DBoW2) and the similarity scores are calculated. If it is judged as a similar frame, the relative pose of the two frames is calculated and the loop constraint is inserted as an edge into the pose graph to connect the current frame with the candidate frame, thereby correcting the overall pose.

[0182] Superpixels are constructed by extracting groups of pixels of similar color, texture, and spatial position from visual images to build facets, i.e. each superpixel is fitted with a local plane by back-projecting a set of 3D points from the depth map, and then it is determined whether it can be successfully associated with an existing facet by judging the distance threshold of the center position and the angle threshold of the normal vector. If the association is successful, the facet information is updated by weighted fusion, and when the inertial navigation system detects it, all facets need to be rigidly transformed according to the pose graph optimization results. The final map is the collection of all facets, which can be further converted into a dense point cloud.

[0183] 3. Offline construction of fine-grained map

[0184] The pose obtained by SfM and the initial 3D points in the foregoing are input to construct an initial 3D Gaussian scene together with the pictures, and then the scene is slowly refined by copying and splitting the Gaussian sphere. Since the color of the underwater scene is disturbed by multiple factors, we use the underwater image formation model to constrain the rendering process to restore the true color of the scene. The formation process of the underwater image can be represented by the following formula:

[0185] ;

[0186] First, the scene depth map is obtained by rendering the initial 1000 rounds of 3D GS , and then the attenuation coefficient , the scattering coefficient , and the background color are obtained according to the attenuation effect, i.e. the attenuation and scattering effects are obtained:

[0187] ;

[0188] ;

[0189] ​Superimpose the above two influences to the estimated underwater real image The above, the subsequent first round of training directly Directly set to the original image, the resulting image With the original image to construct the reconstruction loss so that they are as close as possible, after several iterations can be obtained close to the underwater real image of the image J and restore the real color scene.

[0190] The above-mentioned embodiment number of the present application is only for description, not represent the pros and cons of the embodiment. In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the part not described in detail in a certain embodiment can be referred to the related description of other embodiments. In the several embodiments provided in the present application, it should be understood that the disclosed technical content can be realized by other ways.

[0191] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitution for part or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for underwater multimodal intelligent sensing for marine ranches, characterized in that, Includes the following steps: Step 1: Based on the YOLO underwater multimodal detection module, sonar sensing modes are introduced to construct a dynamically adaptive multimodal sensing system; Step 1 includes the following steps: Step 11: Evaluate the quality of the visual data and establish a visual quality evaluation system based on the average value of the brightness gradient amplitude; the formula for calculating the brightness gradient amplitude is: ； in, Indicates the image in coordinates Pixel intensity at that location and These represent the horizontal and vertical gradients calculated using the Sobel operator, respectively. Step 12, the quantitative evaluation indicators are: ； in, Indicates image resolution, Indicates the visual image quality score; Step 13: Determine if the visual image quality score is too low; if the visual score is too low, increase the weight of the sonar detection result, and vice versa. The final detection result is corrected by the visual data quality score; the formula for calculating the visual and sonar detection weights is as follows: ； in, Indicates visual clarity score, and Represents the modal sensitivity coefficient; Step 2: Perform semantic SLAM and dense coarse mapping; the process of performing semantic SLAM and dense coarse mapping includes: preprocessing of visual-inertial data, initialization and joint optimization, dense mapping based on surface elements, and semantic fusion mapping; Step 3: Perform physical-driven 3DGS fine reconstruction, including: initial 3DGS modeling and underwater medium estimation and compensation; Step 4: Output a multi-scale semantic map.

2. The underwater multimodal intelligent sensing method for marine ranching according to claim 1, characterized in that, In step 2, the preprocessing of the visual-inertial data is as follows: ； in, Represents the rotational pre-integral quantity, i.e., from time t. i arrive j The relative rotational change is obtained by integrating the continuous IMU angular velocity measurements; Indicates the first k Angular velocity measured by each IMU; This represents the velocity pre-integral quantity, that is, in i The velocity change obtained by integration in the time-time coordinate system; Indicates the first k Linear acceleration measured by an IMU; Indicates the position pre-integral quantity, that is, in i The change in displacement in the time coordinate system.

3. The underwater multimodal intelligent sensing method for marine ranching according to claim 1, characterized in that, The initialization and joint optimization adopt a loosely coupled approach, aligning the IMU's pre-integral values ​​with the visual trajectory: ； in, s represents the scale factor, and g represents the direction of gravity; When using a sliding window to perform local optimization on the visual-inertial navigation system, the state variables are: ； in, Indicate pose, velocity, and IMU bias; then the optimized objective function is: ； in, This represents the IMU pre-integration residual. This indicates visual reprojection error. Represents Huber robust kernel coefficient; The visual bag-of-words model is used to detect loop closures during semantic SLAM and dense coarse-scale graph construction. Feature points in the current frame are matched with historical keyframes, and similarity scores are calculated. If similar frames are identified, they are marked as loop closures. If loop closures exist, global optimization is performed, i.e., loop closure constraints are added to the pose graph for optimization. ； in This represents the solution obtained by minimizing the objective function. i The optimal pose estimation for each keyframe is calculated; otherwise, the pose with 6 degrees of freedom is directly output.

4. The underwater multimodal intelligent sensing method for marine ranching according to claim 1, characterized in that, Each cell in the cell-based dense mapping is represented by the following parameters: ； in, Indicates the center position of the face element. The normal vector of a surface element. Indicates the radius of the element. Indicates color information, The timestamp of the last update of the face element; First, pixels with similar colors, textures, and spatial locations are extracted from the visual image to form superpixels, which are then associated with surface elements. The method involves backprojecting each superpixel through a depth map to obtain a set of 3D points and fitting a local plane. A threshold for the distance between the center position and a threshold for the angle between the normal vectors are used to determine whether the superpixel can be successfully associated with existing surface elements. If the association is successful, the surface element information is updated through weighted fusion. ； in, and This represents the updated face element position and normal vector. Indicates the center position of the superpixel. The normal vector of a superpixel. and All represent confidence levels; Update the color and radius of the opposite element: ； in, Indicates the expansion coefficient; If association with an existing facet fails, the center point, normal vector, and color information parameters of the superpixel fitting are stored in the new facet. Modify the timestamp to the most recently updated timestamp; when the SLAM system detects a loop closure, all facets need to undergo rigid body transformation based on the pose graph optimization results; if there is no loop closure, no processing is required. ； in, Indicates from keyframe i arrive j The pose transformation matrix optimized by loop closure; Global consistency is ensured through pose graph optimization: ； in, This represents the set of edges constrained by loops; the final map is a set of all polygons. If a dense point cloud map is required, only the center point position parameter is retained to convert the polygon map into a dense point cloud map.

5. The underwater multimodal intelligent sensing method for marine ranching according to claim 1, characterized in that, The semantic fusion mapping method in step 2 is as follows: Semantic information is integrated during the mapping process. During semantic SLAM and dense coarse mapping, the frames captured by the camera are sent in real time to the underwater multimodal detection module based on YOLO to obtain the semantic labels and location information of the target objects. Since the detection results are based on the camera coordinate system, they are transformed to the world coordinate system through pose transformation. The transformation formula is as follows: ； in, Represents a 3D point in the camera coordinate system. Represents the corresponding point in the world coordinate system. This represents a 3×3 rotation matrix, which is the pose transformation from the camera coordinate system to the world coordinate system. This represents a 3×1 translation vector, which is the position of the camera's optical center in the world coordinate system; Obtained directly from the output of the SLAM module The target's position in the world coordinate system is calculated from this. After time and spatial filtering, it can be determined whether the target exists stably and whether the target has been recorded. If it has not been recorded, the target's label and position information in the world coordinate system are recorded. If it has been recorded, the recording is skipped. Temporal filtering is used to filter transient noise to ensure temporal consistency across multiple detection frames; sliding window detection is employed, with temporal stability based on the following criteria: ； in, Indicates the length of the sliding window. Indicates the first Frame detection confidence, and These represent the confidence threshold and the minimum pass rate, respectively. Finally, the recorded target locations and label information are represented in the map construction described above, thus completing the semantic fusion map construction.

6. The underwater multimodal intelligent sensing method for marine ranching according to claim 1, characterized in that, In step 3, the physical imaging model is integrated into the 3DGS mapping process. By analyzing the depth map generated by 3DGS, the water attenuation coefficient and scattering parameters are estimated in real time, and an underwater optical compensation model is constructed accordingly. Given a viewpoint, an image is rendered efficiently by sorting Gaussians from front to back, projecting them onto the camera plane, and performing alpha synthesis; for a pixel x, the color of pixel x is calculated as: ； in, and Indicates the first i Given a 3D Gaussian color and density, the underwater image formation process can be represented as follows: ； in, Indicates the actual captured image. This represents a realistic underwater scene image after removing the influence of the medium. Indicates the attenuation coefficient. Represents the backscattering coefficient. Representing the backscattered color at infinity, it is assumed that... Used as the background color.

7. The underwater multimodal intelligent sensing method for marine ranching according to claim 6, characterized in that, The method for embedding underwater image physical modeling models into 3DGS is as follows: First, the depth map of the scene is obtained from the initial few rounds of 3DGS rendering. ,pass Estimate attenuation coefficient scattering coefficient and background color The effects of attenuation and scattering can then be obtained: Then, the estimated underwater real image is superimposed using the following formula. Up, the first round can Set it directly to the original image; The resulting image will then be generated. By constructing a reconstruction loss model that closely approximates the original image, and through multiple iterations, an image close to the actual underwater image can be obtained. J .

Images