A cross-medium cooperative positioning method and system for a dynamic 3D GS driven backhoe dredger

Abstract

The application discloses a kind of dynamic 3DGS driven backhoe dredger cross-media cooperative positioning method and system, the method includes: carrying out multimodal data acquisition;Based on dynamic 3DGS technology respectively constructs underwater scene dynamic 3DGS adaptive model and water scene dynamic 3DGS adaptive model, relies on IMU pre-integral constraint to build multimodal tight coupling positioning framework, complete heterogeneous data depth fusion and conflict resolution, optimize model parameter adaptive rule;Construct IMU error residual function, through IMU drift online correction algorithm to compensate benchmark drift error, combined with graph optimization framework and Gauss-Newton iteration method complete multi-source error joint correction;Based on Transformer network constructs underwater operation scene autonomous prediction model and water operation scene autonomous prediction model, combined with fuzzy logic decision algorithm exports underwater positioning mode and the adaptive selection instruction of water positioning mode;Through cooperative positioning module realizes positioning data coordinate unification and time series smoothing processing, exports global operation space cooperative positioning result.

Description

Technical Field

[0001] This application belongs to the field of marine engineering equipment positioning technology, specifically referring to a dynamic 3DGS driven cross-media collaborative positioning method and system for backhoe dredgers. Background Technology

[0002] Backhoe dredgers, as core equipment in marine engineering such as channel dredging and port construction, involve cross-media scenarios including underwater excavation, surface vessel transfer, and barge docking. Precise positioning of the operating space is a key prerequisite for achieving intelligent dredging. However, backhoe dredgers face many technical challenges in cross-media operations: First, multi-source coupled disturbances such as wind, waves, currents, and underwater acoustic and optical attenuation lead to low positioning accuracy and poor stability of single sensors; second, heterogeneous data from underwater sonar and vision, and from surface lidar and vision, have characteristic differences, which can easily cause fusion conflicts; third, surface disturbances cause fluctuations in the reference position of multimodal sensors, leading to sensor reference drift and further reducing positioning accuracy; fourth, backhoe dredgers will face different underwater and surface operating scenarios at different stages of operation, and fixed positioning modes cannot adapt to the positioning requirements of different scenarios, so it is urgent to achieve precise adaptive selection of positioning modes according to the operating scenario.

[0003] In existing technologies, dredging vessel positioning technologies mostly focus on terrain construction and visualization of a single underwater or above-water scene, such as underwater terrain positioning through multibeam bathymetry and above-water hull positioning based on GPS / RTK. However, they lack collaborative positioning solutions for cross-media operation spaces. Although some 3D visualization systems have achieved the display of above-water and underwater scenes, they are only based on the simple superposition of raw sensor data and have not solved the problems of heterogeneous data fusion conflicts and sensor reference drift. Although the slope precision excavation control method of unmanned dredging vessels involves bucket positioning, it has not introduced dynamic 3DGS technology to achieve deep fusion of multimodal data, nor has it proposed an adaptive selection scheme for positioning mode based on scene prediction. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this application provides a dynamic 3DGS-driven cross-media cooperative positioning method and system for backhoe dredgers, which solves technical problems such as low positioning accuracy, heterogeneous data fusion conflicts, sensor reference drift, and positioning modes that cannot be accurately adapted to underwater / above-water scenarios during cross-media operations of backhoe dredgers.

[0005] This invention provides a dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers, the method comprising: Multimodal data acquisition is performed, including underwater scanning profile sonar data, underwater binocular vision data, surface lidar data, and surface binocular vision data. Equipment motion data is also acquired through the IMU module. We constructed and fused dynamic 3DGS models for both underwater and above-water scenarios. Based on dynamic 3DGS technology, we built dynamic 3DGS adaptation models for underwater and above-water scenarios respectively. We built a multimodal tightly coupled positioning framework based on IMU pre-integration constraints, completed deep fusion and conflict resolution of heterogeneous data, and optimized the adaptive rules for model parameters. IMU drift correction and multi-source error joint correction are performed. An IMU error residual function is constructed. The baseline drift error is compensated by the online IMU drift correction algorithm. The multi-source error joint correction is completed by combining the graph optimization framework and the Gauss-Newton iteration method. The system performs operation scenario prediction and positioning mode decision-making. It constructs an autonomous prediction model for underwater operation scenarios and an autonomous prediction model for surface operation scenarios based on the Transformer network, and outputs adaptive selection instructions for underwater positioning mode and surface positioning mode by combining fuzzy logic decision-making algorithm. Cross-media collaborative positioning is performed. The collaborative positioning module realizes the unification of positioning data coordinates and smoothing of time series, and outputs the collaborative positioning results of the entire operation space.

[0006] Furthermore, according to the dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers provided by the present invention, the underwater scene dynamic 3DGS adaptation model is based on a three-dimensional Gaussian kernel function to construct a Gaussian splash representation model of point cloud features, with the structure as follows: and ;in, The coordinates of the underwater point. The coordinates of feature points acquired by underwater sonar. The coordinates of feature points acquired by underwater binocular vision. , For Gaussian kernel weights, The width of the Gaussian kernel function. It is a 3-order identity matrix. The underwater sound propagation attenuation coefficient is... The underwater light coefficient; The structure of the dynamic 3DGS adaptation model for the water scene is as follows: ;in, The coordinates of the underwater point. The coordinates of the feature points acquired by LiDAR / vision. For water surface light coefficient, The weights are Gaussian kernel weights.

[0007] Furthermore, according to the dynamic 3DGS-driven cross-media collaborative positioning method for backhoe dredgers provided by the present invention, the adaptive rules for the optimized model parameters include: the underwater scene dynamic 3DGS adaptation model adjusts the width of the Gaussian kernel function according to the underwater turbidity value, the above-water scene dynamic 3DGS model adjusts the feature extraction threshold according to the above-water light intensity, and the model update frequency is dynamically adapted to the equipment movement speed.

[0008] Furthermore, according to the dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers provided by the present invention, the IMU drift online correction algorithm includes real-time correction of its output data through IMU static zero drift statistics, characterizing the reference position deviation of multi-modal sensors through the corrected IMU sensor data, incorporating multi-modal sensor errors into a unified optimization objective through a graph optimization framework, and completing the joint correction of multi-source errors through the Gauss-Newton iteration method. The Gauss-Newton iterative method is set to 15-25 iterations, and the iteration convergence threshold is set to 0.001-0.01m according to the positioning accuracy requirements to ensure the accuracy and efficiency of multi-source error correction.

[0009] Furthermore, according to the dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers provided by the present invention, the operation scenario is divided into underwater excavation operation scenario and above-water operation scenario, which is above-water docking of mud barge scenario, and the classification accuracy of the scenario prediction model is not less than 95%; The specific logic of adaptive selection is as follows: Using scene prediction results, real-time positioning accuracy of each positioning mode, and sensor operating status as three input variables, each input variable is divided into different fuzzy subsets. The fuzzy subset of the scene prediction results is {underwater high confidence, underwater low confidence, surface high confidence, surface low confidence}, the fuzzy subset of the real-time positioning accuracy is {high accuracy, medium accuracy, low accuracy}, and the fuzzy subset of the sensor operating status is {normal, abnormal, standby}. Multiple sets of fuzzy rules are preset, and the logic of the fuzzy rules is: when the scene prediction result is underwater high confidence / low confidence and... When the underwater positioning mode is set to high / medium accuracy and the underwater sensor is functioning normally, the underwater positioning mode is activated. When the scene prediction result is high / low confidence on the surface and the surface positioning mode is set to high / medium accuracy and the surface sensor is functioning normally, the surface positioning mode is activated. When the sensor function of a certain mode is abnormal, that positioning mode is disabled. When both modes are high accuracy and the scene prediction is at a fuzzy boundary, the positioning mode with higher adaptability is selected based on the device's movement trend. Finally, the defuzzification process is completed using the center of gravity method, and a clear underwater / surface positioning mode selection command is output.

[0010] The present invention also provides a dynamic 3DGS driven cross-media cooperative positioning system for backhoe dredgers, the positioning system comprising: a multimodal perception module, an error correction module, a data fusion module, a scene prediction and decision-making module, and a cooperative positioning module; The multimodal perception module includes an underwater scanning profile sonar, an underwater binocular vision camera, a surface lidar, and a surface binocular vision camera, used to collect multimodal perception data of the underwater-surface operation space; The error correction module collects motion attitude, speed and acceleration data of the backhoe dredger through IMU sensors, thereby providing multi-modal sensor position fluctuation compensation information; the error correction module is used for IMU zero drift correction, and obtains the reference position deviation of the multi-modal sensor through the corrected IMU, thereby combining the graph optimization framework and Gauss-Newton iteration method to complete the joint correction of multi-source errors; The data fusion module is used to construct underwater scene dynamic 3DGS adaptation models and above-water scene dynamic 3DGS adaptation models based on dynamic 3DGS technology, build a multimodal tightly coupled positioning framework, and complete the deep fusion and conflict resolution of heterogeneous data. The scenario prediction and decision module is used to build an autonomous prediction model for underwater operation scenarios and an autonomous prediction model for surface operation scenarios based on the Transformer network, and to achieve adaptive selection of underwater positioning mode and surface positioning mode by combining fuzzy logic decision algorithm. The collaborative positioning module is used to establish a coordinate unification mechanism for underwater-surface positioning data, integrate dual-scene positioning results and mode selection instructions, and output collaborative positioning data for the entire operational space.

[0011] Furthermore, according to the dynamic 3DGS-driven cross-media cooperative positioning system for backhoe dredgers provided by the present invention, the optimization parameters of the underwater scene dynamic 3DGS adaptation model constructed in the data fusion module include point cloud sampling density, Gaussian kernel function width, and update frequency, in order to adapt to underwater sound and light propagation characteristics; the optimization parameters of the surface scene dynamic 3DGS adaptation model include feature extraction threshold and Gaussian kernel function center coordinate update rules, in order to adapt to surface light propagation characteristics. The multimodal tightly coupled positioning framework relies on IMU pre-integration constraints to provide motion prior information, uses a feature-level fusion method to extract key features of each modality's data, and resolves heterogeneous data conflicts through Euclidean distance matching and feature similarity screening.

[0012] Furthermore, according to the dynamic 3DGS-driven cross-media collaborative positioning system for backhoe dredgers provided by the present invention, the autonomous prediction model of the operation scenario in the scenario prediction and decision module takes the main pump pressure data, pilot control signal data, equipment motion attitude data, and environmental perception data of the backhoe operation system as input features, and extracts spatiotemporal correlation features through the multi-head attention mechanism of the Transformer network to achieve binary classification prediction of underwater and above-water operation scenarios. The fuzzy logic decision algorithm takes the scene prediction results, the real-time positioning accuracy of each positioning mode, and the sensor working status as input variables. Through preset fuzzy rules and defuzzification methods, it outputs an adaptive selection instruction for underwater positioning mode and surface positioning mode.

[0013] Furthermore, according to the dynamic 3DGS-driven cross-media cooperative positioning system for backhoe dredgers provided by the present invention, the specific logic of adaptive selection is as follows: taking the scene prediction result, the real-time positioning accuracy of each positioning mode, and the sensor working status as three input variables, each input variable is divided into different fuzzy subsets. The fuzzy subset of the scene prediction result is {underwater high confidence, underwater low confidence, surface high confidence, surface low confidence}, the fuzzy subset of the real-time positioning accuracy is {high accuracy, medium accuracy, low accuracy}, and the fuzzy subset of the sensor working status is {normal, abnormal, standby}; and multiple sets of fuzzy rules are preset, the logic of which is: when the... When the scene prediction result is high / low confidence underwater, the underwater positioning mode is high / medium accuracy, and the underwater sensor is functioning normally, the underwater positioning mode is activated. When the scene prediction result is high / low confidence above water, the above water positioning mode is high / medium accuracy, and the above water sensor is functioning normally, the above water positioning mode is activated. When the sensor functioning of a certain mode is abnormal, that positioning mode is disabled. When both modes are high accuracy and the scene prediction is at a fuzzy boundary, the positioning mode with higher adaptability is selected based on the device's movement trend. Finally, the defuzzification process is completed using the center of gravity method, and a clear underwater / above water positioning mode selection command is output.

[0014] Furthermore, according to the dynamic 3DGS-driven cross-media collaborative positioning system for backhoe dredgers provided by the present invention, the collaborative positioning module establishes a unified coordinate transformation based on the WGS84 geographic coordinate system, transforming underwater and surface positioning data to the same reference; and eliminates abrupt errors in positioning data through time series smoothing processing to achieve continuous positioning across the entire domain.

[0015] The beneficial effects of this invention are as follows: The dynamic 3DGS-driven cross-media cooperative positioning method and system for backhoe dredgers provided by this invention constructs a non-general-purpose, customized underwater and surface 3DGS adaptation model based on dynamic 3DGS technology. This model has an innovative structure, designing differentiated Gaussian splash characterization rules for the characteristics of underwater and surface media. Relying on IMU pre-integration constraints, it achieves deep fusion and conflict resolution of multimodal data through a standardized process of temporal and spatial synchronization, feature extraction, matching and filtering, and fusion characterization. This solves the problems of low positioning accuracy of single sensors and difficulty in heterogeneous data fusion, significantly improving the positioning accuracy in dual scenarios; and it also proposes to address IMU error... The differential residual function and the clearly defined IMU drift online correction algorithm, combined with the graph optimization framework and the Gauss-Newton iterative method, complete the joint correction of multi-source errors, effectively compensating for sensor reference drift errors caused by water surface disturbances, and improving the stability and anti-interference capability of the positioning system. Based on the Transformer network, a binary classification prediction model for underwater and surface operation scenarios is constructed. Combined with the fuzzy logic decision algorithm, the specific selection logic of input variable fuzzification, preset fuzzy rules, and defuzzification processing realizes the adaptive and accurate selection of positioning mode, solving the problem that the fixed positioning mode cannot adapt to different underwater / surface scenarios, and greatly improving the scenario adaptability of the positioning system. Attached Figure Description

[0016] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating the cross-media collaborative positioning method for a dynamic 3DGS-driven backhoe dredger provided in this embodiment.

[0018] Figure 2 This is a schematic diagram of the structure of the dynamic 3DGS driven cross-media cooperative positioning system for backhoe dredgers provided in this embodiment.

[0019] The components in the diagram are labeled as follows: Multimodal perception module 1, Error correction module 2, Data fusion module 3, Scene prediction and decision-making module 4, Cooperative localization module 5, Data storage module 6, Communication module 7; Underwater scanning profile sonar 11, Underwater binocular vision camera 12, Surface lidar 13, Surface binocular vision camera 14, IMU 15. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] The following disclosure provides many different embodiments or examples for implementing different structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. In addition, various specific examples of processes and materials are provided in this application, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0023] The embodiments of this application will now be further described in conjunction with the accompanying drawings and specific implementation details.

[0024] Figure 1 This is a flowchart illustrating the cross-media collaborative positioning method for a dynamic 3DGS-driven backhoe dredger provided in this embodiment.

[0025] like Figure 1 As shown, the dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers includes: Step S1: Perform multimodal data acquisition, including underwater scanning profile sonar data, underwater binocular vision data, surface lidar data, and surface binocular vision data, and acquire equipment motion data through the IMU module; Specifically, the multimodal perception module synchronously collects bucket excavation contour data from underwater scanning profile sonar, underwater terrain texture data from underwater binocular vision, point cloud data around the hull from surface lidar, and relative position data of the mud barge from surface binocular vision. The IMU module collects hull motion attitude, velocity, and acceleration data. After preprocessing (filtering and denoising), the data is transmitted to the data fusion module.

[0026] Step S2: Construct and fuse dynamic 3DGS models for both underwater and above-water scenes. Based on dynamic 3DGS technology, construct dynamic 3DGS adaptation models for underwater and above-water scenes respectively. Build a multimodal tightly coupled positioning framework based on IMU pre-integration constraints to complete deep fusion and conflict resolution of heterogeneous data and optimize the adaptive rules for model parameters. Specifically, customized 3DGS adaptation models for underwater and surface water were constructed, key features of each modal data were extracted, equipment motion offset was compensated through time and space synchronization, Euclidean distance between feature points was calculated and feature pairs less than 0.08m were selected, and erroneous matching pairs with similarity less than 0.8 were eliminated to achieve heterogeneous data conflict resolution and deep fusion. The model parameters were dynamically optimized based on underwater turbidity value (when turbidity > 50 NTU, the width of the Gaussian kernel function was adjusted to 0.12) and surface light intensity (when light intensity < 5000 lux, the feature extraction threshold was adjusted to 0.75).

[0027] Specifically, in this embodiment, the customized underwater and above-water 3DGS adaptation model is an innovative model structure. The underwater scene dynamic 3DGS adaptation model is based on a three-dimensional Gaussian kernel function to construct a Gaussian splash representation model of point cloud features, with the following structure: and ;in, The coordinates of the underwater point. The coordinates of feature points acquired by underwater sonar. The coordinates of feature points acquired by underwater binocular vision. , For Gaussian kernel weights, The width of the Gaussian kernel function. It is a 3-order identity matrix. The underwater sound propagation attenuation coefficient is... The underwater light coefficient; The structure of the dynamic 3DGS adaptation model for the water scene is as follows: ;in, The coordinates of the underwater point. The coordinates of the feature points acquired by LiDAR / vision. For water surface light coefficient, The model uses Gaussian kernel weights. Its innovation lies in designing differentiated Gaussian splash characterization rules for the physical properties of different media, both underwater and above water. It introduces media characteristic coefficients to achieve dynamic adaptation between the model and the operating environment, breaking through the limitations of the generalized design of traditional 3DGS models.

[0028] Specifically, in this embodiment, the deep fusion and conflict resolution of heterogeneous data are as follows: First, based on the IMU pre-integration constraints, prior information on device motion is obtained, and motion compensation and time synchronization are performed on the original data of each modality; second, a feature-level fusion method is used to extract the contour features of underwater scanning profile sonar, the texture features of underwater binocular vision, the 3D point cloud features of surface lidar, and the image features of surface binocular vision, respectively; then, the matching degree between feature points is calculated through Euclidean distance, and features with Euclidean distance less than a set threshold are selected. d th For each feature pair, the similarity between the feature pairs is calculated, and features with similarity below a threshold are discarded. s th The incorrect matching pairs are identified; finally, the selected valid feature pairs are input into the customized 3DGS adaptation model, and the heterogeneous features are fused and represented by superposition and optimization of Gaussian kernel functions, thus completing the resolution of heterogeneous data conflicts.

[0029] The adaptive rules for the optimized model parameters include: the underwater scene dynamic 3DGS adaptation model adjusts the width of the Gaussian kernel function according to the underwater turbidity value, the above-water scene dynamic 3DGS model adjusts the feature extraction threshold according to the above-water light intensity, and the model update frequency is dynamically adapted to the device movement speed.

[0030] Step S3: Perform IMU drift correction and multi-source error joint correction, construct the IMU error residual function, compensate the reference drift error through the online IMU drift correction algorithm, and complete the multi-source error joint correction by combining the graph optimization framework and the Gauss-Newton iteration method. The IMU drift online correction algorithm includes real-time correction of its output data based on the statistical results of IMU static zero drift, and characterization of the reference position deviation of multimodal sensors through the corrected IMU sensor data. The graph optimization framework incorporates multimodal sensor errors into a unified optimization objective and completes joint correction of multi-source errors through the Gauss-Newton iteration method. The Gauss-Newton iterative method is set to 15-25 iterations, and the iteration convergence threshold is set to 0.001-0.01m according to the positioning accuracy requirements to ensure the accuracy and efficiency of multi-source error correction.

[0031] Step S4: Perform operation scenario prediction and positioning mode decision-making. Based on the Transformer network, construct an autonomous prediction model for underwater operation scenarios and an autonomous prediction model for surface operation scenarios. Combine the fuzzy logic decision-making algorithm to output adaptive selection instructions for underwater positioning mode and surface positioning mode. The scenario prediction and decision-making module's autonomous prediction model for the operation scenario takes the main pump pressure data, pilot control signal data, equipment motion posture data, and environmental perception data of the backhoe operation system as input features. It extracts spatiotemporal correlation features through the multi-head attention mechanism of the Transformer network to achieve binary classification prediction of underwater and above-water operation scenarios. The fuzzy logic decision algorithm takes the scene prediction results, the real-time positioning accuracy of each positioning mode, and the sensor working status as input variables. Through preset fuzzy rules and defuzzification methods, it outputs an adaptive selection instruction for underwater positioning mode and surface positioning mode.

[0032] Specifically, in this embodiment, the operation scenarios are divided into underwater excavation operation scenarios and above-water operation scenarios, which are above-water docking of mud barges. The classification accuracy of the scenario prediction model is not less than 95%. The specific logic of adaptive selection is as follows: Using scene prediction results, real-time positioning accuracy of each positioning mode, and sensor operating status as three input variables, each input variable is divided into different fuzzy subsets. The fuzzy subset of the scene prediction results is {underwater high confidence, underwater low confidence, surface high confidence, surface low confidence}, the fuzzy subset of the real-time positioning accuracy is {high accuracy, medium accuracy, low accuracy}, and the fuzzy subset of the sensor operating status is {normal, abnormal, standby}. Multiple sets of fuzzy rules are preset, and the logic of the fuzzy rules is: when the scene prediction result is underwater high confidence / low confidence and... When the underwater positioning mode is set to high / medium accuracy and the underwater sensor is functioning normally, the underwater positioning mode is activated. When the scene prediction result is high / low confidence on the surface and the surface positioning mode is set to high / medium accuracy and the surface sensor is functioning normally, the surface positioning mode is activated. When the sensor function of a certain mode is abnormal, that positioning mode is disabled. When both modes are high accuracy and the scene prediction is at a fuzzy boundary, the positioning mode with higher adaptability is selected based on the device's movement trend. Finally, the defuzzification process is completed using the center of gravity method, and a clear underwater / surface positioning mode selection command is output.

[0033] Step S5: Perform cross-media collaborative positioning. The collaborative positioning module unifies the positioning data coordinates and smooths the time series, and outputs the collaborative positioning results of the entire operation space.

[0034] Specifically, in this embodiment, underwater and surface positioning data are converted to the WGS84 geographic coordinate system to achieve coordinate unification. The positioning data is smoothed using the 5-point moving average method, and a collaborative positioning result with a positioning accuracy of ±0.08m is output, providing precise positioning support for intelligent dredging operations of backhoe dredgers.

[0035] Figure 2 This is a schematic diagram of the structure of the dynamic 3DGS driven cross-media cooperative positioning system for backhoe dredgers provided in this embodiment.

[0036] like Figure 2 As shown, the positioning system includes: a multimodal perception module, an error correction module, a data fusion module, a scene prediction and decision-making module, and a collaborative positioning module; The multimodal perception module includes an underwater scanning profile sonar, an underwater binocular vision camera, a surface lidar, and a surface binocular vision camera, used to collect multimodal perception data of the underwater-surface operation space; The error correction module collects motion attitude, speed and acceleration data of the backhoe dredger through IMU sensors, thereby providing multi-modal sensor position fluctuation compensation information; the error correction module is used for IMU zero drift correction, and obtains the reference position deviation of the multi-modal sensor through the corrected IMU, thereby combining the graph optimization framework and Gauss-Newton iteration method to complete the joint correction of multi-source errors; The data fusion module is used to construct underwater scene dynamic 3DGS adaptation models and above-water scene dynamic 3DGS adaptation models based on dynamic 3DGS technology, build a multimodal tightly coupled positioning framework, and complete the deep fusion and conflict resolution of heterogeneous data. The scenario prediction and decision module is used to build an autonomous prediction model for underwater operation scenarios and an autonomous prediction model for surface operation scenarios based on the Transformer network, and to achieve adaptive selection of underwater positioning mode and surface positioning mode by combining fuzzy logic decision algorithm. The collaborative positioning module is used to establish a coordinate unification mechanism for underwater-surface positioning data, integrate dual-scene positioning results and mode selection instructions, and output collaborative positioning data for the entire operational space.

[0037] Specifically, in the multimodal perception module, an underwater scanning profile sonar and an underwater binocular vision camera are installed at the front end of the bucket to collect sonar contour data and visual texture data of the underwater working space; a surface lidar and a surface binocular vision camera are installed at the excavation operating room to collect point cloud data and visual image data of the surface working space. The acquisition frequency of each perception device is kept synchronized and matched with the acquisition frequency of the IMU module (not less than 10Hz).

[0038] The parameters of each component in the multimodal sensing module are as follows: underwater scanning profile sonar range 0-50m, resolution 0.01m; underwater binocular vision camera waterproof rating IP68, resolution 1920×1080; surface lidar range 0-100m, point cloud frequency 100,000 points / second; surface binocular vision camera resolution 1920×1080, and the acquisition frequency of each device is set to 20Hz.

[0039] In the error correction module, the output data is corrected in real time using the IMU static zero drift statistics. The corrected IMU sensor data is used to characterize the reference position deviation of the multimodal sensor. The graph optimization framework incorporates the multimodal sensor error into a unified optimization objective. The Gauss-Newton iteration method (15-25 iterations, convergence threshold 0.001-0.01m) is used to complete the joint correction of multi-source errors and compensate for the sensor reference drift error.

[0040] The error correction module uses a high-precision inertial navigation IMU, with gyroscope zero bias ≤0.01° / h, accelerometer zero bias ≤10μg, acquisition frequency 20Hz, Gauss-Newton iteration number set to 20 times, and convergence threshold 0.005m.

[0041] The optimization parameters of the underwater scene dynamic 3DGS adaptation model constructed in the data fusion module include point cloud sampling density, Gaussian kernel function width, and update frequency to adapt to the underwater sound and light propagation characteristics; the optimization parameters of the surface scene dynamic 3DGS adaptation model include feature extraction threshold and Gaussian kernel function center coordinate update rules to adapt to the surface light propagation characteristics; the multimodal tightly coupled positioning framework relies on IMU pre-integration constraints to provide motion prior information, uses feature-level fusion to extract key features of each modality data, and achieves heterogeneous data conflict resolution through Euclidean distance matching and feature similarity screening.

[0042] Specifically, in this embodiment, the core function of the data fusion module is to construct customized underwater and surface 3DGS adaptation models and build a multimodal tightly coupled positioning framework. This customized 3DGS adaptation model is an innovative structure, distinct from traditional general-purpose 3DGS models. It is designed differently for the physical characteristics of different underwater and surface media. The underwater scene dynamic 3DGS adaptation model constructs a Gaussian splash characterization model of point cloud features, with the following structure: and ;in, The coordinates of the underwater point. The coordinates of feature points acquired by underwater sonar. The coordinates of feature points acquired by underwater binocular vision. , For Gaussian kernel weights, The width of the Gaussian kernel function. It is a 3-order identity matrix. The underwater sound propagation attenuation coefficient is... The underwater light coefficient; The structure of the dynamic 3DGS adaptation model for the water scene is as follows: ;in, The coordinates of the underwater point. The coordinates of the feature points acquired by LiDAR / vision. This is the water surface light coefficient, which is dynamically adjusted according to the light intensity. The weights are Gaussian kernel weights.

[0043] In this embodiment, the sampling density of the point cloud of the underwater scene dynamic 3DGS adaptation model is 0.05m / point, the width of the Gaussian kernel function is 0.1, and the initial update frequency is 10Hz. and To dynamically adjust based on underwater turbidity, values ​​of 0.8 and 0.6 are used when turbidity > 50 NTU, and values ​​of 1.0 and 0.8 are used when turbidity ≤ 50 NTU.

[0044] The sampling density of the point cloud of the water scene model is 0.03m / point, the width of the Gaussian kernel function is 0.08, and the initial update frequency is 20Hz. The threshold is dynamically adjusted based on light intensity: 0.9 for light intensity < 5000 lux and 1.0 for light intensity ≥ 5000 lux; the feature similarity threshold is set to 0.8, and the Euclidean distance matching threshold is... d th Set it to 0.08m.

[0045] To address the underwater sound propagation attenuation characteristics, the point cloud sampling density (0.04-0.06 m / point), Gaussian kernel width (0.08-0.12), and update frequency (8-12 Hz) of the dynamic 3DGS model were optimized. To address the water surface light variation characteristics, the feature extraction threshold (0.7-0.9), Gaussian kernel center coordinate update rule, and update frequency (15-25 Hz) of the model were optimized.

[0046] The process for deep fusion and conflict resolution of heterogeneous data is as follows: 1. Time and space synchronization: Time alignment of sensing data of each modality is performed based on the timestamp of the IMU, and spatial coordinate compensation is performed on the data collected by each sensor according to the motion pre-integration result of the IMU to eliminate feature point offset caused by device motion. 2. Feature extraction: Feature-level fusion is used to extract the contour features of underwater scanning profile sonar, the SIFT texture features of underwater binocular vision, the FPFH 3D point cloud features of surface lidar, and the ORB image features of surface binocular vision. 3. Feature matching and filtering: Calculate the Euclidean distance between feature points and filter those with an Euclidean distance less than a set threshold.d th Feature pairs (with values ​​ranging from 0.05 to 0.1m) are selected, and their similarity is calculated using a cosine similarity algorithm. Pairs with similarity values ​​below a threshold are discarded. s th (Values ​​range from 0.75 to 0.85) of incorrect matching pairs, achieving initial conflict resolution of heterogeneous data; 4. Feature fusion representation: The selected effective feature pairs are input into the customized 3DGS adaptation model for the corresponding scene. Through the superposition of Gaussian kernel functions and weight optimization, Gaussian splash fusion representation of heterogeneous features is achieved, completing the deep fusion of heterogeneous data and simultaneously optimizing the adaptive rules of model parameters.

[0047] In this embodiment, in the scenario prediction and decision-making module, an autonomous prediction model for underwater and above-water operation scenarios is constructed based on a Transformer network. The model uses the main pump pressure data (0-31.5MPa), pilot control signal data (0-5V), equipment motion attitude data, and environmental perception data of the backhoe system as input features. A multi-head attention mechanism is used to extract spatiotemporal correlation features, achieving binary classification prediction of underwater and above-water operation scenarios (accuracy not less than 95%). Combined with a fuzzy logic decision-making algorithm, the model uses the scenario prediction result, the real-time positioning accuracy of each positioning mode, and the sensor operating status as three core input variables. Through preset fuzzy rules and defuzzification methods, an adaptive selection command for the underwater and above-water positioning modes is output.

[0048] The specific selection logic is as follows: 1. Input variables are fuzzified, and the scene prediction results are divided into four fuzzy subsets: {underwater high confidence, underwater low confidence, surface high confidence, surface low confidence}. A confidence level ≥ 90% is considered high confidence, and 60% ≤ confidence level < 90% is considered low confidence. The real-time positioning accuracy of each positioning mode is divided into three fuzzy subsets: {high accuracy (positioning error ≤ 0.1m), medium accuracy (0.1m < positioning error ≤ 0.3m), and low accuracy (positioning error > 0.3m)}. The sensor operating status is also considered. The system is divided into three fuzzy subsets: {Normal (no fault, stable data acquisition), Abnormal (fault alarm, data loss), Standby (not started)}. Triangular membership functions are used to fuzzify each input variable. 2. Preset fuzzy rules: Multiple sets of fuzzy rules are preset based on the adaptability of the operating scenario and positioning mode. Core rules include: Rule 1: If the scenario prediction is underwater with high confidence and the underwater positioning mode is high / medium precision and the underwater sensor is functioning normally, then the underwater positioning mode is activated; Rule 2: If... If the scene prediction is underwater with low confidence, the underwater positioning mode is high precision, and the underwater sensor is functioning normally, then the underwater positioning mode is enabled. Rule 3: If the scene prediction is above water with high confidence, the above water positioning mode is high / medium precision, and the above water sensor is functioning normally, then the above water positioning mode is enabled. Rule 4: If the scene prediction is above water with low confidence, the above water positioning mode is high precision, and the above water sensor is functioning normally, then the above water positioning mode is enabled. Rule 5: If the underwater sensor is in an abnormal / standby state, then the underwater positioning mode is disabled. Rule 6: If the above water sensor is in an abnormal / standby state, then the above water positioning mode is disabled. Rule 7: If both modes are high precision and the scene prediction is at a fuzzy boundary (underwater and above water confidence difference < 10%), then the positioning mode with higher adaptability is selected based on the equipment movement trend (dredging / moving the boat). 3. Defuzzification: The centroid method is used to defuzzify the inference results of the fuzzy rules, outputting a clear, non-fuzzy underwater / above water positioning mode selection command.

[0049] In the collaborative positioning module, a coordinate unified transformation mechanism based on the WGS84 geographic coordinate system is established to transform underwater and above-water positioning data to the same reference. The moving average method (window size 3-8 data points) is used to smooth the positioning data over time to eliminate random errors and sudden interference, and output continuous and high-precision collaborative positioning results for the entire operating space.

[0050] like Figure 2As shown, the dynamic 3DGS-driven cross-media collaborative positioning system for backhoe dredgers also includes a data storage module and a communication module. The data storage module uses an industrial-grade solid-state drive to classify and store multimodal sensing data, IMU data, positioning result data, and scene prediction data, and supports historical data query. The communication module uses a combination of industrial Ethernet and CAN bus to realize data interaction between modules and communication with the dredger's main control system, ensuring the real-time performance and stability of data transmission (transmission delay ≤50ms).

[0051] This invention provides a dynamic 3DGS-driven cross-media collaborative positioning method and system for backhoe dredgers. Based on dynamic 3DGS technology, it constructs a non-general-purpose, customized underwater and surface 3DGS adaptation model. This model features an innovative structure, designing differentiated Gaussian splash characterization rules for the characteristics of underwater and surface media. Relying on IMU pre-integration constraints, it achieves deep fusion and conflict resolution of multimodal data through a standardized process of temporal and spatial synchronization, feature extraction, matching and filtering, and fusion characterization. This solves the problems of low positioning accuracy from a single sensor and difficulty in fusion of heterogeneous data, significantly improving the positioning accuracy in dual-scene scenarios. Furthermore, it proposes an IMU error residual function. A clear IMU drift online correction algorithm, combined with a graph optimization framework and the Gauss-Newton iterative method, achieves joint correction of multi-source errors, effectively compensating for sensor reference drift errors caused by water surface disturbances and improving the stability and anti-interference capability of the positioning system. A binary classification prediction model for underwater and surface operation scenarios is constructed based on a Transformer network. Combined with a fuzzy logic decision algorithm, the system achieves adaptive and accurate selection of the positioning mode through specific selection logic of input variable fuzzification, preset fuzzy rules, and defuzzification processing. This solves the problem that fixed positioning modes cannot adapt to different underwater / surface scenarios, significantly improving the scenario adaptability of the positioning system. Simultaneously, by unifying the coordinates and smoothing the time series of underwater and surface positioning data, a cross-media collaborative positioning system is constructed, achieving high-precision, continuous positioning across the entire underwater-surface operation space. This provides core technical support for intelligent dredging operations of backhoe dredgers and meets the needs of intelligent upgrades for marine engineering equipment.

[0052] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the present invention. Finally, it should be noted that in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0053] The foregoing has provided a detailed description of a dynamic 3DGS-driven cross-media cooperative positioning method and system for backhoe dredgers provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers, characterized in that, The method includes: Multimodal data acquisition is performed, including underwater scanning profile sonar data, underwater binocular vision data, surface lidar data, and surface binocular vision data. Equipment motion data is also acquired through the IMU module. We constructed and fused dynamic 3DGS models for both underwater and above-water scenarios. Based on dynamic 3DGS technology, we built dynamic 3DGS adaptation models for underwater and above-water scenarios respectively. We built a multimodal tightly coupled positioning framework based on IMU pre-integration constraints, completed deep fusion and conflict resolution of heterogeneous data, and optimized the adaptive rules for model parameters. IMU drift correction and multi-source error joint correction are performed. An IMU error residual function is constructed. The baseline drift error is compensated by the online IMU drift correction algorithm. The multi-source error joint correction is completed by combining the graph optimization framework and the Gauss-Newton iteration method. The system performs operation scenario prediction and positioning mode decision-making. It constructs an autonomous prediction model for underwater operation scenarios and an autonomous prediction model for surface operation scenarios based on the Transformer network, and outputs adaptive selection instructions for underwater positioning mode and surface positioning mode by combining fuzzy logic decision-making algorithm. Cross-media collaborative positioning is performed. The collaborative positioning module realizes the unification of positioning data coordinates and smoothing of time series, and outputs the collaborative positioning results of the entire operation space.

2. The dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers according to claim 1, characterized in that, in, The underwater scene dynamic 3DGS adaptation model is based on a three-dimensional Gaussian kernel function to construct a Gaussian splash representation model of point cloud features, with the following structure: and ;in, The coordinates of the underwater point. The coordinates of feature points acquired by underwater sonar. The coordinates of feature points acquired by underwater binocular vision. , For Gaussian kernel weights, The width of the Gaussian kernel function. It is a 3-order identity matrix. The underwater sound propagation attenuation coefficient is... The underwater light coefficient; The structure of the dynamic 3DGS adaptation model for the water scene is as follows: ;in, The coordinates of the underwater point. The coordinates of the feature points acquired by LiDAR / vision. For water surface light coefficient, The weights are Gaussian kernel weights.

3. The dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers according to claim 2, characterized in that, The adaptive rules for the optimized model parameters include: the underwater scene dynamic 3DGS adaptation model adjusts the width of the Gaussian kernel function according to the underwater turbidity value, the above-water scene dynamic 3DGS model adjusts the feature extraction threshold according to the above-water light intensity, and the model update frequency is dynamically adapted to the device movement speed.

4. The dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers according to claim 1, characterized in that, The IMU drift online correction algorithm includes real-time correction of its output data based on the statistical results of IMU static zero drift, and characterization of the reference position deviation of multimodal sensors through the corrected IMU sensor data. The graph optimization framework incorporates multimodal sensor errors into a unified optimization objective and completes joint correction of multi-source errors through the Gauss-Newton iteration method. The Gauss-Newton iterative method is set to 15-25 iterations, and the iteration convergence threshold is set to 0.001-0.01m according to the positioning accuracy requirements to ensure the accuracy and efficiency of multi-source error correction.

5. The dynamic 3DGS-driven cross-media cooperative positioning method for backhoe dredgers according to claim 1, characterized in that, The operational scenarios are divided into underwater excavation scenarios and above-water operation scenarios, which are above-water docking of mud barges. The classification accuracy of the scenario prediction model is no less than 95%. The specific logic of adaptive selection is as follows: Using scene prediction results, real-time positioning accuracy of each positioning mode, and sensor operating status as three input variables, each input variable is divided into different fuzzy subsets. The fuzzy subset of the scene prediction results is {underwater high confidence, underwater low confidence, surface high confidence, surface low confidence}, the fuzzy subset of the real-time positioning accuracy is {high accuracy, medium accuracy, low accuracy}, and the fuzzy subset of the sensor operating status is {normal, abnormal, standby}. Multiple sets of fuzzy rules are preset, and the logic of the fuzzy rules is: when the scene prediction result is underwater high confidence / low confidence and... When the underwater positioning mode is set to high / medium accuracy and the underwater sensor is functioning normally, the underwater positioning mode is activated. When the scene prediction result is high / low confidence on the surface and the surface positioning mode is set to high / medium accuracy and the surface sensor is functioning normally, the surface positioning mode is activated. When the sensor function of a certain mode is abnormal, that positioning mode is disabled. When both modes are high accuracy and the scene prediction is at a fuzzy boundary, the positioning mode with higher adaptability is selected based on the device's movement trend. Finally, the defuzzification process is completed using the center of gravity method, and a clear underwater / surface positioning mode selection command is output.

6. A dynamic 3DGS-driven cross-media cooperative positioning system for a backhoe dredger, characterized in that, The positioning system includes: a multimodal perception module, an error correction module, a data fusion module, a scene prediction and decision-making module, and a collaborative positioning module; The multimodal perception module includes an underwater scanning profile sonar, an underwater binocular vision camera, a surface lidar, and a surface binocular vision camera, used to collect multimodal perception data of the underwater-surface operation space; The error correction module collects motion attitude, speed and acceleration data of the backhoe dredger through IMU sensors, thereby providing multi-modal sensor position fluctuation compensation information; the error correction module is used for IMU zero drift correction, and obtains the reference position deviation of the multi-modal sensor through the corrected IMU, thereby combining the graph optimization framework and Gauss-Newton iteration method to complete the joint correction of multi-source errors; The data fusion module is used to construct underwater scene dynamic 3DGS adaptation models and above-water scene dynamic 3DGS adaptation models based on dynamic 3DGS technology, build a multimodal tightly coupled positioning framework, and complete the deep fusion and conflict resolution of heterogeneous data. The scenario prediction and decision module is used to build an autonomous prediction model for underwater operation scenarios and an autonomous prediction model for surface operation scenarios based on the Transformer network, and to achieve adaptive selection of underwater positioning mode and surface positioning mode by combining fuzzy logic decision algorithm. The collaborative positioning module is used to establish a coordinate unification mechanism for underwater-surface positioning data, integrate dual-scene positioning results and mode selection instructions, and output collaborative positioning data for the entire operational space.

7. The dynamic 3DGS-driven cross-media cooperative positioning system for backhoe dredgers according to claim 6, characterized in that, The optimization parameters of the underwater scene dynamic 3DGS adaptation model constructed in the data fusion module include point cloud sampling density, Gaussian kernel function width and update frequency, in order to adapt to the underwater sound and light propagation characteristics; the optimization parameters of the surface scene dynamic 3DGS adaptation model include feature extraction threshold and Gaussian kernel function center coordinate update rules, in order to adapt to the surface light propagation characteristics. The multimodal tightly coupled positioning framework relies on IMU pre-integration constraints to provide motion prior information, uses a feature-level fusion method to extract key features of each modality's data, and resolves heterogeneous data conflicts through Euclidean distance matching and feature similarity screening.

8. The dynamic 3DGS-driven cross-media cooperative positioning system for backhoe dredgers according to claim 6, characterized in that, The autonomous prediction model for the operation scenario in the scenario prediction and decision-making module takes the main pump pressure data, pilot control signal data, equipment motion posture data, and environmental perception data of the backhoe operation system as input features, and extracts spatiotemporal correlation features through the multi-head attention mechanism of the Transformer network to achieve binary classification prediction of underwater and above-water operation scenarios. The fuzzy logic decision algorithm takes the scene prediction results, the real-time positioning accuracy of each positioning mode, and the sensor working status as input variables. Through preset fuzzy rules and defuzzification methods, it outputs an adaptive selection instruction for underwater positioning mode and surface positioning mode.

9. The dynamic 3DGS-driven cross-media cooperative positioning system for backhoe dredgers according to claim 8, characterized in that, The specific logic of adaptive selection is as follows: taking the scene prediction result, the real-time positioning accuracy of each positioning mode, and the sensor working status as three input variables, each input variable is divided into different fuzzy subsets. The fuzzy subset of the scene prediction result is {underwater high confidence, underwater low confidence, surface high confidence, surface low confidence}, the fuzzy subset of the real-time positioning accuracy is {high accuracy, medium accuracy, low accuracy}, and the fuzzy subset of the sensor working status is {normal, abnormal, standby}. Multiple sets of fuzzy rules are preset. The logic of the fuzzy rules is as follows: when the scene prediction result is high confidence / low confidence underwater and the underwater positioning mode is high precision / medium precision, and the underwater sensor is working normally, the underwater positioning mode is enabled; when the scene prediction result is high confidence / low confidence above water and the above water positioning mode is high precision / medium precision, and the above water sensor is working normally, the above water positioning mode is enabled; when the sensor working state of a certain mode is abnormal, the positioning mode is disabled; when both modes are high precision and the scene prediction is at a fuzzy boundary, the positioning mode with higher adaptability is selected according to the device's movement trend; finally, the defuzzification process is completed using the center of gravity method, and a clear underwater / above water positioning mode selection command is output.

10. The dynamic 3DGS-driven cross-media cooperative positioning system for backhoe dredgers according to claim 6, characterized in that, The collaborative positioning module establishes a unified coordinate transformation based on the WGS84 geographic coordinate system, converting underwater and surface positioning data to the same reference; it eliminates abrupt errors in positioning data through time series smoothing processing, achieving continuous positioning across the entire domain.

Images