Underwater robot simulation training method and system based on visual and sonar data

Abstract

The application discloses a simulation training method and system for an underwater robot based on visual and sonar data, models an underwater operation scene, simulates kinematic operation logic of an underwater robot and an operation panel, and configures physical parameters of an underwater camera and a three-dimensional sonar sensor; a generative adversarial network is used to inject deep-sea backscattering and light attenuation features into a visual image, and environmental bottom noise and speckle noise are injected into sonar echo intensity in a bottom acoustic signal rendering pipeline; an underwater robot in artificial teleoperation simulation searches and interacts with the operation panel, an asynchronous alignment mechanism is used to record synchronous multi-modal sensor data and artificial instructions to build a data set; the data set is used to build global multi-modal conditional representation through space-time alignment and feature splicing, and an end-to-end action generation network is trained by using a diffusion strategy algorithm; in a simulation scene, a trained network is used to output a three-dimensional action trajectory sequence of a robot body and a mechanical arm to drive a bottom control.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence and underwater robot simulation technology, specifically relating to underwater robot simulation training methods and systems based on vision and sonar data. Background Technology

[0002] As the development of offshore oil and gas resources continues to expand into deeper waters, underwater robots have become indispensable core equipment in complex deep-sea engineering projects such as subsea pipeline inspection, subsea production tree panel monitoring, and interactive operations. Currently, such complex underwater precision operations heavily rely on experienced navigators conducting remote manual operation from a surface mother ship. However, remote manual operation is not only limited by the operator's experience but is also prone to errors when facing poor visibility in deep water and fatigue. Therefore, embodied intelligence technology, which endows underwater robots with autonomous perception and operation capabilities, has become a critical technological bottleneck that urgently needs to be overcome in the field of marine robotics.

[0003] The biggest bottleneck in engineering practice for training underwater embodied agents with end-to-end operational capabilities using deep learning and imitation learning is the lack of data. Training large-scale end-to-end embodied models requires massive amounts of multimodal data sequences with precise timestamps (including acoustic and optical perception, pose, depth, force feedback, and maneuvering), which typically needs to be generated by human operators demonstrating on a real machine. However, data acquisition in real deep-sea environments is extremely costly, and underwater environments cannot be physically reset and randomized as quickly as terrestrial environments. Therefore, there is a strong reliance on simulation platforms to generate high-quality teaching data.

[0004] Existing underwater simulation platforms primarily focus on verifying traditional control algorithms or performing physical simulations of single states. Examples include simulations mainly targeting robot dynamics or semi-physical testing systems built to identify software and hardware vulnerabilities. However, these existing technologies have significant shortcomings when it comes to training multimodal embodied intelligence: Firstly, traditional simulations overly idealize the underlying sensors, failing to realistically reproduce optical degradation phenomena and complex noise interference from multibeam sonar in the complex deep-sea environment, resulting in severe inter-domain differences between the generated simulation data and the real deep-sea environment. Secondly, existing platforms lack spatiotemporal alignment and joint representation mechanisms for multimodal perception data, making it impossible to directly output high-quality multimodal teaching datasets that meet the training requirements of embodied intelligence models, thus hindering the closed-loop training and deployment of end-to-end embodied large-scale models. Summary of the Invention

[0005] To address the shortcomings of existing technologies and achieve high-fidelity multimodal physical degradation simulation and spatiotemporal alignment of heterogeneous sensing data, thereby generating simulation teaching data that can be directly used for closed-loop training of end-to-end embodied intelligent models, this invention adopts the following technical solution:

[0006] The underwater robot simulation training method based on visual and sonar data is as follows:

[0007] Model underwater operation scenarios and construct kinematic simulation models of underwater robots;

[0008] By operating underwater robots to obtain operating instructions and motion data, and acquiring visual and sonar data of underwater operation scenarios, a teaching dataset of operating instructions and corresponding multimodal data is constructed.

[0009] From the teaching dataset, multimodal historical observation sequence data over a past period are extracted and fused into multimodal features. A future real joint operation instruction sequence is extracted as a real action trajectory sequence, thus forming a sampling for sequence generation. A heterogeneous data hierarchical storage structure is adopted to construct an expert experience playback buffer for multimodal time series slices. A training batch is constructed through a sliding window sampling strategy, thus forming an expert experience playback for sequence generation.

[0010] End-to-end training of the diffusion model based on trajectory prediction objective function: Construct a diffusion strategy conditional action generation network, and under the guidance of the diffusion time step and the multimodal features, perform multi-step denoising on the noisy action trajectory to obtain the predicted noiseless action trajectory. Construct a loss by comparing the predicted noiseless action trajectory with the future real action trajectory sequence to train the action generation network.

[0011] Furthermore, a pre-trained generative adversarial network is used to inject deep-sea backscattering and light attenuation features into RGB image visual data to synthesize underwater-style image data as visual data.

[0012] By injecting environmental background noise and speckle noise into the sonar data, noisy acoustic data is obtained as sonar data, thereby generating three-dimensional point cloud data with real physical degradation characteristics.

[0013] The difference between the low-frequency timestamps of sonar data and the high-frequency timestamps of visual data is compared with a set time matching tolerance window threshold to perform asynchronous alignment of visual data and sonar data; motion data is generated by operating an underwater robot to construct corresponding motion sequences, thereby forming synchronized operation commands and multimodal data.

[0014] Furthermore, data spatial alignment and multimodal point cloud-level fusion are performed: noisy sonar point cloud data in the local coordinate system is acquired and transformed to the underwater robot body coordinate system to obtain a set of three-dimensional coordinate points; based on the underwater style image, the three-dimensional coordinate points are assigned corresponding visual color attributes, thereby splicing the spatial geometric coordinates and visual color attributes to obtain an audio-visual fusion feature with color information.

[0015] Furthermore, based on the latent space mapping of the three-dimensional feature network, point cloud features are extracted from the sound-light fusion features to obtain one-dimensional sound-light fusion latent space features.

[0016] The ontological state representing the kinematic state of the underwater robot is extracted from the motion data and fused with the acoustic-optical latent space features. The multimodal features are obtained through channel splicing and mapping.

[0017] Furthermore, the diffusion time step is randomly sampled from a uniform distribution and processed by a sinusoidal position encoding and a multilayer perceptron module; random Gaussian noise of equal dimension is sampled from a standard normal distribution, and the random Gaussian noise is injected into the real motion trajectory sequence through a closed-form solution to synthesize a noisy motion trajectory with a specific noise intensity.

[0018] Furthermore, the loss is a composite objective loss function that combines score matching and physical smoothing constraints for the original trajectory; the composite objective loss function includes the sum of a smoothing regularization term with smoothness weights and a mean squared error loss term.

[0019] The mean squared error loss term is constructed by the difference between the predicted noiseless motion trajectory and the future actual motion trajectory.

[0020] The smoothing regularization term is a first-order difference action sequence smoothing regularization term calculated based on the second norm square sum of the differences between adjacent single-step actions in the predicted trajectory.

[0021] Furthermore, in the diffusion strategy conditional action generation network, the multimodal features are represented as a multimodal sequence feature matrix, and the multimodal sequence feature matrix is ​​mapped to a key matrix and a value matrix respectively. Through independent linear mapping weight matrices, the temporal action features are mapped to a query matrix. Based on the cross-attention mechanism, the dynamic attention weights of the action trajectory at each time step to the multimodal environment features are calculated through the query matrix and the key matrix, and then weighted and summed with the value matrix to obtain the conditional injection features.

[0022] The dimensions of the conditionally injected features are aligned back to the number of feature channels of the current temporal action features by using a linear projection layer, and then the residual connection is used to add the original temporal action features at the pixel level to inject the multimodal conditional features into the temporal action features.

[0023] By repeatedly interspersing the aforementioned cross-attention mechanism between downsampling blocks, deep temporal feature processing blocks, and upsampling blocks, the denoising network can extract relevant spatial environment geometric and visual texture features at each local time step of the time axis, thereby generating a joint kinematic trajectory.

[0024] The underwater robot simulation method based on vision and sonar data employs the aforementioned underwater robot simulation training method based on vision and sonar data. The trained construct diffusion strategy conditional action generation network is deployed in a closed-loop simulation scenario. Through the multi-dimensional kinematic command sequence of the underwater robot body and the multi-degree-of-freedom manipulator, the underlying controller is driven to complete the interactive operation.

[0025] The underwater robot simulation training system based on vision and sonar data includes an environment and model construction module, a physical degradation simulation module, a teaching data acquisition module, and an end-to-end model training module. It adopts the underwater robot simulation training method based on vision and sonar data to sequentially execute the construction of the kinematic simulation model of the underwater operation scenario and the underwater robot, the physical degradation simulation of the data, the construction of the teaching dataset, and the construction and training of the diffusion strategy conditional action generation network.

[0026] The underwater robot simulation system based on vision and sonar data includes a simulation training system and a closed-loop deployment and control module. Through the closed-loop deployment and control module, the underwater robot simulation training system based on vision and sonar data deploys the trained construct diffusion strategy conditional action generation network in the closed-loop simulation scene. The underlying controller is driven to complete the interactive operation through the multi-dimensional kinematic command sequence of the underwater robot body and the multi-degree-of-freedom manipulator.

[0027] The advantages and beneficial effects of this invention are as follows:

[0028] This invention uses an adversarial network to transfer the light attenuation and backscattering features of real deep-sea images to forward synthesize degraded visual images. At the same time, it incorporates environmental background noise and speckle measurement noise into the underlying acoustic rendering pipeline by combining the underwater acoustic propagation equation. This overcomes the shortcomings of traditional simulation environment sensor outputs being too idealized, effectively compensates for the inter-domain differences between simulation data and the real deep-sea environment, and greatly improves the realism of simulation teaching data.

[0029] This invention effectively solves the spatiotemporal misalignment problem caused by different sampling frequencies of acousto-optic sensors by setting an asynchronous alignment mechanism with a time-matching tolerance window and stitching together three-dimensional spatial coordinates and color attributes in the body coordinate system. This provides a high-quality data foundation for spatial alignment for the training of subsequent multimodal network models, ensuring the spatiotemporal consistency and high-quality fusion of heterogeneous acousto-optic data.

[0030] This invention integrates the 6-DOF pose of the robot body with the robotic arm's... Each joint command is incorporated into a unified 6+ This architecture constructs a multi-dimensional action space and innovatively introduces a cross-attention mechanism as a multimodal conditional injection module into the denoising backbone network of the diffusion model. This architecture not only enables the agent to accurately extract the geometric and textural features of the environment to adapt to complex obstacle avoidance constraints, but also, through training with a composite loss function including first-order differential smoothing constraints, ensures that the directly generated end-to-end motion trajectories possess extremely high temporal smoothness while conforming to physical constraints. This achieves full-body cooperative and smooth motion generation for underwater robots in highly complex environments. Attached Figure Description

[0031] Figure 1 This is a flowchart of the simulation training method in an embodiment of the present invention.

[0032] Figure 2 This is a flowchart of the simulation method in an embodiment of the present invention.

[0033] Figure 3 This is a flowchart of the fusion of visual and sonar features in an embodiment of the present invention.

[0034] Figure 4 This is a schematic diagram of the structure of the diffusion strategy conditional action generation network in an embodiment of the present invention.

[0035] Figure 5 This is a schematic diagram of the simulation system in this embodiment of the invention. Detailed Implementation

[0036] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0037] To address the challenges of high cost and long cycle time for multimodal data acquisition of underwater robots due to the complex environment of deep water, as well as difficulties in environment reset and randomization, this invention provides an underwater robot simulation training method based on visual and sonar data. This method enables high-fidelity multimodal data-driven end-to-end operational simulation, significantly improving the realism of 3D simulation data and drastically shortening the development and verification cycle of real underwater equipment. Figure 1 As shown, the specific steps include the following:

[0038] Step S1: Model the underwater operation scenario based on the simulation physics engine and construct the kinematic simulation model of the underwater operation robot.

[0039] like Figure 2As shown, in this embodiment, the specific modeling process includes: importing the three-dimensional models of the seabed topography, ROV (Remotely Operated Vehicle), robotic arm, and underwater work panel into the Gazebo physics engine, configuring the model, lighting, and sensor plugins through an SDF file, setting the kinematic collision and interaction logic of the ROV and underwater work panel through the SDF file, and configuring the physical parameters and data transmission frequency of the underwater camera plugin, sonar plugin, coordinate and inertial measurement sensor plugin.

[0040] Preferably, based on the actual ROV equipment parameters, the simulation parameters in this embodiment are configured as follows:

[0041] ROV body parameters: geometric dimensions are 3.2m × 1.8m × 1.8m; thruster configuration consists of 4 horizontal vector thrusters and 4 vertical thrusters; onboard dual robotic arm parameters: a 4-DOF grasping robotic arm on the left and a 6-DOF dexterous working robotic arm on the right. The DH parameters of the left 4-DOF grasping robotic arm are shown in Table 1.

[0042] Table 1. Parameters of the 4-DOF Robotic Arm (DH)

[0043]

[0044] The DH parameter configuration of the 6-DOF robotic arm is shown in Table 2:

[0045] Table 2 Parameter Table for 6-DOF Robotic Arm (DH)

[0046]

[0047] The maximum output torque of the 6-DOF robotic arm is set to 170. .

[0048] The valve rotation setting on the underwater operation panel has a dynamic torque threshold: the static friction over-torque threshold is set to 120. The dynamic friction torque threshold is set to 80. It should be noted that when the end effector torque of a 6-DOF robotic arm is less than 120... At this time, the Gazebo physics engine will determine that the valve remains stationary. Its mechanical limit configuration is as follows: the maximum clockwise / counterclockwise rotation angle is set to 90°, and after exceeding the limit, it will be physically rigidly blocked by a collision.

[0049] For the sensor plugin, the underwater camera plugin has an image resolution of 2 megapixels, a maximum clear imaging distance of 5m, a field of view of 50° horizontal × 40° vertical, and an output frequency of 10Hz; the sonar plugin (preferably the DaveProject plugin) has a transmission frequency of 1.2MHz, a coverage angle of 90° × 40°, a beam count of 150 × 50, an angular beamwidth of 0.6° × 1°, a maximum detection range of 15m, a minimum detection range of 30cm, a distance resolution of 3mm, and a real-time output frequency of 5Hz.

[0050] Step S2: Generative adversarial network based on cycle consistency loss is used to inject deep-sea physical degradation features into the visual image.

[0051] like Figure 3 As shown, since the realism of the perception data directly output by the Gazebo physics engine is limited, this embodiment introduces a pre-trained generative adversarial network (preferably a UWGAN network). The RGB and depth images output by the underwater camera plugin are acquired and input into the generative adversarial network, which then generates global background lighting parameters. With transmittance characteristic map Substitute the following underwater physical imaging formula into the equation. The goal is to synthesize underwater-style images. Simultaneously, at the underlying physical acoustic signal level, the average ambient noise level is set. Standard deviation of speckle noise Inject it into the ideal echo intensity output by the sonar plug-in. In this context, the noise addition formula is:

[0052]

[0053] in, This indicates the echo intensity after noise addition. This represents the speckle noise term, characterizing the granular speckle disturbance caused by underwater coherent interference.

[0054] This results in the output of 3D point cloud data with real physical degradation characteristics.

[0055] Step S3: Using the heterogeneous data asynchronous alignment module, the underwater robot is manually operated to perform search and interaction with underwater equipment; a time matching tolerance window threshold is set, and an asynchronous alignment mechanism based on the sonar low-frequency timestamp is executed to record synchronized multimodal sensor data and manual instructions to construct a teaching dataset.

[0056] like Figure 2 and Figure 3As shown, this step involves an underwater robot in a remotely operated simulation searching for and interacting with the underwater operation panel. A multi-topic subscription mechanism using middleware communication frameworks such as ROS2 is employed to record synchronized multimodal sensor data and human commands to construct the dataset. The time-matching tolerance window threshold is set to... A buffer space is set, with a synchronization tolerance of 0.1s. This is for low-frequency sonar timestamps. Visual high-frequency timestamps Perform the following constraint checks to complete the asynchronous alignment: .

[0057] like Figure 2 As shown, the coordinates and the manually operated data output by the inertial measurement sensor plug-in 24 are unified in data format, specifically unified as: 6-DOF ROV center motion pose and N-DOF (N=10 in this embodiment) dual robotic arm joint angle motion sequence. The above-aligned sensing data and unified formatted motion commands are integrated to complete the dataset construction.

[0058] Step S4: Perform preprocessing on the teaching dataset, and perform cross-modal stitching and fusion of the generated two-dimensional visual image and the spatiotemporally aligned three-dimensional sonar point cloud to construct a global multimodal conditional representation; guided by the global multimodal conditional representation, use the diffusion strategy conditional action generation network to output a smooth multidimensional action trajectory sequence through an iterative denoising process.

[0059] End-to-end training of the diffusion model, such as Figures 2 to 4 As shown, it includes the following steps:

[0060] Step S4.1: Data Spatial Alignment and Multimodal Point Cloud Fusion. Noisy sonar point cloud data in the sonar local coordinate system is acquired. Using the extrinsic parameter matrix, current rotation matrix, and translation vector of the underwater robot, it is transformed to the underwater robot's body coordinate system (ROV coordinate system) to obtain... A set of three-dimensional coordinate points Simultaneously, underwater-style images that retain physical degradation features, output by a generative adversarial network, are acquired; projection mapping is performed using the intrinsic and extrinsic parameter matrices of the underwater camera plugin, for... Each three-dimensional coordinate point is assigned a corresponding visual color attribute. By combining spatial geometric coordinates with visual color attributes, a dimension of [missing information] is constructed. The sound and light fusion features with color information.

[0061] Step S4.2: Based on the latent space mapping of the 3D feature network, the dimension is... The overall acoustic-optical fusion features are input into a point cloud feature extraction network (preferably PointNet++ architecture), which outputs a one-dimensional acoustic-optical fusion latent space feature vector. , dimension 1*D_cond.

[0062] Step S4.3: Constructing the multimodal feature vector C. Extract the ontological state vector representing the current kinematic state of the underwater robot. (Dimension 1*D_state) and the latent space feature vector of audio-visual fusion Inputting into the channel dimension of a tensor, the process involves concatenating and mapping channels using a channel concatenation + MLP module, outputting a multimodal feature vector C (dimension 1*d) generated by the end-to-end actions of the guided diffusion strategy.

[0063]

[0064] in, This represents the concatenation operator. The multimodal feature vector C will be used as global context information and injected into the subsequent diffusion strategy conditional action generation network.

[0065] Step S4.4: Expert Experience Replay and Sampling for Sequence Generation. An expert experience playback buffer for multimodal time series slices is constructed. This buffer adopts a heterogeneous data hierarchical storage structure, and a sliding window sampling strategy is used to construct training batches for imitation learning. For the baseline time step of sampling... The system extracts past The multimodal observation history sequence of the steps is fused into the aforementioned multimodal feature vector C, while simultaneously extracting a length of [missing information] from the future. The real joint action instruction sequence is used as the real action trajectory sequence required for training the network. .

[0066] Step S4.5: End-to-end training of the diffusion model based on the trajectory prediction objective function. During the training phase, diffusion time steps are randomly selected from a uniform distribution. After processing by a sinusoidal position encoding + MLP module, random Gaussian noise of equal dimension is sampled from a standard normal distribution. The noise is injected into the real motion trajectory sequence through closed-form solution. In the process, synthesized noisy motion trajectories with specific noise intensities are generated. Guided by the multimodal feature vector C, a diffusion strategy conditional action generation network is trained. (Including network layers such as one-dimensional convolutional linear projection and one-dimensional convolutional projection mapping) Predict and reconstruct noiseless predicted motion trajectories. Its network training employs a composite objective loss function that combines score matching for the original trajectory with physical smoothing constraints. Perform parameter optimization:

[0067]

[0068] in, This represents the mean squared error loss in trajectory prediction. Indicates the network's diffusion time step With multimodal feature vectors Guided Noise Addition Motion Trajectory Perform multi-step denoising to output a noise-free predicted motion trajectory. ; This indicates the use of single-step actions in the predicted trajectory. The calculation of the first-order difference action sequence smoothing regularization term, This represents the smoothness weight coefficient. During the inference phase, the network starts from a Gaussian noise sequence and, guided by the multimodal feature vector C, performs iterative denoising calculations for a specified number of steps, ultimately outputting a denoised 6+N dimensional (N=10 in this embodiment) motion trajectory. .

[0069] Before performing cross-modal feature fusion in the denoising backbone network, the current diffusion time step k needs to be conditionally injected into the 1DUNet feature map to provide a unified time step benchmark for subsequent multimodal conditional injection. Specifically, after the diffusion time step k is processed by sinusoidal position encoding and the MLP module, the time step embedded feature vector is obtained. Subsequently, A set of channel-wise scaling parameters γ and offset parameters β are generated via a linear mapping layer:

[0070] [γ, β] = Linear( )

[0071] For feature maps After broadcasting γ and β along the time-series dimension, the operation is performed at each time step:

[0072] = γ· + β

[0073] This mechanism is executed intermittently between the downsampling blocks of the 1D U-Net, injecting the prior information of the diffusion time step into the feature representations at different time resolutions step by step, providing a unified time step benchmark for the deep injection of multimodal conditional features C in subsequent steps.

[0074] Step S4.6: Multimodal conditional temporal injection based on cross-attention mechanism. This is performed in the diffusion policy conditional action generation network. In this paper, a cross-attention mechanism is introduced as a deep injection module for conditional features. Specifically, the multimodal feature vector C is represented as a vector of length [missing information]. Channel dimension is Sequence feature matrix In the denoising backbone network (1D U-Net), any _th _ Layer temporal action feature map ,in This indicates the time step at the current network level. This represents the current number of feature channels. The specific operational logic of the cross-attention mechanism is as follows:

[0075] First, through independent linear mapping weight matrices Generate query matrices respectively Key matrix AND-value matrix :

[0076]

[0077]

[0078]

[0079] in, Generated from one-dimensional temporal action features; and It is generated from the multimodal feature vector C. Subsequently, the dynamic attention weights of the action trajectory at each time step to the multimodal environmental features are calculated and compared with the value matrix. Weighted summation is performed to obtain the conditional injection features. :

[0080]

[0081] in, Indicates the scaling factor. The operation is normalized in the spatial dimension. Finally, conditionally injected features are applied through a linear projection layer. Align the dimensions back to the current number of feature channels. And use residual connections with the original temporal action feature map. Perform pixel-level addition to complete one deep cross-modal feature fusion:

[0082]

[0083] By repeatedly interspersing the aforementioned cross-attention mechanism in the downsampling block (1D CNN Down), deep temporal feature processing block (bottleneck), and upsampling block (1D CNN Up) of 1D U-Net, the denoising network can extract relevant spatial environment geometric and visual texture features at various local time steps on the time axis, thereby generating joint kinematic trajectories.

[0084] This invention also proposes an underwater robot simulation method based on visual and sonar data, adding step S5 to the above training method:

[0085] Using a whole-body co-kinematic control module, a trained diffusion strategy model is deployed in a closed-loop simulation scenario for testing. This module jointly outputs a multi-dimensional kinematic command sequence for the underwater robot body and its multi-degree-of-freedom manipulator, driving the underlying controller to complete interactive operations. During the closed-loop testing phase, the trained embodied intelligent agent control model jointly outputs a smooth 6+N dimensional (N=10 in this embodiment) motion trajectory sequence. The first 6 dimensions guide the spatial translation and rotation of the ROV body, while the latter 10 dimensions control the target angles of each joint of the mounted manipulator. To cope with water flow disturbances and computational latency, the system adopts a rolling time-domain control (RHC) cascade architecture, extracting only the generated future... The first instruction in a continuous instruction sequence is sent to the underlying PID controller. The PID controller, combined with the actual feedback from the physics engine, drives the virtual thrusters and joint motors to complete a high-fidelity operation interaction.

[0086] like Figure 5 As shown, this invention also provides an underwater robot simulation system based on vision and sonar data, including an environment and model construction module, a physical degradation simulation module, a teaching data acquisition module, an end-to-end model training module, and a closed-loop deployment and control module. Using the above-mentioned underwater robot simulation method based on vision and sonar data, the system sequentially executes the construction of the underwater operation scenario and the kinematic simulation model of the underwater robot, the physical degradation simulation of the data, the construction of the teaching dataset, and the construction and training of the diffusion strategy conditional action generation network. The trained diffusion strategy conditional action generation network is then deployed in the closed-loop simulation scenario. Through the multi-dimensional kinematic command sequence of the underwater robot body and the multi-degree-of-freedom manipulator, the underlying controller is driven to complete the interactive operation.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A simulation training method for underwater robots based on visual and sonar data, characterized in that: Model underwater operation scenarios and construct kinematic simulation models of underwater robots; By operating underwater robots to obtain operating instructions and motion data, and acquiring visual and sonar data of underwater operation scenarios, a teaching dataset of operating instructions and corresponding multimodal data is constructed. From the teaching dataset, multimodal historical observation data over a past period are extracted and fused into multimodal features, and a future real joint operation command is extracted as the real action trajectory. A diffusion strategy conditional action generation network is constructed. Under the guidance of the diffusion time step and the multimodal features, the noisy action trajectory is denoised to obtain the predicted noiseless action trajectory. The loss is constructed by comparing the predicted noiseless action trajectory with the future real action trajectory to train the action generation network.

2. The underwater robot simulation training method based on vision and sonar data according to claim 1, characterized in that: A pre-trained generative adversarial network is used to inject deep-sea backscattering and light attenuation features into RGB image visual data to synthesize underwater-style image data as visual data. Environmental background noise and speckle noise are injected into the sonar data to obtain noisy acoustic data as sonar data.

3. The underwater robot simulation training method based on vision and sonar data according to claim 2, characterized in that: Noisy sonar point cloud data in the local coordinate system is acquired and transformed to the underwater robot body coordinate system to obtain a set of three-dimensional coordinate points; based on the underwater style image, the three-dimensional coordinate points are assigned corresponding visual color attributes to obtain sound and light fusion features.

4. The underwater robot simulation training method based on vision and sonar data according to claim 3, characterized in that: Based on the latent space mapping of the three-dimensional feature network, point cloud features are extracted from the sound-light fusion features to obtain one-dimensional sound-light fusion latent space features. The ontological state representing the kinematic state of the underwater robot is extracted from the motion data and fused with the acoustic-optical latent space features. The multimodal features are obtained through channel splicing and mapping.

5. The underwater robot simulation training method based on vision and sonar data according to claim 1, characterized in that: The diffusion time step is randomly sampled from a uniform distribution and processed by a sinusoidal position encoding and a multilayer perceptron module; random Gaussian noise of equal dimension is sampled from a standard normal distribution, and the random Gaussian noise is injected into the real motion trajectory sequence through a closed-form solution to synthesize a noisy motion trajectory with a specific noise intensity.

6. The underwater robot simulation training method based on vision and sonar data according to claim 1, characterized in that: The loss is a composite objective loss function that combines score matching and physical smoothing constraints on the original trajectory; the composite objective loss function includes a smoothing regularization term with smoothness weights and a mean squared error loss term. The mean squared error loss term is constructed by the difference between the predicted noiseless motion trajectory and the future actual motion trajectory. The smoothing regularization term is a first-order difference action sequence smoothing regularization term calculated based on the second norm square sum of the differences between adjacent single-step actions in the predicted trajectory.

7. The underwater robot simulation training method based on vision and sonar data according to claim 1, characterized in that: In the diffusion strategy conditional action generation network, the multimodal features are represented as a multimodal sequence feature matrix, and the multimodal sequence feature matrix is ​​mapped to a key matrix and a value matrix respectively. Through independent linear mapping weight matrices, the temporal action features are mapped to a query matrix. Based on the cross-attention mechanism, the dynamic attention weights of the action trajectory at each time step to the multimodal features are calculated through the query matrix and the key matrix, and then weighted and summed with the value matrix to obtain the conditional injection features. Align the dimension of the conditionally injected feature back to the number of feature channels of the current temporal action feature, and use residual connection to add it to the temporal action feature at the pixel level, thereby injecting the multimodal conditional feature into the temporal action feature.

8. An underwater robot simulation method based on visual and sonar data, characterized in that: The underwater robot simulation training method based on visual and sonar data as described in any one of claims 1 to 7 is used to deploy the trained construct diffusion strategy conditional action generation network in a closed-loop simulation scenario, and drive the underlying controller to complete interactive operations through the underwater robot body and kinematic command sequence.

9. An underwater robot simulation training system based on vision and sonar data, comprising an environment and model building module, a physical degradation simulation module, a teaching data acquisition module, and an end-to-end model training module, characterized in that: The underwater robot simulation training method based on visual and sonar data as described in any one of claims 1 to 7 is used to sequentially perform the construction of the kinematic simulation model of the underwater operation scenario and the underwater robot, the physical degradation simulation of the data, the construction of the teaching dataset, and the construction and training of the diffusion strategy conditional action generation network.

10. An underwater robot simulation system based on vision and sonar data, comprising a simulation training system and a closed-loop deployment and control module, characterized in that: Through the closed-loop deployment and control module, using the underwater robot simulation training system based on vision and sonar data as described in claim 9, the trained construct diffusion strategy conditional action generation network is deployed in the closed-loop simulation scenario, and the underwater robot body and kinematic command sequence drive the underlying controller to complete the interactive operation.

Images