A ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction

Abstract

The application provides a ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction, and belongs to the technical field of digital twinning. The method comprises the following steps: collecting operation instructions and multi-source sensor data; predicting the ship state and trajectory based on an MMG model, and calculating the propeller control amount to drive the physical equipment; synchronously fusing visual and radar data to generate environment point clouds; generating a ship contour prior using the predicted state, and registering the contour prior with real-time point clouds to obtain a pose deviation; taking the deviation as a visual observation, fusing the deviation with GPS, inertial navigation and other data, and correcting the predicted state through a state estimator; and finally superimposing and displaying the corrected contour and trajectory in a video and visual system. The application solves the problems of asynchronization between simulation and entity and lack of real-time correction, and significantly improves the intuitiveness, accuracy and training efficiency of ship operation simulation.

Description

Technical Field

[0001] This invention relates to a method for simulating and visualizing ship motion based on multi-source data fusion and closed-loop correction, belonging to the field of digital twin technology. Background Technology

[0002] Waterjet-propelled vessels are increasingly widely used in modern shipbuilding due to their superior maneuverability and handling. Compared to traditional propeller-rudder propulsion, waterjet propulsion systems achieve flexible motion control of the vessel in multiple degrees of freedom by adjusting the direction and magnitude of the jet vector, making them particularly suitable for scenarios requiring frequent maneuvering, such as near-shore areas, harbor basins, and narrow waterways. However, the control system of waterjet-propelled vessels is complex, involving the coordinated adjustment of multiple control variables such as main engine power, inverted bucket angle, and nozzle angle, making operation more difficult. Currently, the main problems in actually operating waterjet-propelled vessels are as follows: Waterjet-propelled vessels involve more control variables than traditional propeller-rudder vessels, increasing the difficulty of operation. Direct operation of waterjet-propelled vessels is very costly. The status of key components cannot be observed in real time; the status of the spray pump and inverted bucket cannot be directly observed during actual operation, making it difficult to grasp the equipment's operating status. Due to the lack of visual status feedback for the spray pump and inverted bucket, it is impossible to effectively correlate the vessel's trajectory with the state of the waterjet propulsion system. In actual ship operations, ship operation and homing require adjustments to the ship's attitude and position, but this often requires operator experience and lacks accurate short-term trajectory prediction and virtual trajectory prediction methods for designated areas.

[0003] Therefore, there is an urgent need for a digital twin data processing method and system that can synchronize the simulation of waterjet-propelled ship motion with the visualization of the propeller status, and has real-time environmental perception and closed-loop correction capabilities, so as to reduce training costs, improve operational intuitiveness, and provide high-precision auxiliary decision support for ship berthing, obstacle avoidance and other scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method for simulating and visualizing ship motion based on multi-source data fusion and closed-loop correction, so as to solve the problems of asynchronous simulation and physical reality, lack of real-time closed-loop correction, high operating cost and poor intuitiveness in the existing technology.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A method for simulating and visualizing ship motion based on multi-source data fusion and closed-loop correction includes the following steps:

[0007] The analog voltage signal output from the acquisition console is converted from analog to digital and then filtered by low-pass to obtain digital control commands, which are then timestamped.

[0008] Collect ship observation data, including GPS, inertial navigation, rudder angle and bow angle data, to form an observation package in a unified coordinate system;

[0009] Using the observation package from the previous cycle and the digital control commands, the current state of the ship is predicted based on the MMG model, and the short-term predicted trajectory is extrapolated. The predicted state and trajectory of the ship are then output.

[0010] The camera images, depth maps, and radar point cloud data are read, unified to the world coordinate system through projection model and coordinate transformation, and fused. After downsampling and noise reduction processing, the dynamic point cloud of the scene is obtained.

[0011] Based on the predicted ship state and ship geometric parameters, the rectangular four-point profile prior of the ship in the world coordinate system is calculated, and the four-corner rectangular profile is obtained based on the rectangular four-point profile prior of each corner.

[0012] The dynamic point cloud of the scene is registered with the four-cornered rectangle contour based on the planar rigid body transformation parameters to obtain the registration transformation parameters;

[0013] The registration transformation parameters and GPS, inertial navigation, and gyroscope data are input into the state estimator to update the predicted state of the ship, and the corrected ship state is output.

[0014] The corrected ship state is recalculated to obtain the ship's rectangular outline, which is then overlaid on the monitoring video in conjunction with the predicted trajectory, and the virtual ship model in the visual system is updated simultaneously.

[0015] Preferably, the observation package includes:

[0016] The ship's position coordinates are obtained from GPS, and the ship's forward angular acceleration, lateral angular acceleration, turning angular acceleration, and turning angular velocity are obtained from inertial navigation, as well as the ship's rudder angle and heading angle parameters.

[0017] Preferably, the specific steps for obtaining the scene dynamic point cloud include:

[0018] Acquire camera images and their corresponding depth maps, as well as radar point cloud data, wherein the radar point cloud data is LiDAR or millimeter-wave radar point cloud;

[0019] The depth corresponding to the image pixels is back-projected to the camera coordinate system and then transformed to the world coordinate system; for radar point cloud data, it is directly transformed to the world coordinate system through sensor extrinsic parameters; the image depth and radar point cloud data in the world coordinate system are merged to obtain the initial scene point cloud.

[0020] The initial scene point cloud is downsampled using voxels and outliers are removed to obtain the dynamic scene point cloud.

[0021] Preferably, the method for generating the outline of the four-cornered rectangle is as follows:

[0022] The bow angle of the ship at the current moment is obtained by double integral of the ship's turning angular velocity in the predicted state, and the ship's length and width are obtained.

[0023] In the ship's coordinate system, a set of relative vectors is constructed at the four corners of a rectangle with the ship's length and width as the origin and the ship's center as the origin.

[0024] A two-dimensional rotation matrix is ​​constructed using the heading angle obtained from the predicted state. The relative vector set of the four corners of the rectangle is rotated to the world coordinate system using the two-dimensional rotation matrix and added to the predicted ship trajectory in the world coordinate system to obtain the coordinates of the four points in the world coordinate system, forming the four-point outline prior of the rectangle.

[0025] The four-cornered rectangle profile in the world coordinate system is obtained based on the prior of the four-point profile of each corner.

[0026] Preferably, the registration transformation parameters are obtained in the following ways:

[0027] A gated area is set with the outline of the four-cornered rectangle as the center. Points falling around the gated area are selected from the dynamic point cloud of the scene as the gated cloud. Specifically, the perimeter of the gated area is 8% of the ship's length in the length direction of the gated area and 16% of the ship's width in the width direction of the gated area.

[0028] The gated cloud is clustered, and a comprehensive consistency scoring function is calculated based on size matching, PCA principal direction consistency, rectangle occupancy rate and boundary fitting features to select the best hull point cluster;

[0029] The planar rigid body transformation parameters are solved by iterative optimization to make the optimal hull point cluster fit the four-cornered rectangular contour, and the registration transformation parameters are output.

[0030] Preferably, the overall consistency scoring function is:

[0031] ,

[0032] ,

[0033] ,

[0034] ,

[0035] ,

[0036] in, For size matching features, Rectangular occupancy characteristics For boundary fitting features, This is a consistent feature of the main direction of PCA. , , , These are the weighting coefficients; The length of the candidate point cluster, The actual length of the ship. The width of the candidate point cluster, The actual width of the ship. The number of points in the current candidate point cluster. For points in the candidate point cluster, This indicates the point that falls within the predicted rectangle. The total number of candidate point clusters. This is the difference between the PCA principal axis orientation angle of the candidate point cluster and the heading angle of the predicted profile. The median of the signed distance quantifies the degree of fit between all points in the candidate point cluster and the predicted rectangular contour boundary. is the signed distance from a 3D point in the candidate point cluster to the predicted rectangular profile.

[0037] Preferably, the state estimator is based on the ship's predicted state and employs a Kalman filter, an extended Kalman filter, or an information filter.

[0038] Preferably, the video overlay display includes:

[0039] Using camera calibration parameters, the rectangular outline of the ship in world coordinates and the predicted trajectory are projected onto image coordinates;

[0040] The updated rectangular outline of the ship and the predicted trajectory curve are plotted on the video frames, and key state parameters and thruster state information are labeled.

[0041] The advantages of this invention are as follows: This invention employs a model predictive control method, combining the ship's dynamic model, sensor data, and environmental perception to optimize control inputs in real time, dynamically adjusting the ship's path and speed during navigation. This method effectively avoids path deviation and collision risks by predicting future trajectories and continuously adjusting control inputs, while simultaneously improving the accuracy and flexibility of path planning. It enhances the ship's adaptability and intelligence in complex environments, enabling the ship to adjust its course and speed in real time in dynamic environments, ensuring accurate trajectory tracking and obstacle avoidance.

[0042] This invention utilizes multi-sensor data fusion, combining information from multiple sensors such as GPS, INS, rudder angle sensors, cameras, and LiDAR. Through methods like Kalman filtering, deep learning, and image-depth map fusion, it accurately acquires the ship's state information and perceives the surrounding environment in real time. By fusing data from different sensors, the system can compensate for the limitations of a single sensor, improving the overall reliability of the data. Multi-sensor fusion enables the system to more accurately identify surrounding obstacles, especially in complex and dynamic environments with other ships, buoys, and port facilities. Through real-time perception and data fusion, the system can dynamically update the ship's trajectory and surrounding environment, providing more precise obstacle avoidance and path planning.

[0043] This invention acquires sensor data and estimates the ship's state in real time, then overlays the predicted trajectory with the real-time updated ship outline onto a video monitoring screen. This allows operators to intuitively and in real-time observe the ship's motion and surrounding environment. This method combines real-time data updates and visual feedback, significantly improving the operator's experience and decision-making efficiency. By dynamically displaying the ship's predicted trajectory and outline in the video monitoring, operators can understand the ship's position, heading, and surrounding environment in real time. Operators can intuitively see the relative position of the ship to obstacles through video monitoring, allowing for timely adjustments to heading and control strategies. The real-time updated trajectory and outline help operators make better obstacle avoidance decisions and avoid collision risks. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0045] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Example 1

[0048] like Figure 1 As shown, a method for simulating and visualizing ship motion based on multi-source data fusion and closed-loop correction includes the following steps:

[0049] S1: Acquire the analog voltage signal output from the control console, convert it to digital signal and low-pass filter it to obtain digital control commands, and add a timestamp.

[0050] S2: Collect ship observation data, including GPS, inertial navigation, rudder angle and bow angle data, to form an observation package in a unified coordinate system;

[0051] S3: Using the observation packet from the previous cycle and the digital control command, predict the current state of the ship based on the MMG model and extrapolate the short-term predicted trajectory, outputting the ship's predicted state and predicted trajectory.

[0052] S4: Read camera images, depth maps and radar point cloud data, unify them to the world coordinate system through projection model and coordinate transformation and fuse them, and obtain the scene dynamic point cloud after downsampling and noise reduction processing;

[0053] S5: Based on the predicted state of the ship and the ship's geometric parameters, calculate the rectangular four-point profile prior of the ship in the world coordinate system, and obtain the four-corner rectangular profile based on the rectangular four-point profile prior of each corner.

[0054] S6: Based on the planar rigid body transformation parameters, register the scene dynamic point cloud with the four-corner rectangle contour to obtain the registration transformation parameters;

[0055] S7: Input the registration transformation parameters and GPS, inertial navigation, and gyroscope data into the state estimator to update the predicted state of the ship and output the corrected ship state;

[0056] S8: Recalculate the rectangular outline of the ship after the correction of the ship state, and combine it with the predicted trajectory to overlay it on the monitoring video, and update the virtual ship model in the visual system at the same time.

[0057] As a refinement of the above embodiments, step S1 specifically includes:

[0058] The operator controls the console. The console subsystem acquires the analog voltage signals output from the handwheel / handle / vector unit, and obtains digital control commands through ADC sampling and quantization. During this process, low-pass filtering and debouncing are applied to the ADC sampling to prevent manipulation jitter from being amplified into control jitter, and each frame of commands is timestamped. This is used for subsequent multi-source time synchronization. This scheme is based on... time, Provide examples to illustrate the iterative progression of time.

[0059] As a refinement of the above embodiments, step S2 includes:

[0060] The system collects GPS (positioning, speed, attitude), inertial navigation / inertial navigation, rudder angle, and bow angle data of the ship to form observation inputs. The time observation package completes coordinate system one:

[0061] ,

[0062] in, , For GPS-based systems The ship's position coordinates at that moment. , , , For INS inertial navigation system The ship's longitudinal velocity, lateral velocity, angular acceleration, and angular velocity at each moment. , for The ship's rudder angle and bow angle parameters at any given time.

[0063] As a refinement of the above embodiments, step S3 includes:

[0064] Utilizing the previous cycle The system continuously monitors the status of the data packet and digital control commands, predicts the ship's current position, speed, heading, and roll rate based on the MMG model, extrapolates the short-term predicted trajectory, and outputs the predicted ship status. and predicted trajectory ,in, For prediction The longitudinal speed of the ship at that moment, For prediction The ship's lateral speed at any given moment, For prediction The ship's turning angle acceleration at time t, For the derivation (prediction) based on the MMG motion equations The x-coordinate value at time [time]. Derived from the MMG equations of motion The y-coordinate value at time [time].

[0065] As a refinement of the above embodiment, step S4 reads images from multiple cameras and LiDAR point cloud depth maps, unifies them to the geodetic coordinate system based on the projection model and coordinate transformation, and fuses them to obtain scene point cloud data. Subsequently, voxel downsampling and outlier removal reduce noise and computing power.

[0066] Specifically, it includes:

[0067] S401: Acquire camera images and their corresponding depth maps, as well as radar point cloud data, wherein the radar point cloud data is LiDAR or millimeter-wave radar point cloud.

[0068] S402: For images and depth maps, backproject the depth corresponding to the image pixels to the camera coordinate system and then transform it to the world coordinate system to combine the pixel points with the depth data. Based on the camera intrinsic parameters, the 2D points are transformed into 3D points, and then aligned with the world coordinate system for use.

[0069] For radar point clouds, the transformation to the world coordinate system is directly performed using the sensor extrinsic parameter matrix.

[0070] By merging image depth and radar point cloud data in the world coordinate system, an initial scene point cloud is obtained, resulting in a denser and more complete 3D environmental point cloud.

[0071] If there are multiple cameras or sensors, the points are converted and then merged in the same coordinate system to obtain the initial scene point cloud.

[0072] S403: Perform voxel downsampling and outlier removal on the initial scene point cloud. That is, set the voxel grid size in the world coordinate system, aggregate / take representative points within the same voxel to reduce the number of points, and then use statistical outlier / radius outlier methods to remove obvious noise points, suppress discrete points caused by waves, reflections, occlusions, etc., reduce subsequent registration errors and computational load, and obtain the scene dynamic point cloud.

[0073] As a refinement of the above embodiments, step S5 includes:

[0074] To convert the simulated predicted state into a rectangular four-point profile prior in the world coordinate system, serving as initial values / prior constraints for registration, this step involves determining the ship's predicted state. The bow angle is obtained by double integral of the ship's turning angular acceleration at the current moment. and the geometric parameters of the ship's foundation, including the ship's length. ,width .

[0075] In the ship's coordinate system, based on geometric parameters and with the ship's center as the origin, construct a set of relative vectors at the four corners of a rectangle. Then use heading angle Constructing a two-dimensional rotation matrix Rotate the four relative vectors to the world coordinate system, and add the predicted trajectory in the world coordinate system. The coordinates of the four corner points in the world coordinate system are obtained, forming a four-point profile prior. .

[0076] The coordinates of the four corner points can be represented as: top left corner Top right corner bottom left corner bottom right corner .

[0077] The corresponding calculation formula is as follows:

[0078] ,

[0079] ,

[0080] ,

[0081] ,

[0082] ,

[0083] ,

[0084] ,

[0085] ,

[0086] in, for The heading angle of the time-predicted value.

[0087] The outline of the four-cornered rectangle in the prior world coordinate system is calculated sequentially.

[0088] As a refinement of the above embodiments, step S6 includes:

[0089] To register the dynamic point cloud of the scene with the rectangular outline, and to correct the 3-DOF pose in a 2D plane, it is necessary to solve for the planar rigid body transformation parameters. ,in To predict the deviation correction of the center of the rectangular profile on the x-axis of the world coordinate system. To predict the deviation correction of the center of the rectangular profile on the y-axis of the world coordinate system. To predict the rotational correction of the rectangular profile around its own center, ensuring that the predicted ship attitude and trajectory of the digital twin remain consistently aligned with the real scene over the long term, and using this consistency to correct model errors, this step employs spatial gating, clustering filtering, and rigid body registration methods for fitting.

[0090] S601: Space Gating

[0091] Centered on a rectangular outline, a gated region with a safety margin is set. Only points falling near the gated region are retained from the scene's dynamic point cloud, forming a gated cloud. Gating filters out splashes, reflections, and occlusions unrelated to the ship's hull, improving clustering and registration stability.

[0092] Specifically, taking a small waterjet propulsion vessel as an example, the perimeter of the gated area extends 8% of the vessel's length in the length direction of the gated area and 16% of the vessel's width in the width direction of the gated area.

[0093] S602: Clustering and Candidate Selection

[0094] Point cloud clustering is performed on the gated cloud to obtain a set of candidate point clusters. A comprehensive score is calculated for each candidate point cluster. The scoring indicators include size matching features, PCA main direction consistency features, rectangle occupancy features and boundary fitting features for discrimination.

[0095] (1) Size matching feature

[0096] The axis-aligned bounding boxes of the clusters are selected, and here only the clusters with the best matching point cloud cluster size are selected based on the size error function:

[0097] ,

[0098] Define dimensional error as a dimensional matching feature:

[0099] ,

[0100] in, The length of the candidate point cluster, The actual length of the ship. The width of the candidate point cluster, The actual width of the ship. This represents a candidate point cluster.

[0101] (2) Consistency characteristics of PCA principal direction

[0102] If we perform PCA on the principal axis of a cluster of two-dimensional points, then the PCA principal axis consistency characteristic is:

[0103] ,

[0104] in, It is the difference between the PCA principal axis orientation angle of the candidate point cluster and the heading angle of the predicted profile.

[0105] (3) Rectangular occupancy characteristics

[0106] The proportion of rectangles falling within the predicted rectangle is calculated as the rectangle occupancy rate feature:

[0107] ,

[0108] in, The number of points in the current candidate point cluster. For points in the candidate point cluster, This indicates the point that falls within the predicted rectangle. This represents the total number of points in the candidate point cluster.

[0109] (4) Boundary fitting characteristics

[0110] Define the signed distance from a point to a rectangle, let , ,in , These are the half-length and half-width of the rectangle in the x and y directions, respectively. If it's outside the rectangle, i.e. or ,but

[0111] ,

[0112] If it is inside the rectangle, that is and ,but

[0113] ,

[0114] Define signed distance:

[0115] ,

[0116] in, Let be the distance from a point inside the rectangle to the nearest edge. Let be the shortest distance from a point outside the rectangle to the rectangle. The absolute value of the distance from the point to the y-axis is derived from the property of a rectangle being symmetric about its center. is the perpendicular distance from the point to the x-axis.

[0117] The median is used as the more robust boundary fitting error as the boundary fitting feature:

[0118] ,

[0119] in, The median of the signed distance quantifies the degree of fit between all points in the candidate point cluster and the predicted rectangular contour boundary. is the signed distance from a 3D point in the candidate point cluster to the predicted rectangular profile.

[0120] Based on the above characteristics, a comprehensive consistency scoring function is constructed and the optimal hull point cluster is selected. The comprehensive consistency scoring function is as follows:

[0121] ,

[0122] in, For size matching features, Rectangular occupancy characteristics For boundary fitting features, This is a consistent feature of the main direction of PCA. It is a natural constant. , , , These are the weighting coefficients.

[0123] S603: Solving Rigid Body Registration

[0124] Before completing the registration solution, first find the planar rigid body transformation. To ensure the optimal hull point cluster fits the rectangular outline, an objective function is constructed that penalizes the distance from points to the interior of the rectangle, and the solution is obtained through iterative optimization. Robust processing is introduced to suppress the influence of noise points, improving stability under conditions of waves, reflections, and occlusion, and finally outputting registration transformation parameters. .

[0125] As a refinement of the above embodiments, step S7 includes:

[0126] In order to adjust the registration transformation parameters As a new visual observation measure, it is used together with GPS (ship's two-dimensional position coordinates), inertial navigation (ship's forward speed, lateral speed, turning angular velocity and turning angular acceleration), and gyroscope data (providing ship's heading angle information) to update state estimation, forming a closed loop of prediction-observation-correction-repreneurial, so that the profile and trajectory continuously follow the real observation.

[0127] S701: Visual observation structure

[0128] The registration output is defined as a visual measurement, representing the deviation of the predicted profile from the true position and attitude of the hull in the visual point cloud:

[0129] ,

[0130] S702 Integration Update (Basic Implementation)

[0131] Ship Predicted Status Based on MMG Prediction As a priori state, The data, along with observations from GPS, inertial navigation, and gyroscope, are input into the state estimator for updating. The updated output includes the corrected ship position, heading, and other states, which are then used as the initial state for the next cycle prediction, thus forming a closed-loop correction.

[0132] In the basic scheme, the state estimator can be implemented using any of the following methods: Kalman filtering, extended Kalman filtering, information filtering, etc. The key step in this process is to... Incorporate state updates, rather than limiting them to specific filter types.

[0133] As a refinement of the above embodiments, step S8 includes:

[0134] The updated ship outline and trajectory are overlaid on the monitoring video, and the virtual ship model and propeller status are updated simultaneously to achieve visual twin feedback.

[0135] S801: Generation of superimposed elements

[0136] The ship's rectangular profile is recalculated using the corrected ship state, the predicted trajectory is read as the trajectory curve, and key state parameters such as position, heading, and speed, as well as the propeller state of speed, nozzle angle, and reverse bucket state are obtained for display.

[0137] S802: Video Coordinate Mapping and Overlay Drawing

[0138] For each frame of monitoring video, the world coordinate information such as the ship's rectangular outline and trajectory curve is projected onto the image coordinates using camera calibration parameters. The updated ship rectangular outline and short-term predicted trajectory curve are then drawn on the video frame, and key status parameters and thruster status text / icon information are labeled, thus forming a digital twin prediction fusion display.

[0139] S803: Visual System Synchronous Update

[0140] The virtual ship model's attitude and thruster status are updated synchronously in the visual system to ensure consistency between the virtual model and the physical thruster's movements and the video overlay results.

[0141] Specifically, the method for obtaining the thruster status is as follows:

[0142] (1) Map the desired longitudinal and lateral thrust and steering torque into a combination of control quantities for the left and right thrusters, and apply physical constraints and limits to generate the desired sum of thruster control quantities; that is:

[0143] Based on target force and torque ,

[0144] Adjust thruster control quantity ,

[0145] in, The equations of motion derived from MMG are used to derive the following: The force on the x-axis at time t. The equations of motion derived from MMG are used to derive the following: The force on the y-axis at time [time]. The torque acting on the ship is derived from the MMG equations of motion. , This indicates the parameters of the left and right thrusters. , , These represent the rotational speed, nozzle angle, and inverted bucket angle of the waterjet propulsion system, respectively. Here, the decoupling of the propulsion system's control variables and data acquisition are achieved based on the head model and the MMG model.

[0146] (2) Transmit the sum of the desired thruster control quantities to the thruster host computer to drive the physical thruster to move and read the thruster feedback parameters.

[0147] Adjust the thruster control quantity The data is transmitted to the host computer of the thruster via CAN bus, completing the digital-to-voltage conversion and driving the movement of the physical thruster. This achieves synchronization between the physical thruster and the virtual simulation. At the same time, the thruster feedback (speed, nozzle angle, and reversing bucket angle) is read back as the basis for closed-loop control and display.

[0148] It should be noted that this invention dynamically adjusts the ship's path and profile by updating the ship's outline position and predicted trajectory in real time, combined with environmental data, to avoid collisions and optimize navigation. This invention does not use the registration results solely for video overlay display; instead, it incorporates the obtained pose deviation as a visual observation into the state estimation process, working in conjunction with data from GPS, inertial navigation, and other sensors to complete state updates. By acquiring sensor data and motion state estimation information in real time and fusing data from different sensors, the predicted trajectory and dynamic profile of the ship are overlaid onto the video monitoring screen in real time, providing operators with intuitive feedback and decision support.

[0149] This invention implements a real-time environmental perception method that fuses multi-angle cameras and depth sensors, constructing a three-dimensional environmental model around a ship in real time and achieving dynamic environmental perception. A real-time environmental model is generated by weighted fusion of image and depth data from multiple cameras.

[0150] This invention establishes a closed-loop mechanism of prediction-observation-correction-repreneurial prediction, ensuring that the virtual state of the ship, its outline display, and propeller control remain consistent over the long term. This constitutes the essential characteristic that distinguishes this invention from traditional simulation display systems or purely sensing systems, and is also the key to the significance of digital twins. Patent protection is sought for the data processing relationships and the closed-loop structure itself.

[0151] Example 2

[0152] This disclosure also provides a ship motion simulation and visualization device based on multi-source data fusion and closed-loop correction, including a processor and a memory. Optionally, the device may further include a communication interface and a bus. The processor, communication interface, and memory can communicate with each other via the bus. The communication interface can be used for information transmission. The processor can call logical instructions in the memory to execute the ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction described in the above embodiments.

[0153] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0154] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes the program instructions / modules stored in the memory to perform functional applications and data processing, thereby realizing the ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction in the above embodiments.

[0155] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory.

[0156] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to execute the aforementioned ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction.

[0157] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0158] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code. It can also be a transient storage medium.

[0159] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for simulating and visualizing ship motion based on multi-source data fusion and closed-loop correction, characterized in that, Includes the following steps: The analog voltage signal output from the acquisition console is converted from analog to digital and then filtered by low-pass to obtain digital control commands, which are then timestamped. Collect ship observation data, including GPS, inertial navigation, rudder angle and bow angle data, to form an observation package in a unified coordinate system; Using the observation package from the previous cycle and the digital control commands, the current state of the ship is predicted based on the MMG model, and the short-term predicted trajectory is extrapolated. The predicted state and trajectory of the ship are then output. The camera images, depth maps, and radar point cloud data are read, unified to the world coordinate system through projection model and coordinate transformation, and fused. After downsampling and noise reduction processing, the dynamic point cloud of the scene is obtained. Based on the predicted ship state and ship geometric parameters, the rectangular four-point profile prior of the ship in the world coordinate system is calculated, and the four-corner rectangular profile is obtained based on the rectangular four-point profile prior of each corner. The dynamic point cloud of the scene is registered with the four-cornered rectangle contour based on the planar rigid body transformation parameters to obtain the registration transformation parameters; The registration transformation parameters and GPS, inertial navigation, and gyroscope data are input into the state estimator to update the predicted state of the ship, and the corrected ship state is output. The corrected ship state is used to recalculate the ship's rectangular outline, which is then overlaid on the monitoring video in conjunction with the predicted trajectory, and the virtual ship model in the visual system is updated synchronously. The method for generating the outline of the four-cornered rectangle is as follows: The bow angle of the ship at the current moment is obtained by double integral of the ship's turning angular velocity in the predicted state, and the ship's length and width are obtained. In the ship's coordinate system, a set of relative vectors is constructed at the four corners of a rectangle with the ship's length and width as the origin and the ship's center as the origin. A two-dimensional rotation matrix is ​​constructed using the heading angle obtained from the predicted state. The relative vector set of the four corners of the rectangle is rotated to the world coordinate system using the two-dimensional rotation matrix and added to the predicted ship trajectory in the world coordinate system to obtain the coordinates of the four points in the world coordinate system, forming the four-point outline prior of the rectangle. The four-point rectangle profile in the world coordinate system is obtained based on the prior knowledge of the rectangle's four-point profile at each corner. The registration transformation parameters are obtained in the following ways: A gated area is set with the outline of the four-cornered rectangle as the center. Points falling around the gated area are selected from the dynamic point cloud of the scene as the gated cloud. Specifically, the perimeter of the gated area is 8% of the ship's length in the length direction of the gated area and 16% of the ship's width in the width direction of the gated area. The gated cloud is clustered, and a comprehensive consistency scoring function is calculated based on size matching, PCA principal direction consistency, rectangle occupancy rate and boundary fitting features to select the best hull point cluster; The planar rigid body transformation parameters are solved by iterative optimization, so that the optimal hull point cluster fits the quadrangular rectangular contour, and the registration transformation parameters are output. The overall consistency scoring function is as follows: ， ， ， ， in, For size matching features, Rectangular occupancy characteristics For boundary fitting features, This is a consistent feature of the main direction of PCA. , , , These are the weighting coefficients; The length of the candidate point cluster, The actual length of the ship. The width of the candidate point cluster, The actual width of the ship. The number of points in the current candidate point cluster. For points in the candidate point cluster, This indicates the point that falls within the predicted rectangle. The total number of candidate point clusters. This is the difference between the PCA principal axis orientation angle of the candidate point cluster and the heading angle of the predicted profile. The median of the signed distance quantifies the degree of fit between all points in the candidate point cluster and the predicted rectangular contour boundary. is the signed distance from a 3D point in the candidate point cluster to the predicted rectangular profile.

2. The ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction according to claim 1, characterized in that, The observation package includes: The ship's position coordinates are obtained from GPS, and the ship's forward angular acceleration, lateral angular acceleration, turning angular acceleration, and turning angular velocity are obtained from inertial navigation, as well as the ship's rudder angle and heading angle parameters.

3. The ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction according to claim 1, characterized in that, The specific steps for obtaining the dynamic point cloud of the scene include: Acquire camera images and their corresponding depth maps, as well as radar point cloud data, wherein the radar point cloud data is LiDAR or millimeter-wave radar point cloud; The depth corresponding to the image pixels is back-projected to the camera coordinate system and then transformed to the world coordinate system; for radar point cloud data, it is directly transformed to the world coordinate system through sensor extrinsic parameters; the image depth and radar point cloud data in the world coordinate system are merged to obtain the initial scene point cloud. The initial scene point cloud is downsampled using voxels and outliers are removed to obtain the dynamic scene point cloud.

4. The method for ship motion simulation and visualization based on multi-source data fusion and closed-loop correction according to claim 1, characterized in that, The state estimator is based on the ship's predicted state prior to the state estimator, and employs a Kalman filter, an extended Kalman filter, or an information filter.

5. The method for ship motion simulation and visualization based on multi-source data fusion and closed-loop correction according to claim 1, characterized in that, The overlay rendering on the surveillance video includes: Using camera calibration parameters, the rectangular outline of the ship in world coordinates and the predicted trajectory are projected onto image coordinates; The updated rectangular outline of the ship and the predicted trajectory curve are plotted on the video frames, and key state parameters and thruster state information are labeled.

6. A ship motion simulation and visualization device based on multi-source data fusion and closed-loop correction, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute, when running the program instructions, the ship motion simulation and visualization method based on multi-source data fusion and closed-loop correction as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a ship motion simulation and visualization method for multi-source data fusion and closed-loop correction as described in any one of claims 1-5.

Images