A ship multi-modal perception intelligent monitoring system and method based on data analysis

Abstract

This invention relates to the field of ship perception, specifically to a data analysis-based intelligent monitoring system and method for multimodal ship perception, comprising: an image processing module, a sample calibration module, a modality fusion module, a target localization module, and an automatic formation module. The image processing module is used to acquire multimodal images; the sample calibration module is used to extract ROIs and obtain standard samples for each modality; the modality fusion module is used to construct a three-modal dataset; the target localization module is used to perform three-dimensional localization of targets; and the automatic formation module is used to control ships to avoid obstacles and form formations. This invention can improve the robustness and accuracy of shipborne obstacle detection, enhance the tracking performance of target tracking algorithms, improve the stability of obstacle target detection, improve the quality of high-resolution point clouds, enhance image fusion effects, improve the response speed of ship navigation, and improve the operational efficiency and safety of ship swarms.

Description

Technical Field

[0001] This invention relates to the field of ship sensing, specifically to a ship multimodal sensing intelligent monitoring system and method based on data analysis. Background Technology

[0002] Unmanned surface vessels (USVs) are small vessels that operate on water using remote control or automated control devices. They are typically equipped with sensors, control systems, and communication equipment, and possess functions such as target recognition, attitude awareness, and underwater monitoring. They are commonly used for surface navigation and underwater mapping operations in rivers, lakes, and oceans. As surface operations become increasingly complex, USVs are evolving from single-vehicle navigation to multi-vehicle swarms. Through collaborative formation control, multiple USVs can achieve efficient collaborative operations, gaining benefits in information acquisition, operational range, and mission efficiency.

[0003] Commonly used unmanned surface vessel (USV) target recognition methods rely solely on visible light or radar single-modal image features. However, during surface operations, weather, lighting, and wave interference can lead to poor target recognition performance under single-modal conditions. Multimodal recognition systems, on the other hand, suffer from difficulties in registration, poor data analysis capabilities, sparse point clouds, insufficient training data for target perception, and detection performance that is easily affected by the environment and data functionality. This can result in modality dependency issues, poor long-term tracking capabilities, and difficulties in target recognition.

[0004] In addition, during multi-ship formation, existing formation algorithms are not flexible enough due to the limitations of water surface obstacles and ship attitudes. The formation is easily disrupted during navigation, making it difficult to meet the needs of automated operation scenarios. The modal characteristics of different ship equipment are quite different, which increases the registration error in the formation process and limits the efficiency of operation. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal intelligent monitoring system and method for ships based on data analysis, so as to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a ship multimodal perception intelligent monitoring system based on data analysis, comprising: an image processing module, a sample calibration module, a modal fusion module, a target localization module, and an automatic formation module; The image processing module is used to acquire fused images, infrared images, and radar images, establish a reflection model based on visible light positioning, simulate multipath reflection, attenuation, and reflection noise of radar signals, fit an attenuation-free radar image, use a regression forest model to learn the differences in radar signals of obstacles between different backgrounds, detect the areas where obstacles differ from the background echo, and construct the recognition area of ​​the radar point cloud. The sample calibration module is used to extract ROI images in the recognition area of ​​radar point cloud, oversample the ROI areas of fused image and infrared image, group the sampled signals, calculate timing deviation using the grouped signals, calculate interpolation base point and interpolation coefficient through timing deviation, interpolate the original oversampled signal to obtain sampled dataset, simulate environmental occlusion and inter-sample occlusion effects, filter invalid labels, and obtain standard samples for each modality. The modality fusion module is used to extract semantic information from standard samples of the fused image, detect target boundaries, map them to radar point clouds, determine target edges, label standard samples, construct visible light and radar dual-modality datasets, fuse visible light features and infrared features according to weights to generate fused features, register images based on fused features, generate dual-light fused images using the TIF algorithm, fuse the dual-modality datasets, and obtain infrared, visible light and radar three-modality datasets. The target localization module is used to generate a three-dimensional spatial point cloud based on the features of the trimodal dataset, encode the point cloud based on voxels, extract the features of the three-dimensional spatial point cloud using dynamic graph convolution, fuse the radar point cloud, expand and reconstruct the coordinates of the fused point cloud to obtain a multi-dimensional point cloud, use the multi-dimensional point cloud and the dual-light fusion image to detect and segment the target, determine the target coordinates, integrate the depth information of the coordinate region into the coordinates, and perform three-dimensional localization of the target. The automatic formation module is used to identify all obstacle targets on the movement path of the vessel cluster, perform motion modeling on each vessel, take the vessel with the smallest obstacle area in the field of view as the center, establish a metric model based on spatial density and turning similarity, cluster turning points, determine the connectivity between turning points through the trajectory, output a directed trajectory graph, control the unmanned vessels to avoid obstacles and reach the center, determine the formation spacing based on the maximum gap between obstacles, form a formation, and adjust the formation width according to obstacle information during formation navigation.

[0007] Furthermore, the image processing module includes: a hardware acquisition unit and a radar processing unit; The hardware acquisition unit is used to set up a depth camera, an infrared camera, and a lidar on the vessel, and simultaneously acquire fused images, infrared images, and radar images. It performs joint calibration of the three sensors, unifies the multimodal images to the same coordinate system, and compensates for image distortion caused by the movement of the vessel based on the ship's IMU data. The radar processing unit is used to dynamically columnarize the original point cloud in the radar image, eliminate hard voxelization of the image, use median filtering and morphological processing to reduce clutter interference, detect the areas where obstacles differ from the background echo, and obtain the radar image ROI.

[0008] Furthermore, the sample calibration module includes: a signal sampling unit and a tag filtering unit; The signal sampling unit is used to project the bounding box of the radar identification area onto the fused image and infrared image plane using the extrinsic parameter matrix calibrated by the sensor, obtain the ROI of the corresponding image, refine the encoding of the ROI, perform pixel oversampling from the images of each modality, and group the oversampled signals according to the time window so that each group of signals is continuous at the time point. The label filtering unit is used to extract texture features from the depth map ROI and the infrared map ROI. It uses a Transformer model to fuse texture features, obtains a sampled dataset, trains an occlusion simulation model using the sampled dataset, fits the number of sample points in the dataset to the offset distance of the sample center, filters invalid samples, and obtains standard samples.

[0009] Furthermore, the modality fusion module includes: a semantic fusion unit, an environment adjustment unit, and a multimodal data unit; The semantic fusion unit is used to extract feature points based on the geometric features of the target, register the radar point cloud, extract the target contour on the mask of semantic segmentation of the fused image using an edge detection algorithm, transform the target boundary points extracted in the fused image to the radar coordinate system, and determine the corresponding edge points in the radar point cloud through local search. The environment adjustment unit is used to train a CNN architecture using radar data, output obstacle features, register radar point clouds of consecutive frames, and annotate obstacle target edge data on standard samples to obtain a dual-modal dataset. The multimodal data unit is used to input the fused image and the infrared image into the ECA attention model, output the weights corresponding to different modes according to the illumination conditions, use the TIF algorithm to decompose the overlapping area of ​​the infrared image and the fused image, perform fusion based on the local energy maximum, generate a dual-light fused image, and spatially align it with the radar point cloud to obtain a three-modal dataset.

[0010] Furthermore, the target localization module includes: a point cloud expansion unit and an obstacle segmentation unit; The point cloud extension unit is used to input the three-modal dataset into a conditional generative adversarial network to generate a three-dimensional spatial point cloud, convert the three-dimensional spatial point cloud into a voxel representation, extract features using a dynamic graph convolutional network, perform point-level concatenation of the three-dimensional spatial point cloud features and radar point cloud features, input them into a multilayer perceptron to obtain a fused point cloud, and extend and reconstruct the coordinates of the fused point cloud to obtain a multivariate point cloud. The obstacle segmentation unit is used to project the multi-dimensional point cloud onto the image plane and align it with the dual-light fusion image. Using the point cloud clusters of the target in the multi-dimensional point cloud, the three-dimensional position of the target is obtained by fitting a 3D bounding box. The position is input into the point cloud segmentation model, and the two-dimensional bounding box and segmentation mask of the target are output.

[0011] Furthermore, the automatic formation module includes: a motion modeling unit, a track recognition unit, and a path control unit; The motion modeling unit is used to determine all obstacle targets on the movement path of the boat group, uses GPS and INS combined navigation to determine the boat attitude, and models each boat based on the boat position and heading information. The track identification unit is used to establish a metric model based on spatial density and turning similarity, cluster turning points, determine the connectivity between turning points, and generate a directed track graph. In the directed track graph, nodes are turning points, edges represent passable paths, and edge weights are the distances between nodes. The path control unit is used to find the shortest path from each vessel's position to the center of the circle in the directed track graph. Based on the directed track graph, it executes the USV longitudinal control algorithm to make the vessels sail along the planned path and avoid obstacles. Based on the multi-dimensional point cloud, it detects the gaps between obstacles and selects the largest gap as the safe distance between vessels in the formation to form a stable formation.

[0012] A data analysis-based multimodal sensing intelligent monitoring method for ships includes the following steps: Step S1. Acquire fused images of visible light and depth, infrared images, and radar images; establish a radar reflection model based on visible light positioning; fit the radar image without attenuation; detect the areas where obstacles differ from the background echo; and construct the recognition area of ​​the radar point cloud. Step S2. Extract ROI images from the recognition area of ​​the radar point cloud, oversample the ROI areas of the fused image and infrared image, group the sampled signals, interpolate the original oversampled signals to obtain the sampled dataset, filter out invalid labels, and obtain standard samples for each modality. Step S3. Extract semantic information from the standard samples of the fused image, detect the target boundary, map it to the radar point cloud, determine the target edge, label the standard samples, construct a dual-modal dataset, weighted fuse visible light features and infrared features to generate fused features, generate a dual-light fused image based on the fused features, and obtain a tri-modal dataset after sampling and merging. Step S4. Generate a three-dimensional spatial point cloud based on the features of the three-modal dataset, fuse the radar point cloud and expand and reconstruct the coordinates to obtain a multi-dimensional point cloud, use the multi-dimensional point cloud and the dual-light fusion image to detect and segment obstacle targets, and perform three-dimensional localization of the targets; Step S5. Perform motion modeling on each vessel, establish a metric model, cluster turning points, determine the connectivity between turning points through the trajectory, output a directed trajectory graph, take the vessel with the smallest obstacle area in the field of view as the center, control the unmanned vessel to avoid obstacles and reach the center based on all obstacle targets on the vessel's motion path, form a formation, and adjust the formation width according to obstacle information during formation navigation.

[0013] Furthermore, step S1 includes: Step S11. Set up a depth camera, an infrared camera, and a lidar on the vessel to simultaneously acquire fused images of visible light and depth, infrared images, and lidar images. Perform joint calibration on the three sensors to unify the multimodal images into the same coordinate system. Based on the shipborne IMU data, compensate for image distortion caused by the movement of the hull. Step S12. Determine the region where the obstacle is located by segmenting the obstacle in the fused image. In the radar image, the original radar point cloud is dynamically columnarized to eliminate hard voxelization of the image. Median filtering and morphological processing are used to reduce clutter interference. The location of the obstacle that differs from the background echo is detected in the region where the obstacle is located, and the obstacle location is used as the ROI of the radar image.

[0014] Furthermore, step S2 includes: Step S21. Using the extrinsic parameter matrix calibrated by the sensor, project the bounding box of the radar identification area onto the fused image and infrared image plane to obtain the ROI of the corresponding image. Refine the encoding of the ROI, perform pixel oversampling from the images of each modality, and group the oversampled signals according to the time window so that each group of signals is continuous at the time point. Step S22. Extract texture features from the fused image ROI and the infrared image ROI, fuse the texture features using the Transformer model, adjust the sampling frequency, obtain the sampling dataset, train the occlusion simulation model using the sampling dataset, fit the number of sample points in the dataset to the offset distance of the sample center, filter invalid samples, and obtain standard samples.

[0015] Furthermore, step S3 includes: Step S31. Extract feature points based on the target's geometric features, register the radar point cloud, use an edge detection algorithm to extract the target contour on the mask of semantic segmentation of the fused image, transform the target boundary points extracted from the fused image to the radar coordinate system, and determine the corresponding edge points in the radar point cloud through local search. Step S32. Train a CNN architecture using radar data, output obstacle features, register radar point clouds of consecutive frames, and annotate obstacle target edge data on standard samples to obtain a dual-modal dataset. Step S33. Input the fused image and the infrared image into the ECA attention model, output the weights corresponding to different modes according to the illumination conditions, use the TIF algorithm to decompose the overlapping area of ​​the infrared image and the fused image, fuse them based on the local energy maximum, generate a dual-light fused image, spatially align it with the radar point cloud, and obtain a three-modal dataset.

[0016] Furthermore, step S4 includes: Step S41. Input the trimodal dataset into the conditional generative adversarial network to generate a three-dimensional spatial point cloud. Convert the three-dimensional spatial point cloud into a voxel representation. Use a dynamic graph convolutional network to extract features. Perform point-level concatenation of the three-dimensional spatial point cloud features and radar point cloud features. Input the concatenation into a multilayer perceptron to obtain a fused point cloud. Expand and reconstruct the coordinates of the fused point cloud to obtain a multivariate point cloud. Step S42. Project the multi-dimensional point cloud onto the image plane and align it with the dual-light fusion image. Using the point cloud clusters of the target in the multi-dimensional point cloud, obtain the three-dimensional position of the target through 3D bounding box fitting. Input the position into the point cloud segmentation model and output the two-dimensional bounding box and segmentation mask of the target.

[0017] Furthermore, step S5 includes: Step S51. Determine all obstacles on the movement path of the boat cluster, use GPS and INS combined navigation to determine the boat attitude, model each boat based on its position and heading information, establish a metric model based on spatial density and turning similarity, cluster the turning points, determine the connectivity between the turning points, and generate a directed track graph. In the directed track graph, the nodes are turning points, the edges represent passable paths, and the edge weight is the distance between the nodes. Step S52. Using the path optimization algorithm, with the current position of the vessel as the starting point and the directed graph track as the orbit, plan the shortest path from each vessel's position to the center of the circle in the directed graph of the track, and execute the USV longitudinal control algorithm to make the vessel sail along the planned path, avoid obstacles, and reach the formation area. Step S53. During formation navigation, continuously update the position of obstacles. When the gap between obstacles cannot be passed by the formation, detect the gap between obstacles based on multi-dimensional point cloud, select the largest gap as the safe distance between ships in the formation, adjust the formation width according to the safe distance, form a stable formation that can pass through obstacles, and restore the original formation after passing through obstacles.

[0018] Compared with the prior art, the beneficial effects achieved by the present invention are: 1. This invention constructs the recognition region of radar point cloud by fitting non-attenuated radar images, extracts ROI images for oversampling, and uses standard samples from each modality to fuse multimodal data from depth cameras, infrared cameras, and lidar. This improves the robustness and accuracy of shipborne obstacle detection, enhances the tracking performance of target tracking algorithms in complex environments, strengthens the stability and generalization ability of obstacle target detection, and improves the computational efficiency of multimodal processes.

[0019] 2. This invention extracts semantic information, detects target boundaries, determines target edges, registers radar point clouds, and constructs a three-modal dataset of infrared, visible light, and radar features by weighted fusion of visible light and infrared features. This achieves high-precision cross-modal data annotation, is suitable for target perception tasks in complex water environments, improves the quality of high-resolution point clouds in noisy environments, enhances image fusion effects in complex environments, and improves the response speed of ship navigation.

[0020] 3. This invention determines the attitude of each vessel by performing motion modeling, determines the connectivity between turning points by using the trajectory, controls the unmanned vessels to avoid obstacles and reach the center of the circle, determines the formation spacing and forms a formation, and dynamically adjusts the formation state according to the gaps between obstacles. This enhances the environmental perception capability, realizes the autonomous coordination and automated formation maintenance of the vessel cluster, and improves the operational efficiency and safety of the vessel cluster. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of a ship multimodal perception intelligent monitoring system based on data analysis according to the present invention; Figure 2 This is a schematic diagram illustrating the steps of a data analysis-based intelligent monitoring method for multimodal sensing of ships according to the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figures 1 to 2 The present invention provides a technical solution: a ship multimodal perception intelligent monitoring system based on data analysis, comprising: an image processing module, a sample calibration module, a modal fusion module, a target localization module, and an automatic formation module; The image processing module is used to acquire fused images, infrared images, and radar images, establish a reflection model based on visible light positioning, simulate multipath reflection, attenuation, and reflection noise of radar signals, fit an attenuation-free radar image, use a regression forest model to learn the differences in radar signals of obstacles between different backgrounds, detect the areas where obstacles differ from the background echo, and construct the recognition area of ​​the radar point cloud. The image processing module includes: a hardware acquisition unit and a radar processing unit; The hardware acquisition unit is used to set up a depth camera, an infrared camera, and a lidar on the vessel, and simultaneously acquire fused images, infrared images, and radar images. It performs joint calibration of the three sensors, unifies the multimodal images to the same coordinate system, and compensates for image distortion caused by the movement of the vessel based on the ship's IMU data. The radar processing unit is used to dynamically columnarize the original point cloud in the radar image, eliminate hard voxelization of the image, use median filtering and morphological processing to reduce clutter interference, detect the areas where obstacles differ from the background echo, and obtain the radar image ROI.

[0024] The sample calibration module is used to extract ROI images in the recognition area of ​​radar point cloud, oversample the ROI areas of fused image and infrared image, group the sampled signals, calculate timing deviation using the grouped signals, calculate interpolation base point and interpolation coefficient through timing deviation, interpolate the original oversampled signal to obtain sampled dataset, simulate environmental occlusion and inter-sample occlusion effects, filter invalid labels, and obtain standard samples for each modality. The sample calibration module includes: a signal sampling unit and a tag filtering unit; The signal sampling unit is used to project the bounding box of the radar identification area onto the fused image and infrared image plane using the extrinsic parameter matrix calibrated by the sensor, obtain the ROI of the corresponding image, refine the encoding of the ROI, perform pixel oversampling from the images of each modality, and group the oversampled signals according to the time window so that each group of signals is continuous at the time point. The label filtering unit is used to extract texture features from the depth map ROI and the infrared map ROI. It uses a Transformer model to fuse texture features, obtains a sampled dataset, trains an occlusion simulation model using the sampled dataset, fits the number of sample points in the dataset to the offset distance of the sample center, filters invalid samples, and obtains standard samples.

[0025] The modality fusion module is used to extract semantic information from standard samples of the fused image, detect target boundaries, map them to radar point clouds, determine target edges, label standard samples, construct visible light and radar dual-modality datasets, fuse visible light features and infrared features according to weights to generate fused features, register images based on fused features, generate dual-light fused images using the TIF algorithm, fuse the dual-modality datasets, and obtain infrared, visible light and radar three-modality datasets. The modality fusion module includes: a semantic fusion unit, an environment adjustment unit, and a multimodal data unit; The semantic fusion unit is used to extract feature points based on the geometric features of the target, register the radar point cloud, extract the target contour on the mask of semantic segmentation of the fused image using an edge detection algorithm, transform the target boundary points extracted in the fused image to the radar coordinate system, and determine the corresponding edge points in the radar point cloud through local search. The environment adjustment unit is used to train a CNN architecture using radar data, output obstacle features, register radar point clouds of consecutive frames, and annotate obstacle target edge data on standard samples to obtain a dual-modal dataset. The multimodal data unit is used to input the fused image and the infrared image into the ECA attention model, output the weights corresponding to different modes according to the illumination conditions, use the TIF algorithm to decompose the overlapping area of ​​the infrared image and the fused image, perform fusion based on the local energy maximum, generate a dual-light fused image, and spatially align it with the radar point cloud to obtain a three-modal dataset.

[0026] The target localization module is used to generate a three-dimensional spatial point cloud based on the features of the trimodal dataset, encode the point cloud based on voxels, extract the features of the three-dimensional spatial point cloud using dynamic graph convolution, fuse the radar point cloud, expand and reconstruct the coordinates of the fused point cloud to obtain a multi-dimensional point cloud, use the multi-dimensional point cloud and the dual-light fusion image to detect and segment the target, determine the target coordinates, integrate the depth information of the coordinate region into the coordinates, and perform three-dimensional localization of the target. The target localization module includes: a point cloud expansion unit and an obstacle segmentation unit; The point cloud extension unit is used to input the three-modal dataset into a conditional generative adversarial network to generate a three-dimensional spatial point cloud, convert the three-dimensional spatial point cloud into a voxel representation, extract features using a dynamic graph convolutional network, perform point-level concatenation of the three-dimensional spatial point cloud features and radar point cloud features, input them into a multilayer perceptron to obtain a fused point cloud, and extend and reconstruct the coordinates of the fused point cloud to obtain a multivariate point cloud. The obstacle segmentation unit is used to project the multi-dimensional point cloud onto the image plane and align it with the dual-light fusion image. Using the point cloud clusters of the target in the multi-dimensional point cloud, the three-dimensional position of the target is obtained by fitting a 3D bounding box. The position is input into the point cloud segmentation model, and the two-dimensional bounding box and segmentation mask of the target are output.

[0027] The automatic formation module is used to identify all obstacle targets on the movement path of the vessel cluster, perform motion modeling on each vessel, take the vessel with the smallest obstacle area in the field of view as the center, establish a metric model based on spatial density and turning similarity, cluster turning points, determine the connectivity between turning points through the trajectory, output a directed trajectory graph, control the unmanned vessels to avoid obstacles and reach the center, determine the formation spacing based on the maximum gap between obstacles, form a formation, and adjust the formation width according to obstacle information during formation navigation.

[0028] The automatic formation module includes: a motion modeling unit, a trajectory recognition unit, and a path control unit; The motion modeling unit is used to determine all obstacle targets on the movement path of the boat group, uses GPS and INS combined navigation to determine the boat attitude, and models each boat based on the boat position and heading information. The track identification unit is used to establish a metric model based on spatial density and turning similarity, cluster turning points, determine the connectivity between turning points, and generate a directed track graph. In the directed track graph, nodes are turning points, edges represent passable paths, and edge weights are the distances between nodes. The path control unit is used to find the shortest path from each vessel's position to the center of the circle in the directed track graph. Based on the directed track graph, it executes the USV longitudinal control algorithm to make the vessels sail along the planned path and avoid obstacles. Based on the multi-dimensional point cloud, it detects the gaps between obstacles and selects the largest gap as the safe distance between vessels in the formation to form a stable formation.

[0029] A data analysis-based multimodal sensing intelligent monitoring method for ships includes the following steps: Step S1. Acquire fused images, infrared images, and radar images, establish a ray-tracing-based reflection model, fit the non-attenuated radar image, detect the areas where obstacles differ from the background echo, and construct the recognition area of ​​the radar point cloud. Step S1 includes: Step S11. Set up a depth camera, an infrared camera, and a lidar on the vessel to simultaneously acquire fused images of visible light and depth, infrared images, and lidar images. Perform joint calibration on the three sensors to unify the multimodal images into the same coordinate system. Based on the shipborne IMU data, compensate for image distortion caused by the movement of the hull. Step S12. Determine the region where the obstacle is located by segmenting the obstacle in the fused image. In the radar image, the original radar point cloud is dynamically columnarized to eliminate hard voxelization of the image. Median filtering and morphological processing are used to reduce clutter interference. The location of the obstacle that differs from the background echo is detected in the region where the obstacle is located, and the obstacle location is used as the ROI of the radar image.

[0030] Step S2. Extract ROI images from the recognition area of ​​the radar point cloud, oversample the ROI areas of the fused image and infrared image, group the sampled signals, interpolate the original oversampled signals to obtain the sampled dataset, filter out invalid labels, and obtain standard samples for each modality. Step S2 includes: Step S21. Using the extrinsic parameter matrix calibrated by the sensor, project the bounding box of the radar identification area onto the fused image and infrared image plane to obtain the ROI of the corresponding image. Refine the encoding of the ROI, perform pixel oversampling from the images of each modality, and group the oversampled signals according to the time window so that each group of signals is continuous at the time point. Step S22. Extract texture features from the fused image ROI and the infrared image ROI, fuse the texture features using the Transformer model, adjust the sampling frequency, obtain the sampling dataset, train the occlusion simulation model using the sampling dataset, fit the number of sample points in the dataset to the offset distance of the sample center, filter invalid samples, and obtain standard samples.

[0031] Step S3. Extract semantic information from the standard samples of the fused image, detect the target boundary, map it to the radar point cloud, determine the target edge, label the standard samples, construct a dual-modal dataset, weighted fuse visible light features and infrared features to generate fused features, generate a dual-light fused image based on the fused features, and obtain a tri-modal dataset after sampling and merging. Step S3 includes: Step S31. Extract feature points based on the target's geometric features, register the radar point cloud, use an edge detection algorithm to extract the target contour on the mask of semantic segmentation of the fused image, transform the target boundary points extracted from the fused image to the radar coordinate system, and determine the corresponding edge points in the radar point cloud through local search. Step S32. Train a CNN architecture using radar data, output obstacle features, register radar point clouds of consecutive frames, and annotate obstacle target edge data on standard samples to obtain a dual-modal dataset. Step S33. Input the fused image and the infrared image into the ECA attention model, output the weights corresponding to different modes according to the illumination conditions, use the TIF algorithm to decompose the overlapping area of ​​the infrared image and the fused image, fuse them based on the local energy maximum, generate a dual-light fused image, spatially align it with the radar point cloud, and obtain a three-modal dataset.

[0032] Step S4. Generate a three-dimensional spatial point cloud based on the features of the three-modal dataset, fuse the radar point cloud and expand and reconstruct the coordinates to obtain a multi-dimensional point cloud, use the multi-dimensional point cloud and the dual-light fusion image to detect and segment obstacle targets, and perform three-dimensional localization of the targets; Step S4 includes: Step S41. Input the trimodal dataset into the conditional generative adversarial network to generate a three-dimensional spatial point cloud. Convert the three-dimensional spatial point cloud into a voxel representation. Use a dynamic graph convolutional network to extract features. Perform point-level concatenation of the three-dimensional spatial point cloud features and radar point cloud features. Input the concatenation into a multilayer perceptron to obtain a fused point cloud. Expand and reconstruct the coordinates of the fused point cloud to obtain a multivariate point cloud. Step S42. Project the multi-dimensional point cloud onto the image plane and align it with the dual-light fusion image. Using the point cloud clusters of the target in the multi-dimensional point cloud, obtain the three-dimensional position of the target through 3D bounding box fitting. Input the position into the point cloud segmentation model and output the two-dimensional bounding box and segmentation mask of the target.

[0033] Step S5. Perform motion modeling on each vessel, establish a metric model, cluster turning points, determine the connectivity between turning points through the trajectory, output a directed trajectory graph, take the vessel with the smallest obstacle area in the field of view as the center, control the unmanned vessel to avoid obstacles and reach the center of the circle based on all obstacle targets on the vessel's motion path, and form a formation.

[0034] Step S5 includes: Step S51. Determine all obstacles on the movement path of the boat cluster, use GPS and INS combined navigation to determine the boat attitude, model each boat based on its position and heading information, establish a metric model based on spatial density and turning similarity, cluster the turning points, determine the connectivity between the turning points, and generate a directed track graph. In the directed track graph, the nodes are turning points, the edges represent passable paths, and the edge weight is the distance between the nodes. Step S52. Using the path optimization algorithm, with the current position of the vessel as the starting point and the directed graph track as the orbit, plan the shortest path from each vessel's position to the center of the circle in the directed graph of the track, and execute the USV longitudinal control algorithm to make the vessel sail along the planned path, avoid obstacles, and reach the formation area. Step S53. During formation navigation, continuously update the position of obstacles. When the gap between obstacles cannot be passed by the formation, detect the gap between obstacles based on multi-dimensional point cloud, select the largest gap as the safe distance between ships in the formation, adjust the formation width according to the safe distance, form a stable formation that can pass through obstacles, and restore the original formation after passing through obstacles.

[0035] Example: Capture trimodal images from a depth camera, a thermal infrared camera, and a radar. Output time-aligned RGB images, thermal images, and a radar data matrix. Divide the radar 3D point cloud into dynamic sector columns according to azimuth angle. Project the point cloud energy within each column onto a plane to generate a corresponding radar reflection intensity map. Simulate radar reflection. Obtain the core region through constant false alarm rate detection. Extract multimodal ROIs. Group and oversample the signals. Calculate timing deviations and interpolate the signals to generate standard samples. Perform image semantic segmentation. Extract various targets such as vehicles and pedestrians from the segmentation results. Extract target boundaries and map semantic boundaries onto the radar point cloud to determine target edges. Align dual-modal data. Use a dual-stream network to fuse visible light texture features and infrared temperature features to construct a trimodal dataset. Assign surface roughness and hardness probabilities to the point cloud to generate a 3D spatial point cloud. Extract dynamic image convolution features. Perform target detection and segmentation based on multimodal point cloud and dual-light fusion image.

[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0037] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data analysis-based multimodal sensing intelligent monitoring method for ships, characterized in that, The method includes the following steps: Step S1. Acquire fused images of visible light and depth, infrared images, and radar images; establish a radar reflection model based on visible light positioning; fit the radar image without attenuation; detect the areas where obstacles differ from the background echo; and construct the recognition area of ​​the radar point cloud. Step S2. Extract ROI images from the recognition area of ​​the radar point cloud, oversample the ROI areas of the fused image and infrared image, group the sampled signals, interpolate the original oversampled signals to obtain the sampled dataset, filter out invalid labels, and obtain standard samples for each modality. Step S3. Extract semantic information from the standard samples of the fused image, detect the target boundary, map it to the radar point cloud, determine the target edge, label the standard samples, construct a dual-modal dataset, weighted fuse visible light features and infrared features to generate fused features, generate a dual-light fused image based on the fused features, and obtain a tri-modal dataset after sampling and merging. Step S4. Generate a three-dimensional spatial point cloud based on the features of the three-modal dataset, fuse the radar point cloud and expand and reconstruct the coordinates to obtain a multi-dimensional point cloud, use the multi-dimensional point cloud and the dual-light fusion image to detect and segment obstacle targets, and perform three-dimensional localization of the targets; Step S5. Perform motion modeling on each vessel, establish a metric model, cluster turning points, determine the connectivity between turning points through the trajectory, output a directed trajectory graph, take the vessel with the smallest obstacle area in the field of view as the center, control the unmanned vessel to avoid obstacles and reach the center based on all obstacle targets on the vessel's motion path, form a formation, and adjust the formation width according to obstacle information during formation navigation.

2. The intelligent monitoring method for multimodal perception of ships based on data analysis according to claim 1, characterized in that: Furthermore, step S1 includes: Step S11. Set up a depth camera, an infrared camera, and a lidar on the vessel to simultaneously acquire fused images of visible light and depth, infrared images, and lidar images. Perform joint calibration on the three sensors to unify the multimodal images into the same coordinate system. Based on the shipborne IMU data, compensate for image distortion caused by the movement of the hull. Step S12. Determine the region where the obstacle is located by segmenting the obstacle in the fused image. In the radar image, the original radar point cloud is dynamically columnarized to eliminate hard voxelization of the image. Median filtering and morphological processing are used to reduce clutter interference. The location of the obstacle that differs from the background echo is detected in the region where the obstacle is located. The location of the obstacle is used as the ROI of the radar image. Step S2 includes: Step S21. Using the extrinsic parameter matrix calibrated by the sensor, project the bounding box of the radar identification area onto the fused image and infrared image plane to obtain the ROI of the corresponding image. Refine the encoding of the ROI, perform pixel oversampling from the images of each modality, and group the oversampled signals according to the time window so that each group of signals is continuous at the time point. Step S22. Extract texture features from the fused image ROI and the infrared image ROI, fuse the texture features using the Transformer model, adjust the sampling frequency, obtain the sampling dataset, train the occlusion simulation model using the sampling dataset, fit the number of sample points in the dataset to the offset distance of the sample center, filter invalid samples, and obtain standard samples.

3. The intelligent monitoring method for multimodal perception of ships based on data analysis according to claim 2, characterized in that: Step S3 includes: Step S31. Extract feature points based on the target's geometric features, register the radar point cloud, use an edge detection algorithm to extract the target contour on the mask of semantic segmentation of the fused image, transform the target boundary points extracted from the fused image to the radar coordinate system, and determine the corresponding edge points in the radar point cloud through local search. Step S32. Train a CNN architecture using radar data, output obstacle features, register radar point clouds of consecutive frames, and annotate obstacle target edge data on standard samples to obtain a dual-modal dataset. Step S33. Input the fused image and the infrared image into the ECA attention model, output the weights corresponding to different modes according to the illumination conditions, use the TIF algorithm to decompose the overlapping area of ​​the infrared image and the fused image, fuse them based on the local energy maximum, generate a dual-light fused image, spatially align it with the radar point cloud, and obtain a three-modal dataset.

4. The intelligent monitoring method for multimodal perception of ships based on data analysis according to claim 3, characterized in that: Step S4 includes: Step S41. Input the trimodal dataset into the conditional generative adversarial network to generate a three-dimensional spatial point cloud. Convert the three-dimensional spatial point cloud into a voxel representation. Use a dynamic graph convolutional network to extract features. Perform point-level concatenation of the three-dimensional spatial point cloud features and radar point cloud features. Input the concatenation into a multilayer perceptron to obtain a fused point cloud. Expand and reconstruct the coordinates of the fused point cloud to obtain a multivariate point cloud. Step S42. Project the multi-dimensional point cloud onto the image plane and align it with the dual-light fusion image. Using the point cloud clusters of the target in the multi-dimensional point cloud, obtain the three-dimensional position of the target through 3D bounding box fitting. Input the position into the point cloud segmentation model and output the two-dimensional bounding box and segmentation mask of the target.

5. The intelligent monitoring method for multimodal perception of ships based on data analysis according to claim 4, characterized in that: Step S5 includes: Step S51. Determine all obstacles on the movement path of the boat cluster, use GPS and INS combined navigation to determine the boat attitude, model each boat based on its position and heading information, establish a metric model based on spatial density and turning similarity, cluster the turning points, determine the connectivity between the turning points, and generate a directed track graph. In the directed track graph, the nodes are turning points, the edges represent passable paths, and the edge weight is the distance between the nodes. Step S52. Using the path optimization algorithm, with the current position of the vessel as the starting point and the directed graph track as the orbit, plan the shortest path from each vessel's position to the center of the circle in the directed graph of the track, and execute the USV longitudinal control algorithm to make the vessel sail along the planned path, avoid obstacles, and reach the formation area. Step S53. During formation navigation, continuously update the position of obstacles. When the gap between obstacles cannot be passed by the formation, detect the gap between obstacles based on multi-dimensional point cloud, select the largest gap as the safe distance between ships in the formation, adjust the formation width according to the safe distance, form a stable formation that can pass through obstacles, and restore the original formation after passing through obstacles.

6. A ship multimodal sensing intelligent monitoring system based on data analysis, characterized in that, The system includes the following modules: image processing module, sample calibration module, modality fusion module, target localization module, and automatic formation module; The image processing module is used to acquire fused images, infrared images, and radar images, establish a reflection model based on visible light positioning, simulate multipath reflection, attenuation, and reflection noise of radar signals, fit an attenuation-free radar image, use a regression forest model to learn the differences in radar signals of obstacles between different backgrounds, detect the areas where obstacles differ from the background echo, and construct the recognition area of ​​the radar point cloud. The sample calibration module is used to extract ROI images in the recognition area of ​​radar point cloud, oversample the ROI areas of fused image and infrared image, group the sampled signals, calculate timing deviation using the grouped signals, calculate interpolation base point and interpolation coefficient through timing deviation, interpolate the original oversampled signal to obtain sampled dataset, simulate environmental occlusion and inter-sample occlusion effects, filter invalid labels, and obtain standard samples for each modality. The modality fusion module is used to extract semantic information from standard samples of the fused image, detect target boundaries, map them to radar point clouds, determine target edges, label standard samples, construct visible light and radar dual-modality datasets, fuse visible light features and infrared features according to weights to generate fused features, register images based on fused features, generate dual-light fused images using the TIF algorithm, fuse the dual-modality datasets, and obtain infrared, visible light and radar three-modality datasets. The target localization module is used to generate a three-dimensional spatial point cloud based on the features of the trimodal dataset, encode the point cloud based on voxels, extract the features of the three-dimensional spatial point cloud using dynamic graph convolution, fuse the radar point cloud, expand and reconstruct the coordinates of the fused point cloud to obtain a multi-dimensional point cloud, use the multi-dimensional point cloud and the dual-light fusion image to detect and segment the target, determine the target coordinates, integrate the depth information of the coordinate region into the coordinates, and perform three-dimensional localization of the target. The automatic formation module is used to identify all obstacle targets on the movement path of the vessel cluster, perform motion modeling on each vessel, take the vessel with the smallest obstacle area in the field of view as the center, establish a metric model based on spatial density and turning similarity, cluster turning points, determine the connectivity between turning points through the trajectory, output a directed trajectory graph, control the unmanned vessels to avoid obstacles and reach the center, determine the formation spacing based on the maximum gap between obstacles, form a formation, and adjust the formation width according to obstacle information during formation navigation.

7. The intelligent monitoring system for multimodal sensing of ships based on data analysis according to claim 6, characterized in that: The image processing module includes: a hardware acquisition unit and a radar processing unit; The hardware acquisition unit is used to set up a depth camera, an infrared camera, and a lidar on the vessel, and simultaneously acquire fused images, infrared images, and radar images. It performs joint calibration of the three sensors, unifies the multimodal images to the same coordinate system, and compensates for image distortion caused by the movement of the vessel based on the ship's IMU data. The radar processing unit is used to dynamically columnarize the original point cloud in the radar image, eliminate hard voxelization of the image, use median filtering and morphological processing to reduce clutter interference, detect the area where the obstacle is located that differs from the background echo, and obtain the radar image ROI. The sample calibration module includes: a signal sampling unit and a tag filtering unit; The signal sampling unit is used to project the bounding box of the radar identification area onto the fused image and infrared image plane using the extrinsic parameter matrix calibrated by the sensor, obtain the ROI of the corresponding image, refine the encoding of the ROI, perform pixel oversampling from the images of each modality, and group the oversampled signals according to the time window so that each group of signals is continuous at the time point. The label filtering unit is used to extract texture features from the depth map ROI and the infrared map ROI. It uses a Transformer model to fuse texture features, obtains a sampled dataset, trains an occlusion simulation model using the sampled dataset, fits the number of sample points in the dataset to the offset distance of the sample center, filters invalid samples, and obtains standard samples.

8. The ship multimodal perception intelligent monitoring system based on data analysis according to claim 7, characterized in that: The modality fusion module includes: a semantic fusion unit, an environment adjustment unit, and a multimodal data unit; The semantic fusion unit is used to extract feature points based on the geometric features of the target, register the radar point cloud, extract the target contour on the mask of semantic segmentation of the fused image using an edge detection algorithm, transform the target boundary points extracted in the fused image to the radar coordinate system, and determine the corresponding edge points in the radar point cloud through local search. The environment adjustment unit is used to train a CNN architecture using radar data, output obstacle features, register radar point clouds of consecutive frames, and annotate obstacle target edge data on standard samples to obtain a dual-modal dataset. The multimodal data unit is used to input the fused image and the infrared image into the ECA attention model, output the weights corresponding to different modes according to the illumination conditions, use the TIF algorithm to decompose the overlapping area of ​​the infrared image and the fused image, perform fusion based on the local energy maximum, generate a dual-light fused image, and spatially align it with the radar point cloud to obtain a three-modal dataset.

9. The ship multimodal perception intelligent monitoring system based on data analysis according to claim 8, characterized in that: The target localization module includes: a point cloud expansion unit and an obstacle segmentation unit; The point cloud extension unit is used to input the three-modal dataset into a conditional generative adversarial network to generate a three-dimensional spatial point cloud, convert the three-dimensional spatial point cloud into a voxel representation, extract features using a dynamic graph convolutional network, perform point-level concatenation of the three-dimensional spatial point cloud features and radar point cloud features, input them into a multilayer perceptron to obtain a fused point cloud, and extend and reconstruct the coordinates of the fused point cloud to obtain a multivariate point cloud. The obstacle segmentation unit is used to project the multi-dimensional point cloud onto the image plane and align it with the dual-light fusion image. Using the point cloud clusters of the target in the multi-dimensional point cloud, the three-dimensional position of the target is obtained by fitting a 3D bounding box. The position is input into the point cloud segmentation model, and the two-dimensional bounding box and segmentation mask of the target are output.

10. A ship multimodal sensing intelligent monitoring system based on data analysis according to claim 9, characterized in that: The automatic formation module includes: a motion modeling unit, a trajectory recognition unit, and a path control unit; The motion modeling unit is used to determine all obstacle targets on the movement path of the boat group, uses GPS and INS combined navigation to determine the boat attitude, and models each boat based on the boat position and heading information. The track identification unit is used to establish a metric model based on spatial density and turning similarity, cluster turning points, determine the connectivity between turning points, and generate a directed track graph. In the directed track graph, nodes are turning points, edges represent passable paths, and edge weights are the distances between nodes. The path control unit is used to find the shortest path from each vessel's position to the center of the circle in the directed track graph. Based on the directed track graph, it executes the USV longitudinal control algorithm to make the vessels sail along the planned path and avoid obstacles. Based on the multi-dimensional point cloud, it detects the gaps between obstacles and selects the largest gap as the safe distance between vessels in the formation to form a stable formation.

Images