A Multimodal Data Collaborative Intelligent Detection and 3D Positioning Method for Ships at Sea

Abstract

This application discloses a multimodal data collaborative intelligent detection and 3D localization method for maritime vessels, comprising: fine preprocessing of collected multi-source heterogeneous data and preliminary target detection; dynamic weighted fusion of the preprocessed multi-source heterogeneous data, combined with depth data from lidar, to achieve accurate coordinate recovery of the vessel target in 3D space; and prediction and updating of the motion state of the identified target based on a multi-tracking mechanism collaborative strategy. By fully utilizing the rich texture information of visible light images, the anti-interference capability of thermal imaging, and the precise spatial depth data of lidar through multimodal data perception and preliminary processing, heterogeneous data deep fusion and 3D localization, and advanced multi-target continuous tracking and state estimation, this method integrates a multi-tracking mechanism collaborative strategy to predict and update the motion state of the identified target, effectively addressing the problem of tracking interruption caused by nonlinear motion or temporary occlusion of maritime targets.

Description

Technical Field

[0001] This application relates to the field of target detection and multi-sensor data fusion technology, specifically to a method for intelligent detection and three-dimensional positioning of ships at sea using multimodal data collaboration. Background Technology

[0002] Maritime target detection is a crucial foundational technology for ensuring navigation safety, enabling intelligent navigation, and naval surveillance. Traditional methods primarily rely on visible light images or radar data, employing image processing or machine learning algorithms to identify and track vessel targets. However, single-modal perception capabilities face multiple challenges in complex maritime conditions. First, the maritime environment is characterized by unstable lighting, variable weather, and a wide range of target sizes. Traditional RGB image-based detection methods are prone to missed or false detections under low visibility conditions such as nighttime, dense fog, and strong backlighting. Second, while thermal imaging offers some adaptability to adverse lighting conditions, it lacks clear texture and boundary information, making it difficult to accurately locate target outlines. Although lidar can provide target depth and spatial location information, its capabilities in small target detection and semantic segmentation are limited.

[0003] Current maritime target detection systems largely rely on single-modal sensing methods, making it difficult to cope with diverse scenarios under complex sea conditions. While RGB images contain rich texture and semantic information, their image quality deteriorates sharply in low-visibility environments such as nighttime, dense fog, and strong backlighting, leading to blurred target segmentation and significant risks of missed detections and false identifications. Although thermal imaging is resistant to illumination interference, it lacks clear structural features and high-frequency information, making it prone to boundary blurring and positioning misalignment in densely packed multi-target scenarios. While lidar can provide accurate spatial depth data, it is limited by viewpoint obstruction and reflection interference, resulting in insufficient accuracy for detecting long-range or small targets. The inherent limitations of single-modal sensing severely restrict the robustness and accuracy of detection.

[0004] Existing multimodal fusion methods mostly remain at the shallow-level feature stitching or decision-level weighted fusion stage, lacking an effective cross-modal consistency modeling mechanism. This simple fusion approach fails to fully exploit the complementary characteristics of each modality, easily leading to target re-identification failures or semantic conflicts—for example, RGB and thermal imaging detection results cannot be effectively correlated due to feature differences, or there are deviations between LiDAR depth data and visual target localization, further reducing the reliability of fused detection. While some depth detection models based on complex structures can improve accuracy, they suffer from slow inference speed and high computational resource consumption. Maritime unmanned platforms have strict engineering deployment requirements for real-time performance and low power consumption; such models cannot meet the rapid response requirements in real-world scenarios, limiting their application in intelligent navigation systems.

[0005] In recent years, deep learning methods (such as the YOLO series of networks) have made significant progress in visual object detection, exhibiting good end-to-end detection performance and real-time capabilities. However, their ability to handle high-quality image input and adapt to different environments still needs improvement, particularly in the area of ​​multi-source sensor fusion, where a systematic engineering implementation path is still lacking. Therefore, how to combine the advantages of thermal imaging, LiDAR, and RGB vision sensors to achieve all-weather, all-scenario ship detection and 3D localization has become a key technical challenge for current intelligent navigation systems. Summary of the Invention

[0006] In order to solve the above-mentioned technical problems, this application proposes the following technical solution:

[0007] In a first aspect, embodiments of this application provide a multimodal data collaborative intelligent detection and three-dimensional positioning method for maritime vessels, including:

[0008] The collected multi-source heterogeneous data is subjected to refined preprocessing and preliminary target detection.

[0009] The preprocessed multi-source heterogeneous data is dynamically weighted and fused, and combined with the depth data from the lidar, to achieve accurate coordinate recovery of the ship target in three-dimensional space.

[0010] The motion state of identified targets is predicted and updated based on a multi-tracking mechanism collaborative strategy.

[0011] In one possible implementation, the refined preprocessing and preliminary target detection of the collected multi-source heterogeneous data includes:

[0012] Multi-sensor devices are used to synchronously acquire multi-modal data and obtain multi-source heterogeneous data;

[0013] The multi-source heterogeneous data is preprocessed to obtain an enhanced dataset;

[0014] Construct a YOLOX multi-branch parallel sensing network based on adaptive optimization;

[0015] The YOLOX multi-branch parallel sensing network based on adaptive optimization is trained using the augmented dataset.

[0016] Initial target detection was performed using a YOLOX multi-branch parallel sensing network that had been trained and optimized adaptively.

[0017] In one possible implementation, the step of using multi-sensor devices to synchronously acquire multi-modal data and obtain multi-source heterogeneous data includes:

[0018] Multimodal data is simultaneously acquired using both a light camera and a thermal imaging camera, wherein the light camera acquires RGB images and depth information, and the thermal imaging camera acquires thermal images.

[0019] The intrinsic parameter matrices of the visible light camera and the thermal imaging camera were obtained using the checkerboard calibration method.

[0020] A unified clock module is introduced to achieve timestamp synchronization of multi-sensor sampling, ensuring precise alignment between RGB images, thermal imaging, and LiDAR data frames.

[0021] In one possible implementation, the preprocessing of the multi-source heterogeneous data to obtain the augmented dataset includes:

[0022] The Mixup data augmentation strategy is adopted, which randomly selects two images and their corresponding labels, and then linearly combines their pixel values ​​and labels in proportion to generate training samples;

[0023] A linear combination of Mixups can be represented as:

[0024] ;

[0025] ;

[0026] in: , It is the original image. , These are the corresponding tags. It is the mixing coefficient;

[0027] The thermal imager's image frame was adjusted to a preset multiple of its original size to achieve precise matching of image frames from multiple devices.

[0028] ;

[0029] ;

[0030] Where W and H represent the width and height of the image, respectively, and 0.7 represents the preset multiplier.

[0031] In one possible implementation, the adaptively optimized YOLOX multi-branch parallel perception network is based on the YOLOX network, and ECANet and SENet channel attention mechanisms are incorporated into the YOLOX backbone network or neck network. The channel attention mechanism establishes feature dependencies in the channel dimension, dynamically learns and adjusts the weights of each channel, so that the network can actively focus on the effective features of key targets of the ship, while suppressing the interference of redundant information.

[0032] The adaptively optimized YOLOX multi-branch parallel perceptual network training process introduces a CIoU loss function, which considers the overlap area between the predicted and ground truth boxes, the distance between their center points, and the consistency of the aspect ratio of the bounding boxes; the total loss function... The composition remains as classification loss. Regression loss and targeted loss Weighted sum:

[0033] ;

[0034] in: , , These are weighting coefficients used to balance the various losses. The classification loss is typically binary cross-entropy (BCE) loss or Focal loss; for each predicted bounding box and the ground truth label, the BCE loss is calculated as follows:

[0035] ;

[0036] in: It is a real label (0 or 1). It is the probability predicted by the model; It is the regression loss, calculated using the CIoU loss function.

[0037] In one possible implementation, the adaptively optimized YOLOX multi-branch parallel sensing network performs thermal imaging heat source identification and binarization optimization based on a refined strategy of connected component analysis, including:

[0038] First, the raw thermal imaging data is binarized using a temperature threshold to separate the ship's thermal radiation area above the threshold, while the water background below the threshold is filtered out. The binarization function is:

[0039] ;

[0040] in: It is a pixel. Temperature value at that location, It is the set temperature threshold (20℃). These are the pixel values ​​after binarization;

[0041] Subsequently, through connected component analysis of two scans, all connected white pixel regions are identified and marked, thereby accurately calculating the number of heat source targets and generating the minimum bounding rectangle for them.

[0042] In one possible implementation, the process of identifying and marking all connected white pixel regions through connected component analysis in two scans, thereby accurately calculating the number of heat source targets and generating a minimum bounding rectangle for them, includes:

[0043] First, the binary image is traversed pixel by pixel. For white pixels encountered, the existing label or a new label is assigned to the pixel depending on whether there is a labeled connected component in its neighborhood.

[0044] Traverse the image again and merge the labels of the same connected components based on the equivalent label table established in the first scan;

[0045] Finally, the set of pixels with the same label constitutes a connected component, which the system uses to determine the number of ship targets; the minimum bounding rectangle is calculated for each connected component, and anchor box labels are drawn as the detection results of the thermal imaging branch.

[0046] In one possible implementation, the step of dynamically weighting and fusing the preprocessed multi-source heterogeneous data, and combining it with lidar depth data, to achieve accurate coordinate recovery of the ship target in three-dimensional space includes:

[0047] IoU is used as the basic tool to measure the overlap between RGB and thermal imaging detection boxes. The overlap relationship is quantified by calculating the ratio of the intersection area to the union area of ​​the two bounding boxes, which provides a basis for subsequent fusion decisions.

[0048] We design a dynamic weighted fusion and adaptive switching strategy, which utilizes the complementary characteristics between different data features to evaluate the effectiveness of each modality in real time and optimize the fusion results.

[0049] Finally, the depth image of the target is acquired using an optical radar camera, and the precise coordinates of the target in three-dimensional space are recovered through camera intrinsic parameters and back projection algorithms.

[0050] In one possible implementation, the design of the dynamic weighted fusion and adaptive switching strategy utilizes the complementary characteristics between different data features to evaluate the effectiveness of each modality in real time to optimize the fusion result, including:

[0051] For each detected target, the respective detection confidence scores CRGB and RGB are obtained from the RGB and thermal imaging branches. The final fusion confidence level The weighted average is obtained as follows:

[0052] ;

[0053] in: and It is a dynamic weight, and ;

[0054] Set dynamic thresholds and priority rules to achieve intelligent switching between modalities:

[0055] Normal mode: When visibility is good, the RGB visual branch is the dominant modality, and its detection results have high priority. Approaching 1, Approaching 0;

[0056] Assisted Enhancement Mode: When the detection confidence of the RGB visual branch... Below the preset threshold At this time, the system enters the auxiliary enhancement mode, and the weight of the thermal imaging branch is adjusted accordingly. It will depend on its own confidence level The system dynamically adjusts to adapt to changing environmental conditions in order to compensate for the limitations of RGB vision.

[0057] Thermal imaging-dominant mode: In extremely harsh environments, if If the value is far below the threshold or the RGB branch is completely disabled, the system will switch to thermal imaging-dominated mode. Approaching 1, The accuracy is close to zero, mainly relying on the detection results of thermal imaging and three-dimensional positioning through lidar;

[0058] LiDAR Priority: As an independent branch providing accurate 3D positioning, LiDAR 3D coordinate information is correlated with the fused 2D detection results to provide the final 3D positioning output.

[0059] In one possible implementation, the final step involves acquiring a depth image of the target using an optical radar camera and recovering the target's precise coordinates in three-dimensional space through camera intrinsic parameters and a back-projection algorithm, including:

[0060] An optical radar camera directly outputs the depth value of each pixel using the Time-of-Flight (ToF) principle, forming a depth image. Each pixel value in the depth image represents the distance from that pixel to the camera plane. The camera intrinsic parameter matrix K describes the projection relationship of the camera from the three-dimensional camera coordinate system to the two-dimensional pixel coordinate system. For an ideal pinhole camera model, the intrinsic parameter matrix is ​​usually expressed as:

[0061] ;

[0062] in: and These are the focal lengths along the x-axis and y-axis, respectively. These are the coordinates of the principal point;

[0063] Given a pixel and the corresponding depth values ​​obtained from the depth map. The following formula is used to back-project it to three-dimensional coordinates in the camera coordinate system. :

[0064] ;

[0065] ;

[0066] ;

[0067] in: These are the pixel coordinates in the image. It is the depth value of that pixel in the depth map. , , , These are the camera's intrinsic parameters;

[0068] For each ship target bounding box output by the RGB detection branch First, determine its center pixel coordinates. Then, the depth value of the center pixel is obtained from the synchronously acquired depth image. Using the above back projection formula, Convert to 3D coordinates in camera coordinate system If you need to convert 3D coordinates in the camera coordinate system to 3D coordinates in the world coordinate system... Then, the camera's extrinsic parameters, rotation matrix R and translation vector T, are required, and their relationship is as follows:

[0069] ;

[0070] in It is a point in the camera coordinate system. It is a point in the world coordinate system.

[0071] In one possible implementation, the prediction and updating of the motion state of the identified target based on the multi-tracking mechanism collaborative strategy includes:

[0072] Based on a multi-algorithm collaborative multi-target continuous tracking module, Kalman filtering and the Hungarian algorithm are used as the core of the multi-target tracking framework. Kalman filtering predicts the possible position and state of the target in the next frame based on the target's historical trajectory and a preset motion model. The mathematical model of the prediction stage can be expressed as:

[0073] ;

[0074] ;

[0075] Subsequently, the Hungarian algorithm was used to optimally match the detection results of the current frame with the prediction results of the Kalman filter. By solving a cost matrix, the matching scheme that minimizes the total cost was found, thereby ensuring that each detected target was correctly assigned to the corresponding historical trajectory.

[0076] A robust tracker is designed with particle filtering as an auxiliary method, which approximates the posterior probability distribution of the target by using a set of weighted random samples.

[0077] When the prediction confidence of the Kalman filter falls below a certain threshold, the particle filter is dynamically activated. The particle filter generates a large number of particles in the region where the target may exist and updates their weights based on the latest observations to recover targets that have temporarily disappeared or whose motion patterns have changed, thus ensuring the continuity of the trajectory. The particle weight update formula is as follows:

[0078] ;

[0079] in: It is the first Individual particles Weight of time, yes The observed value at time, It is the first Individual particles The state at any given moment, It is the likelihood function;

[0080] Resampling is performed based on the particle weights, eliminating low-weight particles and replicating high-weight particles to prevent particle degradation.

[0081] In this embodiment, multimodal data perception and preliminary processing, heterogeneous data deep fusion and 3D localization, and advanced multi-target continuous tracking and state estimation are employed. By constructing a multi-branch parallel perception architecture, the rich texture information of visible light images, the anti-interference capability of thermal imaging, and the precise spatial depth data of lidar are fully utilized to achieve all-weather, highly robust, and high-precision intelligent perception of maritime targets. An integrated multi-tracking mechanism collaborative strategy predicts and updates the motion state of identified targets, effectively addressing the problem of tracking interruptions caused by nonlinear motion or temporary obstruction of maritime targets. Attached Figure Description

[0082] Figure 1 A flowchart illustrating a multimodal data collaborative intelligent detection and 3D positioning method for maritime vessels provided in this application embodiment;

[0083] Figure 2 A schematic diagram of the overall network framework provided for embodiments of this application;

[0084] Figure 3This is a diagram of the YOLOX network structure provided in the embodiments of this application;

[0085] Figure 4 A schematic diagram of connected component analysis provided in an embodiment of this application;

[0086] Figure 5 This is a schematic diagram of IoU calculation provided in an embodiment of this application;

[0087] Figure 6 This is a schematic diagram of the back projection from 2D pixels to 3D spatial points provided in an embodiment of this application. Detailed Implementation

[0088] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0089] See Figure 1 and Figure 2 The multimodal data collaborative intelligent detection and three-dimensional positioning method for maritime vessels provided in this embodiment includes:

[0090] S101 performs refined preprocessing and preliminary target detection on the collected multi-source heterogeneous data.

[0091] This stage aims to refine the collected multimodal data and perform preliminary target detection and feature extraction on RGB images and thermal imaging data, laying the foundation for subsequent deep fusion.

[0092] In this embodiment, the following sensor devices are used for multimodal data synchronous acquisition:

[0093] RGB Image and Depth Information Acquisition: The system uses an Intel RealSense L515 LiDAR camera to acquire RGB images and depth information. This device employs MEMS solid-state LiDAR technology, featuring high laser power efficiency and low power consumption (less than 3.5W), and integrates a full HD RGB camera and an IMU (Inertial Measurement Unit). Its operating range is 0.25 meters to 9 meters, with a depth resolution of 1280×720, a frame rate of 30 FPS, and a field of view of 70°×55°.

[0094] Thermal Infrared Data Acquisition: The system uses a Hikvision HM-TD3117T-2 / Q monocular card-type thermal imager to synchronously acquire thermal infrared data. This thermal imager has a response band of 8-14μm, a resolution of 160×120, and a frame rate of 25 FPS. It has a wide temperature measurement range, switching between -20°C to 150°C or 0°C to 550°C, with a measurement accuracy of ±2°C or ±2% of the reading.

[0095] The intrinsic parameter matrices of the visible light camera and the thermal imaging camera were obtained using the checkerboard calibration method.

[0096] RGB camera intrinsic parameter matrix:

[0097] ;

[0098] Thermal imaging camera intrinsic parameter matrix:

[0099] ;

[0100] A unified clock module is introduced to synchronize the timestamps of multi-sensor sampling, ensuring precise alignment between RGB, thermal imaging, and LiDAR data frames.

[0101] To address the complex scenarios in sea surface target detection tasks, this embodiment performs data augmentation on approximately 6,700 manually labeled images to construct an augmented dataset for training the model.

[0102] The Mixup data augmentation strategy is employed, randomly selecting two images and their corresponding labels, and then linearly combining their pixel values ​​and labels proportionally to generate "non-existent training samples." This method mixes small targets with large targets, forcing the model to learn the features of small targets, thereby enhancing the algorithm's detection accuracy and generalization ability.

[0103] A linear combination of Mixups can be represented as:

[0104] ;

[0105] ;

[0106] in: , It is the original image. , These are the corresponding tags. It is the mixing coefficient.

[0107] During multimodal data acquisition, there is a significant difference in the imaging field of view between the LiDAR camera and the thermal imager, resulting in inconsistent image sizes. To solve this problem, the team experimentally determined that adjusting the thermal imager's field of view to 0.7 times its original size successfully achieved precise matching of the imaging field of view between the two devices.

[0108] ;

[0109] ;

[0110] Where W and H represent the width and height of the image, respectively.

[0111] The core detection module in this embodiment is based on the YOLOX_M model and incorporates two key optimizations for maritime vessel detection tasks. The YOLOX network, with its anchor-free mechanism, decoupled detection head, and SimOTA dynamic label allocation strategy, has demonstrated excellent real-time performance and high accuracy in general object detection. The anchor-free design avoids the limitations of preset anchor boxes, and the decoupled head structure allows for independent optimization of classification, regression, and confidence tasks, significantly improving convergence speed and accuracy. The SimOTA strategy dynamically allocates positive and negative samples based on target features, resolving the issue of unreasonable matching caused by manually set thresholds in traditional methods.

[0112] The YOLOX network structure consists of four core modules: the input, the backbone, the neck, and the prediction head. Figure 3 The network structure of YOLOX is shown in detail, which is fundamental to understanding its working principle.

[0113] (1) The input terminal receives an RGB image with a size of 640×640×3. All input images will be uniformly adjusted to this size, which helps to simplify network calculations and reduce computational complexity and instability caused by differences in image size.

[0114] (2) The backbone network is built on Darknet53. Its main function is to extract features at different levels in the image, from low-level edge and texture features to high-level semantic features, to provide rich feature representations for subsequent object detection tasks. The core of the backbone network is the combination of "Conv+ResBlock".

[0115] ① The CBL module consists of convolution (Conv), batch normalization (BN), and LeakyReLU activation functions, responsible for initial feature extraction and nonlinear transformation. The output Y of a convolutional layer can be expressed as:

[0116] ;

[0117] in, It is the input feature map. It is a convolution kernel. It is a bias term. This represents the convolution operation. Batch normalization normalizes the input for each batch, using the following formula:

[0118] ;

[0119] ;

[0120] in, and These are the batch mean and variance. and These are learnable scaling and translation parameters. It is a small constant to prevent division by zero. The LeakyReLU activation function introduces nonlinearity:

[0121] ;

[0122] in It is a small positive number (usually 0.01).

[0123] ② Residual blocks, by introducing residual connections, effectively solve the vanishing gradient problem commonly encountered in deep network training, allowing networks to be stacked deeper and thus extract more complex and abstract features. The output of a residual block... It can be represented as:

[0124] ;

[0125] in, These are input features. It is the output after a series of convolutional layers and activation function transformations.

[0126] ③ The Spatial Pyramid Pooling (SPP) module performs multi-scale max pooling on the deep feature maps output by the backbone network, then concatenates and fuses them. This aggregates multi-scale contextual information, enhances the network's receptive field, and allows it to better adapt to targets of different sizes, improving its ability to capture features from small and irregular targets. SPP uses pooling kernels of different sizes (e.g., ...) , Max pooling is performed on the input feature map, and then all pooling results are flattened and concatenated.

[0127] (3) The neck network fuses and further processes the multi-scale features output by the backbone to enhance the representational power of the features, thereby better adapting to the target detection task. It uses a Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) structure, employing upsampling, downsampling, and concatenation operations to fuse features from different stages of the backbone network across scales. This fusion strategy ensures that the network retains the fine-grained information required for small targets while incorporating the rich semantic information required for large targets, providing the prediction head with more comprehensive and discriminative feature inputs. For example, the upsampling and concatenation operations of the FPN can be expressed as:

[0128] ;

[0129] in, It is the feature map of the i-th layer of the backbone network. Indicates an upsampling operation. This indicates a convolution operation.

[0130] (4) The prediction head outputs the final detection result through multi-scale prediction branches based on the fusion features output by Neck.

[0131] To address the issue of abundant background clutter at sea (such as waves and fog), this embodiment independently designed a channel attention feature interaction enhancement module. It integrates ECANet (Efficient Channel Attention Network) and SENet (Squeeze-and-Excitation Network) channel attention mechanisms into the YOLOX backbone or neck network. This mechanism establishes feature dependencies along the channel dimension, dynamically learning and adjusting the weights of each channel. This allows the network to proactively focus on effective features of key targets such as ships, while suppressing interference from redundant information such as the sea surface background and sky. This adaptive feature weighting strategy, especially when the target and background are similar, or when there is significant island interference, can significantly reduce false alarm and false negative rates, thereby improving detection accuracy and anti-interference capabilities.

[0132] To make bounding box regression more accurate and stable, this embodiment replaces the traditional GIoU or IoU loss with CIoU loss. CIoU loss not only considers the overlap area and center point distance between the predicted and ground truth boxes, but also incorporates consideration of the consistency of the bounding box's aspect ratio. Total Loss Function The composition remains as classification loss. Regression loss and targeted loss Weighted sum:

[0133] ;

[0134] in: , , These are weighting coefficients used to balance the various losses. The classification loss is typically binary cross-entropy (BCE) loss or Focal loss; for each predicted bounding box and the ground truth label, the BCE loss is calculated as follows:

[0135] ;

[0136] in It is a real label (0 or 1). It is the probability predicted by the model.

[0137] This is the regression loss, calculated using the CIoU loss function. CIoU loss forces the model to not only have greater overlap with the ground truth boxes when predicting bounding boxes, but also to have a more similar aspect ratio. This allows the model to converge faster and more stably during training, further improving localization accuracy and regression performance.

[0138] The thermal imaging branch, serving as an independent sensing guarantee under low visibility conditions in this embodiment, focuses on efficiently and accurately identifying heat source targets. This embodiment employs a refined strategy based on connected component analysis. First, the raw thermal imaging data is binarized using a temperature threshold (20°C) to separate the ship's thermal radiation area above the threshold, while the water background below the threshold is filtered out. Binarization function:

[0139] ;

[0140] in, It is a pixel. Temperature value at that location, It is the set temperature threshold (20℃). It is the pixel value after binarization.

[0141] Subsequently, the system identifies and marks all connected white pixel regions through connected component analysis of two scans, thereby accurately calculating the number of heat source targets and generating a minimum bounding rectangle for them. This strategy can serve as an independent, highly robust sensing modality when the RGB branch fails due to dense fog or darkness, ensuring minimum sensing capability in all weather conditions.

[0142] See Figure 4 The specific process is as follows:

[0143] (1) Initial scan: The system traverses the binary image (thermal imaging image after temperature thresholding) pixel by pixel. For white pixels encountered, the system determines whether to retain the existing label or assign a new label based on whether there is a labeled connected component in its neighborhood (e.g., 8-neighborhood or 4-neighborhood). During the scan, different connected components may be assigned different labels due to discontinuous pixel distribution. In this case, the system will establish an equivalent label table to record the label relationships of these components that actually belong to the same connected component.

[0144] (2) Second scan: The system will traverse the image again, merging labels of the same connected component based on the equivalent label table established in the first scan. For example, if the label and tags If something is marked as equivalent in the equivalence table, then all things marked as equivalent... All pixels will be relabeled as (Usually, the smallest equivalent label is chosen as the final label).

[0145] (3) Final output: Finally, the set of pixels with the same label constitutes a connected component, and the system determines the number of ship targets based on this. The minimum bounding rectangle is calculated for each connected component, and the anchor box is drawn as the detection result of the thermal imaging branch.

[0146] S102, the preprocessed multi-source heterogeneous data is dynamically weighted and fused, and combined with the depth data of the lidar, to realize the accurate coordinate recovery of the ship target in three-dimensional space.

[0147] This stage aims to intelligently fuse RGB visual detection results with thermal imaging heat source information, and combine this with LiDAR depth data to achieve accurate coordinate recovery of the ship target in three-dimensional space. The fusion strategy proposed in this embodiment goes beyond traditional shallow stitching, employing a more robust dynamic weighted fusion mechanism. Its core idea is to adaptively adjust the contribution weights of different modalities in the final decision based on real-time environmental conditions and the real-time confidence level of each modality.

[0148] IoU (Intersection over Union) will serve as a fundamental tool to measure the overlap between RGB and thermal imaging bounding boxes. This mechanism quantifies the overlap by calculating the ratio of the intersection area to the union area of ​​the two bounding boxes, providing a basis for subsequent fusion decisions. When the RGB branch (output by the YOLOX network) detects the anchor box... Anchor frames detected by thermal imaging branch (output from connected component analysis) When the IoU value exceeds a preset threshold (e.g., 0.5), the system determines that the two detected the same target and enters the deep fusion process. Figure 5 This is a schematic diagram for IoU calculation.

[0149] To achieve more intelligent perception in complex and ever-changing maritime environments, this embodiment designs a dynamic weighted fusion and adaptive switching strategy. Visible light images are clear during the day, but their performance deteriorates sharply at night or in foggy weather; thermal imaging performs well at night and in fog, but has low resolution and lacks texture detail; lidar provides accurate depth, but its detection range is limited in dense fog. This strategy aims to leverage these complementary characteristics to optimize the fusion result by evaluating the effectiveness of each modality in real time. This strategy effectively overcomes the limitations of relying on a single fixed threshold, making the system more adaptable to uncertainties.

[0150] (1) Confidence-weighted fusion:

[0151] For each detected target, the respective detection confidence scores CRGB and RGB are obtained from the RGB and thermal imaging branches. The final fusion confidence level The weighted average is obtained as follows:

[0152] ;

[0153] in, and It is a dynamic weight, and These weights can be adjusted based on factors such as, for example, during the day when visibility is good. Higher; at night or in heavy fog, Dynamically added to compensate for the shortcomings of RGB visuals.

[0154] (2) Intelligent mode switching logic:

[0155] This embodiment achieves intelligent switching between modalities by setting dynamic thresholds and priority rules:

[0156] Normal mode: In good visibility conditions (e.g., daytime), the RGB visual branch is the dominant modality, and its detection results have high priority. At this time, Approaching 1, Close to 0.

[0157] Assisted Enhancement Mode: When the detection confidence of the RGB visual branch... Below the preset threshold When the weight is 0.6 (for example), the system enters the auxiliary enhancement mode. At this time, the weight of the thermal imaging branch... It will depend on its own confidence level The weights are dynamically adjusted based on environmental conditions to compensate for the limitations of RGB vision. For example, the sigmoid function or linear interpolation can be used to smoothly adjust the weights.

[0158] ;

[0159] in It is a coefficient that adjusts the steepness of the weight change.

[0160] Thermal imaging-dominant mode: In extremely harsh environments (such as dense fog or complete darkness), if If the value is significantly below a threshold (e.g., below 0.3), or if the RGB branch is completely disabled, the system will switch to thermal imaging-dominated mode. At this time, Approaching 1, The accuracy is close to zero, mainly relying on thermal imaging detection results and three-dimensional positioning through lidar.

[0161] LiDAR Priority: As an independent branch providing precise 3D positioning, LiDAR data always has high priority, especially in obstacle avoidance decision-making. Its 3D coordinate information is correlated with the fused 2D detection results to provide the final 3D positioning output.

[0162] This dynamic modal fusion and decision-making strategy enables the system to intelligently adjust its perception strategy based on real-time environmental conditions and sensor performance. This significantly improves the accuracy and robustness of vessel detection in various complex maritime scenarios, especially at night and in low visibility conditions, while reducing false alarms and missed detections. It achieves a more refined decision-making ratio and more flexible modal switching, making the system more adaptable to uncertainty.

[0163] S103, based on a multi-tracking mechanism collaborative strategy, predicts and updates the motion state of the identified target.

[0164] This embodiment uses an optical radar camera (Intel RealSense L515) to acquire a depth image of the target, and uses camera intrinsics and back projection algorithms to recover the target's precise coordinates (X, Y, Z) in three-dimensional space.

[0165] The Intel RealSense L515 uses the Time-of-Flight (ToF) principle to directly output the depth value of each pixel, forming a depth image. Each pixel value in the depth image represents the distance from that pixel to the camera plane.

[0166] The camera intrinsic parameter matrix K describes the projection relationship of the camera from the 3D camera coordinate system to the 2D pixel coordinate system. For an ideal pinhole camera model, the intrinsic parameter matrix is ​​usually expressed as:

[0167] ;

[0168] in, and These are the focal lengths along the x-axis and y-axis, respectively (in pixels). These are the coordinates of the principal point (usually close to the center of the image).

[0169] Back projection algorithms are the core of the transformation from 2D pixel coordinates to 3D camera coordinates, such as Figure 6 As shown. Given a pixel point and its corresponding depth value (Originally obtained from the depth map), it can be back-projected to three-dimensional coordinates in the camera coordinate system using the following formula. :

[0170] ;

[0171] ;

[0172] ;

[0173] in: These are the pixel coordinates in the image. It is the depth value of the pixel in the depth map (usually in meters). , , , These are the camera's intrinsic parameters.

[0174] For each ship target bounding box output by the RGB detection branch First, determine its center pixel coordinates. Then, the depth value of the center pixel is obtained from the synchronously acquired depth image. Using the above back projection formula, Convert to 3D coordinates in camera coordinate system If it is necessary to convert 3D coordinates in the camera coordinate system to 3D coordinates in the world coordinate system... Then the camera's extrinsic parameters (rotation matrix R and translation vector T) are needed, and their relationship is as follows:

[0175] ;

[0176] in It is a point in the camera coordinate system. It is a point in the world coordinate system.

[0177] In complex and dynamic maritime scenarios, ship tracking may be interrupted due to nonlinear motion or brief obstructions. To address this issue, this embodiment does not use Kalman filtering, Hungarian algorithm, or particle filtering independently, but rather organically integrates them into a collaborative and complementary robust tracking framework.

[0178] This embodiment designs a multi-object continuous tracking module with multi-algorithm collaboration. Kalman filtering and the Hungarian algorithm are used as the core of the multi-object tracking (MOT) framework.

[0179] Kalman filtering is a probability-based recursive estimation algorithm. Its core function is to predict the target's possible position and state in the next frame based on its historical trajectory and a pre-defined motion model (such as a uniform velocity model). The mathematical model for the prediction phase can be expressed as:

[0180] ;

[0181] ;

[0182] Subsequently, the Hungarian algorithm was used to optimally match the detection results of the current frame with the prediction results of the Kalman filter. This algorithm finds the matching scheme that minimizes the total cost by solving a cost matrix, thus ensuring that each detected target is correctly assigned to its corresponding historical trajectory. This combination can efficiently solve the data association problem in maritime scenes where multiple vessels are operating in parallel, intersecting, or experiencing brief occlusion, effectively maintaining the continuity of target identities.

[0183] However, in a marine environment, a ship's motion may change nonlinearly due to factors such as wind, waves, and turning, or it may become undetectable by the detector due to temporary obstruction by other large ships or objects. In such cases, Kalman filtering based on the linear motion assumption may fail.

[0184] To address this challenge, this embodiment designs a particle filter as an auxiliary robust tracker. Particle filtering is a Monte Carlo method that approximates the posterior probability distribution of a target using a set of weighted random samples (particles), making it particularly suitable for nonlinear, non-Gaussian systems. The system dynamically activates the particle filter when the prediction confidence of the Kalman filter falls below a certain threshold (e.g., the target is not detected for several consecutive frames). By generating a large number of particles in areas where the target might exist and updating their weights based on the latest observations (such as low-confidence detection boxes), particle filtering effectively "recovers" targets that have temporarily disappeared or whose motion patterns have changed, thus ensuring trajectory continuity and significantly improving the robustness of the tracking framework.

[0185] Particle weight update formula:

[0186] ;

[0187] in: It is the first Individual particles Weight of time, yes The observed value at time, It is the first Individual particles The state at any given moment, It is the likelihood function.

[0188] Resampling is performed based on the particle weights, eliminating low-weight particles and replicating high-weight particles to avoid particle degradation issues.

[0189] For complex situations such as ship obstruction, particle filtering covers multiple possible locations along the target's historical trajectory, with each particle carrying a weight indicating the probability of the target appearing at that location. Then, based on the match with low-confidence detection boxes, the particle weights are reassigned; particles with higher weights are retained, while those with lower weights are discarded. Finally, by analyzing the distribution of these particles, the actual location of the target can be found more reliably, reconnecting broken trajectories. This effectively maintains tracking continuity when ships at sea temporarily disappear or their appearance changes due to waves, other vessels, or environmental factors.

[0190] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0191] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A multimodal data collaborative intelligent detection and three-dimensional positioning method for maritime vessels, characterized in that, include: The collected multi-source heterogeneous data is subjected to refined preprocessing and preliminary target detection. The preprocessed multi-source heterogeneous data is dynamically weighted and fused, and combined with the depth data from lidar, to achieve accurate coordinate recovery of the ship target in three-dimensional space, including: IoU is used as the basic tool to measure the overlap between RGB and thermal imaging detection boxes. The overlap relationship is quantified by calculating the ratio of the intersection area to the union area of ​​the two bounding boxes, which provides a basis for subsequent fusion decisions. The design incorporates a dynamic weighted fusion and adaptive switching strategy, leveraging the complementary characteristics of different data features to evaluate the effectiveness of each modality in real time and optimize the fusion results. This includes: For each detected target, the detection confidence C is obtained from the RGB and thermal imaging branches. RGB and The final fusion confidence level The weighted average yields: ； in: and It is a dynamic weight, and ; Set dynamic thresholds and priority rules to achieve intelligent switching between modalities: Normal mode: When visibility is good, the RGB visual branch is the dominant modality, and its detection results have high priority. Approaching 1, Approaching 0; Assisted Enhancement Mode: When the detection confidence of the RGB visual branch... Below the preset threshold At this time, the system enters the auxiliary enhancement mode, and the weight of the thermal imaging branch is adjusted accordingly. It will depend on its own confidence level The system dynamically adjusts to adapt to changing environmental conditions in order to compensate for the limitations of RGB vision. Thermal imaging-dominant mode: In extremely harsh environments, if If the value is far below the threshold or the RGB branch is completely disabled, the system will switch to thermal imaging-dominated mode. Approaching 1, Approaching zero, relying on thermal imaging detection results and using lidar for three-dimensional positioning; LiDAR Priority: As an independent branch providing accurate 3D positioning, LiDAR 3D coordinate information is correlated with the fused 2D detection results to provide the final 3D positioning output; Finally, the depth image of the target is obtained using an optical radar camera, and the precise coordinates of the target in three-dimensional space are recovered through camera intrinsic parameters and back projection algorithm. A multi-tracking mechanism collaborative strategy is used to predict and update the motion state of identified targets. This includes a multi-target continuous tracking module based on a multi-algorithm collaboration, with Kalman filtering and the Hungarian algorithm as the core of the multi-target tracking framework; a particle filter is designed as an auxiliary robust tracker, which approximates the posterior probability distribution of the target through a set of weighted random samples; and the particle filter is dynamically activated when the prediction confidence of the Kalman filter is lower than a certain threshold.

2. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 1, characterized in that, The process of performing refined preprocessing and preliminary target detection on the collected multi-source heterogeneous data includes: Multi-sensor devices are used to synchronously acquire multi-modal data and obtain multi-source heterogeneous data; The multi-source heterogeneous data is preprocessed to obtain an enhanced dataset; Construct a YOLOX multi-branch parallel sensing network based on adaptive optimization; The YOLOX multi-branch parallel sensing network based on adaptive optimization is trained using the augmented dataset. Preliminary target detection was performed using the YOLOX multi-branch parallel sensing network, which was trained and optimized adaptively.

3. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 2, characterized in that, The method of acquiring multi-source heterogeneous data through multi-sensor synchronous acquisition includes: Multimodal data is simultaneously acquired using both a light camera and a thermal imaging camera, wherein the light camera acquires RGB images and depth information, and the thermal imaging camera acquires thermal images. The intrinsic parameter matrices of the visible light camera and the thermal imaging camera were obtained using the checkerboard calibration method. A unified clock module is introduced to achieve timestamp synchronization of multi-sensor sampling, ensuring precise alignment between RGB images, thermal imaging, and LiDAR data frames.

4. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 2, characterized in that, The preprocessing of the multi-source heterogeneous data to obtain the augmented dataset includes: The Mixup data augmentation strategy is adopted, which randomly selects two images and their corresponding labels, and then linearly combines their pixel values ​​and labels in proportion to generate training samples; A linear combination of Mixup can be represented as: ； ； in: , It is the original image. , These are the corresponding tags. It is the mixing coefficient; The thermal imager's image frame was adjusted to a preset multiple of its original size to achieve precise matching of image frames from multiple devices. ； ； Where W and H represent the width and height of the image, respectively, and 0.7 represents the preset multiplier.

5. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 2, characterized in that, The adaptive optimization-based YOLOX multi-branch parallel perception network is based on the YOLOX network. ECANet and SENet channel attention mechanisms are incorporated into the YOLOX backbone network or neck network. The channel attention mechanism establishes feature dependencies in the channel dimension, dynamically learns and adjusts the weights of each channel, enabling the network to actively focus on the effective features of key targets of the ship, while suppressing the interference of redundant information. The adaptively optimized YOLOX multi-branch parallel perceptual network training process introduces a CIoU loss function, which considers the overlap area between the predicted and ground truth boxes, the distance between their center points, and the consistency of the aspect ratio of the bounding boxes; the total loss function... The composition remains as classification loss. Regression loss and targeted loss Weighted sum: ； in: , These are weighting coefficients used to balance the various losses.

6. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 5, characterized in that, The adaptively optimized YOLOX multi-branch parallel sensing network performs thermal imaging heat source identification and binarization optimization based on a refined strategy of connected component analysis, including: First, the raw thermal imaging data is binarized using a temperature threshold to separate the ship's thermal radiation area above the threshold, while the water background below the threshold is filtered out. The binarization function is: ； in: It is a pixel. Temperature value at that location, It is the set temperature threshold. These are the pixel values ​​after binarization; Subsequently, through connected component analysis of two scans, all connected white pixel regions are identified and marked, thereby accurately calculating the number of heat source targets and generating the minimum bounding rectangle for them.

7. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 6, characterized in that, The process involves analyzing connected components through two scans to identify and mark all connected white pixel regions, thereby accurately calculating the number of heat source targets and generating their minimum bounding rectangles. First, the binary image is traversed pixel by pixel. For white pixels encountered, the existing label or a new label is assigned to the pixel depending on whether there is a labeled connected component in its neighborhood. Traverse the image again and merge the labels of the same connected components based on the equivalent label table established in the first scan; Finally, the set of pixels with the same label constitutes a connected component, which the system uses to determine the number of ship targets; the minimum bounding rectangle is calculated for each connected component, and anchor box labels are drawn as the detection results of the thermal imaging branch.

8. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 7, characterized in that, The final step involves acquiring a depth image of the target using an optical radar camera, and then recovering the target's precise coordinates in three-dimensional space using camera intrinsic parameters and a back-projection algorithm. This includes: An optical radar camera directly outputs the depth value of each pixel using the Time-of-Flight (ToF) principle, forming a depth image. Each pixel value in the depth image represents the distance from that pixel to the camera plane. The camera intrinsic parameter matrix K describes the projection relationship of the camera from the three-dimensional camera coordinate system to the two-dimensional pixel coordinate system. For an ideal pinhole camera model, the intrinsic parameter matrix is ​​usually expressed as: ； in: and These are the focal lengths along the x-axis and y-axis, respectively. These are the coordinates of the principal point; Given a pixel and the corresponding depth values ​​obtained from the depth map. The following formula is used to back-project it to three-dimensional coordinates in the camera coordinate system. : ； ； ； in: These are the pixel coordinates in the image. It is the depth value of that pixel in the depth map. , , , These are the camera's intrinsic parameters; For each ship target bounding box output by the RGB detection branch First, determine its center pixel coordinates. Then, the depth value of the center pixel is obtained from the synchronously acquired depth image. Using the above back projection formula, Convert to 3D coordinates in camera coordinate system If you need to convert 3D coordinates in the camera coordinate system to 3D coordinates in the world coordinate system... Then, the camera's extrinsic parameters, rotation matrix R and translation vector T, are required, and their relationship is as follows: ； in It is a point in the camera coordinate system. It is a point in the world coordinate system.

9. The method for intelligent detection and three-dimensional positioning of maritime vessels based on multimodal data collaboration according to claim 1, characterized in that, The multi-tracking mechanism collaborative strategy for predicting and updating the motion state of identified targets includes: Kalman filtering predicts the target's possible position and state in the next frame based on its historical trajectory and a pre-defined motion model. The mathematical model for the prediction phase can be expressed as: ； ； Subsequently, the Hungarian algorithm was used to optimally match the detection results of the current frame with the prediction results of the Kalman filter. By solving a cost matrix, the matching scheme that minimizes the total cost was found, thereby ensuring that each detected target was correctly assigned to the corresponding historical trajectory. Particle filtering generates a large number of particles in the region where the target may exist and updates their weights based on the latest observations to recover targets that have temporarily disappeared or whose motion patterns have changed, thus ensuring the continuity of the trajectory. The particle weight update formula is as follows: ； in: It is the first Individual particles Weight of time, yes The observed value at time, It is the first Individual particles The state at any given moment, It is the likelihood function; Resampling is performed based on the particle weights, eliminating low-weight particles and replicating high-weight particles to prevent particle degradation.

Images