A multi-modal fusion-based maritime target situation awareness method and system

Abstract

The application discloses a kind of based on multimodal fusion's sea target situation awareness method and system, wherein method includes: by first perception mode, obtain first perception data, carry out target identification processing, obtain initial classification result;When judging the target meets the threat trigger condition of pre-set, by second perception mode, the region where target is located is detected, obtains second perception data, determines the spatial position information of the target, obtains the radar classification result of target;By third perception mode, with the region indicated by the spatial position information as center, detection is carried out, obtains third perception data, resolves the azimuth and distance information of the target, obtains the sonar classification result of target;Adaptive weighted fusion algorithm is used, fusion processing is carried out, generates the three-dimensional motion trajectory of the target, fuses sonar classification result, outputs the final category attribute information of the target.The application can realize long-distance, high-precision multimodal collaborative target detection.

Description

Technical Field

[0001] This invention relates to the field of maritime multi-source target situational awareness technology, and in particular to a maritime target situational awareness method and system based on multimodal fusion. Background Technology

[0002] With the increasing complexity of tasks such as safeguarding maritime rights, controlling maritime traffic, and exploring deep-sea resources, maritime target detection and situational awareness technologies face significant challenges related to long-range, high-precision, and highly contested operations. Existing technologies mainly suffer from deficiencies in the following three areas: The physical limitations and environmental vulnerability of single-modal sensors. Traditional target detection methods often rely on a single type of sensor (such as pure optics, pure radar, or pure sonar). However, in the complex marine environment, single-modal data has inherent physical limitations: optical sensors are susceptible to interference from changes in sea surface lighting, fog, haze, and camouflaged targets, and their detection capability drops sharply at night or in severe weather; radar, while capable of all-weather operation, is prone to false alarms or missed detections when facing low-altitude sea-skimming targets or complex sea surface clutter; sonar equipment, although capable of acquiring fine features of underwater and surface targets, has a range greatly affected by hydrological conditions and a low data update rate. These limitations of single-modal sensors make it difficult for existing systems to balance detection range and identification accuracy under complex operating conditions.

[0003] There is a conflict between computational load and distance in multimodal fusion technology. To address the shortcomings of single-modal data, existing technologies attempt to introduce multimodal data fusion. However, most existing solutions adopt a "full acquisition, synchronous fusion" mode, that is, during system operation, regardless of the environment or the presence of targets, all sensors such as image, radar, and sonar are always kept at the highest sampling rate, and all data are fused in real time with coordinate system unification, time synchronization, and high-dimensional features.

[0004] This indiscriminate, end-to-end data processing approach has a fatal flaw: the consumption of system computing resources increases exponentially with the number of sensors and detection range. When increasing detection range or introducing high-resolution sensors to improve situational awareness, the enormous computational burden of synchronous fusion leads to a sharp increase in processing latency, forcing the system to sacrifice resolution or reduce fusion dimensions to achieve real-time performance. Therefore, existing technologies struggle to strike a balance between "long-range" and "high precision, low latency."

[0005] The lack of event-driven dynamic perception mechanisms is a significant drawback. Existing fusion systems typically lack adaptive triggering mechanisms based on scene understanding. The system cannot proactively adjust the detection strategies of backend sensors based on preliminary results from front-end perception (such as determining the presence of a specific type of target). For example, in vast, untapped sea areas, radar and sonar still perform omnidirectional, full-power scanning and data fusion, resulting in a significant waste of computational resources and energy. This "blind" perception mode not only limits the system's endurance during long-range detection but also makes it difficult to concentrate computing power for refined identification and tracking of key targets in adversarial scenarios. Summary of the Invention

[0006] The main objective of this invention is to achieve long-distance, high-precision multimodal collaborative target detection while maintaining low computational overhead, and to establish an intelligent perception mechanism that responds on demand.

[0007] The technical solution adopted in this invention is: a maritime target situational awareness method based on multimodal fusion, comprising: First sensing data is acquired through the first sensing mode, and target recognition processing is performed on the first sensing data to determine whether the maritime target exists and obtain the initial classification result; Based on the initial classification results, when it is determined that the target meets the preset threat triggering conditions, the area where the target is located is detected through the second perception mode to obtain the second perception data, determine the spatial location information of the target in a unified geographic coordinate system and extract features, and obtain the radar classification result of the target; Once the spatial location information of the target is determined, the third sensing mode is used to detect the area indicated by the spatial location information as the center, acquire the third sensing data, calculate the orientation and distance information of the target in a unified geographic coordinate system and extract features, and obtain the sonar classification result of the target. An adaptive weighted fusion algorithm is used to fuse the first, second, and third sensing data to generate the three-dimensional motion trajectory of the target. The initial classification result, radar classification result, and sonar classification result are fused together to output the final category attribute information of the target.

[0008] According to the above technical solution, the first sensing mode is an optical imaging mode, and the first sensing data is sea surface image data; the method for target recognition processing of the first sensing data specifically includes: An improved YOLOv8 model is used to identify targets in the sea surface image data; the improved YOLOv8 model includes an E-ELAN module for enhancing multi-scale feature fusion. The improved YOLOv8 model is optimized through adaptive mixed precision training; during the target recognition inference process, the input image is dynamically adjusted in resolution, and the high-interest region is adaptively cropped according to the target size. The loss function for training the improved YOLOv8 model includes classification loss, bounding box loss, and confidence loss; wherein, the classification loss is used to calculate the cross-entropy loss of the target classification, the bounding box loss is used to measure the difference between the predicted box and the ground truth box through mean squared error, and the confidence loss is used to represent the confidence error of whether the predicted box contains the target.

[0009] According to the above technical solution, the preset threat triggering conditions include: the initial classification result belongs to a preset threat type; the preset threat type is set by the user in advance, including submarines and high-speed assault boats.

[0010] According to the above technical solution, the second sensing mode is a radar detection mode, and the second sensing data is radar echo data; determining the spatial location information of the target in a unified geographic coordinate system based on the second sensing data specifically includes: The radar echo data is subjected to DBSCAN clustering and track initiation logic processing, and the M / N rule and kinematic constraints are combined to distinguish between real targets and false alarms. The target's trajectory is predicted based on the extended Kalman filter, and the latitude and longitude coordinates in a unified geographic coordinate system are output as the target's spatial location information. The radar cross section features of the target are extracted from the radar echo data, and the target is classified based on the radar cross section features using a support vector machine to obtain the radar classification results.

[0011] According to the above technical solution, the third sensing mode is a sonar detection mode, and the third sensing data is a sonar echo signal; directional area detection is performed centered on the area indicated by the spatial location information, specifically including: Based on the distance between the target and the detection platform indicated by the spatial location information, a sound wave pulse with a corresponding frequency and pulse width is set; The sonar scanning angle is controlled by beamforming technology to cover a fan-shaped area centered on the latitude and longitude coordinates indicated by the spatial location information and with a preset radius.

[0012] According to the above technical solution, the third sensing mode adopts a multi-element sonar array, and by adjusting the transmission phase difference of each element, a directional beam with a fan-shaped coverage area is formed. Acoustic features, motion features, and geometric features are extracted from the sonar echo signal. A support vector machine is then used to classify the target based on the acoustic features, motion features, and geometric features to obtain the sonar classification result.

[0013] According to the above technical solution, the location and distance information of the target in a unified geographic coordinate system are calculated based on the third sensing data, specifically including: Based on the speed of sound propagation and the time difference between transmitting and receiving sound waves, the distance information of the target relative to the detection platform is calculated. By separating the signal subspace and the noise subspace, the signal phase difference between multiple receiving channels is analyzed to determine the azimuth information of the target relative to the detection platform. The distance and azimuth information are converted to a unified geographic coordinate system.

[0014] According to the above technical solution, the method for generating the three-dimensional motion trajectory of the target and outputting the final category attribute information of the target specifically includes: Based on the calibration experiment, a first set of basic weights for category fusion of the first sensing data, the second sensing data, and the third sensing data are predetermined, as well as a second set of basic weights for trajectory fusion of the second sensing data and the third sensing data are predetermined. Based on the real-time signal-to-noise ratio of each of the three types of sensing data, the first set of basic weights is corrected to obtain the first set of corrected weights; Based on the real-time signal-to-noise ratios of the second and third sensing data, the second set of basic weights is corrected to obtain the second set of corrected weights. The first set of corrected weights is normalized to obtain the final category fusion weights corresponding to the first, second, and third perception data. The second set of corrected weights is normalized to obtain the final trajectory fusion weights corresponding to the second and third sensing data. Based on the final weight of the trajectory fusion, the positioning and trajectory information of the second and third sensing data are weighted and fused to generate the three-dimensional motion trajectory of the target. Based on the final weight of the category fusion, the initial classification result, radar classification result, and sonar classification result are weighted and fused to output the final category attribute information of the target.

[0015] According to the above technical solution, the step of correcting the basic weights based on the real-time signal-to-noise ratio specifically includes: multiplying the basic weights of each sensing data by the real-time signal-to-noise ratio of the corresponding sensing data, and then multiplying by a preset signal-to-noise ratio correction coefficient to obtain the corrected weight values.

[0016] Another aspect of the present invention provides a maritime target situational awareness system based on multimodal fusion, comprising: a first perception acquisition and identification module, used to acquire first perception data through a first perception mode, perform target identification processing on the first perception data, determine whether a maritime target exists and acquire an initial classification result; The second perception detection and positioning module is connected to the first perception acquisition and identification module. It is used to detect the area where the target is located through the second perception mode when the target is determined to meet the preset threat triggering conditions based on the initial classification result, acquire the second perception data, determine the spatial location information of the target in a unified geographic coordinate system and extract features, and acquire the radar classification result of the target. The third sensing detection and calculation module is connected to the second sensing detection and positioning module. After determining the spatial location information of the target, it uses the third sensing mode to detect the area indicated by the spatial location information as the center, obtains the third sensing data, calculates the orientation and distance information of the target in a unified geographic coordinate system and extracts features, and obtains the sonar classification result of the target. The fusion output module is connected to the first perception acquisition and recognition module, the second perception detection and positioning module, and the third perception detection and calculation module, respectively. It is used to perform fusion processing on the first perception data, the second perception data, and the third perception data using an adaptive weighted fusion algorithm to generate the three-dimensional motion trajectory of the target, fuse the initial classification result, the radar classification result, and the sonar classification result, and output the final category attribute information of the target.

[0017] The beneficial effects of this invention are as follows: It employs a threat-triggered, three-level heterogeneous modal cascaded detection architecture. After initial target screening by the first sensing mode, subsequent detection processes are initiated only for targets meeting preset threat triggering conditions. This is complemented by directional focusing detection of the target's location by the second sensing mode and directional area detection centered on the target's locked location by the third sensing mode. This avoids the inefficient computational power consumption associated with full-area scanning, achieving long-range target detection with low computational power consumption. Simultaneously, it effectively suppresses the impact of sea surface clutter and environmental interference on the detection results, reducing false alarms and missed detections. The entire process is based on… The system calculates the target's spatial location, orientation, and distance information using a unified geographic coordinate system, providing a consistent spatial benchmark for multimodal data fusion and effectively improving the accuracy and reliability of data fusion. Finally, through an adaptive weighted fusion algorithm, it achieves complementary fusion of multimodal perception data, generating accurate three-dimensional motion trajectories of the target and outputting high-confidence final target category attribute information. This comprehensively improves the accuracy, real-time performance, and environmental adaptability of target situational awareness in complex maritime dynamic environments and strong interference scenarios, and can be widely adapted to various maritime application scenarios such as maritime monitoring, maritime security, and marine exploration.

[0018] Furthermore, the improved YOLOv8 model with the addition of the E-ELAN module enhances target recognition accuracy and inference efficiency, optimizes the loss function, and provides a high-confidence initial judgment basis for subsequent triggering.

[0019] Furthermore, a dual-path adaptive weighted fusion algorithm is adopted to dynamically adjust the weights based on real-time detection quality, leveraging the complementary advantages of multimodal approaches to enhance the robustness of perception in complex environments.

[0020] Furthermore, the quantitative calculation method for weight correction is clarified to ensure the feasibility and stability of the fusion algorithm and improve the accuracy of multimodal data fusion results. Attached Figure Description

[0021] Figure 1 This is a flowchart of a maritime target situational awareness method based on multimodal fusion according to an embodiment of the present invention; Figure 2 This is an architecture diagram of the E-ELAN module for YOLOv8 model extension according to an embodiment of the present invention; Figure 3 This is a data processing flowchart of an embodiment of the present invention; Figure 4 This is a flowchart of another maritime target situational awareness method based on multimodal fusion according to an embodiment of the present invention; Figure 5 This is an architecture diagram of a maritime target situational awareness system based on multimodal fusion according to an embodiment of the present invention; Detailed Implementation To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Example 1 This embodiment provides a maritime target situational awareness method based on multimodal fusion, aiming to solve the technical problems in existing technologies where excessive computational load and difficulty in balancing long-range detection and real-time performance result from simultaneous fusion of multiple sensors. This method constructs a cascaded triggering architecture of visual initial screening—radar locking—sonar fine measurement, achieving on-demand perception and adaptive fusion, thereby improving the accuracy and reliability of long-range target detection and identification while controlling computational overhead. Figure 1 As shown, the method includes the following steps: S1. Obtain first perception data through the first perception mode, perform target recognition processing on the first perception data, determine whether the maritime target exists and obtain the initial classification result.

[0023] In a specific implementation, the first sensing mode is an optical imaging mode, and the first sensing data is sea surface image data. Sea surface image data can be acquired through various means, such as high-resolution cameras mounted on UAVs, high-resolution cameras mounted on ships, high-resolution cameras mounted on unmanned surface vessels, or shore-based cameras, to adapt to different deployment scenarios.

[0024] The method for performing target recognition processing on the first perceived data specifically includes: An improved YOLOv8 model was used to identify targets in the sea surface image data.

[0025] YOLOv8, the latest version in the YOLO series, maintains the advantage of fast inference and improves the accuracy of small object detection and robustness to complex backgrounds through improved convolutional modules and a new object detection head. The improved YOLOv8 model introduces the E-ELAN module to enhance multi-scale feature fusion capabilities. The structure of this module is as follows: Figure 2 As shown, by using group convolution to increase the cardinality of newly added features, and by combining features from different groups in a shuffling and merging manner, the features learned by different feature maps are enhanced, thereby improving the efficiency of parameter and computation utilization.

[0026] The improved YOLOv8 model optimizes itself through adaptive mixed precision training to improve inference speed. During target recognition inference, the input image is dynamically adjusted in resolution, and high-interest regions are adaptively cropped according to the target size to further optimize computational efficiency.

[0027] The loss function used in training the improved YOLOv8 model includes classification loss, bounding box loss, and confidence loss. The classification loss is used to calculate the cross-entropy loss for target classification; the bounding box loss measures the difference between the predicted and ground truth bounding boxes using mean squared error; and the confidence loss represents the confidence error in whether the predicted bounding box contains the target. Through joint optimization of these loss functions, the model can output the predicted target's class, bounding box location, and confidence score.

[0028] S2. Based on the initial classification result, when it is determined that the target meets the preset threat triggering conditions, the area where the target is located is detected through the second perception mode, the second perception data is obtained, the spatial location information of the target in a unified geographic coordinate system is determined and the features are extracted, and the radar classification result of the target is obtained.

[0029] The preset threat triggering conditions include: the initial classification result belonging to a preset threat type. The preset threat type is set by the user in advance, such as target types with potential threats like submarines or high-speed assault boats. When the image recognition algorithm determines that the target belongs to the above-mentioned threat type, the system triggers a radar scan command, thereby achieving on-demand activation and avoiding unnecessary energy and computational consumption.

[0030] In its specific implementation, the second sensing mode is a radar detection mode, and the second sensing data is radar echo data. The radar is an S-band phased array radar that uses beamforming technology to focus on the target area and suppress sea clutter interference, thereby improving the reliability of target detection.

[0031] Determining the spatial location information of the target in a unified geographic coordinate system based on the second sensing data specifically includes: First, the radar echo data is processed by DBSCAN clustering and track initiation logic. Combining the M / N rule (such as confirming a real target only after multiple consecutive detections of the same target) and kinematic constraints (such as speed and turning angle limits), real targets are distinguished from false alarms, and false tracks are eliminated.

[0032] Then, the target's trajectory is predicted based on the extended Kalman filter to obtain polar coordinate data (including range and azimuth), which is then converted into latitude and longitude coordinates in a unified geographic coordinate system. The latitude and longitude coordinates with an error within 100 meters are output as the target's spatial location information. Factors such as the radar platform's position, attitude, and the Earth's curvature are considered during the conversion.

[0033] Simultaneously, the radar cross-section features of the target are extracted from the radar echo data, and a support vector machine is used to classify the target based on the radar cross-section features to obtain the radar classification results.

[0034] S3. After determining the spatial location information of the target, the third sensing mode is used to detect the area indicated by the spatial location information as the center, obtain the third sensing data, calculate the orientation and distance information of the target in a unified geographic coordinate system and extract features, and obtain the sonar classification result of the target.

[0035] In a specific implementation, the third sensing mode is a sonar detection mode, and the third sensing data is a sonar echo signal. The sonar detector can be mounted on a ship and perform precise detection based on the target position locked by radar.

[0036] Directional area detection is performed centered on the area indicated by the spatial location information, specifically including: Based on the distance between the target and the detection platform indicated by the spatial location information, a corresponding frequency and pulse width of acoustic pulses are set. Specifically, a higher frequency and a narrower pulse width are used for closer targets to improve resolution; a lower frequency and a wider pulse width are used for farther targets to increase detection distance. The settings are determined by those skilled in the art to balance resolution and propagation distance according to actual needs.

[0037] The sonar scanning angle is controlled by beamforming technology to cover a fan-shaped area centered on the latitude and longitude coordinates indicated by the spatial location information, with a radius of a preset value. In this embodiment, the third sensing mode uses a multi-element sonar array. By adjusting the transmission phase difference of each element, a directional beam with a fan-shaped coverage area is formed, achieving precise area detection.

[0038] Based on the third sensing data, the location and distance information of the target in a unified geographic coordinate system are calculated, specifically including: Based on the speed of sound propagation and the time difference between the emitted and received sound waves, the distance information of the target relative to the detection platform is calculated. During data acquisition, the emission time, reception time, signal strength, and position and attitude information of the sonar platform are recorded. Adaptive filtering is used to remove noise, and matched filtering is used to enhance the target echo signal, thereby improving signal quality.

[0039] By separating the signal subspace and the noise subspace, and analyzing the signal phase difference between multiple receiving channels, the azimuth information of the target relative to the detection platform is determined. Specifically, the direction of arrival is calculated based on the acoustic wavelength and the phase difference between the receiving channels.

[0040] The distance and azimuth information are converted to a unified geographic coordinate system to achieve spatial alignment with radar positioning data.

[0041] Simultaneously, acoustic signature features, motion features, and geometric features are extracted from the sonar echo signal. A support vector machine (SVM) is then used to classify the target based on these features, obtaining the sonar classification result. In this embodiment, radar data classification and sonar data classification can use separate SVMs, or a unified SVM can be used to simultaneously input both types of features for classification prediction.

[0042] S4. An adaptive weighted fusion algorithm is used to fuse the first perception data, the second perception data, and the third perception data to generate the three-dimensional motion trajectory of the target. The initial classification result, radar classification result, and sonar classification result are fused together to output the final category attribute information of the target.

[0043] The method for generating the three-dimensional motion trajectory of the target and outputting the final category attribute information of the target specifically includes: During system construction, calibration experiments were conducted to pre-determine a first set of basic weights for category fusion using the first, second, and third sensing data, and a second set of basic weights for trajectory fusion using the second and third sensing data. The determination of the basic weights was based on the inherent characteristics of each sensor and the calibration results.

[0044] In real-time signal processing, the real-time signal-to-noise ratio (SNR) of each of the three types of sensing data is acquired. Based on the real-time SNR of each of the three types of sensing data, the first set of basic weights is corrected to obtain the first set of corrected weights; based on the real-time SNR of the second and third types of sensing data, the second set of basic weights is corrected to obtain the second set of corrected weights. Specifically, the correction method is as follows: the basic weight of each sensing data is multiplied by the real-time SNR of the corresponding sensing data, and then multiplied by a preset SNR correction coefficient to obtain the corrected weight value.

[0045] The first set of corrected weights is normalized to obtain the final category fusion weights corresponding to the first, second, and third sensing data; the second set of corrected weights is normalized to obtain the final trajectory fusion weights corresponding to the second and third sensing data.

[0046] Based on the final weight of the trajectory fusion, the positioning and trajectory information of the second and third sensing data are weighted and fused to generate the three-dimensional motion trajectory of the target. The second sensing data provides the target's spatial position and motion vector, while the third sensing data provides the target's azimuth and distance information. The weighted fusion of these two data forms a continuous three-dimensional motion trajectory.

[0047] Based on the final weights of the category fusion, the initial classification results, radar classification results, and sonar classification results are weighted and fused to output the final category attribute information of the target. This final category attribute information integrates the multi-dimensional judgment results of visual recognition, radar classification, and sonar classification, and has higher accuracy and robustness.

[0048] Finally, based on the determined final category attribute information, the corresponding icon is selected, and a simulated state diagram with the icon as the main body and the three-dimensional motion trajectory as the path is generated according to the generated three-dimensional motion trajectory. This is used to provide real-time feedback on the target status to the user, making it easier for the user to intuitively understand the dynamic behavior of the target.

[0049] The data processing flow in this method is as follows: Figure 3 As shown: First, the optical equipment mounted on shore-based, shipborne, and drone-borne devices actively acquires sea surface image data across the entire area, simultaneously completing target classification and recognition at the image end, determining whether the sea target exists, and outputting initial category attributes.

[0050] When the image terminal identifies and determines that the target belongs to a preset threat type, it immediately triggers the directional acquisition of radar data. The radar focuses on scanning only the area where the target is located, rather than conducting indiscriminate detection of the entire sea area, thereby greatly reducing the consumption of ineffective computing power.

[0051] Meanwhile, the acquired radar data serves two purposes: firstly, it completes the secondary classification and verification of targets at the radar end, complementing the initial classification results of the images to reduce the probability of misjudgment; secondly, it calculates the target motion vector and trajectory, converting them into target latitude and longitude coordinates under a unified geographic coordinate system.

[0052] Once the radar locks onto the target's latitude and longitude, it triggers directional acquisition of sonar data. The sonar scans only a limited fan-shaped area centered on the target's locked latitude and longitude, further reducing computational overhead and improving the signal-to-noise ratio of the target area. The acquired sonar data is also processed in two parallel branches. On one hand, it performs three-stage target classification verification at the sonar end, forming a three-level heterogeneous complement with the classification results from the image and radar, significantly reducing the probability of missing underwater concealed targets. On the other hand, it calculates the target's distance and azimuth relative to the detection platform and simultaneously converts it to a unified geographic coordinate system, providing accurate positioning supplementary data for trajectory fusion.

[0053] The detection results of the three modes are fused in a dual-path separation manner. On the one hand, the target classification results of images, radar and sonar are collected and a comprehensive judgment is made through an adaptive weighted fusion algorithm. On the other hand, the target trajectory latitude and longitude of radar and the target azimuth distance data of sonar are collected to complete the spatial dimension fusion calculation. Finally, a high-confidence target final category and accurate target three-dimensional motion trajectory are output simultaneously, so as to fully realize target situational awareness in complex maritime environments.

[0054] This method constructs a cascaded sensing architecture of image-triggered radar and radar-triggered sonar, achieving on-demand detection and on-demand fusion, effectively reducing computational load. It can achieve efficient detection and identification of long-range targets within a range of 1-10 kilometers with relatively low computational overhead. For areas within 1 kilometer, it can be combined with conventional sonar, radar, and image independent detection and fusion, and the specific application scenario is not limited to this.

[0055] This embodiment also provides a maritime target situational awareness system based on multimodal fusion, used to execute the above method. Its architecture is as follows: Figure 5 As shown, the system includes: The first perception acquisition and recognition module is used to acquire first perception data through the first perception mode, perform target recognition processing on the first perception data, determine whether the maritime target exists and obtain the initial classification result; The second perception detection and positioning module is connected to the first perception acquisition and identification module. It is used to detect the area where the target is located through the second perception mode when the target is determined to meet the preset threat triggering conditions based on the initial classification result, acquire the second perception data, determine the spatial location information of the target in a unified geographic coordinate system and extract features, and acquire the radar classification result of the target. The third sensing detection and calculation module is connected to the second sensing detection and positioning module. After determining the spatial location information of the target, it uses the third sensing mode to detect the area indicated by the spatial location information as the center, obtains the third sensing data, calculates the orientation and distance information of the target in a unified geographic coordinate system and extracts features, and obtains the sonar classification result of the target. The fusion output module is connected to the first perception acquisition and recognition module, the second perception detection and positioning module, and the third perception detection and calculation module, respectively. It is used to perform fusion processing on the first perception data, the second perception data, and the third perception data using an adaptive weighted fusion algorithm to generate the three-dimensional motion trajectory of the target, fuse the initial classification result, the radar classification result, and the sonar classification result, and output the final category attribute information of the target.

[0056] Example 2 Based on Example 1, this example provides another maritime target situational awareness method based on multimodal fusion. This method achieves long-range, high-precision target detection and identification while controlling computational overhead by constructing a cascaded triggered perception architecture and an adaptive weighted fusion mechanism. The method includes the following steps: T1. Acquire first perception data through the first perception mode, perform target recognition processing on the first perception data, determine whether the maritime target exists and obtain the initial classification result.

[0057] In a specific implementation, the first sensing mode is an optical imaging mode, and the first sensing data is sea surface image data. The sea surface image data can be acquired through high-resolution cameras mounted on UAVs, high-resolution cameras mounted on ships, high-resolution cameras mounted on unmanned surface vessels, or shore-based cameras.

[0058] The first perceived data is processed for target recognition using an improved YOLOv8 model. This improved YOLOv8 model introduces an E-ELAN module to enhance multi-scale feature fusion capabilities, optimizes inference speed through adaptive mixed-precision training, and dynamically adjusts the resolution of the input image during inference, adaptively cropping high-interest regions based on the target size.

[0059] The loss function for training the improved YOLOv8 model includes classification loss, bounding box loss, and confidence loss: the classification loss uses cross-entropy loss to calculate the error in target classification.

[0060] Represents classification loss, Indicates the bounding box loss. This represents the confidence loss.

[0061] Through joint optimization of the above loss functions, the model outputs the predicted target category, bounding box location, and confidence level.

[0062] T2. Based on the initial classification results, when it is determined that the target meets the preset threat triggering conditions, the area where the target is located is detected through the second perception mode to obtain the second perception data, determine the spatial location information of the target in a unified geographic coordinate system and extract features, and obtain the radar classification result of the target.

[0063] The preset threat triggering conditions include: the initial classification result belongs to a preset threat type, such as submarine, high-speed assault boat, etc.

[0064] In its specific implementation, the second sensing mode is a radar detection mode, which uses an S-band phased array radar and focuses on the target area through beamforming technology to suppress sea surface clutter interference.

[0065] Determining the spatial location information of the target in a unified geographic coordinate system based on the second sensing data specifically includes: The radar echo data is subjected to DBSCAN clustering and track initiation logic processing. The M / N rule and kinematic constraints are combined to distinguish between real targets and false alarms.

[0066] The target trajectory is predicted based on the extended Kalman filter to obtain polar coordinate data (distance and azimuth), which is then converted into latitude and longitude coordinates in a unified geographic coordinate system.

[0067]

[0068] d represents the distance from the target to the radar; The azimuth of the target is indicated by R; the Earth's radius is indicated by R; and the altitude of the radar platform is indicated by h. and These are the longitude and latitude of the radar platform, respectively.

[0069] The latitude and longitude coordinates of the target are determined using the methods described above.

[0070] Meanwhile, the radar cross-section features of the target are extracted from the radar echo data, and the target is classified based on the radar cross-section using a support vector machine to obtain the radar classification results.

[0071] T3. After determining the spatial location information of the target, the third sensing mode is used to detect the area indicated by the spatial location information as the center, obtain the third sensing data, calculate the orientation and distance information of the target in a unified geographic coordinate system and extract features to obtain the sonar classification result of the target.

[0072] In a specific implementation, the third sensing mode is a sonar detection mode, which uses a multi-element sonar array. By adjusting the transmission phase difference of each element, a directional beam with a fan-shaped coverage area is formed.

[0073] Directional area detection is performed centered on the area indicated by the spatial location information, specifically including: setting the corresponding frequency and pulse width of the acoustic pulse according to the distance between the target and the detection platform; controlling the sonar scanning angle through beamforming technology to cover a fan-shaped area with a radius of 200 meters centered on the locked target's latitude and longitude coordinates.

[0074] Based on the third sensing data, the location and distance information of the target in a unified geographic coordinate system are calculated, specifically including: Calculate the target distance based on the speed of sound propagation and the time difference between the emitted and received sound waves:

[0075] v represents the speed of sound. This is the time difference between transmitting and receiving sound waves.

[0076] The azimuth angle of the target is calculated by analyzing the signal phase difference between multiple receiving channels.

[0077] λ is the wavelength of the sound wave, Δ It is the phase difference between multiple receiving channels.

[0078] Transform distances and azimuths to a unified geographic coordinate system.

[0079] Simultaneously, acoustic features, motion features, and geometric features are extracted from the sonar echo signal, and a support vector machine is used to classify the target based on the above features to obtain the sonar classification result.

[0080] T4. An adaptive weighted fusion algorithm is used to fuse the first, second, and third sensing data to generate the three-dimensional motion trajectory of the target. The initial classification result, radar classification result, and sonar classification result are fused together to output the final category attribute information of the target.

[0081] During the system construction process, basic weights are pre-determined through calibration experiments: a first set of basic weights for category fusion; and a second set of basic weights for trajectory fusion.

[0082] In real-time signal processing, the real-time signal-to-noise ratio (SNR) of various sensing data is acquired; based on the real-time SNR, the first set of basic weights is corrected to generate the first set of corrected weights.

[0083]

[0084]

[0085] The real-time signal-to-noise ratio of the sonar data is used as the basic weight. For radar base weights, Sonar base weights; Real-time signal-to-noise ratio for image data. For the real-time signal-to-noise ratio of radar data, Real-time signal-to-noise ratio for sonar data; , , These are the signal-to-noise ratio correction coefficients.

[0086] The first set of corrected weights is normalized to obtain the final weights for category fusion:

[0087]

[0088]

[0089] The second set of basic weights is adjusted based on the real-time signal-to-noise ratio to generate the second set of adjusted weights:

[0090]

[0091] As the basic weights for radar trajectory fusion, The basic weights for sonar trajectory fusion; This is the radar signal-to-noise ratio correction factor. This is the sonar signal-to-noise ratio correction factor; For the real-time signal-to-noise ratio of radar echo data, This represents the real-time signal-to-noise ratio of sonar echo data.

[0092] The second set of corrected weights is normalized to obtain the final weights for trajectory fusion:

[0093]

[0094] Based on the final weight of trajectory fusion, the positioning and trajectory information of the second and third sensing data are weighted and fused to generate the three-dimensional motion trajectory of the target.

[0095] Based on the final weight of category fusion, the initial classification results, radar classification results, and sonar classification results are weighted and fused to output the final category attribute information of the target.

[0096] Finally, based on the determined final category attribute information, the corresponding icon is selected, and a simulated state diagram with the icon as the main body and the three-dimensional motion trajectory as the path is generated according to the generated three-dimensional motion trajectory, which is used to provide real-time feedback on the target status to the user.

[0097] The general flow of the method in this embodiment is as follows: Figure 4 As shown: First, sea surface image data is acquired, and image recognition algorithms are used to make an initial classification of maritime targets. When the identification result meets the preset category judgment conditions, the radar is immediately triggered to conduct directional detection on the target in order to lock the target's latitude and longitude coordinates; After accurately locking onto the target's latitude and longitude coordinates based on radar echo data, the system responds to the radar's lock signal, triggering the sonar to emit detection pulses. It then conducts directional sonar detection centered on the area where the radar-locked target's latitude and longitude coordinates are located. Based on the sonar echo data, it calculates and obtains the target's azimuth and distance information. Finally, the system collects three types of heterogeneous sensing data—image, radar, and sonar—in real time, and completes the fusion processing of multi-source data through an adaptive weighted fusion algorithm. It generates the target's three-dimensional motion trajectory and simultaneously outputs the final target category determination result, thus fully realizing an on-demand intelligent maritime target situational awareness closed loop of "full-domain initial screening - step-by-step triggering of directional detection - multi-modal fusion output".

[0098] In summary, this invention provides a method and system for maritime target situational awareness based on multimodal fusion, which achieves long-range, high-precision multimodal cooperative target detection while maintaining low computational overhead.

[0099] The various modules or mechanisms of the system are mainly used to implement the various steps of the above method embodiments, and will not be described in detail here.

[0100] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0101] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for situational awareness of maritime targets based on multimodal fusion, characterized in that, include: First sensing data is acquired through the first sensing mode, and target recognition processing is performed on the first sensing data to determine whether the maritime target exists and obtain the initial classification result; Based on the initial classification results, when it is determined that the target meets the preset threat triggering conditions, the area where the target is located is detected through the second perception mode to obtain the second perception data, determine the spatial location information of the target in a unified geographic coordinate system and extract features, and obtain the radar classification result of the target; Once the spatial location information of the target is determined, the third sensing mode is used to detect the area indicated by the spatial location information as the center, acquire the third sensing data, calculate the orientation and distance information of the target in a unified geographic coordinate system and extract features, and obtain the sonar classification result of the target. An adaptive weighted fusion algorithm is used to fuse the first, second, and third sensing data to generate the three-dimensional motion trajectory of the target. The initial classification result, radar classification result, and sonar classification result are fused together to output the final category attribute information of the target.

2. The maritime target situational awareness method based on multimodal fusion according to claim 1, characterized in that, The first sensing mode is an optical imaging mode, and the first sensing data is sea surface image data; The method for performing target recognition processing on the first perceived data specifically includes: An improved YOLOv8 model is used to identify targets in the sea surface image data; the improved YOLOv8 model includes an E-ELAN module for enhancing multi-scale feature fusion. The improved YOLOv8 model is optimized through adaptive mixed precision training; during the target recognition inference process, the input image is dynamically adjusted in resolution, and the high-interest region is adaptively cropped according to the target size. The loss function for training the improved YOLOv8 model includes classification loss, bounding box loss, and confidence loss; wherein, the classification loss is used to calculate the cross-entropy loss of the target classification, the bounding box loss is used to measure the difference between the predicted box and the ground truth box through mean squared error, and the confidence loss is used to represent the confidence error of whether the predicted box contains the target.

3. The maritime target situational awareness method based on multimodal fusion according to claim 1, characterized in that, The preset threat triggering conditions include: the initial classification result belongs to a preset threat type; the preset threat type is set by the user in advance, including submarines and high-speed assault boats.

4. The maritime target situational awareness method based on multimodal fusion according to claim 1, characterized in that, The second sensing mode is the radar detection mode, and the second sensing data is the radar echo data; Determining the spatial location information of the target in a unified geographic coordinate system based on the second sensing data specifically includes: The radar echo data is subjected to DBSCAN clustering and track initiation logic processing, and the M / N rule and kinematic constraints are combined to distinguish between real targets and false alarms. The target's trajectory is predicted based on the extended Kalman filter, and the latitude and longitude coordinates in a unified geographic coordinate system are output as the target's spatial location information. The radar cross section features of the target are extracted from the radar echo data, and the target is classified based on the radar cross section features using a support vector machine to obtain the radar classification results.

5. The maritime target situational awareness method based on multimodal fusion according to claim 4, characterized in that, The third sensing mode is a sonar detection mode, and the third sensing data is a sonar echo signal; Directional area detection is performed centered on the area indicated by the spatial location information, specifically including: Based on the distance between the target and the detection platform indicated by the spatial location information, a sound wave pulse with a corresponding frequency and pulse width is set; The sonar scanning angle is controlled by beamforming technology to cover a fan-shaped area centered on the latitude and longitude coordinates indicated by the spatial location information and with a preset radius.

6. The maritime target situational awareness method based on multimodal fusion according to claim 5, characterized in that, The third sensing mode employs a multi-element sonar array, which forms a directional beam with a fan-shaped coverage area by adjusting the transmission phase difference of each element. Acoustic features, motion features, and geometric features are extracted from the sonar echo signal. A support vector machine is then used to classify the target based on the acoustic features, motion features, and geometric features to obtain the sonar classification result.

7. The maritime target situational awareness method based on multimodal fusion according to claim 6, characterized in that, Based on the third sensing data, the location and distance information of the target in a unified geographic coordinate system are calculated, specifically including: Based on the speed of sound propagation and the time difference between transmitting and receiving sound waves, the distance information of the target relative to the detection platform is calculated. By separating the signal subspace and the noise subspace, the signal phase difference between multiple receiving channels is analyzed to determine the azimuth information of the target relative to the detection platform. The distance and azimuth information are converted to a unified geographic coordinate system.

8. The maritime target situational awareness method based on multimodal fusion according to claim 1, characterized in that, The method for generating the three-dimensional motion trajectory of the target and outputting the final category attribute information of the target specifically includes: Based on the calibration experiment, a first set of basic weights for category fusion of the first sensing data, the second sensing data, and the third sensing data are predetermined, as well as a second set of basic weights for trajectory fusion of the second sensing data and the third sensing data are predetermined. Based on the real-time signal-to-noise ratio of each of the three types of sensing data, the first set of basic weights is corrected to obtain the first set of corrected weights; Based on the real-time signal-to-noise ratios of the second and third sensing data, the second set of basic weights is corrected to obtain the second set of corrected weights. The first set of corrected weights is normalized to obtain the final category fusion weights corresponding to the first, second, and third perception data. The second set of corrected weights is normalized to obtain the final trajectory fusion weights corresponding to the second and third sensing data. Based on the final weight of the trajectory fusion, the positioning and trajectory information of the second and third sensing data are weighted and fused to generate the three-dimensional motion trajectory of the target. Based on the final weight of the category fusion, the initial classification result, radar classification result, and sonar classification result are weighted and fused to output the final category attribute information of the target.

9. The maritime target situational awareness method based on multimodal fusion according to claim 8, characterized in that, The step of correcting the basic weights based on the real-time signal-to-noise ratio specifically includes: multiplying the basic weights of each sensing data by the real-time signal-to-noise ratio of the corresponding sensing data, and then multiplying by a preset signal-to-noise ratio correction coefficient to obtain the corrected weight values.

10. A maritime target situational awareness system based on multimodal fusion, characterized in that, include: The first perception acquisition and recognition module is used to acquire first perception data through the first perception mode, perform target recognition processing on the first perception data, determine whether the maritime target exists and obtain the initial classification result; The second perception detection and positioning module is connected to the first perception acquisition and identification module. It is used to detect the area where the target is located through the second perception mode when the target is determined to meet the preset threat triggering conditions based on the initial classification result, acquire the second perception data, determine the spatial location information of the target in a unified geographic coordinate system and extract features, and acquire the radar classification result of the target. The third sensing detection and calculation module is connected to the second sensing detection and positioning module. After determining the spatial location information of the target, it uses the third sensing mode to detect the area indicated by the spatial location information as the center, obtains the third sensing data, calculates the orientation and distance information of the target in a unified geographic coordinate system and extracts features, and obtains the sonar classification result of the target. The fusion output module is connected to the first perception acquisition and recognition module, the second perception detection and positioning module, and the third perception detection and calculation module, respectively. It is used to perform fusion processing on the first perception data, the second perception data, and the third perception data using an adaptive weighted fusion algorithm to generate the three-dimensional motion trajectory of the target, fuse the initial classification result, the radar classification result, and the sonar classification result, and output the final category attribute information of the target.

Images