Multi-source heterogeneous data fusion method and device, equipment and storage medium

Abstract

The invention relates to a multi-source heterogeneous data fusion method and device, equipment and a storage medium, and the method comprises the steps: obtaining to-be-processed multi-source data, the multi-source data comprising an AI track and a plurality of pieces of to-be-aligned data; performing space-time alignment on the plurality of pieces of data to be aligned and the AI track in sequence to obtain associated target data; and performing multi-modal fusion processing on the associated target data to obtain target fusion data. According to the multi-source heterogeneous data fusion method provided by the invention, effective fusion of multi-source data can be realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for fusing multi-source heterogeneous data. Background Technology

[0002] With the increasing demand for marine security, underwater security systems are gradually developing towards multi-source sensing.

[0003] Currently, typical underwater security systems usually integrate a variety of heterogeneous sensors, including sonar, radar, optical cameras, AIS (Automatic Identification System), and various environmental parameter sensors. These sensors acquire target information from different dimensions and have their own advantages and applicable scenarios.

[0004] However, in practical applications, due to issues such as inconsistent data formats, different sampling frequencies, and inconsistent spatiotemporal references among various sensors, it is difficult to effectively fuse multi-source data. Summary of the Invention

[0005] This application provides a method, apparatus, computer equipment, and storage medium for fusing multi-source heterogeneous data, aiming to solve the problem that existing underwater security systems have difficulty in effectively fusing multi-source data.

[0006] In a first aspect, embodiments of this application provide a method for fusing multi-source heterogeneous data, the method comprising:

[0007] Acquire multi-source data to be processed, wherein the multi-source data includes AIS trajectory and multiple data to be aligned;

[0008] The multiple data to be aligned are sequentially spatiotemporally aligned with the AIS trajectory to obtain the associated target data;

[0009] The associated target data is subjected to multimodal fusion processing to obtain target fused data.

[0010] In some possible implementations, there are multiple AIS tracks, including a first seabed target AIS track, a second seabed target AIS track, and a third seabed target AIS track; the associated target data includes first associated target data, second associated target data, and third associated target data.

[0011] The step of spatiotemporally aligning the AIS trajectory with the data to be aligned to obtain the associated target data includes:

[0012] The data to be aligned is spatiotemporally aligned with the AIS trajectory of the first seabed target to obtain the first associated target data;

[0013] The data to be aligned is spatiotemporally aligned with the AIS trajectory of the second seabed target to obtain the second associated target data;

[0014] The data to be aligned is spatiotemporally aligned with the AIS trajectory of the third seabed target to obtain the third associated target data.

[0015] In some possible implementations, the data to be aligned includes: optical data, sonar data, environmental parameters, and positioning data of mobile devices.

[0016] In some possible implementations, the data to be aligned includes multiple trajectories to be matched, and the step of spatiotemporally aligning the AIS trajectories with the data to be aligned to obtain associated target data includes:

[0017] The path distance between each of the tracks to be matched and the AIS track is calculated sequentially to obtain multiple path distance values;

[0018] The trajectory to be matched corresponding to the smallest path distance value among multiple path distance values ​​is selected as the associated target data.

[0019] In some possible implementations, the step of performing multimodal fusion processing on the associated target data to obtain target fused data includes:

[0020] Determine the attribute feature data of the associated target data, wherein the attribute feature data includes sonar images, optical images, target trajectory data, and environmental parameters;

[0021] The attribute feature data is input into a multi-channel convolutional neural network for processing to obtain multiple convolution results output by the multi-channel convolutional neural network.

[0022] The multiple convolution results are input into the attention fusion layer for weight fusion processing to obtain the weighted features corresponding to each convolution result.

[0023] In some possible implementations, before inputting the convolution result into the attention fusion layer for weight fusion processing, the method further includes:

[0024] The multiple convolution results are input into the feature extraction layer for feature processing to obtain the feature vector corresponding to each convolution result output by the feature extraction layer.

[0025] The step of inputting multiple convolutional results into an attention fusion layer for weight fusion processing to obtain the weighted features corresponding to each convolutional result includes:

[0026] Multiple feature vectors are input into an attention fusion layer for weight fusion processing to obtain a weighted feature corresponding to each feature vector.

[0027] In some possible implementations, after obtaining the weighted features corresponding to each of the convolution results, the method further includes:

[0028] Multiple weighted features are input into a fully connected layer to obtain target fusion data output by the fully connected layer, wherein the target fusion data includes target type results, threat level results, and behavior prediction results.

[0029] Secondly, embodiments of this application also provide a fusion apparatus for multi-source heterogeneous data, which includes a unit for performing the above-described method.

[0030] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0031] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0032] This application provides a method, apparatus, computer device, and storage medium for fusing multi-source heterogeneous data. The method includes: acquiring multi-source data to be processed, wherein the multi-source data includes AIS trajectories and data to be aligned; sequentially aligning multiple data to be aligned with the AIS trajectories spatiotemporally to obtain associated target data; and performing multimodal fusion processing on the associated target data to obtain target fused data.

[0033] This application embodiment uses AIS trajectory as a spatiotemporal reference to perform spatiotemporal alignment processing with the data to be aligned (such as sonar images, radar signals, optical video, etc.). This effectively solves the problem of data mismatch caused by inconsistent sensor sampling frequencies, asynchronous timestamps, and inconsistent spatial coordinate systems in traditional methods, improving the alignment accuracy and efficiency of multi-source heterogeneous data. Furthermore, based on the completion of spatiotemporal alignment, multimodal fusion processing is performed on the associated target data, which can integrate the advantages of each sensor, eliminate noise interference and misjudgment risks caused by a single sensor, thereby improving the accuracy and stability of target recognition, tracking, and behavior prediction. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0037] Figure 1 A flowchart illustrating a method for fusing multi-source heterogeneous data provided in an embodiment of this application;

[0038] Figure 2 This is a flowchart illustrating the multimodal semantic fusion processing provided in an embodiment of this application.

[0039] Figure 3 This is a schematic diagram of the human-computer interaction visualization interface provided in the embodiments of this application;

[0040] Figure 4 This is a schematic diagram of the computer device structure provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0043] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0044] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0045] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0046] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0047] With the increasing demand for national marine security and energy infrastructure protection, the construction of underwater security systems in key locations such as ports, hydraulic platforms, and submarine pipelines has become a crucial technological direction. Especially in preventing potential threats from divers and underwater vehicles (such as AUVs, UUVs, and SDVs), building an all-weather, three-dimensional, and automated underwater monitoring system is of paramount importance. To improve detection accuracy and response efficiency, the system needs to utilize multiple heterogeneous sensing devices simultaneously, including radar, sonar, cameras, AIS, and environmental sensors. Existing underwater monitoring systems typically employ a multi-node deployment approach, including fixed outposts (such as underwater support stations), mobile outposts (such as underwater buoy arrays), and mobile outposts (such as AUVs equipped with multi-beam sonar), with a shore-based control center at its core, responsible for data reception, fusion processing, and command issuance. These nodes form a complete underwater defense network through underwater acoustic communication networks or cable connections, enabling the detection, identification, tracking, and repelling of targets.

[0048] However, existing underwater monitoring systems have some problems:

[0049] First, underwater communication and network construction is challenging. In the complex underwater marine environment, underwater acoustic communication suffers from low bandwidth, high latency, and high error rates, leading to unstable communication networks and making it difficult to meet the needs of long-distance, multi-node network interconnection. Furthermore, the coordination between multiple sub-devices (fixed sentry posts, mobile sentry posts, and mobile sentry posts) in security systems requires efficient and stable network support.

[0050] Secondly, the detection capability of underwater targets is limited. Current mid-to-long-range underwater mobile target detection mainly relies on acoustic images, which suffers from problems such as low image resolution and poor target recognition accuracy. In particular, it is difficult to accurately identify and continuously track small, low-noise targets (such as frogmen and underwater robots).

[0051] Secondly, heterogeneous sensor data is difficult to fuse. Underwater security systems use various types of sensors (sonar, radar, cameras, AIS, environmental parameter sensors, etc.). These data have different formats, time asynchrony, and spatial inconsistencies, making it impossible to achieve effective alignment and semantic association between multiple sensors, thus affecting the overall monitoring effect.

[0052] Furthermore, existing systems generally lack intuitive presentation and effective interaction of fusion results, making it difficult for monitoring personnel to understand system early warning information and target status in a timely manner, thus affecting command and response efficiency. At the same time, the high deployment cost and difficult recovery and maintenance of underwater engineering systems also limit their scalability and long-term operational capabilities.

[0053] To address the aforementioned technical problems in the prior art, this application provides a method for fusing multi-source heterogeneous data, which can achieve effective fusion of multi-source data.

[0054] See Figure 1 , Figure 1 A flowchart illustrating a method for fusing multi-source heterogeneous data provided in this application embodiment, the method comprising:

[0055] Step 110: Obtain the multi-source data to be processed.

[0056] The multi-source data includes AIS trajectories and multiple data to be aligned.

[0057] Step 120: Align the multiple data to be aligned with the AIS trajectory in a spatiotemporal manner to obtain the associated target data.

[0058] Step 130: Perform multimodal fusion processing on the associated target data to obtain target fusion data.

[0059] This embodiment uses the AIS trajectory as a spatiotemporal reference to perform spatiotemporal alignment processing with the data to be aligned (such as sonar images, radar signals, optical video, etc.). This effectively solves the problem of data mismatch caused by inconsistent sensor sampling frequencies, asynchronous timestamps, and inconsistent spatial coordinate systems in traditional methods. It improves the alignment accuracy and efficiency of multi-source heterogeneous data. Furthermore, based on the completion of spatiotemporal alignment, multimodal fusion processing is performed on the associated target data, which can integrate the advantages of each sensor and eliminate noise interference and misjudgment risks caused by a single sensor, thereby improving the accuracy and stability of target recognition, tracking, and behavior prediction.

[0060] In some possible implementations, there are multiple AIS tracks, including a first seabed target AIS track, a second seabed target AIS track, and a third seabed target AIS track; the associated target data includes first associated target data, second associated target data, and third associated target data.

[0061] The step of spatiotemporally aligning the AIS trajectory with the data to be aligned to obtain the associated target data includes:

[0062] Step 121: Perform spatiotemporal alignment of the data to be aligned with the AIS trajectory of the first seabed target to obtain the first associated target data.

[0063] Step 122: Align the data to be aligned with the AIS trajectory of the second seabed target in time and space to obtain the second associated target data.

[0064] Step 123: Perform spatiotemporal alignment of the data to be aligned with the AIS trajectory of the third seabed target to obtain the third associated target data.

[0065] In some possible implementations, the seabed target can be a ship, submarine, UUV, frogman, etc. For example, the first seabed target can be a ship, the second seabed target can be a submarine, and the third seabed target can be a UUV.

[0066] In some possible implementations, the data to be aligned includes: optical data, sonar data, environmental parameters, and positioning data of mobile devices.

[0067] In some possible implementations, the data to be aligned includes multiple trajectories to be matched, and the step of spatiotemporally aligning the AIS trajectories with the data to be aligned to obtain associated target data includes:

[0068] The path distance between each of the tracks to be matched and the AIS track is calculated sequentially to obtain multiple path distance values;

[0069] The trajectory to be matched corresponding to the smallest path distance value among multiple path distance values ​​is selected as the associated target data.

[0070] In some possible implementations, the step of performing multimodal fusion processing on the associated target data to obtain target fused data includes:

[0071] Determine the attribute feature data of the associated target data, wherein the attribute feature data includes sonar images, optical images, target trajectory data, and environmental parameters;

[0072] The attribute feature data is input into a multi-channel convolutional neural network for processing to obtain multiple convolution results output by the multi-channel convolutional neural network.

[0073] The multiple convolution results are input into the attention fusion layer for weight fusion processing to obtain the weighted features corresponding to each convolution result.

[0074] In some possible implementations, before inputting the convolution result into the attention fusion layer for weight fusion processing, the method further includes:

[0075] The multiple convolution results are input into the feature extraction layer for feature processing to obtain the feature vector corresponding to each convolution result output by the feature extraction layer.

[0076] The step of inputting multiple convolutional results into an attention fusion layer for weight fusion processing to obtain the weighted features corresponding to each convolutional result includes:

[0077] Multiple feature vectors are input into an attention fusion layer for weight fusion processing to obtain a weighted feature corresponding to each feature vector.

[0078] In some possible implementations, after obtaining the weighted features corresponding to each of the convolution results, the method further includes:

[0079] Multiple weighted features are input into a fully connected layer to obtain target fusion data output by the fully connected layer, wherein the target fusion data includes target type results, threat level results, and behavior prediction results.

[0080] Based on the above embodiments, this application provides a method for fusing multi-source heterogeneous data, which mainly includes the following steps:

[0081] 1) Data collection is carried out using multi-source sensing devices.

[0082] Among them, the multi-source sensing device can be a sensor, and the connected sensors include, but are not limited to, the following:

[0083] Water surface: Radar, AIS equipment, optical video surveillance equipment;

[0084] Underwater: Active sonar, passive sonar, multibeam imaging sonar, underwater camera, environmental parameter detection sensors (temperature, salinity, turbidity);

[0085] Mobile platforms: AUVs, underwater mooring vessels, and other mobile patrol devices equipped with forward-looking sonar, navigation, and positioning modules.

[0086] Each sensor is managed by a shore-based control center or an underwater substation, supporting multi-node access and asynchronous data acquisition, and possessing the capabilities of timestamp recording, preliminary classification and labeling, and status packaging.

[0087] The collected multi-source data can be divided into AIS trajectory and data to be aligned. The data to be aligned includes active sonar data, passive sonar data, radar data, optical video, monitoring data, environmental parameter data, etc.

[0088] 2) Data preprocessing;

[0089] Data preprocessing can include deduplication, handling missing or incomplete data, handling data with inconsistent time intervals and formats, and imputing or filling missing data. The specific processing can be carried out according to the actual situation.

[0090] 3) Perform spatiotemporal alignment between the AIS trajectory and the data to be aligned to obtain the associated target data.

[0091] For example, the AIS trajectory includes the AIS trajectory of the first seabed target, the AIS trajectory of the second seabed target, and the AIS trajectory of the third seabed target. The data to be aligned includes optical data, active sonar data, passive sonar data, and environmental parameters. Each data includes multiple trajectories to be matched.

[0092] Multiple tracks of optical data to be matched are sequentially matched with the AIS track of the first seabed target to obtain the path distance value between each track to be matched and the AIS track of the first seabed target.

[0093] The minimum value among multiple path distance values ​​is selected as the target data for the association between the optical data and the AIS trajectory of the first seabed target, i.e., the first associated target data;

[0094] Similarly, multiple tracks of optical data to be matched are sequentially matched with the AIS track of the second seabed target to obtain the path distance value between each track to be matched and the AIS track of the second seabed target.

[0095] The minimum value among multiple path distance values ​​is selected as the target data for the association between the optical data and the AIS trajectory of the second seabed target, i.e., the second associated target data.

[0096] Similarly, multiple tracks of optical data to be matched are sequentially matched with the AIS track of the third seabed target to obtain the path distance value between each track to be matched and the AIS track of the third seabed target.

[0097] The minimum value among multiple path distance values ​​is selected as the target data for the association between the optical data and the AIS trajectory of the third seabed target, i.e., the third associated target data.

[0098] Similarly, the active sonar data is spatiotemporally aligned with the AIS trajectories of the first, second, and third seabed targets to obtain associated target data.

[0099] The passive sonar data is spatiotemporally aligned with the AIS trajectories of the first, second, and third seabed targets to obtain associated target data.

[0100] The environmental parameters were spatiotemporally aligned with the AIS trajectories of the first, second, and third seabed targets to obtain associated target data.

[0101] In some possible implementations, spatiotemporal alignment can be achieved using a vector distance matrix.

[0102] In some possible implementations, the DTW algorithm can be used for spatiotemporal alignment.

[0103] 4) Perform multimodal semantic fusion processing on the attribute feature data of the associated target data;

[0104] The attribute feature data includes sonar images, optical images, target trajectory data, and environmental parameters.

[0105] Specifically, in combination Figure 2 The multimodal semantic fusion processing provided in this application may include the following:

[0106] 4-1) Input the sonar image, optical image, target trajectory data and environmental parameters into the multi-channel convolutional neural network to obtain the convolution result corresponding to each attribute feature data output by the multi-channel convolutional neural network;

[0107] 4-2) Input the convolution result corresponding to each attribute feature data output by the multi-channel convolutional neural network into the feature extraction layer for feature extraction, and obtain the feature vector corresponding to each attribute feature data output by the feature extraction layer;

[0108] For example, the first eigenvector, the second eigenvector, the third eigenvector, and the fourth eigenvector.

[0109] 4-3) Input the first feature vector, the second feature vector, the third feature vector, and the fourth feature vector into the attention fusion layer for weight calculation to obtain the first weighted feature, the second weighted feature, the third weighted feature, and the fourth weighted feature output by the attention fusion layer.

[0110] 4-4) Input the first weighted feature, the second weighted feature, the third weighted feature, and the fourth weighted feature into the fully connected layer for prediction processing to obtain the target type, threat level, and behavior prediction results output by the fully connected layer.

[0111] 5) Visualization and human-computer interaction.

[0112] Among them, the human-computer interaction visualization interface diagram can be as follows: Figure 3 As shown.

[0113] In other words, this application establishes an end-to-end technical path from data acquisition, spatiotemporal registration, semantic fusion to visualization interaction, to achieve efficient perception and intelligent monitoring of complex underwater environments.

[0114] Specifically, firstly, underwater monitoring involves various data types, such as images, sonar, sound waves, and temperature, salinity, and depth parameters, which are inconsistent in time, space, and modality. Without a unified preprocessing mechanism and registration method, the system will struggle to obtain a consistent and clear underwater situation map. Therefore, this invention effectively eliminates the time misalignment and coordinate offset problems between different sensors by introducing a unified time reference and spatial mapping system, enabling fusion processing to be conducted under the premise of "same time and same space," thereby improving fusion accuracy.

[0115] Secondly, traditional methods often remain at the level of data overlay or feature splicing, lacking the ability to mine the semantic information behind the data, making it difficult to distinguish between noise interference and real threats. Based on this, this application designs a semantic fusion mechanism for target behavior recognition and risk assessment. Combining deep learning models and trajectory fusion methods, it not only fuses static information about "what" (such as target location and shape) but also dynamic semantics about "what it is" (such as path trends, speed changes, and abnormal behavior), better meeting the actual underwater security needs. This fusion method far surpasses traditional solutions in terms of understanding ability, processing efficiency, and decision-making value.

[0116] Furthermore, in the expression and use of fusion results, existing systems often simply overlay trajectory points or hotspot areas onto electronic nautical charts, which fails to reflect the rich information and semantic hierarchy after multi-source fusion, affecting the intuitiveness and decision-making efficiency of manual operations. This invention, however, constructs a structured, visual human-computer interaction platform that displays fusion results in various forms such as image overlay, event warnings, and target analysis, enabling operators to quickly perceive target status, assess risk levels, and issue response instructions, thereby improving the system's real-time response and task command capabilities.

[0117] Furthermore, considering the limited communication capabilities in underwater environments, this invention employs edge processing and distributed task decomposition to ensure stable system operation even under low-bandwidth, high-latency underwater acoustic communication conditions. Each sub-node possesses preliminary fusion and filtering functions, uploading only high-value information, reducing bandwidth consumption, and improving system energy efficiency and autonomous operation capabilities. This design significantly enhances the system's engineering practicality, robustness, and scalability.

[0118] Corresponding to the above-described method for fusing multi-source heterogeneous data, this application also provides a device for fusing multi-source heterogeneous data. This device includes a unit for executing the aforementioned method for fusing multi-source heterogeneous data, and can be configured in a desktop computer, tablet computer, laptop computer, or other terminal.

[0119] Specifically, the fusion device for multi-source heterogeneous data includes multi-source sensing equipment, a data preprocessing module, a spatiotemporal registration and target data association module, a multimodal semantic fusion processing module, a visualization and human-computer interaction module, and a system deployment and communication support module.

[0120] The data preprocessing module mainly performs the following tasks:

[0121] 1) Unify the time base and align the data of each node in time;

[0122] 2) Spatial coordinate standardization: Projecting spatial data from different devices onto a unified reference coordinate system;

[0123] 3) Denoising and interpolation processing to remove abnormal data and complete lost packets;

[0124] 4) Data structure encapsulation and format conversion to establish a unified data description model.

[0125] By preprocessing the data, we can ensure that the subsequent multimodal semantic fusion processing module has the ability to process consistent inputs.

[0126] In some possible implementations, to achieve effective fusion of multiple detection data for the same target, this application introduces a spatiotemporal registration and target data association module, employing a spatiotemporal registration mechanism combined with the following key technologies:

[0127] The relative spatial relationship between the sensor and the target is established using underwater acoustic positioning systems (such as ultra-short baseline positioning and acoustic beacon trilateration).

[0128] A joint probabilistic data association algorithm (JPDA) is introduced to perform multi-sensor matching of track-level targets;

[0129] For image data such as sonar and video, an image projection and point cloud overlap method is used to assist in spatial registration;

[0130] For passive sonar information, targets are classified according to time windows and frequency characteristics to assist in multi-target discrimination.

[0131] After target association and registration are completed, various types of data are sent to the multimodal semantic fusion processing module, which integrates two mechanisms: the feature layer and the decision layer.

[0132] Specifically, in terms of feature fusion, the system uses a multi-channel neural network structure to extract features from image, waveform, and time series data in parallel, and uses an attention mechanism to weight and select the main signal sources.

[0133] In terms of decision fusion, a convex combination track fusion method is introduced to perform state estimation weighting among different sources to improve fusion accuracy;

[0134] The feature fusion results are combined with the target behavior model to perform threat level inference, behavior trend prediction and target identification (e.g., frogmen, small underwater vehicles, support mother ships, etc.);

[0135] It can simultaneously record the fusion results, processing paths, and risk assessment information of each objective.

[0136] In some possible implementations, the fusion results can be centrally displayed in a command and control center visualization interface, which includes:

[0137] 1) Multi-source layer overlay electronic nautical charts (including target trajectory and type labeling);

[0138] 2) The integrated situation dynamically refreshes the window;

[0139] 3) Classified early warning alerts for suspicious targets;

[0140] 4) Interactive event playback and historical trajectory query;

[0141] 5) Task execution feedback window (such as expulsion command, response status).

[0142] Users can confirm, provide feedback on, and issue task instructions for the fusion results, thus building a semi-automatic collaborative operation mode that is human-led and AI-assisted.

[0143] For the system deployment and communication support module, this application mainly adopts a distributed processing and communication layered strategy to take into account the limited underwater communication conditions, as follows:

[0144] 1) The underwater node has local preprocessing and compression capabilities, and only uploads important features and event triggering information;

[0145] 2) Data synchronization between support stations, detection stations, and patrol stations is achieved through an underwater acoustic communication network;

[0146] 3) High-bandwidth connections (such as submarine optical cables) are used for full communication between key points and shore-based control centers;

[0147] 4) The system can run basic monitoring and emergency functions without relying on global communication, thereby improving the system's fault tolerance and independent operation capabilities.

[0148] like Figure 4 As shown in the figure, this application provides a computer device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0149] Memory 113 is used to store computer programs;

[0150] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the control method for multi-source heterogeneous data fusion for underwater security provided in any of the foregoing method embodiments, including:

[0151] Acquire multi-source data to be processed, wherein the multi-source data includes AIS trajectory and data to be aligned;

[0152] The multiple data to be aligned are sequentially spatiotemporally aligned with the AIS trajectory to obtain the associated target data;

[0153] The associated target data is subjected to multimodal fusion processing to obtain target fused data.

[0154] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0155] Therefore, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the multi-source heterogeneous data fusion method provided in any of the foregoing method embodiments, including:

[0156] Acquire multi-source data to be processed, wherein the multi-source data includes AIS trajectory and data to be aligned;

[0157] The multiple data to be aligned are sequentially spatiotemporally aligned with the AIS trajectory to obtain the associated target data;

[0158] The associated target data is subjected to multimodal fusion processing to obtain target fused data.

[0159] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0162] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0164] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0165] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.

[0166] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for fusing multi-source heterogeneous data, characterized in that, The method includes: Acquire multi-source data to be processed, wherein the multi-source data includes AIS trajectory and multiple data to be aligned; The multiple data to be aligned are sequentially spatiotemporally aligned with the AIS trajectory to obtain the associated target data; The associated target data is subjected to multimodal fusion processing to obtain target fused data.

2. The method according to claim 1, characterized in that, The AIS tracks are multiple, including a first seabed target AIS track, a second seabed target AIS track, and a third seabed target AIS track; the associated target data includes first associated target data, second associated target data, and third associated target data. The step of spatiotemporally aligning the AIS trajectory with the data to be aligned to obtain the associated target data includes: The data to be aligned is spatiotemporally aligned with the AIS trajectory of the first seabed target to obtain the first associated target data; The data to be aligned is spatiotemporally aligned with the AIS trajectory of the second seabed target to obtain the second associated target data; The data to be aligned is spatiotemporally aligned with the AIS trajectory of the third seabed target to obtain the third associated target data.

3. The method according to claim 1, characterized in that, The data to be aligned includes: optical data, sonar data, environmental parameters, and positioning data of mobile devices.

4. The method according to claim 1, characterized in that, The data to be aligned includes multiple trajectories to be matched. The step of spatiotemporally aligning the AIS trajectories with the data to be aligned to obtain associated target data includes: The path distance between each of the tracks to be matched and the AIS tracks is calculated sequentially to obtain multiple path distance values; The trajectory to be matched corresponding to the smallest path distance value among multiple path distance values ​​is selected as the associated target data.

5. The method according to claim 1, characterized in that, The step of performing multimodal fusion processing on the associated target data to obtain target fused data includes: The attribute feature data of the associated target data is determined, wherein the attribute feature data includes sonar images, optical images, target trajectory data, and environmental parameters; The attribute feature data is input into a multi-channel convolutional neural network for processing to obtain multiple convolution results output by the multi-channel convolutional neural network. The multiple convolution results are input into the attention fusion layer for weight fusion processing to obtain the weighted features corresponding to each convolution result.

6. The method according to claim 5, characterized in that, Before inputting the convolution result into the attention fusion layer for weight fusion processing, the method further includes: The multiple convolution results are input into the feature extraction layer for feature processing to obtain the feature vector corresponding to each convolution result output by the feature extraction layer. The step of inputting multiple convolutional results into an attention fusion layer for weight fusion processing to obtain the weighted features corresponding to each convolutional result includes: Multiple feature vectors are input into an attention fusion layer for weight fusion processing to obtain a weighted feature corresponding to each feature vector.

7. The method according to claim 5, characterized in that, After obtaining the weighted features corresponding to each of the convolution results, the method further includes: Multiple weighted features are input into a fully connected layer to obtain target fusion data output by the fully connected layer, wherein the target fusion data includes target type results, threat level results, and behavior prediction results.

8. A device for fusing multi-source heterogeneous data, characterized in that, Includes a unit for performing the method as described in any one of claims 1-7.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images