Radar-sonar-video fusion association method and system based on cross-modal semantic mapping

Abstract

The invention discloses a radar-sonar-video fusion association method and system based on cross-modal semantic mapping. The method comprises the steps that modal specific feature vectors are extracted from three heterogeneous sensors including a radar sensor, a sonar sensor and an optical video sensor respectively; constructing a three-branch deep metric learning network, and mapping the heterogeneous features to a unified d-dimensional semantic embedding space; a tripartite graph model is constructed, and semantic similarity and kinematics consistency constraints are comprehensively considered for edge weights; an improved Kuhn-Munkres algorithm is applied to solve the global optimal matching of the tripartite graph; and outputting a fusion target state and matching confidence evaluation. Cross-physical-domain invariant feature representation is automatically extracted through deep metric learning, three-mode global optimal association is achieved in combination with a graph optimization algorithm, the technical problem that cross-physical-domain data fusion is difficult to process through a traditional method is solved, and the method has high accuracy, strong generalization ability and real-time performance in application scenes such as ocean monitoring and port security and protection and is suitable for being used in the field of ocean monitoring and port security and protection. And an effective technical scheme is provided for multi-sensor heterogeneous data fusion.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor information fusion technology, specifically to a method for cross-modal association and fusion of target detection data from three heterogeneous sensors: radar, sonar, and optical video. It is particularly suitable for applications such as marine environmental monitoring, port security, and unmanned systems that require multi-physical domain collaborative perception of the same target. Background Technology

[0002] In complex scenarios such as marine monitoring, port security, and unmanned systems, it is often necessary to use multiple heterogeneous sensors, such as radar, sonar, and optical video, in a coordinated manner to detect and track targets. Radar operates in the electromagnetic wave band and excels at long-range ranging and velocity measurement; sonar, based on the underwater propagation of sound waves, is the primary means of underwater target detection; and optical video can provide rich visual information such as texture and shape. However, these three types of sensors operate in the electromagnetic, acoustic, and optical domains, respectively, and their output data differ significantly in feature representation, data format, accuracy, and error characteristics. This cross-physical domain "feature heterogeneity" is a fundamental challenge faced by traditional multi-sensor data fusion methods.

[0003] Existing multi-sensor correlation and fusion methods have the following main limitations: Feature level: Traditional methods often rely on manually designed features (such as geometric center, motion trajectory) or simple data-level fusion, which makes it difficult to effectively extract and utilize deep, semantically consistent common representations in cross-modal data, resulting in a decrease in association accuracy under heterogeneous feature conditions.

[0004] At the association strategy level: Common methods employ a "serial matching" strategy, such as first associating radar and sonar, and then associating the result with video. This strategy leads to the propagation and accumulation of early matching errors, making it impossible to achieve globally optimal association, and it is difficult to handle association ambiguities in multi-target dense scenes.

[0005] In terms of generalization ability: Existing algorithms are usually customized for specific sensor combinations or environments, and their rules and thresholds rely on human experience. When the sensor model is changed, environmental conditions (such as weather and water quality) change, or the target type becomes diverse, the robustness and generalization ability of the algorithm often decrease significantly.

[0006] Therefore, how to automatically learn invariant semantic features across physical domains and achieve globally optimal association of multimodal data within a unified framework has become a key technical challenge for improving the performance of multi-sensor fusion systems. This invention aims to propose a novel fusion association method that fundamentally addresses the aforementioned challenges. Summary of the Invention

[0007] The purpose of this invention is to provide a radar-sonar-video fusion association method and system based on cross-modal semantic mapping to address the aforementioned problems.

[0008] The technical solution of the present invention is as follows: A radar-sonar-video fusion association method based on cross-modal semantic mapping includes the following steps: Feature extraction: Modality-specific feature vectors are extracted from the raw data collected by radar sensors, sonar sensors, and optical video sensors, respectively, denoted as... , and ; Semantic mapping: Constructing a three-branch deep metric learning network: Modality-specific feature vectors , and Mapped to a unified The semantic embedding space is used to obtain the corresponding semantic embedding vector. , and ; Graph construction: In In the semantic embedding space, construct a set of radar nodes. Sonar node set and video node set The three-part diagram edge set Only connect nodes of different modalities, each edge Connecting nodes With nodes Its edge weight It is calculated based on the semantic similarity and kinematic consistency between nodes; Global matching: Based on tripartite graph The edge weights are determined, and an improved assignment algorithm is used to solve for the globally optimal matching relationship across the three modalities of radar, sonar, and video. ; State fusion: based on the globally optimal matching relationship It fuses the states of the same target from different sensors and outputs the fused target state and the corresponding matching confidence assessment.

[0009] The above method, employing a three-branch deep metric learning network, effectively maps the physical domain features of three heterogeneous sensors—radar, sonar, and optical video—to a unified semantic embedding space, fundamentally solving the feature heterogeneity problem in cross-physical domain data association. Through the triplet loss function and hard example mining strategy, the cross-modal representations of the same target form a clear intra-class aggregation and inter-class separation structure, significantly improving the accuracy of multi-sensor target association.

[0010] Furthermore, the three-branch deep metric learning network is trained using a triplet loss function, which... The definition of is: , in, For the embedding vector of the anchor sample, This is the embedding vector of a positive sample that belongs to the same target as the anchor sample but comes from a different sensor modality. This is the embedding vector of negative samples that belong to different targets from the anchor samples. These are preset boundary hyperparameters, with values ​​ranging from 0.2 to 1.0. The number of triples in the training batch; Let be the Euclidean distance norm.

[0011] Furthermore, the training process of the three-branch deep metric learning network includes: Construct a labeled dataset containing the features and target IDs of the same target under three different sensors; A semi-hard negative sample mining strategy is used for triple sampling. The online semi-hard negative sample mining strategy is as follows: That is, selecting samples that are farther from the anchor point than the positive samples, but still within the boundary. negative samples within the range ; Use inclusion Total loss function of regularization term Optimization was carried out, including The regularization coefficient is... for Regularization term; The network is considered converged when the average distance between positive samples on the validation set is less than 0.3 and the difference between the average distances between positive and negative samples is greater than 0.5.

[0012] Furthermore, the edge weights The calculation formula is: , , , in, These are the motion constraint weighting coefficients. It degenerates into pure semantic matching. Equal weighting of temporal semantics and motion; This is a spatial scale parameter that controls the sensitivity of similarity. , For nodes , The velocity vector corresponding to the target; For velocity scale parameters.

[0013] Furthermore, the improved assignment algorithm is the Kuhn-Munkres algorithm employing a hierarchical matching strategy, specifically including: Phase 1: In the radar node set With sonar node set Between, based on the weight matrix Solving for maximum weight matching ; Phase 2: For each matched radar-sonar pair obtained in Phase 1, construct a fusion node and calculate its relationship with the video node set. The combined weights of all nodes are used to form a new weight matrix. The Kuhn-Munkres algorithm is applied again to solve the matching problem. This leads to the final global trimodal matching. .

[0014] The adaptive feature learning mechanism based on contrastive learning can automatically extract cross-physical domain invariant feature representations without requiring the design of specific matching rules for different sensor combinations. This design significantly reduces reliance on human prior knowledge and ensures strong generalization ability of the method under different environmental conditions, target types, and sensor configurations.

[0015] Furthermore, the feature extraction specifically includes: Extracted radar feature vector Includes range, azimuth, radial velocity, and radar cross section (RCS) parameters; Extracted sonar feature vector It includes distance, azimuth, and peak frequency, centroid frequency, -3dB bandwidth, and total energy extracted after performing a short-time Fourier transform on the echo spectrum; Extracted video feature vector It includes the target centroid coordinates, the HOG texture descriptor reduced by PCA, and the color histogram reduced by mean pooling.

[0016] Furthermore, the fusion of successfully matched triples includes: Location fusion: using weighted average , The weights are related to the sensor accuracy; , These are the standard deviations of position measurements for radar, sonar, and video, respectively. Matching confidence calculation: by edge weights The confidence score is obtained by performing Z-score normalization. ,in , Given the mean and standard deviation of all edge weights in the current frame, and based on... The value categorizes the matching quality into three levels: high, medium, and low.

[0017] Furthermore, it also includes timing tracing steps: Define the target state vector as This includes position, velocity, and acceleration; Establish state transition equations With observation equation ; The state of the fusion target is continuously estimated and tracked using a Kalman filter.

[0018] Furthermore, the global three-modal matching The triangular consistency constraint must be satisfied: if the radar node With sonar nodes Matching, and sonar nodes With video nodes If a match is found, then the radar node... Must be with video node match.

[0019] By constructing a tripartite graph and applying the Hungarian method to solve for the globally optimal matching, this invention achieves trimodal globally optimal association, avoiding the error accumulation problem of traditional two-sequence matching methods. The edge weight calculation mechanism, which comprehensively considers semantic similarity and motion consistency, ensures the global optimality of the matching results, improving target association accuracy by 15%-30% compared to traditional methods.

[0020] A hierarchical matching strategy is adopted to transform the NP-hard three-dimensional assignment problem into a multi-dimensional assignment problem. The time complexity of the first method is O((m+n)^3), which can meet the real-time requirements of practical scenarios. Combined with confidence assessment and hierarchical quality judgment mechanism, manual review or downgrading processing is provided for low-confidence matching, further improving the reliability and security of the system.

[0021] This application also includes a radar-sonar-video fusion association system based on cross-modal semantic mapping, characterized in that it includes a central processing unit, a feature extraction module, a deep learning inference module and a graph optimization module, and is equipped with a millimeter-wave radar, an active sonar and an optical camera. The central processing unit is used to execute a radar-sonar-video fusion association method based on cross-modal semantic mapping.

[0022] Compared with existing technologies, the advantages of this invention are: 1. This invention employs a three-branch deep metric learning network, which can automatically learn cross-physical domain invariant features, effectively solving the problem of traditional methods relying on manual rules and improving the discriminativeness of feature representation; 2. By constructing a tripartite graph model and applying an improved Hungarian algorithm, this invention can achieve global optimal association in three modes, effectively avoiding the error accumulation problem of serial matching and improving the accuracy of association. 3. Based on cross-modal semantic mapping design, the similarity of features in different physical domains can be measured in a unified semantic space, making the invention stable under various environments and target types, and improving generalization ability; 4. The time complexity of the algorithm is optimized by adopting a hierarchical matching strategy, which enables the invention to meet the real-time processing requirements in complex scenarios and provides effective technical support for multi-sensor fusion tracking. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the method in this application. Detailed Implementation

[0024] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0025] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0026] Please see Figure 1This paper presents a radar-sonar-video fusion association method based on cross-modal semantic mapping. This method aims to achieve globally optimal target association among three heterogeneous sensors in the electromagnetic, acoustic, and optical domains. The invention is based on the following steps: First, a modal-specific feature extraction module processes the raw data from the three sensors separately to generate structured feature vectors. Next, a three-branch deep metric learning network is used to map the heterogeneous features to a unified semantic space through contrastive learning, ensuring that cross-modal representations of the same target are close to each other in the embedding space. Subsequently, a tripartite graph is constructed in the semantic space, and edge weights are calculated, integrating semantic similarity and kinematic constraints. Finally, the Hungarian algorithm is used to solve for the globally optimal trimodal matching, and the fusion result and confidence assessment are output. Specifically, the method includes the following steps: Multimodal specific feature extraction: Radar feature extraction: For millimeter-wave radar (operating frequency 77GHz, scan period 100ms), at time... The detected first The target has a raw output of a point cloud in polar coordinates: , The physical meaning of each component is as follows: Target distance (unit: meters), calculated by measuring the round-trip time of the electromagnetic wave: , in, It is the speed of light.

[0027] Azimuth (unit: degrees), range of values , representing the horizontal angle of the target relative to the radar normal direction; Radial velocity (unit: ), calculated using Doppler frequency shift: , in: For wavelength, This is the Doppler frequency shift.

[0028] Radar cross section (RCS) (unit: dBsm) reflects the electromagnetic reflection intensity of the target. , in, This refers to the received power.

[0029] Feature vector construction: Convert polar coordinates to Cartesian coordinates and normalize them to construct feature vectors. , in, , , , These are the normalized parameters.

[0030] Sonar Feature Extraction: For an active sonar system (operating frequency 50kHz, detection period 200ms), at time... The detected first The target has the following raw outputs: , in: Target distance (unit: meters).

[0031] Calculated using sound wave propagation time: , in: This is the speed of sound in water.

[0032] Horizontal azimuth (unit: degrees), range of values ; The echo spectrum (frequency range 40-60kHz) is obtained through short-time Fourier transform. , in, In order to receive signals, For Hamming window functions; Echo intensity (unit: dB), the amount of attenuation relative to the transmitted power.

[0033] Dimensionality reduction of spectral features Extracting statistical features: , in, Peak frequency; Centroid frequency, , -3dB bandwidth.

[0034] The complete feature vector is: , in, For the center frequency, , .

[0035] Video feature extraction: For an optical camera (resolution 1920×1080, frame rate 25fps), an improved YOLOv5 object detection algorithm is used to obtain detection boxes. Target extraction: Spatial location features are set as detection boxes Calculate the centroid coordinates and normalized dimensions: , in, , The width and height of the image.

[0036] HOG texture features: Calculate the directional gradient histogram (9 directions) in the detection box region. (8×8 pixel cell), extract 128-dimensional HOG descriptors Dimensionality reduced to 32 dimensions using PCA: , in, This is the pre-trained principal component projection matrix.

[0037] Color histogram features: The histogram is calculated in the HSV color space (H: 8 bins, S: 4 bins, V: 4 bins, total 128 dimensions), and then normalized. , Dimensionality reduced to 16 dimensions using mean pooling: , The complete feature vector is: , in, This indicates a vector concatenation operation.

[0038] Network architecture design: Constructing a three-branch twin network Each branch is an independent feature encoder, but they share the same embedding space.

[0039] Radar branch network : , , , in, , This is the weight matrix; For batch normalization operations: ; The shared embedding layer weight matrix (shared by all three branches).

[0040] Sonar branch network : , , , Note: The embedding layer shares parameters with the radar branch here. .

[0041] Video branch network Due to the high dimensionality of video features (52 dimensions), an additional network layer is added: , , , ; Normalization ensures that the embedding vector lies on the hypersphere, facilitating similarity calculation. Therefore, the output is normalized. , in, The final embedding dimension is set to .

[0042] Triple loss function design: Triple construction strategy: For each target sample in the training set, construct a triplet. : Anchor point Target features obtained from a sensor (such as radar); Positive samples Features of the same target on other sensors (such as sonar or video); negative samples The characteristics of different targets in any sensor; Specific strategies adopted: , , ; Triple loss function: , in, This represents the number of triples in the batch. This is the boundary hyperparameter (margin), typically ranging from 0.2 to 1.0, which controls the separation between positive and negative samples; Let be the Euclidean distance norm.

[0043] Difficult example mining strategy: To accelerate convergence, semi-hard negative sample mining is employed: , That is, selecting samples that are farther from the anchor point than the positive samples, but still within the boundary. Choose negative samples within the range, avoiding those that are too easy (far away) or too difficult (near away).

[0044] Training process: Dataset Construction: Collecting labeled datasets: , in, The total number of samples, For target ID.

[0045] Data augmentation strategies: Radar: Add Gaussian noise Simulated ranging error; Sonar: Random Spectral Shift ; Video: Randomly cropped, flipped, and color jittered.

[0046] Optimizer configuration: Adam optimizer is used, hyperparameter settings: ; Learning rate decays in stages: , , ; Training objective: Total loss function: , in, for Regularization term, This is the regularization coefficient.

[0047] Convergence criterion is the average positive sample distance on the validation set. satisfy: At that time, it was assumed that the network converged.

[0048] Three-modal global association based on graph optimization: Construction of a ternary graph: Formal definition of a graph: Constructing a ternary graph ,in: Node set: , Embedded representation of radar-detected targets; Embedded representation of sonar-detected targets; Embedded representation of the target in video detection; Edge set: , Edge between radar and sonar nodes; Edge between radar and video nodes; Edges between sonar and video nodes; Constraint: Nodes within the same mode are not connected by edges, i.e. .

[0049] Edge weight calculation: For any cross-modal node pair ,in edge weight Taking into account both semantic similarity and motion consistency: Semantic similarity is calculated using a Gaussian kernel that computes the Euclidean distance in the embedding space: , in, This is a spatial scale parameter that controls the sensitivity of similarity; typical values ​​are... .

[0050] Motion consistency: If the sensor can provide speed information (as provided by radar) If the video is estimated using optical flow, then a velocity constraint is introduced: , in, , For nodes , The velocity vector of the corresponding target (must be transformed to a unified coordinate system); For velocity scale parameters, typical values ​​are... For sonar (which lacks direct velocity information), position estimation can be achieved through consecutive frame differential estimation: , Overall edge weights: ,in, These are the motion constraint weighting coefficients. It degenerates into pure semantic matching. Time semantics and motion are weighted equally.

[0051] The edge weight matrix is ​​constructed by defining three sub-matrices: , , .

[0052] Modeling the tripartite graph matching problem: Matching variable definition: Introducing binary indicator variables: .

[0053] The optimization objective is to maximize the sum of global matching weights: .

[0054] The constraint condition for three-mode matching is that it must satisfy the triangular consistency constraint: (A radar node can be matched with a maximum of one sonar node). (A sonar node can be matched with a maximum of 1 radar node).

[0055] Similar constraints are applied and Triangular closure constraint: If Match and If matched, then A match is required.

[0056] Solution using the Hungarian algorithm: Problem Dimensionality Reduction: A hierarchical matching strategy is used to transform the three-dimensional assignment problem into a two-dimensional assignment problem. Phase 1: Radar-Sonar Matching , obtain the set of matching pairs .

[0057] Phase Two: Fusion Node - Video Matching: For each matched radar-sonar pair Construct fusion nodes: ; Calculate the weights of the fused node and the video node: ; Construct a new weight matrix Apply the Hungarian algorithm again: .

[0058] Hungarian algorithm implementation (Kuhn-Munkres algorithm): for weight matrix Solve for the maximum weight matching: The Covenant: ; List reduction: ; Finding zero-element coverage: Use the minimum line coverage algorithm to find the minimum number of row and column lines that cover all zero elements.

[0059] Augmented path search: if the number of covered lines Modify the matrix: subtract the minimum value of the uncovered region from the uncovered elements. Double overlay element plus Return to the step of finding the zero element to cover.

[0060] Extracting the optimal match: when the number of coverage lines When selecting an unmarked zero element from each row, the optimal match is formed. .

[0061] Algorithm time complexity: For real-time applications The requirements can be met.

[0062] Fusion target state calculation: for successfully matched triples , merge target state: Location fusion (weighted average): , The weights are related to the sensor accuracy; , These are the standard deviations of position measurements for radar, sonar, and video (obtained through calibration).

[0063] Velocity fusion: Radial velocity provided by radar It needs to be combined with the azimuth angle Convert to Cartesian velocity: , The image plane velocity is estimated using optical flow, which needs to be combined with the target distance. Back-projection to 3D space. Finally, Kalman filtering is used for fusion. , in This is the Kalman gain matrix.

[0064] Category decision: The video provides the target category (e.g., "ship", "buoy"), with a confidence level of [value missing]. Verification was performed by combining radar RCS and sonar spectral characteristics:

[0065] Match confidence calculation: Z-score normalized confidence: , in , This represents the mean and standard deviation of all edge weights in the current frame.

[0066] Threshold determination: , Low-confidence matching triggers manual review or downgrades to single-sensor tracking.

[0067] Real-time processing and time-series tracing: Kalman filter temporal correlation: Introducing the multiple hypothesis tracking (MHT) framework, the state vector is defined as: , State transition equation: , Where the state transition matrix is: , For process noise, The sampling period.

[0068] Observation equation: , in, , To observe noise.

[0069] Dynamic adjustment strategy: Monitor threat posture changes in real time and reassess when the following conditions are triggered: ① The change in the target quantity exceeds the set threshold; ② The change in distance between targets exceeds a set threshold; ③ Significant changes in environmental conditions; Gradient descent is used to update model parameters, optimize the embedding space representation, and improve the accuracy of risk assessment.

[0070] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A radar-sonar-video fusion association method based on cross-modal semantic mapping, characterized in that, Includes the following steps: Feature extraction: Modality-specific feature vectors are extracted from the raw data collected by radar sensors, sonar sensors, and optical video sensors, respectively, denoted as... , and ; Semantic mapping: Constructing a three-branch deep metric learning network: Modality-specific feature vectors , and Mapped to a unified The semantic embedding space is used to obtain the corresponding semantic embedding vector. , and ; Graph construction: In In the semantic embedding space, construct a set of radar nodes. Sonar node set and video node set The three-part diagram edge set Only connect nodes of different modalities, each edge Connecting nodes With nodes Its edge weight It is calculated based on the semantic similarity and kinematic consistency between nodes; Global matching: Based on tripartite graph The edge weights are determined, and an improved assignment algorithm is used to solve for the globally optimal matching relationship across the three modalities of radar, sonar, and video. ; State fusion: based on the globally optimal matching relationship It fuses the states of the same target from different sensors and outputs the fused target state and the corresponding matching confidence assessment.

2. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 1, characterized in that, The three-branch deep metric learning network is trained using a triplet loss function. The definition of is: ， in, For the embedding vector of the anchor sample, This is the embedding vector of a positive sample that belongs to the same target as the anchor sample but comes from a different sensor modality. This is the embedding vector of negative samples that belong to different targets from the anchor samples. These are preset boundary hyperparameters, with values ​​ranging from 0.2 to 1.

0. The number of triples in the training batch; Let be the Euclidean distance norm.

3. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 2, characterized in that, The training process of the three-branch deep metric learning network includes: Construct a labeled dataset containing the features and target IDs of the same target under three different sensors; A semi-hard negative sample mining strategy is used for triple sampling. The online semi-hard negative sample mining strategy is as follows: That is, selecting samples that are farther from the anchor point than the positive samples, but still within the boundary. negative samples within the range ; Use inclusion Total loss function of regularization term Optimization was carried out, including The regularization coefficient is... for Regularization term; The network is considered converged when the average distance between positive samples on the validation set is less than 0.3 and the difference between the average distances between positive and negative samples is greater than 0.

5.

4. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 1, characterized in that, The edge weight The calculation formula is: ， ， ， in, These are the motion constraint weighting coefficients. It degenerates into pure semantic matching. Temporal semantics and motion weights; This is a spatial scale parameter that controls the sensitivity of similarity. , For nodes , The velocity vector corresponding to the target; For velocity scale parameters.

5. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 1, characterized in that, The improved assignment algorithm is the Kuhn-Munkres algorithm using a hierarchical matching strategy, specifically including: Phase 1: In the radar node set With sonar node set Between, based on the weight matrix Solving for maximum weight matching ; Phase 2: For each matched radar-sonar pair obtained in Phase 1, construct a fusion node and calculate its relationship with the video node set. The combined weights of all nodes are used to form a new weight matrix. The Kuhn-Munkres algorithm is applied again to solve the matching problem. This leads to the final global trimodal matching. .

6. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 1, characterized in that, The feature extraction specifically includes: Extracted radar feature vector Includes range, azimuth, radial velocity, and radar cross section (RCS) parameters; Extracted sonar feature vector It includes distance, azimuth, and peak frequency, centroid frequency, -3dB bandwidth, and total energy extracted after performing a short-time Fourier transform on the echo spectrum; Extracted video feature vector It includes the target centroid coordinates, the HOG texture descriptor reduced by PCA, and the color histogram reduced by mean pooling.

7. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 1, characterized in that, The fusion of successfully matched triples includes: Location fusion: using weighted average , The weights are related to the sensor accuracy; , These are the standard deviations of position measurements for radar, sonar, and video, respectively. Matching confidence calculation: by edge weights The confidence score is obtained by performing Z-score normalization. ,in , Given the mean and standard deviation of all edge weights in the current frame, and based on... The value categorizes the matching quality into three levels: high, medium, and low.

8. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 1, characterized in that, It also includes timing tracing steps: Define the target state vector as This includes position, velocity, and acceleration; Establish state transition equations With observation equation ; The state of the fusion target is continuously estimated and tracked using a Kalman filter.

9. The radar-sonar-video fusion association method based on cross-modal semantic mapping according to claim 6, characterized in that, The global trimodal matching The triangular consistency constraint must be satisfied: if the radar node With sonar nodes Matching, and sonar nodes With video nodes If a match is found, then the radar node... Must be with video node match.

10. A radar-sonar-video fusion correlation system based on cross-modal semantic mapping, characterized in that, It includes a central processing unit, a feature extraction module, a deep learning inference module, and a graph optimization module, and is equipped with millimeter-wave radar, active sonar, and an optical camera. The central processing unit is used to execute a radar-sonar-video fusion association method based on cross-modal semantic mapping as described in any one of claims 1-9.

Images