A multi-source heterogeneous data-oriented fusion processing method

Abstract

This invention belongs to the field of data processing technology, specifically a fusion processing method for multi-source heterogeneous data. The method includes: receiving multi-source heterogeneous data and constructing a set of feature vectors; determining dynamic semantic anchors based on a density peak clustering algorithm; assigning a semantic membership vector to each feature point; constructing a spatiotemporal point cloud based on the semantic membership vectors; performing cross-modal data alignment by calculating the confidence level of data points; dividing the aligned data into feature blocks according to a spatial grid; calculating the Wasserstein distance between the feature distributions of adjacent feature blocks; performing feature fusion based on the calculation results; and verifying the fusion quality of the fusion results; calculating a fusion quality evaluation index based on the deviation between the decision results and the actual observed data; and performing feedback adjustment based on the calculated fusion quality evaluation index. This invention achieves an adaptive mapping effect for multi-source heterogeneous data in a unified semantic space by employing a unified representation construction technology for heterogeneous data based on dynamic anchor semantic indexes.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically a method for fusion processing of multi-source heterogeneous data. Background Technology

[0002] With the diversification of information collection methods, data from different sources exhibits diverse types, significant structural differences, and inconsistent spatiotemporal scales. Multi-source heterogeneous data is gradually becoming the main data form in data-driven systems. Effectively fusing and processing this type of data is of great significance for improving data utilization efficiency, enhancing system perception capabilities, and supporting high-precision decision-making.

[0003] However, existing technologies are mainly limited by the core technical bottleneck of large differences in data structure and lack of unified representation when realizing unified fusion processing of multi-source heterogeneous data. This makes it difficult to balance the accuracy of cross-modal data fusion and the real-time performance of the fusion process in practical applications, thus affecting the overall performance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a fusion processing method for multi-source heterogeneous data, thereby resolving issues such as large differences in data structure and lack of unified representation in existing technologies.

[0005] A fusion processing method for multi-source heterogeneous data includes the following steps: receiving multi-source heterogeneous data, extracting features of each modality and constructing a feature vector set, calculating the local density of each feature point based on the density peak clustering algorithm, and determining the initial anchor point of each modality; calculating the cross-modal semantic similarity between the initial anchor points of different modalities, determining the dynamic semantic anchor point, constructing a unified semantic index map of all dynamic semantic anchor points and their associated original feature points, and assigning a semantic membership vector to each feature point; The semantic membership vector, timestamp, and spatial coordinates of each data point are used to construct a spatiotemporal point cloud with semantic labels. The initial seed point is determined based on the constructed spatiotemporal point cloud, and the candidate alignment point is determined by calculating the temporal offset of the data point. Cross-modal data alignment is performed by calculating the confidence of the data point. The aligned data pairs are input to the shared feature extractor to generate a high-dimensional feature map, and feature blocks are divided according to a spatial grid. The Wasserstein distance between the feature distributions of adjacent feature blocks is calculated. Feature fusion is performed based on the calculation results, and the fusion quality is verified. The fused feature map is applied to downstream tasks, and a fusion quality assessment index is calculated based on the deviation between the decision results and the actual observation data. Feedback adjustment is then performed based on the calculated fusion quality assessment index.

[0006] Preferably, the determination of the initial anchor point for each mode is as follows: Calculate the feature points of each modal feature set The local density is then: ; in, Representing feature points and The Euclidean distance between them This indicates the preset cutoff distance parameter. For indicator functions, Representing feature points The local density is used to filter initial anchor points, specifically: Based on the modality sequence number, calculate the local density of feature points in each modality, and use the feature point corresponding to the maximum local density as the initial anchor point of the current modality.

[0007] Preferably, the determination of the dynamic semantic anchor point is as follows: Determine the semantic boundaries of the initial anchor points for each modality, and calculate the cross-modal semantic similarity between anchor points of different modalities. ; Dynamic semantic anchors are determined based on the calculated cross-modal semantic similarity, specifically as follows: If cross-modal semantic similarity Greater than the preset first threshold Then it means the first Type and first mode The initial anchor points of each modality are semantically equivalent. They are merged into a dynamic semantic anchor point, and the semantic vector of the merged dynamic anchor point is updated to the mean vector. Conversely, if If the two anchor points are semantically inequivalent, they are retained as independent anchor points.

[0008] Preferably, the construction is a spatiotemporal point cloud with semantic labels, as follows: For each original feature point A semantic membership vector is assigned based on the distance to each dynamic semantic anchor point. Then we have: ; in, Indicates the smoothing parameter. Representing feature points With the A dynamic semantic anchor point The Euclidean distance between them Representing feature points Belonging to the Membership value of a dynamic semantic anchor point; The semantic membership vector of each data point As its semantic tag, its original timestamp is also extracted. and spatial coordinates Constructing a spatiotemporal point cloud with semantic labels .

[0009] Preferably, the step of determining the candidate alignment point by calculating the temporal offset of the data points is as follows: From spatiotemporal point clouds Any selected semantic label has a confidence level higher than the second threshold. The data points are used as initial seed points; Determine the initial seed point The time consistency function, and each neighboring data point The timing consistency function is used to calculate the timing offset between two data points. Spatial Euclidean distance Based on the results, cross-modal alignment verification is performed, specifically as follows: like and This indicates the initial seed point. With neighboring data points To satisfy spatiotemporal consistency, Candidate alignment points are marked as semantically associated with the initial seed point; otherwise, it indicates that the alignment conditions between the two data points are not met.

[0010] Preferably, the cross-modal data alignment by calculating the confidence level of data points is specifically as follows: Using the candidate alignment points as second-generation seed points, we continue to calculate the temporal offset and spatial Euclidean distance between the second-generation seed points and unlabeled data points in their spatiotemporal neighborhoods, until all semantic labels in the entire spatiotemporal point cloud have a confidence level higher than [a certain value]. Up to the data points, the length of the alignment chain formed by the propagation of each initial seed point is counted, and confidence propagation verification is performed based on the alignment chain length.

[0011] Preferably, the calculation of the Wasserstein distance between the feature distributions of adjacent feature blocks is specifically as follows: Determine the feature distribution function of the initial fusion region and the feature distribution functions of adjacent feature blocks, and calculate the Wasserstein distance between two feature distributions. Then: ; in, and Let these represent the feature distribution functions of the two feature block regions, respectively. Indicates all and Joint distribution of marginal distributions The set, This represents the calculated Wasserstein distance.

[0012] Preferably, the feature fusion based on the calculation results is performed as follows: If the calculated Wasserstein distance is less than the preset conflict threshold If the initial fusion region has the same features as the adjacent feature blocks, then the fusion is performed directly by averaging the features. like If the two regions have a feature conflict, a conditional generative adversarial network is started. Using the features of the initial fused region as a condition, interpolated features that can be aligned with the feature distribution of the adjacent feature blocks are generated to replace the original features of the adjacent feature blocks.

[0013] Preferably, the verification of the fusion quality of the fusion result is performed as follows: The adjacent feature blocks after conflict resolution are used as the initial fusion regions of the second generation. The Wasserstein distance between the initial fusion region of the second generation and its next adjacent feature block is calculated and conflict resolution is performed again until all feature block regions have been traversed. The total number of conflict resolutions in the entire fused feature map is counted, and the fusion quality is verified based on the total number of times. Specifically: Set a threshold for the maximum number of conflict resolution attempts. If the total number of conflict resolutions counted in the entire fused feature map satisfies the formula when compared with the set maximum conflict resolution threshold... If the quality of the current fused feature map meets the requirements, then the fused feature map is output. Conversely, if If the cross-modal alignment data pair is of insufficient quality, then the adjacent feature block regions where the conflict resolution failed, along with the corresponding original data, will be re-aligned based on confidence propagation.

[0014] Preferably, the calculation-based fusion quality evaluation index is adjusted based on feedback, as follows: Identify the key regions in the fused feature map and their corresponding regions in the actual observation data, and calculate the mutual information between the two regions. The fusion effect is adaptively evaluated based on the results, specifically as follows: If the calculated mutual information Greater than the preset quality qualification threshold If all the indicators in the multi-dimensional fusion quality evaluation vector are higher than their corresponding lower threshold, it means that the current fusion processing effect meets the system requirements and will not trigger re-fusion. like If any dimension of the fusion quality assessment index is lower than its corresponding lower limit threshold, it means that the current fusion processing effect does not meet the requirements and the fusion fails.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention achieves adaptive mapping of multi-source heterogeneous data in a unified semantic space by adopting a heterogeneous data unified representation construction technology based on dynamic anchor semantic index. This effectively solves the problems of large differences in data structure and lack of unified representation in the existing technology, thereby improving the accuracy of cross-modal semantic understanding.

[0016] 2. This invention achieves collaborative alignment of multi-source data in the spatiotemporal and semantic dimensions by adopting cross-modal data alignment technology based on temporal consistency verification and confidence propagation. This effectively solves the problem of insufficient alignment accuracy caused by relying on simple matching rules in the prior art, thereby improving the reliability of cross-modal data association.

[0017] 3. This invention employs conflict resolution and feature interpolation techniques based on Wasserstein distance detection and conditional generative adversarial networks to achieve adaptive correction of conflict regions in multi-source data and high-quality compensation for missing regions. This effectively solves the problems of instability and severe information loss in the fusion process in existing technologies, thereby improving the integrity and consistency of the fusion results.

[0018] 4. This invention achieves dynamic evaluation and adaptive optimization of fusion results by adopting a closed-loop feedback evaluation technology based on multi-dimensional fusion quality indicators and mutual information constraints. This effectively solves the problem of uncontrollable fusion quality caused by the lack of feedback mechanism in the prior art, thereby improving the overall stability and long-term operating performance of the system.

[0019] 5. By constructing a collaborative processing mechanism for unified representation, alignment, fusion, and feedback optimization, this invention achieves collaborative optimization of each processing stage in the fusion of multi-source heterogeneous data, thereby ensuring fusion accuracy while taking into account processing efficiency and effectively improving the overall performance of the system. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall method steps of the present invention for the fusion processing of multi-source heterogeneous data. Detailed Implementation

[0021] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention. Example 1

[0022] Reference Figure 1As an embodiment of the present invention, a fusion processing method for multi-source heterogeneous data is provided, comprising the following steps: S1: Construction of a unified representation of heterogeneous data based on dynamic anchor semantic index.

[0023] Specifically, the construction of a unified representation of heterogeneous data based on dynamic anchor semantic indexing involves receiving multi-source heterogeneous data from satellite remote sensing, ground monitoring cameras, weather stations, and social media texts, constructing a dynamic anchor semantic index, and mapping data from different sources and with different structures to a unified semantic vector space. The specific implementation is as follows: Using the data acquisition interface, satellite multispectral images of the target urban area, ground monitoring video streams, meteorological numerical matrices, and social media text stream data are received. Based on the received multi-source heterogeneous data, a dynamic anchor point semantic index is constructed, specifically as follows: If we perform initial feature extraction on the heterogeneous data of each received modality, then we have: Spectral reflectance and texture features are extracted from satellite multispectral images; moving target and contour features are extracted from ground monitoring video streams; physical features such as temperature, humidity, and air pressure are extracted from meteorological numerical matrices; and semantic features are extracted from social media text streams using word embedding models. If a set of feature vectors is constructed based on the extracted features, then: ; in, Represents the modal feature index sequence number, and satisfies the formula , This represents the features extracted from satellite multispectral images. This represents the features extracted from ground surveillance video streams. This represents the features extracted from the meteorological numerical matrix. This represents the features extracted from social media text stream data. Indicates from the first The first mode extracted Feature vectors of feature points Indicates the first The total number of feature points extracted from each modality is set by the implementer based on the actual application scenario. Indicates from the first The set of feature vectors extracted from a modality.

[0024] The dynamic anchor semantic index is constructed based on the density peak clustering algorithm, and the specific implementation is as follows: From each feature set The local density of feature points is calculated, and the initial anchor points of the feature set are determined based on the calculated local density, specifically as follows: Calculate feature points The local density is then: ; in, Representing feature points and The Euclidean distance between them This indicates that the preset cutoff distance parameter is set by the implementer based on the feature space density. This is an indicator function that takes the value 1 when the independent variable is less than 0, and 0 otherwise. Representing feature points The local density is used to filter initial anchor points, specifically: Based on the modality sequence number, calculate the local density of feature points in each modality, and use the feature point corresponding to the maximum local density as the initial anchor point of the current modality.

[0025] After determining the semantic boundaries of the initial anchor points for each modality and calculating the cross-modal semantic similarity between anchor points of different modalities, we have: ; in, and They represent from the first Type and first mode The semantic vector of the initial anchor point of each modality. The cross-modal semantic similarity between two anchors is used to determine dynamic anchors, specifically: If cross-modal semantic similarity Greater than the preset first threshold Then it means the first Type and first mode If the initial anchors of a modality are semantically equivalent, and these two anchors are merged into a single dynamic semantic anchor, and the semantic vector of the merged dynamic anchor is updated to the mean vector, then: ; Conversely, if If the two anchor points are semantically inequivalent, they are retained as independent anchor points.

[0026] Merge and update all K dynamic semantic anchors Together with their associated original feature points, they form a unified semantic index graph, as follows: Each original feature point A semantic membership vector is assigned based on the distance to each dynamic semantic anchor point. Then we have: ; in, The smoothing parameter is set by the implementer based on the actual application scenario. Representing feature points With the A dynamic semantic anchor point The Euclidean distance between them Representing feature points Belonging to the Membership value of a dynamic semantic anchor point; Set an index reconstruction cycle (in seconds or minutes). Whenever a new batch of multi-source heterogeneous data arrives, re-execute the dynamic semantic anchor identification and merging process, and recalculate the semantic membership vector of all existing data based on the updated semantic index graph, so that the semantic representation space dynamically evolves with the inflow of data.

[0027] S2: Cross-modal data alignment based on temporal consistency verification and confidence propagation.

[0028] Specifically, cross-modal data alignment based on temporal consistency verification and confidence propagation is based on a deterministic unified semantic vector space representation. It performs timestamp alignment and spatial coordinate matching on data points from different modalities that are semantically related, and outputs the aligned multimodal related data pairs and their propagation confidence. The specific implementation is as follows: The semantic membership vector of each data point As its semantic tag, its original timestamp is also extracted. and spatial coordinates Constructing a spatiotemporal point cloud with semantic labels .

[0029] From spatiotemporal point clouds Any selected semantic label has a confidence level higher than the second threshold. Data points as initial seed points The semantic label confidence score is defined as follows: That is, the maximum value in the membership vector, and the radius of the seed point in the preset spatiotemporal neighborhood. All neighboring data points within, For time radius, Let be the spatial radius.

[0030] Determine the initial seed point The time consistency function, and each neighboring data point The timing consistency function is used to calculate the timing offset between two data points. Spatial Euclidean distance Based on the results, cross-modal alignment verification is performed, and then: ; ; in, and These represent the timestamps of the seed point and its neighboring data points, respectively. and These represent the spatial coordinates of the seed point and its neighboring data points, respectively. like and This indicates the initial seed point. With neighboring data points To satisfy spatiotemporal consistency, Candidate alignment points are marked as semantically associated with the initial seed point; otherwise, it indicates that the alignment conditions between the two data points are not met.

[0031] Using the candidate alignment points as second-generation seed points, we continue to calculate the temporal offset and spatial Euclidean distance between the second-generation seed points and unlabeled data points in their spatiotemporal neighborhoods, until all semantic labels in the entire spatiotemporal point cloud have a confidence level higher than [a certain value]. Up to the data points, the length of the alignment chain formed by propagation from each initial seed point is calculated. Confidence propagation verification is then performed based on the alignment chain length, resulting in: Set minimum alignment chain length threshold If the length of the alignment chain formed by the propagation of a certain initial seed point is... With the set minimum alignment chain length threshold The comparison satisfies the formula This indicates that all data points on the alignment chain have high propagation confidence, and outputting them as valid cross-modal alignment data pairs. ,in, This represents the confidence value of the alignment pair, which is the ratio of the alignment chain length to the total number of nodes.

[0032] S3: Feature-level fusion and interpolation based on adversarial generative conflict resolution.

[0033] Specifically, feature-level fusion and interpolation based on adversarial generative conflict resolution involves generatively resolving feature conflicts in overlapping data regions and performing feature-level interpolation in missing data regions, based on determined high-confidence cross-modal aligned data pairs. The output is a fused feature map with complete features and resolved conflicts. The specific implementation is as follows: All high-confidence cross-modal aligned data pairs Input to a shared feature extractor In the process, each generates its own high-dimensional feature map. , , , ; The high-dimensional feature map is divided into spatial grids. Feature block regions ,in, , ; Select any feature block from the divided regions. This serves as the initial fusion region, and simultaneously, it locates spatially adjacent feature blocks within this initial fusion region. ; Determine the characteristic distribution function of the initial fusion region and the feature distribution function of adjacent feature blocks. And calculate the Wasserstein distance between the two characteristic distributions. Based on the results, conflict detection and resolution are performed, resulting in: ; in, and Let these represent the feature distribution functions of the two feature block regions, respectively. Indicates all and Joint distribution of marginal distributions The set, The calculated Wasserstein distance measures the degree of difference between two feature distributions. If the calculated Wasserstein distance Less than the preset conflict threshold If the initial fusion region has the same features as its adjacent feature blocks, then the fusion is performed directly using feature averaging. Therefore: ; in, Indicates falling within the feature block The feature data set of all modalities within, Indicates the first The eigenvalues ​​of a mode within this feature block Represents the eigenvalues ​​after fusion; like This indicates a feature conflict between the two regions. In this case, a Conditional Generative Adversarial Network (CGN) is activated. The CGN consists of a generator with four convolutional layers and a discriminator with three convolutional layers. It uses Wasserstein distance as the adversarial loss function and generates interpolated features that align with the feature distribution of adjacent feature blocks, based on the features of the initial fused region. These interpolated features replace the original features of the adjacent feature blocks, thus resolving the conflict. Therefore: ; in, This indicates the characteristics of the initial fusion region. It is a random noise vector. This represents the generated aligned interpolation feature.

[0034] The adjacent feature blocks after conflict resolution are used as the initial fusion regions of the second generation. The Wasserstein distance between the initial fusion region of the second generation and its next adjacent feature block is calculated and conflict resolution is performed again until all feature block regions have been traversed. The total number of conflict resolutions in the entire fused feature map is counted, and the fusion quality is verified based on the total number of times. Specifically: Set a threshold for the maximum number of conflict resolution attempts. (Usually, 10% to 20% of the total number of feature blocks is taken). If the total number of conflict resolutions counted in the entire fused feature map satisfies the formula when compared with the set maximum conflict resolution threshold... If the quality of the current fused feature map meets the requirements, then the fused feature map is output. ; Conversely, if If the current fusion process is too frequent, it indicates that the cross-modal alignment data pairs output by S2 are of insufficient quality. The adjacent feature block regions where the conflict resolution failed and the corresponding original data are fed back to S2, requiring cross-modal alignment based on confidence propagation to be performed again.

[0035] S4: Adaptive evaluation of fusion effect and triggering of refusion based on closed-loop feedback.

[0036] Specifically, the adaptive evaluation of the fusion effect and the triggering of re-fusion based on closed-loop feedback are based on the fusion feature map output by S3, which is applied to downstream decision-making tasks (such as urban traffic flow prediction or environmental anomaly detection). At the same time, based on the deviation between the decision result and the actual observation data, the fusion quality evaluation index is calculated to determine whether a new round of fusion processing needs to be triggered, forming a closed-loop logic. The specific implementation is as follows: The fused feature map output by S3 is input into a pre-defined downstream task model. In (such as traffic flow predictors based on deep learning), the task decision results are obtained. ; At the same time, obtain the actual observation data within the time period corresponding to the decision result. (Such as actual traffic flow statistics), as a label of authenticity; Task decision results With authenticity tags By comparison, a multi-dimensional fusion quality evaluation vector is calculated. This vector includes at least three dimensions: decision accuracy, feature consistency, and temporal smoothness. Therefore: ; ; ; in, It is a very small positive number, used to prevent division by zero errors. This represents the set of key regions in the fused feature map. Indicates key areas The true feature value at a given point (which can be obtained from a high-precision verification sensor), where T represents the length of the time series. , , These represent the decision accuracy, feature consistency, and time smoothness metrics, respectively, and their values ​​range from [value range missing]. .

[0037] Identify key regions in the fused feature map and the corresponding area in the actual observation data. The mutual information between the two regions is calculated, and the fusion effect is adaptively evaluated based on the results, specifically: ; in, and Representing regions and Marginal probability distribution of eigenvalues Describes the joint probability distribution. The calculated mutual information is used to measure the degree of information sharing between the fused feature map and the actual observed data. Specifically: If the calculated mutual information is greater than the preset quality pass threshold Furthermore, all indicators in the multi-dimensional fusion quality assessment vector are higher than their corresponding lower thresholds, thus satisfying the formula. If the current fusion processing effect meets the system requirements, then re-fusion will not be triggered. like If any dimension of the fusion quality assessment index is lower than its corresponding lower limit threshold, it means that the current fusion processing effect does not meet the requirements and the fusion is judged to have failed.

[0038] The indication signal indicating fusion failure and the corresponding low-quality fusion feature map region will be determined. As feedback information, it is transmitted to S1, S2, and S3 respectively, specifically as follows: The data is transmitted to S1 to guide the incremental updates of the dynamic anchor semantic index, thereby enhancing the semantic representation of low-quality regions. Transmitted to S2 to restart the time-series consistency check and confidence propagation of data in this area; The data is transmitted to S3 to guide the conditional generative adversarial network in adjusting its generation strategy.

[0039] Set the fusion quality monitoring window length (That is, for three consecutive evaluation cycles), the system continuously performs the above evaluation in the form of a sliding window, and only when the evaluation results of three consecutive windows meet the requirements will the evaluation be completed. The final fusion result is only output when all metrics in each dimension are above the lower threshold. Otherwise, repeat steps S1 to S4 until the closed-loop convergence condition is met, then: ; in, , , , They represent the first The mutual information content of each window and the evaluation indicators of the fusion quality of each dimension.

[0040] Furthermore, if the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0041] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0042] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0043] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of protection claimed by the present invention.

Claims

1. A method for fusion processing of multi-source heterogeneous data, characterized in that: Includes the following steps: Receive multi-source heterogeneous data, extract features of each modality and construct a set of feature vectors, calculate the local density of each feature point based on the density peak clustering algorithm, and determine the initial anchor point of each modality; Calculate the cross-modal semantic similarity between initial anchor points of different modalities, determine the dynamic semantic anchor points, construct a unified semantic index graph of all dynamic semantic anchor points and their associated original feature points, and assign a semantic membership vector to each feature point; The semantic membership vector, timestamp, and spatial coordinates of each data point are used to construct a spatiotemporal point cloud with semantic labels. The initial seed point is determined based on the constructed spatiotemporal point cloud, and the candidate alignment point is determined by calculating the temporal offset of the data point. Cross-modal data alignment is performed by calculating the confidence of the data point. The aligned data pairs are input to the shared feature extractor to generate a high-dimensional feature map, and feature blocks are divided according to a spatial grid. The Wasserstein distance between the feature distributions of adjacent feature blocks is calculated. Feature fusion is performed based on the calculation results, and the fusion quality is verified. The fused feature map is applied to downstream tasks, and a fusion quality assessment index is calculated based on the deviation between the decision results and the actual observation data. Feedback adjustment is then performed based on the calculated fusion quality assessment index.

2. The fusion processing method for multi-source heterogeneous data as described in claim 1, characterized in that: The determination of the initial anchor points for each mode is as follows: Calculate the feature points of each modal feature set The local density is then: ； in, Representing feature points and The Euclidean distance between them This indicates the preset cutoff distance parameter. For indicator functions, Representing feature points The local density is used to filter initial anchor points, specifically: Based on the modality sequence number, calculate the local density of feature points in each modality, and use the feature point corresponding to the maximum local density as the initial anchor point of the current modality.

3. The fusion processing method for multi-source heterogeneous data as described in claim 2, characterized in that: The determination of dynamic semantic anchor points is as follows: Determine the semantic boundaries of the initial anchor points for each modality, and calculate the cross-modal semantic similarity between anchor points of different modalities. ; Dynamic semantic anchors are determined based on the calculated cross-modal semantic similarity, specifically as follows: If cross-modal semantic similarity Greater than the preset first threshold Then it means the first Type and first mode The initial anchor points of each modality are semantically equivalent. They are merged into a dynamic semantic anchor point, and the semantic vector of the merged dynamic anchor point is updated to the mean vector. Conversely, if If the two anchor points are semantically inequivalent, they are retained as independent anchor points.

4. The fusion processing method for multi-source heterogeneous data as described in claim 3, characterized in that: The spatiotemporal point cloud constructed with semantic labels is as follows: For each original feature point A semantic membership vector is assigned based on the distance to each dynamic semantic anchor point. Then we have: ； in, Indicates the smoothing parameter. Representing feature points With the A dynamic semantic anchor point The Euclidean distance between them Representing feature points Belonging to the Membership value of a dynamic semantic anchor point; The semantic membership vector of each data point As its semantic tag, its original timestamp is also extracted. and spatial coordinates Constructing a spatiotemporal point cloud with semantic labels .

5. The fusion processing method for multi-source heterogeneous data as described in claim 4, characterized in that: The process of determining candidate alignment points by calculating the temporal offset of data points is as follows: From spatiotemporal point clouds Any selected semantic label has a confidence level higher than the second threshold. The data points are used as initial seed points; Determine the initial seed point The time consistency function, and each neighboring data point The timing consistency function is used to calculate the timing offset between two data points. Spatial Euclidean distance Based on the results, cross-modal alignment verification is performed, specifically as follows: like and This indicates the initial seed point. With neighboring data points To satisfy spatiotemporal consistency, Candidate alignment points are marked as semantically associated with the initial seed point; Conversely, it indicates that the alignment conditions between the two data points are not met.

6. The fusion processing method for multi-source heterogeneous data as described in claim 5, characterized in that: The cross-modal data alignment by calculating the confidence level of data points is as follows: Using the candidate alignment points as second-generation seed points, we continue to calculate the temporal offset and spatial Euclidean distance between the second-generation seed points and unlabeled data points in their spatiotemporal neighborhoods, until all semantic labels in the entire spatiotemporal point cloud have a confidence level higher than [a certain value]. Up to the data points, the length of the alignment chain formed by the propagation of each initial seed point is counted, and confidence propagation verification is performed based on the alignment chain length.

7. The fusion processing method for multi-source heterogeneous data as described in claim 6, characterized in that: The calculation of the Wasserstein distance between the feature distributions of adjacent feature blocks is as follows: Determine the feature distribution function of the initial fusion region and the feature distribution functions of adjacent feature blocks, and calculate the Wasserstein distance between two feature distributions. Then: ； in, and Let these represent the feature distribution functions of the two feature block regions, respectively. Indicates all and Joint distribution of marginal distributions The set, This represents the calculated Wasserstein distance.

8. The fusion processing method for multi-source heterogeneous data as described in claim 7, characterized in that: The feature fusion based on the calculation results is performed as follows: If the calculated Wasserstein distance is less than the preset conflict threshold If the initial fusion region has the same features as the adjacent feature blocks, then the fusion is performed directly by averaging the features. like If the two regions have a feature conflict, a conditional generative adversarial network is started. Using the features of the initial fused region as a condition, interpolated features that can be aligned with the feature distribution of the adjacent feature blocks are generated to replace the original features of the adjacent feature blocks.

9. The fusion processing method for multi-source heterogeneous data as described in claim 8, characterized in that: The fusion quality verification of the fusion results is performed as follows: The adjacent feature blocks after conflict resolution are used as the initial fusion regions of the second generation. The Wasserstein distance between the initial fusion region of the second generation and its next adjacent feature block is calculated and conflict resolution is performed again until all feature block regions have been traversed. The total number of conflict resolutions in the entire fused feature map is counted, and the fusion quality is verified based on the total number of times. Specifically: Set a threshold for the maximum number of conflict resolution attempts. If the total number of conflict resolutions counted in the entire fused feature map satisfies the formula when compared with the set maximum conflict resolution threshold... If the quality of the current fused feature map meets the requirements, then the fused feature map is output. Conversely, if If the cross-modal alignment data pair is of insufficient quality, then the adjacent feature block regions where the conflict resolution failed, along with the corresponding original data, will be re-aligned based on confidence propagation.

10. The fusion processing method for multi-source heterogeneous data as described in claim 9, characterized in that: The computation-based fusion quality assessment index is adjusted based on feedback, as follows: Identify the key regions in the fused feature map and their corresponding regions in the actual observation data, and calculate the mutual information between the two regions. The fusion effect is adaptively evaluated based on the results, specifically as follows: If the calculated mutual information Greater than the preset quality qualification threshold If all the indicators in the multi-dimensional fusion quality evaluation vector are higher than their corresponding lower threshold, it means that the current fusion processing effect meets the system requirements and will not trigger re-fusion. like If any dimension of the fusion quality assessment index is lower than its corresponding lower limit threshold, it means that the current fusion processing effect does not meet the requirements and the fusion fails.

Images