Distributed multi-sensor data fusion unsupervised learning method and system based on multi-dimensional mask

Abstract

The application discloses a kind of distributed multi-sensor data fusion unsupervised learning method and system based on multi-dimension mask, belong to multi-sensor data fusion technical field, including: the feature block of the observation time series data of multi-sensor, block initial embedding, two-dimensional space-time position coding;After the initial feature sequence obtained is parallelly carried out the block random mask of time dimension and space dimension, respectively extract time dimension feature and space dimension feature not visible block by mask and decode to obtain reconstructed time series data, construct the mask data reconstruction loss of mask block and introduce the contrast learning loss based on space position to optimize the two autoencoders of feature extraction;After the adaptive fusion based on confidence of time dimension feature and space dimension feature and the traditional feature of data are used for downstream task prediction.This can effectively learn the space-time heterogeneity of multi-sensor data and other features, complete distributed multi-sensor data fusion task.

Description

Technical Field

[0001] This invention belongs to the field of distributed multi-sensor information fusion technology, specifically relating to a distributed multi-sensor data fusion unsupervised learning method and system based on multi-dimensional masks. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and intelligent sensing technologies, distributed multi-sensor systems are increasingly widely used in various scenarios, including civilian applications. Distributed multi-sensor data naturally possesses spatiotemporal attributes; its temporal dimension reflects dynamic trends, while its spatial dimension reflects the correlation between sensors. Multi-sensor information fusion technology processes data from multiple sensors at multiple levels and dimensions to generate a consistent interpretation and decision support for the observed environment. Its core is to improve the accuracy, robustness, and real-time performance of the system through the complementarity and redundancy of spatiotemporal information. However, existing technologies often fail to fully separate spatiotemporal features when processing distributed multi-sensor data. This leads to insufficient learning of spatiotemporal heterogeneity (the difference between temporal and spatial features) on the one hand, and the coupling of spatiotemporal features significantly increases model complexity on the other, ultimately limiting its effectiveness and performance in complex downstream tasks.

[0003] Patent application CN118364432A proposes a sensor data fusion method based on graph neural networks. This method constructs spatial and temporal relationships between nodes in a target graph, then uses the graph neural network to extract and update the temporal and spatial features of the nodes to obtain a global feature vector. Finally, a decoding operation transforms the global feature vector into environmental perception information. This method involves mixed processing of spatiotemporal features, which blurs the spatiotemporal heterogeneity, and the cross-fusion of spatiotemporal features increases the complexity of the model.

[0004] Patent application CN117540333A proposes a multi-sensor data anomaly detection method based on spatiotemporal information fusion. This method achieves anomaly detection in multi-observation time-series data through spatiotemporal feature extraction and cross-attention fusion. Specifically, it models time-series dependencies using LSTM, extracts spatial correlations of sensor nodes using graph convolutional networks, and finally fuses spatiotemporal features through attention weights. However, this method fuses spatiotemporal features before executing downstream tasks, resulting in underutilization of spatiotemporal heterogeneity and an inability to dynamically adjust the weights of spatiotemporal features for different tasks. Summary of the Invention

[0005] In view of the above, the purpose of this invention is to provide an unsupervised learning method and system for distributed multi-sensor data fusion based on multi-dimensional masks. By independently decoupling the temporal and spatial features in the observed time-series data, and introducing block processing for long-term series data and reconstruction loss only for masked regions, multi-dimensional masking and contrastive learning for the heterogeneous distribution characteristics of distributed data, and a confidence-guided spatiotemporal feature weighted fusion mechanism, the invention achieves a significant reduction in model complexity, effective learning of spatiotemporal heterogeneity, and adaptation to various downstream tasks.

[0006] To achieve the above-mentioned objectives, an embodiment provides a distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks, comprising the following steps: After acquiring the observation time series data collected by distributed multi-sensor, feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding are performed to obtain the initial feature sequence of superimposed spatiotemporal location. The observation time series data includes detected environmental features and / or object features, as well as the corresponding timestamps. After performing random masking on multiple blocks in the temporal and spatial dimensions of the initial feature sequence in parallel, the temporal and spatial features of the unmasked visible blocks are extracted using a temporal autoencoder and a spatial autoencoder, respectively, and then decoded to obtain reconstructed time series data. Based on the reconstructed observation time series data, a masked block masking data reconstruction loss is constructed, and a contrastive learning loss based on spatial location is introduced to optimize the two autoencoders. After extracting temporal and spatial features using optimized dual autoencoders, the fused features are adaptively fused with traditional features from observed time-series data based on confidence level and then used for downstream task prediction.

[0007] Preferably, the observation time-series data is subjected to feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding, including: The observation time series data corresponding to each sensor is divided into non-overlapping feature blocks according to a preset time window size. A fully connected layer is used to map each feature block into an initial embedding representation. Two-dimensional spatiotemporal location encoding is superimposed on the initial embedding representation to obtain an initial feature sequence with superimposed spatiotemporal location.

[0008] Preferably, the initial feature sequence is subjected to random masking of multiple blocks along the time dimension, including: For each sensor in the initial feature sequence, multiple feature blocks are randomly selected along the time dimension for masking.

[0009] Preferably, the initial feature sequence is randomly masked into multiple blocks along the spatial dimension, including: For the initial feature sequence, along the spatial dimension, all feature blocks corresponding to multiple sensors are randomly selected and masked.

[0010] Preferably, the mask data reconstruction loss for masked blocks is constructed based on the reconstructed observation time-series data, including: in, This represents the raw observation time series data corresponding to the i-th sensor. and These are reconstructed time-series data based on temporal autoencoders and spatial autoencoders, respectively. and These are mask indicator matrices, where the masked areas have a value of 1, and the unmasked visible areas have a value of 0. The multiplication is element-wise, where N is the total number of sensors. and These represent the mask data reconstruction losses for the temporal autoencoder and the spatial autoencoder, respectively.

[0011] Preferably, a spatial location-based contrastive learning loss is introduced, including: Constructing positive and negative samples based on physical correlation: Calculating the Pearson correlation coefficient between the observation time series data of any target sensor i and the observation time series data of other sensors j in the distributed network. Set the positive sample threshold to and negative sample threshold is For target sensors extracted from spatial dimensional features Spatial feature vectors ,like If two sensors have the same data acquisition attributes, then the sensors will be... Spatial features As a sensor Positive samples, denoted as ;like Then the sensor Spatial features As a negative sample, it is denoted as , For the index of negative samples, a total of [number] samples were selected. One negative sample; Construction of contrastive learning loss: Based on the InfoNCE framework, for all Each sensor calculates a contrastive learning loss based on positive and negative sample pairs to determine the spatial dimension. .

[0012] Preferably, the optimized two autoencoders extract time-dimensional and spatial-dimensional features, which are then fused with traditional features from the observed time-series data using a confidence-based adaptive fusion process to obtain fused features, including: in, Traditional features are obtained by extracting features from observed time-series data. and These are time-dimensional features and spatial-dimensional features, respectively. This represents element-wise multiplication. Representative feature fusion and splicing operation, and The time confidence and spatial confidence scores corresponding to the time dimension features and spatial dimension features, respectively, are determined in the following way: Time dimension features and spatial dimensional features The input is fed into a confidence evaluation network, which outputs a normalized time confidence matrix. and spatial confidence matrix The confidence evaluation network includes a multilayer perceptron, whose parameters are determined by backpropagation optimization of a supervised loss function constructed based on downstream task prediction using fused features.

[0013] To achieve the above-mentioned objectives, embodiments of the present invention provide a distributed multi-sensor data fusion unsupervised learning system based on multi-dimensional masks, comprising: The data processing module is used to acquire observation time-series data collected by distributed multi-sensor collection, and then perform feature segmentation, initial embedding of the segments, and two-dimensional spatiotemporal location encoding to obtain an initial feature sequence of superimposed spatiotemporal locations. The observation time-series data includes detected environmental features and / or object features, as well as corresponding timestamps. The mask learning module is used to perform random masking on multiple blocks in the time and space dimensions of the initial feature sequence in parallel. Then, the temporal and spatial autoencoders are used to extract the temporal and spatial features of the unmasked visible blocks and decode them to obtain the reconstructed time series data. Based on the reconstructed observation time series data, the mask data reconstruction loss of the mask blocks is constructed and the contrastive learning loss based on spatial location is introduced to optimize the two autoencoders. The feature fusion module is used to extract time-dimensional and spatial-dimensional features using two optimized autoencoders and then perform confidence-based adaptive fusion with traditional features of observed time-series data to obtain fused features for downstream task prediction.

[0014] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks.

[0015] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-mentioned unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks.

[0016] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. The method proposed in this invention extracts and decouples spatiotemporal features separately through time autoencoders and spatial autoencoders. Compared with spatiotemporal feature hybrid extraction methods, it can learn the clear and complete spatiotemporal heterogeneity in sensor data, supporting the improvement of accuracy and precision of downstream tasks.

[0017] 2. The method proposed in this invention embeds time and space information into the input of downstream tasks without changing the network structure of downstream tasks, seamlessly connecting with various downstream tasks and exhibiting good generalization and universality.

[0018] 3. The method proposed in this invention pre-trains on temporal and spatial features, which reduces the complexity of feature extraction for downstream tasks, improves training efficiency, and saves resource costs.

[0019] 4. The method proposed in this invention combines temporal and spatial information with downstream input through a fusion method based on confidence level as weight. It supports dynamic and independent adjustment of the weights of temporal and spatial features according to downstream tasks, which has better flexibility compared to the original method of combining temporal and spatial features with downstream input.

[0020] 5. The method proposed in this invention introduces a feature patching mechanism and two-dimensional spatiotemporal location encoding, and specifically designs a reconstruction loss function for masked regions. Compared with traditional models based on full-sequence computation (such as LSTM or global Transformer), this method greatly reduces computational complexity, can efficiently process ultra-long sensor time series, and forces the model to infer across long-term context gaps, effectively capturing long-term dependencies in the data. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks provided in the embodiment; Figure 2This is a flowchart of the distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks provided in the embodiment; Figure 3 This is a schematic diagram of the structure of the self-encoder provided in the embodiment; Figure 4 This is a flowchart of the time-dimensional and spatial-dimensional feature extraction based on mask learning provided in the embodiment; Figure 5 This is a flowchart of the spatiotemporal feature fusion and downstream task process provided in the embodiment; Figure 6 This is a schematic diagram of the structure of the distributed multi-sensor data fusion unsupervised learning system based on multi-dimensional masks provided in the embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0024] like Figure 1 and Figure 2 As shown in the figure, the embodiment provides a distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks, including the following steps: S1. After acquiring the observation time series data collected by distributed multi-sensor, feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding are performed to obtain the initial feature sequence of superimposed spatiotemporal location. The observation time series data includes detected environmental features and / or object features, as well as the corresponding timestamps.

[0025] In this embodiment, the distributed network contains N sensors, each observing either a static or dynamic target. The time-series data collected by the i-th sensor at m time points is used as... , Timestamps are generally used to represent different moments in observed time series data. This represents the sensor's observation data at time m, which can include detected environmental features and / or object features. The specific dimensions and content vary depending on the sensor type and task type. For example, in a radar target point fusion task... The observation vector, which includes features such as target coordinates, azimuth, and velocity, is ultimately represented by the raw time-series observation data from multiple sensors. .

[0026] Existing point-level masks struggle to handle long-term time-series data; therefore, this invention employs a patch-based masking mechanism. First, the long-term observation data corresponding to a single sensor is divided into non-overlapping feature blocks according to a preset window size L, reducing the sequence length from m to p (…). Subsequently, each feature block is mapped to an initial embedding representation Ep through a fully connected layer. Finally, a two-dimensional spatiotemporal location code Epos is superimposed on the initial embedding representation Ep to simultaneously record the temporal absolute position of the feature block and its spatial position with the sensor. The calculation formula for the two-dimensional spatiotemporal location code is as follows: Where t is the time block index, i is the sensor node index, D is the feature dimension, and a and b are the feature dimension indices for time location encoding and spatial location encoding, respectively.

[0027] S2: After randomly masking multiple blocks in the time and space dimensions of the initial feature sequence in parallel, the temporal and spatial autoencoders are used to extract the temporal and spatial features of the unmasked visible blocks and decode them to obtain the reconstructed time series data. Based on the reconstructed observation time series data, the masked block mask data reconstruction loss is constructed and the spatial location-based contrastive learning loss is introduced to optimize the two autoencoders.

[0028] In the embodiments, such as Figure 4 As shown, temporal autoencoders and spatial autoencoders are used to extract temporal and spatial features for mask-based prediction tasks, respectively.

[0029] To address the issues of excessive computational complexity and difficulty in capturing long-term dependencies caused by ultra-long sensor time series, the Time Auto Encoder (TAE) and the Spatial Auto Encoder (SAE) employ a patch-based mask reconstruction mechanism.

[0030] For temporal feature extraction, a high-proportion temporal block-level mask is first applied to the initial feature sequence after overlay position encoding. This involves randomly selecting multiple feature blocks corresponding to each sensor in the initial feature sequence along the temporal dimension for overall masking. The masked time-series observation data of a single sensor is then represented as follows: The data matrix after masking multiple sensors is represented as The details are as follows: It should be noted that in the formula In, column vector Instead of representing a single sampling point, it represents the p-th feature block, with each feature block containing L consecutive time-step data; The black masking block in the image indicates that the entire feature block is completely masked.

[0031] The initial feature sequence after masking The input is fed into a time autoencoder (TAE) containing a time encoder (TE) and a time decoder (TD). An example of the structure of a time autoencoder (TAE) is given as follows: Figure 3 As shown, the temporal encoder (TE) employs a convolutional network, while the temporal decoder (TD) employs a deconvolutional network. The temporal encoder (TE) extracts features only from the visible, unmasked blocks, outputting temporal dimension features. Then, the time dimension features are processed using the time decoder (TD). Decoding is performed to restore the masked portion of the observed time series data, thus obtaining the reconstructed time series data. .

[0032] For spatial dimension feature extraction, a high-proportion spatial dimension block-level mask is first applied to the initial feature sequence after overlay position encoding. This involves randomly selecting feature blocks corresponding to multiple sensors at different spatial locations along the spatial dimension to mask the initial feature sequence. The spatially masked time-series observation data of a single sensor is then represented as follows: The observation time series data after spatial masking of multiple sensors is represented as follows: : Mode In the middle, the observation time series data of the first sensor It is masked and represented by a black overlay block.

[0033] The initial feature sequence after masking The input is fed into a temporal autoencoder (SAE) containing a spatial encoder (SE) and a spatial decoder (SD). An example of the structure of a spatial autoencoder (SAE) is given as follows: Figure 3 As shown, the spatial encoder (SE) employs a convolutional network, while the spatial decoder (SD) uses a deconvolutional network. The spatial encoder (SE) extracts features only from the visible, unmasked blocks, outputting temporal features. Then, the spatial decoder SD is used to analyze the temporal dimension features. Decoding is performed to restore the masked portion of the observed time series data, thus obtaining the reconstructed time series data. .

[0034] In the embodiment, by minimizing the input raw observation time series data With the output reconstructed time series data and The difference between the two is minimized to perform unsupervised learning. Specifically, to adapt to long time series and significantly reduce computational complexity, the reconstruction loss is calculated only for the masked feature blocks, excluding the reconstruction error in the unmasked regions. That is, the masked data reconstruction loss is used in the following form, thus ensuring that the reconstruction error is calculated only in the masked regions: in, This represents the raw observation time series data corresponding to the i-th sensor. and These are reconstructed time-series data based on temporal autoencoders and spatial autoencoders, respectively. and These are mask indicator matrices, where the masked areas have a value of 1, and the unmasked visible areas have a value of 0. The multiplication is element-wise, where N is the total number of sensors. and These represent the mask data reconstruction losses for the temporal autoencoder and the spatial autoencoder, respectively.

[0035] Traditional mask autoencoders rely solely on reconstruction loss, which easily falls into the trap of "overfitting local heterogeneous noise." This invention introduces a spatial location-based contrastive learning loss, jointly optimizing the mask data reconstruction loss and the contrastive learning loss.

[0036] To address the inherent heterogeneity of distributed multi-sensor networks in terms of geographical location and observed targets, this invention focuses on spatial dimension features. After extraction, an unsupervised node-level contrastive learning constraint flow was added to filter out local heterogeneous noise, specifically including the following sub-steps: Construction of positive and negative samples based on physical correlation: Each sensor in the distributed network is treated as a node. The observation time series data of each node is acquired, and then the Pearson correlation coefficient between the observation time series data of any target sensor i and the observation time series data of other sensors j in the distributed network is calculated. Set the positive sample threshold to and negative sample threshold is For target sensors extracted from spatial dimensional features Spatial feature vectors ,like If two sensors have the same data acquisition attributes, then the sensors will be... Spatial features As a sensor Positive samples, denoted as ;like Then the sensor Spatial features As a negative sample, it is denoted as , For the index of negative samples, a total of [number] samples were selected. One negative sample; Contrastive learning loss construction: After constructing positive and negative sample pairs, the cosine similarity between features is calculated in the latent space. Based on the InfoNCE framework, for all For each sensor node, calculate the contrastive learning loss in the spatial dimension. : in, This is a temperature hyperparameter used to control the model's sensitivity to difficult negative samples. It is achieved by minimizing... The spatial autoencoder is forced to bring closer the feature distances of homogeneous or observation period nodes, while pushing away heterogeneous environmental noise, thereby extracting robust global core physical representations.

[0037] The final overall training loss function Defined as time reconstruction loss Spatial reconstruction loss Comparative learning loss Weighted sum: in, To compare the balance coefficients of the losses. By minimizing The backpropagation process updates the parameters of the temporal autoencoder (TAE) and spatial autoencoder (SAE), ultimately enabling the TAE and SAE to obtain high-quality spatiotemporal feature representations of sensor data. The temporal decoder (TD) and spatial decoder (ST) can accurately reconstruct sensor data (reconstruction constraints) and spontaneously perform clustering and decoupling of heterogeneous multi-sensor distributions in the latent space (contrast constraints). Ultimately, the TAE and SAE can extract long-term, high-quality spatiotemporal feature representations.

[0038] S3 utilizes the optimized two autoencoders to extract time-dimensional and spatial-dimensional features, and then performs confidence-based adaptive fusion with the traditional features of the observed time-series data to obtain fused features, which are then used for downstream task prediction.

[0039] In the embodiments, such as Figure 5As shown, optimized two autoencoders extract temporal and spatial features and fuse them with traditional features to form fused features embedded with spatiotemporal information for downstream task prediction. The downstream task can be a target classification task, prediction task, target recognition task, target tracking task, etc. Specifically, the downstream task network includes a traditional feature extraction layer (FE), a confidence-based feature fusion layer (FF), and a fully connected layer (FC). The traditional feature extraction layer (FE) extracts traditional features from the sensor's observation time-series data; the feature fusion layer (FF) weights and fuses the temporal features extracted by the temporal autoencoder (TAE), the spatial features extracted by the spatial autoencoder (SAE), and the traditional features extracted by the traditional feature extraction layer (FE) based on confidence, forming a fused feature; the fully connected layer (FC) completes the sensor data fusion based on the fused feature and outputs the result. The specific fusion process is as follows: First, traditional feature extraction is performed: the traditional feature extraction layer (FE) extracts features from the acquired sensor observation time-series data to obtain traditional features. .

[0040] Then, confidence-guided feature fusion is performed. Feature fusion is not a simple concatenation or addition of features, but rather a confidence-guided weighted fusion mechanism driven by downstream task supervision. Before entering the fully connected layer of the downstream task, the extracted temporal dimension features are... and spatial dimensional features The input is fed into a confidence evaluation network composed of a multilayer perceptron (MLP). This network outputs a normalized time confidence matrix. and spatial confidence matrix The final fusion features Represented as: in, This represents element-wise multiplication. Representative feature fusion and splicing operation.

[0041] Finally, the fusion features The input is fed into the fully connected layer (FC) for data mapping and fusion, and the fused data is output.

[0042] In this embodiment, the parameters of the confidence evaluation network are optimized based on task-driven supervised iterative training. Based on the fusion features... The system processes time-series observation data from multiple sensors to output results for downstream tasks. In terms of training mechanism: the front-end TAE and SAE are only constrained by unsupervised reconstruction loss and contrastive loss; their parameters are frozen or only fine-tuned at this stage. Meanwhile, the supervised losses for the current downstream tasks (e.g., cross-entropy loss between fused data and real data, or mean squared error prediction loss) are backpropagated. The gradient from this backpropagation will dominate the updating of the parameters of the confidence evaluation network.

[0043] Existing spatiotemporal fusion schemes (such as cross-coupling spatiotemporal features) destroy the spatiotemporal heterogeneity learned independently, and simple addition cannot adapt to complex and ever-changing downstream tasks. In the feature fusion scheme of this invention, the network no longer blindly fuses spatiotemporal features, but allows the backend task to "teach" the model: under the current specific task, which sensor nodes should be assigned higher spatial confidence, and which segment of the historical observation time series should be given higher temporal weight, thereby achieving perfect dynamic adaptation between spatiotemporally decoupled features and backend tasks.

[0044] In this embodiment, the method proposed in this invention is described in detail using the distributed multi-sensor target point fusion task in the field of radar detection as an example. Specifically, it includes: First, the observation time-series data acquired by multiple sensors is target point trace data, which mainly includes four dimensions: time, coordinates, orientation, and velocity. Time is represented by... Representation, coordinates Indication of direction Indicates speed If we express this as follows, then the time series data of a single sensor at m time points can be represented as: .

[0045] The collected target point data is subjected to feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding according to step S1 to obtain the initial feature sequence of superimposed spatiotemporal location; Then, following step S2, the spatial dimension features of the target point trace are extracted from the initial feature sequence. and target point trace time dimension features And decode and restore the target point data and And construct a masked data loss function. and Constructing a comparative learning loss To optimize the autoencoder parameters.

[0046] Next, follow step S3 to analyze the spatial dimension features of the target point. and target point trace time dimension features Compared with traditional features extracted from target point trace data Confidence-based dynamic weighted fusion is performed to obtain the target point trace fusion features. According to fusion characteristics Complete the prediction and fusion output of target point traces. .

[0047] Specifically, the parameters of the confidence evaluation network for predicting confidence are optimized through supervised fine-tuning in downstream tasks. The loss function used in the supervised fine-tuning of downstream tasks is... for With real data The difference between them, according to the loss function This optimizes the confidence assessment network parameters, ultimately enabling the downstream task network to allocate optimal fusion weights to different radar nodes and time periods, thus accurately completing the data fusion of multiple radar sensors.

[0048] like Figure 6 As shown in the embodiment, an unsupervised learning system 60 based on multi-dimensional masks for distributed multi-sensor data fusion is also provided. This system includes a data processing module 61, a mask learning module 62, and a feature fusion module 63. The data processing module 61 acquires observation time-series data collected by distributed multi-sensors, performs feature segmentation, initial embedding of the segments, and two-dimensional spatiotemporal location encoding to obtain an initial feature sequence superimposed with spatiotemporal locations. The observation time-series data includes detected environmental features and / or object features, as well as corresponding timestamps. The mask learning module 62 performs parallel temporal fusion on the initial feature sequence. After random masking of multiple blocks in the dimensional and spatial dimensions, temporal and spatial autoencoders are used to extract the temporal and spatial features of the unmasked visible blocks, respectively, and decode them to obtain reconstructed time-series data. Based on the reconstructed observation time-series data, a masked block reconstruction loss is constructed, and a spatial location-based contrastive learning loss is introduced to optimize the two autoencoders. The feature fusion module 63 is used to extract temporal and spatial features from the optimized two autoencoders and then perform confidence-based adaptive fusion with the traditional features of the observation time-series data to obtain fused features, which are then used for downstream task prediction.

[0049] It should be noted that the distributed multi-sensor data fusion unsupervised learning system based on multi-dimensional masks provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing unsupervised learning of multi-sensor data fusion. The functions described above can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the distributed multi-sensor data fusion unsupervised learning system based on multi-dimensional masks provided in the above embodiments and the distributed multi-sensor data fusion unsupervised learning method embodiments based on multi-dimensional masks belong to the same concept. For details of its specific implementation process, please refer to the distributed multi-sensor data fusion unsupervised learning method embodiments based on multi-dimensional masks, which will not be repeated here.

[0050] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-described unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks, specifically including the following steps: S1. After acquiring the observation time series data collected by distributed multi-sensor, feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding are performed to obtain the initial feature sequence of superimposed spatiotemporal location. The observation time series data includes detected environmental features and / or object features, as well as the corresponding timestamps. S2, after performing random masking on multiple blocks in the time and space dimensions of the initial feature sequence in parallel, the time and space dimensions of the unmasked visible blocks are extracted by the time autoencoder and the space autoencoder respectively and decoded to obtain the reconstructed time series data. Based on the reconstructed observation time series data, the mask data reconstruction loss of the masked blocks is constructed and the contrastive learning loss based on spatial location is introduced to optimize the two autoencoders. S3 utilizes the optimized two autoencoders to extract time-dimensional and spatial-dimensional features, and then performs confidence-based adaptive fusion with the traditional features of the observed time-series data to obtain fused features, which are then used for downstream task prediction.

[0051] The computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks described in S1-S3 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0052] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks, specifically including the following steps: S1. After acquiring the observation time series data collected by distributed multi-sensor, feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding are performed to obtain the initial feature sequence of superimposed spatiotemporal location. The observation time series data includes detected environmental features and / or object features, as well as the corresponding timestamps. S2, after performing random masking on multiple blocks in the time and space dimensions of the initial feature sequence in parallel, the time and space dimensions of the unmasked visible blocks are extracted by the time autoencoder and the space autoencoder respectively and decoded to obtain the reconstructed time series data. Based on the reconstructed observation time series data, the mask data reconstruction loss of the masked blocks is constructed and the contrastive learning loss based on spatial location is introduced to optimize the two autoencoders. S3 utilizes the optimized two autoencoders to extract time-dimensional and spatial-dimensional features, and then performs confidence-based adaptive fusion with the traditional features of the observed time-series data to obtain fused features, which are then used for downstream task prediction.

[0053] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0054] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks, characterized in that, Includes the following steps: After acquiring the observation time series data collected by distributed multi-sensor, feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding are performed to obtain the initial feature sequence of superimposed spatiotemporal location. The observation time series data includes detected environmental features and / or object features, as well as the corresponding timestamps. After performing random masking on multiple blocks in the temporal and spatial dimensions of the initial feature sequence in parallel, the temporal and spatial features of the unmasked visible blocks are extracted using a temporal autoencoder and a spatial autoencoder, respectively, and then decoded to obtain reconstructed time series data. Based on the reconstructed observation time series data, a masked block masking data reconstruction loss is constructed, and a contrastive learning loss based on spatial location is introduced to optimize the two autoencoders. After extracting temporal and spatial features using two optimized autoencoders, the fused features are obtained by adaptively fusing them with traditional features from the observed time-series data based on confidence. These fused features are then used for prediction in downstream tasks. The mask data reconstruction loss, which constructs mask blocks based on reconstructed observation time-series data, includes: in, This represents the raw observation time series data corresponding to the i-th sensor. and These are reconstructed time-series data based on temporal autoencoders and spatial autoencoders, respectively. and These are mask indicator matrices, where the masked areas have a value of 1, and the unmasked visible areas have a value of 0. The multiplication is element-wise, where N is the total number of sensors. and These represent the mask data reconstruction losses for the temporal autoencoder and the spatial autoencoder, respectively. Introduce a spatial location-based contrastive learning loss, including: Constructing positive and negative samples based on physical correlation: Calculating the Pearson correlation coefficient between the observation time series data of any target sensor i and the observation time series data of other sensors j in the distributed network. Set the positive sample threshold to and negative sample threshold is For target sensors extracted from spatial dimensional features Spatial feature vectors ,like If two sensors have the same data acquisition attributes, then the sensors will be... Spatial features As a sensor Positive samples, denoted as ;like Then the sensor Spatial features As a negative sample, it is denoted as , For the index of negative samples, a total of [number] samples were selected. One negative sample; Construction of contrastive learning loss: Based on the InfoNCE framework, for all Each sensor calculates a contrastive learning loss based on positive and negative sample pairs to determine the spatial dimension. .

2. The unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks according to claim 1, characterized in that, The observed time-series data undergoes feature segmentation, initial embedding of segments, and two-dimensional spatiotemporal location encoding, including: The observation time series data corresponding to each sensor is divided into non-overlapping feature blocks according to a preset time window size. A fully connected layer is used to map each feature block into an initial embedding representation. Two-dimensional spatiotemporal location encoding is superimposed on the initial embedding representation to obtain an initial feature sequence with superimposed spatiotemporal location.

3. The unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks according to claim 1, characterized in that, Random masks are applied to multiple time-divided blocks of the initial feature sequence, including: For each sensor in the initial feature sequence, multiple feature blocks are randomly selected along the time dimension for masking.

4. The unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks according to claim 1, characterized in that, Random masks are applied to the initial feature sequence, dividing it into multiple spatial blocks, including: For the initial feature sequence, along the spatial dimension, all feature blocks corresponding to multiple sensors are randomly selected and masked.

5. The unsupervised learning method for distributed multi-sensor data fusion based on multi-dimensional masks according to claim 1, characterized in that, After extracting temporal and spatial features using two optimized autoencoders, the fused features are obtained by adaptively fusing them with traditional features from the observed time-series data based on confidence. These features include: in, Traditional features are obtained by extracting features from observed time-series data. and These are time-dimensional features and spatial-dimensional features, respectively. This represents element-wise multiplication. Representative feature fusion and splicing operation, and The time confidence and spatial confidence scores corresponding to the time dimension features and spatial dimension features, respectively, are determined in the following way: Time dimension features and spatial dimensional features The input is fed into a confidence evaluation network, which outputs a normalized time confidence matrix. and spatial confidence matrix The confidence evaluation network includes a multilayer perceptron, whose parameters are determined by backpropagation optimization of a supervised loss function constructed based on downstream task prediction using fused features.

6. A distributed multi-sensor data fusion unsupervised learning system based on multi-dimensional masks, characterized in that, include: The data processing module is used to acquire observation time-series data collected by distributed multi-sensor collection, and then perform feature segmentation, initial embedding of the segments, and two-dimensional spatiotemporal location encoding to obtain an initial feature sequence of superimposed spatiotemporal locations. The observation time-series data includes detected environmental features and / or object features, as well as corresponding timestamps. The mask learning module is used to perform random masking on multiple blocks in the time and space dimensions of the initial feature sequence in parallel. Then, the temporal and spatial autoencoders are used to extract the temporal and spatial features of the unmasked visible blocks and decode them to obtain the reconstructed time series data. Based on the reconstructed observation time series data, the mask data reconstruction loss of the mask blocks is constructed and the contrastive learning loss based on spatial location is introduced to optimize the two autoencoders. The feature fusion module is used to extract time-dimensional and spatial-dimensional features using two optimized autoencoders and then perform confidence-based adaptive fusion with traditional features of observed time-series data to obtain fused features for downstream task prediction. The mask data reconstruction loss, which constructs mask blocks based on reconstructed observation time-series data, includes: in, This represents the raw observation time series data corresponding to the i-th sensor. and These are reconstructed time-series data based on temporal autoencoders and spatial autoencoders, respectively. and These are mask indicator matrices, where the masked areas have a value of 1, and the unmasked visible areas have a value of 0. The multiplication is element-wise, where N is the total number of sensors. and These represent the mask data reconstruction losses for the temporal autoencoder and the spatial autoencoder, respectively. Introduce a spatial location-based contrastive learning loss, including: Constructing positive and negative samples based on physical correlation: Calculating the Pearson correlation coefficient between the observation time series data of any target sensor i and the observation time series data of other sensors j in the distributed network. Set the positive sample threshold to and negative sample threshold is For target sensors extracted from spatial dimensional features Spatial feature vectors ,like If two sensors have the same data acquisition attributes, then the sensors will be... Spatial features As a sensor Positive samples, denoted as ;like Then the sensor Spatial features As a negative sample, it is denoted as , For the index of negative samples, a total of [number] samples were selected. One negative sample; Construction of contrastive learning loss: Based on the InfoNCE framework, for all Each sensor calculates a contrastive learning loss based on positive and negative sample pairs to determine the spatial dimension. .

7. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the distributed multi-sensor data fusion unsupervised learning method based on multi-dimensional masks as described in any one of claims 1-5.

Images