Time-Frequency Dual-Mode Alignment Method and Apparatus for Multivariable Time-Series Signals

Abstract

This invention relates to the field of signal processing technology, and provides a time-frequency dual-modal alignment method and apparatus for multivariable time-series signals. It utilizes a cross-scale time-series feature extraction network to extract discriminative features containing both time and frequency information from the multivariable time-series signal using cross-variable attention and cross-time attention. Then, it employs a codebook sharing time-frequency feature prototypes across domains for dual-modal vector quantization alignment based on spectral consistency, thereby eliminating domain offset through explicit frequency domain constraints. Furthermore, it uses an adaptive pseudo-label optimization strategy based on channel mutual information to suppress noise channel interference and improve the confidence of the target domain pseudo-label. Finally, under an end-to-end unified framework, it jointly optimizes the global alignment loss, local alignment loss, mutual information weighted maximization of the confusion matrix loss, and source domain cross-entropy loss to determine the target domain pseudo-label, deeply adapting to the physical characteristics of the time-series signal and simultaneously solving the problems of multivariable coupling, long-term time dependence, and time-frequency feature collaborative alignment.

Description

Technical Field

[0001] This invention belongs to the field of signal processing technology and relates to a time-frequency dual-mode alignment method and apparatus for multivariable time-series signals. Background Technology

[0002] In IoT and intelligent monitoring systems, time-series signals are a core data form for sensing the physical world. However, in practical applications, due to heterogeneous acquisition devices, changing working environments, or transmission channel interference, there are often significant distribution differences between labeled source domain data and unlabeled target domain data, leading to a substantial decrease in the model's recognition accuracy in the target domain. Existing technologies, when dealing with such cross-domain problems, mostly directly adopt domain adaptation methods from the field of computer vision, treating multivariate time-series signals as static single-channel or multi-channel images. This simple "image-based" processing approach has significant technical limitations when facing complex time-series signals.

[0003] Multivariate time-series signals differ fundamentally from image data in physical properties and data structure. Different variable channels often correspond to different physical quantities, and complex nonlinear coupling relationships exist between variables. Existing methods struggle to effectively characterize and decouple this cross-variable physical dependency. Furthermore, time-series signals typically exhibit long time spans and non-stationary evolution. Existing convolutional networks (CNNs) or standard Transformers primarily focus on local window features, easily severing the long-term evolution of the signal and making it difficult for models to capture dynamic evolution patterns across long time scales. In addition, typical time-series data exhibits significant frequency sensitivity; key discriminative information is often implicit in the spectral structure. Aligning features only in the time domain easily ignores the consistency of frequency domain distribution, leading to decreased alignment performance in noisy or frequency-shifted scenarios. Therefore, a dedicated unsupervised adaptive analysis technique is urgently needed that can deeply adapt to the physical characteristics of time-series signals and simultaneously address the problems of multivariate coupling, long-term time dependencies, and time-frequency feature co-alignment. Summary of the Invention

[0004] To address the problems existing in the above-mentioned traditional methods, this invention proposes a time-frequency dual-mode alignment method and a time-frequency dual-mode alignment device for multivariable time-series signals. This method can deeply adapt to the physical characteristics of time-series signals and solve the problems of multivariable coupling, long time-series dependence, and time-frequency feature collaborative alignment.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a time-frequency dual-mode alignment method for multivariable time-series signals is provided, including the following steps: Multivariate time series signals are input into a cross-scale time series feature extraction network, and discriminative features containing both time and frequency information are extracted using cross-variable attention and cross-time attention. The multivariate time series signals include labeled source domain data and unlabeled target domain data, and the discriminative features include source domain features and target domain features. The cross-scale time series feature extraction network uses a time series transformer adapted to the signal characteristics as the backbone network. By utilizing the codebook of cross-domain shared time-frequency feature prototypes, bimodal vector quantization alignment based on spectral consistency is performed on source domain features and target domain features. The bimodal vector quantization alignment includes global alignment based on spectral distance and local alignment based on temporal microstructure. An adaptive pseudo-label optimization strategy based on channel mutual information is used to construct a weighted confusion matrix and maximize the trace value of the features after quantization and alignment of dual-modal vectors. After jointly optimizing the total function consisting of global alignment loss, local alignment loss, mutual information weighted maximization of confusion matrix loss, and source domain cross-entropy loss, the pseudo-label of the target domain is determined.

[0006] In one embodiment, the step of inputting a multivariate time-series signal into a cross-scale time-series feature extraction network and extracting discriminative features containing both time-frequency information using cross-variable attention and cross-temporal attention includes: A time-series transformer is used to divide a multivariable time-series signal into time-series segments of fixed size; each time-series segment not only retains the time-domain values, but also undergoes a fast Fourier transform in parallel to form a time-frequency joint input vector; Learnable location codes with three-dimensional attributes are added to each time segment; the three-dimensional attributes include time, variable, and frequency. Based on the temporal segments with added position encoding, the correlation matrix between different sensor channels is calculated using cross-variable attention, and long-term temporal dependencies are captured using cross-temporal attention, generating source domain features and target domain features.

[0007] In one embodiment, the adaptive pseudo-label optimization strategy based on channel mutual information includes: The output probability distribution of the classifier corresponding to the target domain data is normalized. Mutual information is used to measure the importance of each sample for classifying the target domain. After calculating the weight of each sample based on mutual information, the weighted confusion matrix is ​​calculated using the normalized output probability distribution and the sample weight matrix. The mutual information weighted maximum confusion matrix loss is calculated based on the normalized weighted confusion matrix.

[0008] In one embodiment, the adaptive pseudo-label optimization strategy based on channel mutual information further includes: A confidence threshold is set and used to select samples with a predicted probability greater than the confidence threshold to participate in the calculation of local alignment loss; the confidence threshold is 0.8.

[0009] In one embodiment, the global alignment loss is: ; in, The weighting coefficients represent the frequency domain global alignment loss. Represents the L2 norm. It is a frequency domain transform operator. Representing source domain features The quantized time-frequency feature prototype matched in the codebook Representing target domain features The prototype of the quantized time-frequency feature matched in the codebook.

[0010] In one embodiment, the local alignment loss is: ; in, The weighting coefficients represent the temporal local alignment loss. C is the total number of categories, and c is the category index. This indicates that it needs to go through the bottleneck layer. Features belonging to class c in the source domain of the mapping. This indicates that it needs to go through the bottleneck layer. Features belonging to category c in the target domain features of the mapping. This represents the L2 norm.

[0011] In one embodiment, the mutual information weighted maximization of the confusion matrix loss is: ; in, This is the normalized weighted confusion matrix. Used to represent classes in the target domain j and class The correlation between them C The total number of categories.

[0012] In one embodiment, the source domain cross-entropy loss is: ; in, This represents the total number of samples in the source domain. C This indicates the total number of categories. Represents the source domain. i The sample at the th k Real labels on the class, This indicates that the classifier is effective for the source domain.i The sample at the th k Predicted probabilities on a class.

[0013] On the other hand, a time-frequency dual-mode alignment device for multivariable time-series signals is also provided, comprising: The feature extraction module is used to input multivariate time series signals into the cross-scale time series feature extraction network, and extract discriminative features containing both time and frequency information using cross-variable attention and cross-time attention. The multivariate time series signals include labeled source domain data and unlabeled target domain data. The discriminative features include source domain features and target domain features. The cross-scale time series feature extraction network uses a time series transformer adapted to the signal characteristics as the backbone network. The dual-modal alignment module is used to perform dual-modal vector quantization alignment of source domain features and target domain features based on spectral consistency by utilizing a codebook of cross-domain shared time-frequency feature prototypes. The dual-modal vector quantization alignment includes global alignment based on spectral distance and local alignment based on temporal microstructure. The label optimization module is used to construct a weighted confusion matrix and maximize the trace value of the features after quantization and alignment of the dual-modal vectors by adopting an adaptive pseudo-label optimization strategy based on channel mutual information. The joint optimization module is used to jointly optimize the total function composed of global alignment loss, local alignment loss, mutual information weighted maximization of confusion matrix loss, and source domain cross-entropy loss to determine the target domain pseudo-label.

[0014] In one embodiment, the feature extraction module includes: The timing partitioning submodule is used to divide a multivariable time-series signal into fixed-size timing segments using a timing transformer. Each timing segment not only retains the time-domain values ​​but also undergoes a fast Fourier transform in parallel to form a time-frequency joint input vector. The encoding addition submodule is used to add learnable location codes for three-dimensional attributes to each time segment; the three-dimensional attributes include time, variable, and frequency. The feature generation submodule is used to calculate the correlation matrix between different sensor channels using cross-variable attention and capture long-term temporal dependencies using cross-temporal attention based on the temporal segments after adding position encoding, and generate source domain features and target domain features.

[0015] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned time-frequency dual-modal alignment method and apparatus for multivariable time-series signals extracts discriminative features containing both time and frequency information from multivariable time-series signals using a cross-scale time-series feature extraction network with cross-variable attention and cross-time attention. Then, it performs dual-modal vector quantization alignment based on spectral consistency using a codebook of cross-domain shared time-frequency feature prototypes to eliminate domain offset through explicit frequency domain constraints. Next, it suppresses noise channel interference and improves the confidence of the target domain pseudo-label based on an adaptive pseudo-label optimization strategy using channel mutual information. Finally, under an end-to-end unified framework, it jointly optimizes the global alignment loss, local alignment loss, mutual information weighted maximization of the confusion matrix loss, and source domain cross-entropy loss to determine the target domain pseudo-label. This deeply adapts to the physical characteristics of the time-series signal and simultaneously solves the problems of multivariable coupling, long-term time dependence, and time-frequency feature collaborative alignment. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a time-frequency dual-mode alignment method for multivariable time-series signals in one embodiment. Figure 2 This is a schematic diagram illustrating the application of a time-frequency dual-mode alignment method for multivariable time-series signals in one embodiment; Figure 3 This is a schematic diagram of the module architecture for cross-scale feature extraction and bimodal vector quantization alignment in one embodiment; Figure 4 This is a schematic diagram of the module architecture of a time-frequency dual-mode alignment device for multivariable time-series signals in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0019] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0020] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] Time-series signals (especially multivariate non-stationary signals) differ fundamentally from static image data in physical properties, data structure, and feature representation. Existing technologies (formerly TransVQA) target isotropic spatial pixel matrices, which are insufficient to effectively address the unique challenges of time-series signals, such as "long-term time-dependent breaks," "complex multivariate coupling," and "sensitivity to frequency domain features." This invention is not a simple model transfer, but rather addresses the difficulties in time-series signal processing by employing a time-frequency multi-scale fusion feature extraction and alignment architecture centered on a time-series transformer structure designed for multivariate time-series signals.

[0022] In the field of signal processing, directly applying Transformers (such as ViT or Swing) from the image domain faces three major insurmountable challenges, which are also the problems this invention will focus on solving: First, the heterogeneity challenge of multivariate coupling. The RGB channels of an image are isomorphic, while multivariate signals (such as industrial sensors, communication signal I / Q channels) represent completely different physical quantities, and there are complex nonlinear couplings between variables. Traditional image models cannot decouple this cross-variable correlation. Second, the challenge of long-term time-series dependence of non-stationary signals. Signal data is usually extremely long and contains transient changes. Traditional image patch partitioning strategies usually assume that adjacent patches are semantically weakly correlated, while in time-series signals, adjacent time segments often have strong causal and continuous evolutionary relationships. Directly applying this partitioning method easily disrupts the continuity of the temporal context and makes it difficult to capture evolutionary patterns spanning long periods. Finally, the lack of frequency domain information. Radio and sensor signals are "frequency-sensitive". Image models focus only on the spatial domain, ignoring the periodicity, harmonic structure, and spectral sparsity inherent in the signal. This leads to the loss of key discriminative features in low signal-to-noise ratio or cross-domain scenarios.

[0023] Existing technologies, such as the previously researched unsupervised domain adaptation-based transferable vector quantization alignment method and device, provide a basic unsupervised domain adaptation framework. While this invention adopts the overall architecture of "converter + vector quantization alignment," it creatively modifies the specific implementation path to address the high-dimensional coupling, non-stationarity, and frequency domain sensitivity of time-series signal processing. For example, considering the physical characteristics of time-series signals (especially multivariate time-series signals), it makes comprehensive and substantial improvements from feature extraction (introducing the frequency domain), alignment methods (dual-modality), to optimization strategies (channel-level mutual information). Therefore, the substantial improvements made by this invention in feature extraction and alignment mechanisms are as follows: Firstly, the backbone network reconstruction is based on cross-variable and cross-time attention mechanisms: Instead of using a general visual Transformer, this invention designs a two-dimensional attention mechanism adapted to temporal characteristics: Cross-variable attention addresses the coupling problem of multi-source signals by designing a dedicated module to calculate the correlation matrix between different sensors / channels. This allows the model to automatically learn the dynamic dependencies between different physical variables, a capability completely lacking in the original image TransVQA. Cross-time attention addresses long-term temporal problems by not only focusing on local patches but also introducing global temporal attention calculations. By capturing the evolution trend of signals over long time spans, this solves the problem of traditional CNNs or short-window Transformers "seeing the trees but not the forest" when processing non-stationary signals.

[0024] Secondly, it introduces "time-frequency dual-modal" feature perception and alignment: Unlike the image domain which only operates in pixel space, this invention introduces a time-frequency domain multi-scale fusion mechanism before vector quantization alignment (VQA): (1) Explicit encoding of frequency domain features: Considering that the signal may have amplitude changes in the time domain but maintain structural consistency in the frequency domain, this invention integrates spectral information in the feature extraction stage. By transforming the signal patch in the frequency domain or spectral domain, frequency domain structural features (such as periodicity, spectral peak distribution, etc.) that are not sensitive to amplitude changes are introduced. (2) Time-frequency complementary alignment: In the VQA alignment stage, it is no longer a single feature vector matching, but requires the source domain and target domain to achieve distribution alignment in both the "time domain dynamics" and "frequency domain structure" dimensions. This dual-modal constraint significantly improves the model's generalization ability in cross-domain scenarios (such as environments with different sampling rates and different signal-to-noise ratios).

[0025] Thirdly, there is a multi-scale local semantic fusion mechanism: Addressing local transient features in signals (such as sudden interference and brief modulation patterns), this invention improves the patching process. Firstly, it employs a multi-scale patching strategy. Unlike fixed-size image slices, this invention uses a multi-scale sliding window mechanism, capable of capturing both short-lived transient pulses and long-term periodic signals. Secondly, it features fine-grained alignment of local semantics. In the local alignment stage, an improved bottleneck layer forces the model to focus on the signal's local microstructure (such as waveform edges and rising / falling edge features), rather than merely aligning the overall statistical distribution. This directly solves the problem of offsets in local details between signals from different domains, even though they are generally similar.

[0026] In one embodiment, such as Figure 1 As shown, a time-frequency dual-mode alignment method for multivariable time-series signals is provided, which may include the following processing steps: S10, input the multivariate time series signal into the cross-scale time series feature extraction network, and use cross-variable attention and cross-time attention to extract discriminative features containing both time and frequency information; the multivariate time series signal includes labeled source domain data and unlabeled target domain data, the discriminative features include source domain features and target domain features, and the cross-scale time series feature extraction network uses a time series transformer adapted to the signal characteristics as the backbone network; S12 utilizes the codebook of cross-domain shared time-frequency feature prototypes to perform bimodal vector quantization alignment of source domain features and target domain features based on spectral consistency; bimodal vector quantization alignment includes global alignment based on spectral distance and local alignment based on temporal microstructure; S14, an adaptive pseudo-label optimization strategy based on channel mutual information is used to construct a weighted confusion matrix for the features after quantization and alignment of dual-modal vectors and maximize the trace value; S16. After jointly optimizing the total function consisting of global alignment loss, local alignment loss, mutual information weighted maximization of confusion matrix loss, and source domain cross-entropy loss, the target domain pseudo-label is determined.

[0027] Understandable, such as Figure 2As shown, a cross-scale temporal feature extraction network is first constructed: a time-series transformer adapted to signal characteristics is used as the backbone network. Specifically, the input multivariate time-series signal (including labeled source domain data and unlabeled target domain data) is first divided into multi-scale blocks, and the frequency domain amplitude spectrum is extracted using Fast Fourier Transform (FFT). Unlike traditional spatial attention mechanisms, this step utilizes cross-variable attention to calculate the correlation matrix between different sensor channels, automatically learning the coupling relationship between multi-source signals; simultaneously, cross-time attention is used to capture the non-stationary dynamic evolution of the signal over a long time span, thereby extracting discriminative features containing both time and frequency information.

[0028] Then, a bimodal vector quantization alignment based on spectral consistency is constructed: a codebook with cross-domain shared time-frequency feature prototypes is introduced, and domain offset is eliminated through explicit frequency domain constraints.

[0029] Specifically, firstly, global alignment based on spectral distance is performed, mapping the source domain features and target domain features obtained in the previous step to the frequency domain space. Using the codebook as a reference, the consistency loss of spectral amplitude and phase is calculated, forcing the signals in the source and target domains to remain synchronized in the frequency domain distribution, eliminating global drift caused by sampling rate or environmental noise. Secondly, local alignment based on temporal microstructure is performed, using pseudo-labels as guidance to perform fine-grained matching of temporal waveform features of the same category, and solving the inter-class confusion problem in the temporal domain through feature bottleneck layer fusion.

[0030] Next, an adaptive pseudo-label optimization strategy based on channel mutual information is constructed: For cases where some channels in a multivariate time series signal may be faulty or noisy, a channel quality assessment mechanism is introduced. The mutual information between each variable channel and the prediction result is calculated, key channels with high signal-to-noise ratios and high-value samples are selected, a weighted confusion matrix is ​​constructed, and its trace value is maximized, thereby suppressing noise channel interference and improving the confidence of the pseudo-label in the target domain.

[0031] Finally, an end-to-end unified framework joint optimization is performed: the above calculation process is integrated into a unified framework to jointly optimize the source domain cross-entropy loss, global alignment loss, local alignment loss, and mutual information weighted maximization of the confusion matrix loss.

[0032] The aforementioned time-frequency dual-modal alignment method for multivariable time-series signals extracts discriminative features containing both time and frequency information from multivariable time-series signals through a cross-scale time-series feature extraction network using cross-variable attention and cross-time attention. It then utilizes a codebook sharing time-frequency feature prototypes across domains for dual-modal vector quantization alignment based on spectral consistency, thereby eliminating domain offset through explicit frequency domain constraints. Furthermore, an adaptive pseudo-label optimization strategy based on channel mutual information suppresses noise channel interference and improves the confidence of the target domain pseudo-label. Finally, under an end-to-end unified framework, it jointly optimizes the global alignment loss, local alignment loss, mutual information weighted maximization of the confusion matrix loss, and source domain cross-entropy loss to determine the target domain pseudo-label. This deeply adapts to the physical characteristics of time-series signals and simultaneously addresses the problems of multivariable coupling, long-term time dependence, and time-frequency feature collaborative alignment.

[0033] Compared to existing technologies, the above design achieves stable representation of long-range dependencies of multivariate time-series signals through cross-variable and cross-time joint attention modeling; improves cross-domain consistency of feature representation under different signal-to-noise ratios and sampling conditions through time-frequency dual-modal feature constraints; and alleviates the problem of reduced recognition performance caused by local structural misalignment between different domains through a multi-scale local semantic alignment mechanism.

[0034] In one embodiment, regarding step S10 above, the following improved time-series transformer can be used for deep feature mining, and the specific processing procedure is as follows: A time-series transformer is used to divide a multivariable time-series signal into time-series segments of fixed size; each time-series segment not only retains the time-domain values, but also undergoes a fast Fourier transform in parallel to form a time-frequency joint input vector; Learnable location codes with three-dimensional attributes are added to each time segment; the three-dimensional attributes include time, variable, and frequency. Based on the temporal segments with added position encoding, the correlation matrix between different sensor channels is calculated using cross-variable attention, and long-term temporal dependencies are captured using cross-temporal attention, generating source domain features and target domain features.

[0035] Specifically, such as Figure 3 As shown, a time-series transformer (Transformer model) is used as the backbone network to process the input multivariate time-series signals. (in T For time step, VThe multivariate time-series signal is divided into fixed-size time segments (patches) based on the number of variables. The specific value of the fixed size is selected based on factors such as the signal's physical frequency and sampling rate, the task's sensitivity to time scale, model computational and statistical efficiency, and compatibility with subsequent modules. For example, it can be a power of 2 (such as 32, 64, 128, or 256). Considering the two-dimensional structure of the multivariate time-series signal, each time segment not only retains its time-domain values ​​but also undergoes a parallel Fast Fourier Transform (FFT) to extract its frequency-domain amplitude spectrum, forming a joint time-frequency input vector. This means that the global frequency-domain energy distribution (amplitude spectrum) of the signal is obtained through FFT. This frequency-domain feature vector is then concatenated with the original time-domain sequence to form a new feature vector that simultaneously contains time-domain details and frequency-domain structure, serving as the model input. In this way, frequency-domain information (FFT, frequency coding) is deeply integrated, and cross-variable and cross-time attention mechanisms are specifically designed to address the multivariate coupling problem that existing technologies have failed to handle.

[0036] Transformer is a general sequence-to-sequence (seq2seq) model architecture based on a self-attention mechanism. It was originally designed for natural language processing (NLP) but has been widely used in time series prediction, speech recognition, trajectory prediction and other time series tasks.

[0037] Each time-series segment is given a learnable location code containing three-dimensional attributes: time, variables, and frequency. This ensures the model can perceive the temporal evolution logic of the signal and the attributes of each physical variable. The time dimension identifies which window on the time axis the segment belongs to, allowing the model to perceive trend evolution, causal relationships, and sequence. The variable dimension identifies which physical variable (e.g., flow rate, rotational speed, or vibration) the segment belongs to. Different variables have distinct dimensions, dynamic ranges, and physical meanings; this code allows the model to distinguish signal sources and learn dependencies between variables (similar to node identities in a neural network). The frequency dimension identifies which frequency component or band (e.g., fundamental frequency, second harmonic, or high-frequency noise region) the segment corresponds to. This allows the model to determine whether the current segment describes a slowly drifting low-frequency trend or a transient high-frequency detail, facilitating targeted frequency-domain attention interactions.

[0038] Calculate the correlation matrix between different sensor channels using transvariable attention. ( C The weights are used to suppress redundant or faulty channels (total number of categories). It's understandable that when facing data collected simultaneously from multiple sensors, such as an environmental monitoring station with multiple sensor channels for temperature, humidity, air pressure, and wind speed, there are complex interactions between these channels: for example, an increase in temperature can lead to a decrease in humidity, changes in air pressure, etc. Modeling such multivariate dependencies requires a... C ×C The correlation matrix is ​​used to quantitatively describe the dependence of each channel on another. The correlation matrix can be used as input to a graph neural network (adjacency matrix) or directly for feature weighting or interpretability analysis. Traditional attention mechanisms typically operate on the time step (e.g., self-attention in Transformers, where each time step focuses on other time steps); cross-variable attention, on the other hand, shifts attention to the variable (channel) dimension, treating each sensor channel as a token and calculating the attention weights between channels. Thus, the attention matrix naturally forms a... C × C A square matrix, where each element represents a channel. i For the channel j Attention scores, i.e., the strength of their correlation.

[0039] Specifically, an attention mechanism can be used. The query (Q) and key (K) come from the variable dimension. An attention score matrix is ​​calculated, and then, typically, averaging or some aggregation (such as average pooling) is applied to obtain a representation of the correlation between variables. For example, first, embedding is performed on the temporal segments of each channel (e.g., pooling on the time dimension (average pooling or attention pooling) to obtain a global channel representation, where each row is a semantic vector of a channel). Then, a learnable weight matrix is ​​used to perform a linear transformation on the global channel representation to generate Q and K. Next, the dot product is used to calculate the original correlation score between any two channels. Each element in the score matrix formed by the original correlation scores represents the degree of matching between one channel (as Q) and another channel (as K). Finally, Softmax normalization is applied to each row of the score matrix so that the sum of the attention weights of each channel to all other channels is 1, thus obtaining the correlation matrix. .

[0040] By leveraging cross-temporal attention to capture long-term temporal dependencies, source domain features are ultimately generated. and target domain features These feature vectors implicitly contain the time-frequency structure information of the signal. Long-term dependencies in time-series signals (such as slow trends, low-frequency periods, or long-delay causal relationships) are difficult to be effectively captured by the receptive field of convolution or the gradual gradient of recurrent networks. Cross-time attention is introduced to solve this problem: it treats each time segment on the time axis as a token, and through a self-attention mechanism, allows each time position to directly focus on any other time position in the sequence, thereby establishing a global time correlation in one step. This enables the model to capture patterns that are hundreds of time steps apart, such as long-term degradation trends of equipment performance, daily / weekly seasonal patterns, or the long evolution process of a fault from early signs to its final occurrence.

[0041] Specifically, the process of capturing long-term temporal dependencies using cross-temporal attention includes multi-head self-attention, global dependency aggregation, and multi-layer stacking. Multi-head self-attention generates a query (Q), key (K), and value (V) at each time step using a learnable linear transformation. The dot product attention score across all time steps is calculated and normalized using Softmax to obtain an attention weight matrix, where each element represents the degree of attention one time step receives from another. Global dependency aggregation: the output at each time step is a weighted sum of the values ​​(V) at all other times, with weights determined by the attention score. This is equivalent to adaptively aggregating global information in the temporal dimension, directly amplifying any significant long-term time-interval dependencies. Multi-layer stacking, by stacking multiple layers of cross-temporal attention (typically combined with feedforward networks, residual connections, and layer normalization), gradually refines high-order temporal dynamic patterns, enabling the model to perceive local abrupt changes while maintaining long-range contextual consistency. The parameters of cross-temporal attention are completely shared between the source and target domains, meaning the modeling logic for temporal dynamics is identical across both domains. For both source and target domain samples, their respective temporal segments are input and processed by the same cross-temporal attention encoder (a Transformer encoder specifically designed for the time dimension) to generate source domain features. and target domain features .

[0042] Furthermore, such as Figure 3 As shown, regarding step S12 above, the bimodal vector quantization alignment for spectral consistency may specifically include the following processing steps: To address the frequency shift caused by cross-domain signal variations, an updatable time-frequency feature prototype codebook is used. Constraints are applied, and the codebook stores cross-domain shared discretized time-frequency feature prototypes. For any input feature... The nearest neighbor search is used to find the best-matching prototype vector in the codebook. prototype vector index k : ; in, It is a frequency domain transform operator, which requires that the characteristics be closest in the frequency domain, thus ignoring the phase alignment error in the time domain. For the first i Each input feature For the first in the codebook j One prototype vector.

[0043] A global alignment loss is constructed based on spectral distance, forcing source and target domain features to approximate the time-frequency feature prototypes in the codebook in terms of spectral distribution. The global alignment loss is: ; in, The weighting coefficients represent the frequency domain global alignment loss. It is the L2 norm (also known as the Euclidean norm); Representing source domain features The quantized time-frequency feature prototype matched in the codebook Representing target domain features The quantized time-frequency feature prototypes are matched in the codebook. This step ensures the consistency of frequency components in the cross-domain signal. The global alignment loss includes a frequency domain transform operator, which is explicitly calculated in the frequency domain space to ensure spectral consistency.

[0044] Building upon frequency domain alignment, local alignment is performed on the time-domain micro-waveforms of similar signals. Features are then passed through a bottleneck layer. After mapping, the local alignment loss of similar samples is calculated using pseudo-labels as follows: ; in, The weighting coefficients represent the temporal local alignment loss; C The total number of categories; c Indexed by category; This indicates that it needs to go through the bottleneck layer. The source domain features of the mapping belong to the category c Features This indicates that it needs to go through the bottleneck layer. The target domain features of the mapping belong to the category c Features; It is the L2 norm. This step eliminates local differences caused by phase drift or time delay.

[0045] It's important to note that the bottleneck layer is a structure introduced in Deep Residual Networks (ResNet) to reduce the computational complexity of the model and improve feature extraction capabilities. A bottleneck layer typically consists of three convolutional layers: the first 1x1 convolutional layer reduces the number of input channels (dimensionality) to decrease the computational cost of subsequent convolutional layers; the second 3x3 convolutional layer performs convolution operations on the reduced-dimensional feature map to extract features; and the third 1x1 convolutional layer restores the number of channels to the original dimension for use in the next layer.

[0046] In one embodiment, the adaptive pseudo-label optimization strategy based on channel mutual information in step S14 above may include the following processing steps: Output probability distribution of the classifier corresponding to the target domain data Normalization is performed to eliminate dimensional differences between different samples: ; in, Indicates the first j The class of i One sample, here C It is the total number of categories. This is the probability normalization scaling parameter. It should be noted that only one classifier needs to be configured to label or classify samples in the target domain; the source and target domains share the same classifier.

[0047] Utilizing mutual information To measure each sample The importance of mutual information for the classification of the target domain is calculated using the following formula: ; in, Indicates sample The corresponding classifier's output probability distribution, and the average value of the output probability distribution. , This indicates taking the expected value of the random variable in the subscript (i.e., a probability-weighted average). Represents entropy, Represents each sample The output probability distribution of the corresponding classifier.

[0048] Calculate each sample based on mutual information The weights are calculated using the following formula: ; in, For the first i The weights of each sample, and the weight matrix formed by these weights. W It is a diagonal matrix representing the importance of each sample batch to the target domain. The output probability distribution is then normalized. and the weight matrix of the samples W Calculate the weighted confusion matrix: ; in, Used to represent classes in the target domain j and class The correlation between them, for the weighted confusion matrix Normalization yields the normalized weighted confusion matrix. .

[0049] The mutual information weighted maximization of the confusion matrix loss is calculated based on the normalized weighted confusion matrix. This optimizes the output of the classifier corresponding to the target domain, making it closer to one-hot encoding. The function that maximizes the confusion matrix loss using mutual information weighting is defined as: ; Optionally, a confidence threshold can be set to select samples with a predicted probability greater than the confidence threshold to participate in the calculation of local alignment loss; the confidence threshold is 0.8.

[0050] Specifically, to improve the quality of pseudo-labels, a confidence threshold can be set. Select only the predicted probability The samples participate in the local alignment loss The calculation reduces the impact of noise and false labels; Indicates the first in the target domain j One sample, The model represents the target domain. j Each sample in category c The confidence level is measured. Existing technologies focus on sample-level importance, while this embodiment, combined with the improvements described above, focuses on the importance of variable channels, suppressing interference from noisy or faulty channels, and specifically solving the unique problems of multivariable signals.

[0051] In one embodiment, the unified framework joint optimization, conventionally performed in step S16 above, may include the following process: The features mapped by the bottleneck layer are processed through a fully connected layer. Input the classifier C to generate the classification output. Specifically, for the feature data in the source domain, calculate the source domain cross-entropy loss. This is to optimize the classifier's performance. The calculation formula is as follows: ; in, This represents the total number of samples in the source domain. C This indicates the total number of categories. Represents the source domain. i The sample at the th k Real labels on the class, This indicates that the classifier has a certain number of views of the source domain. i The sample at the th k The predicted probability on the class. By minimizing this loss, supervised training of the source domain feature classification ability is achieved.

[0052] Jointly optimize global alignment loss Local alignment loss Mutual information weighted maximization of confusion matrix loss Source domain cross-entropy loss This enables the classification of time-series signals. The unified framework achieves this by jointly optimizing the following total loss function, defined as: ; in, , and These are the balancing parameters for each loss.

[0053] The aforementioned time-frequency bimodal alignment method for multivariate time-series signals breaks through the limitations of traditional image processing. It constructs a cross-scale time-series feature extraction network to deeply mine the spatiotemporal physical dependencies and discriminative features of multivariate time-series signals. It utilizes spectrally consistent bimodal vector quantization alignment to address cross-domain distribution differences, optimizes pseudo-label confidence through channel mutual information weighting, and integrates these into a unified framework to achieve efficient unsupervised domain-adaptive classification. It successfully achieves feature alignment and efficient classification of cross-domain time-series signals. Through bimodal vector quantization alignment and pseudo-label optimization strategies, it significantly reduces the impact of domain offset on classification performance and improves the model's generalization ability in unlabeled target domains.

[0054] It should be understood that, although Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0055] In one embodiment, such as Figure 4As shown, a time-frequency dual-modal alignment device for multivariate time-series signals is also provided, which may include a feature extraction module 11, a dual-modal alignment module 13, a label optimization module 15, and a joint optimization module 17. The feature extraction module 11 is used to input the multivariate time-series signal into a cross-scale time-series feature extraction network, and extract discriminative features containing both time-frequency information using cross-variable attention and cross-time attention. The multivariate time-series signal includes labeled source domain data and unlabeled target domain data. The discriminative features include source domain features and target domain features. The cross-scale time-series feature extraction network uses a time-series transformer adapted to the signal characteristics as its backbone network. The dual-modal alignment module 13 is used to perform dual-modal vector quantization alignment of source domain features and target domain features based on spectral consistency using a codebook of cross-domain shared time-frequency feature prototypes. The dual-modal vector quantization alignment includes global alignment based on spectral distance and local alignment based on temporal microstructure. The label optimization module 15 is used to construct a weighted confusion matrix for the features after dual-modal vector quantization alignment using an adaptive pseudo-label optimization strategy based on channel mutual information and to maximize the trace value. The joint optimization module 17 is used to jointly optimize the total function composed of global alignment loss, local alignment loss, mutual information weighted maximization of confusion matrix loss and source domain cross-entropy loss, and then determine the target domain pseudo-label.

[0056] The aforementioned time-frequency dual-modal alignment device for multivariable time-series signals extracts discriminative features containing both time and frequency information from multivariable time-series signals through a cross-scale time-series feature extraction network using cross-variable attention and cross-time attention. Then, it performs dual-modal vector quantization alignment based on spectral consistency using a codebook of cross-domain shared time-frequency feature prototypes to eliminate domain offset through explicit frequency domain constraints. Next, it suppresses noise channel interference and improves the confidence of target domain pseudo-labels based on an adaptive pseudo-label optimization strategy based on channel mutual information. Finally, under an end-to-end unified framework, it jointly optimizes global alignment loss, local alignment loss, mutual information weighted maximization of confusion matrix loss, and source domain cross-entropy loss to determine target domain pseudo-labels, thereby deeply adapting to the physical characteristics of time-series signals and solving the problems of multivariable coupling, long time-series dependence, and time-frequency feature collaborative alignment.

[0057] In one embodiment, the feature extraction module 11 may specifically include: a time-series segmentation submodule, used to divide the multivariate time-series signal into fixed-size time-series segments using a time-series transformer; each time-series segment not only retains time-domain values ​​but also undergoes a fast Fourier transform in parallel to form a time-frequency joint input vector; an encoding addition submodule, used to add learnable positional codes of three-dimensional attributes to each time-series segment; the three-dimensional attributes include time, variable, and frequency; and a feature generation submodule, used to calculate the correlation matrix between different sensor channels using cross-variable attention and capture long-term temporal dependencies using cross-temporal attention based on the time-series segments with added positional codes, generating source domain features and target domain features.

[0058] It is understood that the specific limitations of the time-frequency dual-mode alignment device for multivariable time-series signals mentioned above can be found in the corresponding limitations of the time-frequency dual-mode alignment method for multivariable time-series signals mentioned above, and will not be repeated here.

[0059] Each module in the aforementioned time-frequency dual-mode alignment device for multivariable time-series signals can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of a device with data processing capabilities, or stored in software within the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of computer devices already existing in the art.

[0060] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus DRAM (RDRAM), and interface DRAM (DRDRAM), etc.

[0061] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0062] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A time-frequency dual modality alignment method for multivariate time series, characterized in that, Including the following steps: Multivariate time series signals are input into a cross-scale time series feature extraction network, and discriminative features containing both time and frequency information are extracted using cross-variable attention and cross-time attention. The multivariate time series signals include labeled source domain data and unlabeled target domain data, and the discriminative features include source domain features and target domain features. The cross-scale time series feature extraction network uses a time series transformer adapted to the signal characteristics as the backbone network. By utilizing the codebook of cross-domain shared time-frequency feature prototypes, bimodal vector quantization alignment based on spectral consistency is performed on source domain features and target domain features. The bimodal vector quantization alignment includes global alignment based on spectral distance and local alignment based on temporal microstructure. An adaptive pseudo-label optimization strategy based on channel mutual information is used to construct a weighted confusion matrix and maximize the trace value of the features after quantization and alignment of dual-modal vectors. After jointly optimizing the total function consisting of global alignment loss, local alignment loss, mutual information weighted maximization of confusion matrix loss, and source domain cross-entropy loss, the pseudo-label of the target domain is determined.

2. The time-frequency dual modality alignment method for multivariate time series signals according to claim 1, characterized in that, The steps of inputting multivariate time-series signals into a cross-scale time-series feature extraction network and extracting discriminative features containing both time-frequency information using cross-variable attention and cross-time attention include: A time-series transformer is used to divide a multivariable time-series signal into time-series segments of fixed size; each time-series segment not only retains the time-domain values, but also undergoes a fast Fourier transform in parallel to form a time-frequency joint input vector; Learnable location codes with three-dimensional attributes are added to each time segment; the three-dimensional attributes include time, variable, and frequency. Based on the temporal segments with added position encoding, the correlation matrix between different sensor channels is calculated using cross-variable attention, and long-term temporal dependencies are captured using cross-temporal attention, generating source domain features and target domain features.

3. The time-frequency dual modality alignment method for multivariate time series signals according to claim 1 or 2, characterized in that, The adaptive pseudo-tag optimization strategy based on channel mutual information includes: The output probability distribution of the classifier corresponding to the target domain data is normalized. Mutual information is used to measure the importance of each sample for classifying the target domain. After calculating the weight of each sample based on mutual information, the weighted confusion matrix is ​​calculated using the normalized output probability distribution and the sample weight matrix. The mutual information weighted maximum confusion matrix loss is calculated based on the normalized weighted confusion matrix.

4. The time-frequency dual modality alignment method for multivariate time series signals according to claim 3, characterized in that, The adaptive pseudo-label optimization strategy based on channel mutual information also includes: A confidence threshold is set and used to select samples with a predicted probability greater than the confidence threshold to participate in the calculation of local alignment loss; the confidence threshold is 0.

8.

5. The time-frequency dual modality alignment method for multivariate time series signals according to claim 3, characterized in that, The global alignment loss is: wherein, denotes a weight coefficient of the frequency domain global alignment loss, denotes an L2 norm, is a frequency domain transform operator, denotes a source domain feature quantized time-frequency feature prototype matched in the codebook, denotes a target domain feature quantized time-frequency feature prototype matched in the codebook.

6. The time-frequency dual modality alignment method for multivariate time series signals according to claim 3, characterized in that, The local alignment loss is: in, The weighting coefficients represent the temporal local alignment loss. C is the total number of categories, and c is the category index. This indicates that it needs to go through the bottleneck layer. Features belonging to class c in the source domain of the mapping. This indicates that it needs to go through the bottleneck layer. Features belonging to category c in the target domain features of the mapping. This represents the L2 norm.

7. The time-frequency dual modality alignment method for multivariate time series signals according to claim 3, characterized in that, The mutual information weighted maximization of the confusion matrix loss is: wherein, is the normalized weighted confusion matrix, the weighted confusion matrix for representing the correlation between classes j and classes in the target domain, C is the total number of classes for classification.

8. The time-frequency dual modality alignment method for multivariate time series signals according to claim 3, characterized in that, The source domain cross-entropy loss is: in, This represents the total number of samples in the source domain. C This indicates the total number of categories. Represents the source domain. i The sample at the th k Real labels on the class, This indicates that the classifier is effective for the source domain. i The sample at the th k Predicted probabilities on a class.

9. A time-frequency dual modality alignment device for multivariate time series, characterized in that, include: The feature extraction module is used to input multivariate time series signals into the cross-scale time series feature extraction network, and extract discriminative features containing both time and frequency information using cross-variable attention and cross-time attention. The multivariate time series signals include labeled source domain data and unlabeled target domain data. The discriminative features include source domain features and target domain features. The cross-scale time series feature extraction network uses a time series transformer adapted to the signal characteristics as the backbone network. The dual-modal alignment module is used to perform dual-modal vector quantization alignment of source domain features and target domain features based on spectral consistency by utilizing a codebook of cross-domain shared time-frequency feature prototypes. The dual-modal vector quantization alignment includes global alignment based on spectral distance and local alignment based on temporal microstructure. The label optimization module is used to construct a weighted confusion matrix and maximize the trace value of the features after quantization and alignment of the dual-modal vectors by adopting an adaptive pseudo-label optimization strategy based on channel mutual information. The joint optimization module is used to jointly optimize the total function composed of global alignment loss, local alignment loss, mutual information weighted maximization of confusion matrix loss, and source domain cross-entropy loss to determine the target domain pseudo-label.

10. The time-frequency dual modality alignment apparatus for multivariate time series signals according to claim 9, wherein, The feature extraction module includes: The timing partitioning submodule is used to divide a multivariable time-series signal into fixed-size timing segments using a timing transformer. Each timing segment not only retains the time-domain values ​​but also undergoes a fast Fourier transform in parallel to form a time-frequency joint input vector. The encoding addition submodule is used to add learnable location codes for three-dimensional attributes to each time segment; the three-dimensional attributes include time, variables, and frequencies. The feature generation submodule is used to calculate the correlation matrix between different sensor channels using cross-variable attention and capture long-term temporal dependencies using cross-temporal attention based on the temporal segments after adding position encoding, and generate source domain features and target domain features.

Images