A method, system and device for spatiotemporal cross-scale dimensionality reduction representation of multi-modal physiological signals

Abstract

This invention discloses a spatiotemporal cross-scale dimensionality reduction representation method, system, and device for multimodal physiological signals. The method includes: multimodal signal temporal alignment, sliding scale segmentation, single-modal spatiotemporal feature pre-training, cross-modal coupling alignment, and model transfer application. In single-modal spatiotemporal feature pre-training, single-modal signal segments under different time processes or spatial arrangements are reconstructed through an encoder-decoder, realizing multi-level spatiotemporal modeling and analysis of the signal source. In cross-modal coupling alignment, the spatiotemporal features of other modal physiological signals are reconstructed through an encoder-decoder, learning the coupling relationships between different modalities of the physiological system. The model after cross-modal coupling alignment can be transferred to specific tasks for fine-tuning, enhancing the comprehensive representation capability of physiological systems by physiological signals from a few modalities, specified spaces, or finite time periods acquired by the backward device. This invention achieves effective representation of complex overall physiological states using easily obtainable signals through low-level dimensionality reduction and transfer.

Description

Technical Field

[0001] This invention relates to the fields of physiological signal processing and artificial intelligence, specifically to a method, system, and apparatus for learning a comprehensive representation of the state of a complex physiological system from easily accessible physiological signals through self-supervised learning. Background Technology

[0002] Comprehensive and accurate assessment of human health status is a primary goal of modern preventive medicine and personalized health management. Complex and close coupling relationships exist between various physiological systems in the human body. These relationships are not only reflected in the interaction of multimodal signals but are also often implicit in the dynamic changes of individual signals. Therefore, even relying on only a small number of easily accessible physiological signal types, it may be possible to infer and characterize higher-dimensional physiological states by mining the cross-modal features they contain. Based on this dimensionality reduction approach, it may be feasible to achieve comprehensive analysis and prediction of complex systems using simple signals.

[0003] Signals such as photoplethysmography (PPG), body temperature, voice, and respiration can be easily, non-invasively, and long-term collected through consumer-grade wearable devices such as smartwatches and wristbands, resulting in massive amounts of data. On the other hand, signals that provide richer and deeper information, such as heart sounds, blood pressure, blood glucose, blood oxygen, blood lipids, multichannel EEG, multi-lead ECG, and various electrical, optical, acoustic, mechanical, biochemical, ultrasound, electromagnetic wave, and imaging signals, typically require specialized medical equipment, complex electrode patches, and a clinical environment. These processes are cumbersome and costly, making them difficult to popularize in daily life. This leads to a core technological contradiction: while we can easily obtain massive amounts of data, the physiological dimensions directly reflected by this data are relatively singular; whereas data that provides comprehensive and in-depth information is extremely difficult to acquire. Therefore, there is an urgent need in this field for a technology that can achieve information dimensionality enhancement—that is, to infer and characterize physiological information that typically requires complex equipment to measure using only easily obtainable, simple signals, thereby achieving a comprehensive, multi-layered, and multi-dimensional understanding of the human body's state. However, the following problems still exist:

[0004] First, there is the challenge of data availability: Building a robust model capable of learning cross-modal physiological coupling theoretically requires a large-scale dataset with all modalities strictly time-synchronized. In the real world, obtaining such a "perfect" dataset is extremely difficult. Large-scale publicly available datasets are typically single-modal, while datasets containing synchronous multimodal signals have much smaller sample sizes and population coverage. This imbalance in data availability constitutes a core bottleneck in the current field of multimodal physiological signal analysis. Existing technological approaches cannot effectively utilize the "non-ideal" data distribution in the real world—namely, a large amount of asynchronous single-modal data and a small amount of synchronous multimodal data.

[0005] Second, the limitation of analytical scale: Physiological processes exhibit multi-scale characteristics over time, ranging from millisecond-level cardiac electrical activity to minute- and hour-level autonomic nervous system regulation, and even day-based circadian rhythms. Existing methods typically process signals using fixed time windows, which restricts the model's ability to capture cross-scale dynamic relationships. For example, the model may struggle to understand how short-term heart rate fluctuations predict long-term stress trends. Therefore, a mechanism is needed that can simultaneously process and correlate information at different temporal granularities to construct more comprehensive physiological dynamic models. Summary of the Invention

[0006] Purpose of the invention: The present invention aims to provide a spatiotemporal cross-scale dimensionality reduction representation method, system and device for multimodal physiological signals. Under the real-world conditions of lacking large-scale, fully synchronized multimodal physiological datasets, based solely on simple and easily obtainable low-dimensional physiological signals, the invention achieves spatiotemporal feature transformation and intermodal information flow representation through scale transformation and mask prediction, and comprehensively and deeply extracts and predicts the overall complex state changes of the physiological system.

[0007] Technical solution: To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a spatiotemporal cross-scale dimensionality reduction characterization method for multimodal physiological signals, comprising the following steps:

[0009] Timing alignment of multimodal signals;

[0010] Single-modal spatiotemporal physiological signals are segmented using a sliding scale over time.

[0011] Single-modal spatiotemporal feature pre-training involves training the model based on a single-modal dataset by performing temporal mask prediction and spatial mask prediction tasks to learn the temporal dependencies and spatial correlations within each modality. For each single modality, a corresponding encoder is trained. The encoder takes into input the spatiotemporal physiological signal sequence of the corresponding modality after sliding scale segmentation and outputs spatiotemporal physiological features. Information at different scales is input into the encoder through splicing, stacking, or parallel multi-branching.

[0012] Cross-modal coupling alignment, which is based on a time-synchronized multimodal dataset, incorporates discrete clinical features, and trains the model to learn the coupling relationship between different physiological signal modalities by performing a modality mask prediction task; during training, the spatiotemporal physiological features output by the encoder corresponding to the source modality are transformed by the cross-modal predictor and used for the target feature prediction task of the target modality that is different from the source modality.

[0013] The encoders corresponding to one or more modalities, which have been trained through cross-modal coupling alignment, are applied to downstream tasks to generate a comprehensive representation of the overall physiological state based on a single input or multiple modal signals, or fewer than the number of modalities used during training.

[0014] Preferably, multimodal signal timing alignment is performed through event anchoring mapping, including: based on the reference signal x(t), the time axis And the signal to be aligned, y(t), with time axis u, and the event anchor points of both. Solve the time mapping function t = g(u) using spline regression:

[0015]

[0016] in, For the timestamp of the i-th event in the reference modality, Let Z be the timestamp of the i-th event in the modality to be aligned, Z be the total number of observable anchor pairs, and i be the index of the event pair. Let w be the function space. i Let be the importance weight of the i-th anchor pair, and λ be the coefficient of the smoothing regularization term, controlling the smoothness of the time transformation function; the aligned signal is:

[0017] y aligned (t)=y(g -1 (t))

[0018] Among them, g -1 This represents the inverse function of the time mapping function.

[0019] Preferably, the slip scale segmentation is achieved by setting a scale set. and the corresponding window length and step length Given the signal S = {s1, s2, ..., s...} N At each scale d k The upper segment is divided into a series of fragments:

[0020]

[0021] in, Representing scale d k The i-th segment, where N is the total length of the current signal, and the step size is defined as... γ∈(0,1).

[0022] Preferably, before pre-training the single-modal spatiotemporal features, signal enhancement is also included, wherein the signal enhancement applies a set of enhancement operators to the segments before or after segmentation; the set of enhancement operators includes one or more of additive noise perturbation, amplitude scaling, time clipping and splicing, time warping, frequency domain filtering, and random occlusion.

[0023] Preferably, the loss in the single-modal spatiotemporal feature pre-training stage comprehensively considers temporal reconstruction loss and spatial reconstruction loss, or further considers prior feature prediction loss, wherein:

[0024] Temporal reconstruction loss L temporal Defined as:

[0025]

[0026] Among them, M temporal Let p represent the set of signal segments that are masked in the time dimension, where p is M. temporal A real signal segment, For the model's reconstruction result of fragment p, |M temporal |For M temporal The total number of elements;

[0027] Spatial reconstruction loss L spatial Defined as:

[0028]

[0029] Among them, C masked Let c represent the set of signal segments that are masked in spatial dimensions, where c is C masked A real signal segment, The model reconstructs fragment c, |C masked |For C masked The total number of elements;

[0030] Prior feature prediction loss L prior Defined as:

[0031]

[0032] Where M is the total number of prior features. For the model to predict priors, f (m) For true prior features, Let be the standard deviation of the m-th prior feature on the training set, used for normalization.

[0033] Preferably, the comprehensive loss of the cross-modal coupling alignment considers feature regression loss and prior feature alignment loss, or further considers cross-modal contrast loss, wherein:

[0034] Feature regression loss Lfeat Defined as:

[0035]

[0036] in, Let be a two-dimensional set of time location u and scale index d. This represents the true feature vector extracted at time position u and scale layer d in the target modality. This represents the estimated value of the target mode features predicted from the feature vector extracted from the source mode at the same position (u,d);

[0037] Prior feature alignment loss L prior_align Defined as:

[0038]

[0039] Among them, R t (·) represents the prior predictor of the target mode. Z source To represent the feature representation extracted from the source modes, Ψ s→t As a cross-modal predictor, its function is to map source modal features to the target modal feature space, [·] m This indicates taking the m-th dimension from the prediction results. Let m be the m-th prior feature in the target mode;

[0040] Cross-modal contrast loss L contrast Defined as:

[0041]

[0042] Where φ(·) is the feature projection normalization operator, which projects the features into the embedding space of the contrastive learning, and τ is a temperature coefficient used to control the sharpness of the contrastive loss distribution. source Z target )∈ For a batch of positive samples paired together, <·,·> denote the inner product between two vectors, Z′ target These are negative sample target modal features. Includes negative samples from the same batch.

[0043] Preferably, when applying encoders corresponding to one or more modes after cross-modal coupling alignment training to downstream tasks, the joint encoder E obtained after cross-modal coupling alignment is used. joint Combined with the classification head, cross-entropy loss is used for fine-tuning to achieve cross-scale and cross-temporal feature mapping and prediction between modalities in downstream tasks:

[0044]

[0045] H = E joint (S′ source ;θ E )

[0046] Among them, the joint encoder E joint S′ integrates the structure and parameters of multiple modal encoders and cross-modal predictors trained through cross-modal coupling alignment. source θ represents a single-mode or multi-mode input signal. E For the joint encoder E joint The parameter set, H = [h1, h2, ..., h T [ ] represents the output time-series feature sequence, where T represents the length of the input sequence and z is the feature convergence vector. These are attention weights, where q is a learnable query vector, and W... cls and b cls For classification header parameters, L task The cross-entropy loss is denoted by C, where C is the number of target categories in the downstream task.

[0047] Secondly, the present invention provides a spatiotemporal cross-scale dimensionality reduction characterization system for multimodal physiological signals, used to implement the aforementioned spatiotemporal cross-scale dimensionality reduction characterization method for multimodal physiological signals, including a forward device and / or a backward device.

[0048] The forward device is used to receive multimodal synchronous data during the training phase, and to perform single-modal feature pre-training and cross-modal coupling alignment operations to learn the spatiotemporal dependencies and feature mapping rules between different modalities and to construct a unified joint encoder.

[0049] The backward device is used during the deployment phase to generate a dimensionality-reduced representation vector that fuses features from multiple modalities based on a single or fewer modal input signal used during training, using a trained joint encoder to support real-time downstream task inference.

[0050] Furthermore, the forward device includes: a multi-channel signal acquisition unit for acquiring one or more multimodal physiological signals; a temporal alignment unit for temporally aligning the multimodal signals; a data preprocessing unit for sliding scale segmentation, or sliding scale segmentation combined with signal enhancement; and a training support unit for providing aligned multimodal data and clinical features to the cross-modal self-supervised learning model.

[0051] The backward device includes: a signal acquisition unit for acquiring continuous physiological signals; a model running and computing unit for running a trained encoder model locally, at the edge, or in the cloud to generate a comprehensive representation vector containing cross-modal spatiotemporal features; and an application interface unit for applying the comprehensive representation vector to downstream tasks.

[0052] Thirdly, the present invention provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals.

[0053] Beneficial effects: Compared with the prior art, the beneficial effects of this invention are:

[0054] (1) Achieving spatiotemporal dimensionality reduction and compact representation of multimodal high-dimensional physiological information. This invention constructs a self-supervised learning model with cross-modal coupling modeling capabilities, which can learn a low-dimensional, cross-scale, and cross-spatiotemporal joint representation vector from high-dimensional and complex multi-source physiological signals such as electroencephalography (EEG), multi-lead / single-lead electrocardiography (ECG), photoplethysmography (PPG), skin conductance / skin resistance, electromyography (EMG), heart sounds, respiration, body temperature, blood pressure, blood glucose, blood oxygen, speech, gait, and various electrical, optical, acoustic, mechanical, biochemical, ultrasound, electromagnetic waves, and imaging signals. While preserving key physiological semantic features, this vector significantly compresses the dimensionality, temporal length, and modal complexity of the original multimodal signals, exhibiting good generalization and expressiveness. The proposed method improves the sharing of features across time, frequency, and modality through steps such as alignment mechanism of multimodal synchronization signals, scale sliding processing, spatiotemporal mask prediction, and modality mask prediction. It can effectively reduce the dependence on the number of sensors, modality integrity, and acquisition quality, and is suitable for lightweight and high-expression representation of physiological systems in consumer or wearable devices. At the same time, it enhances the model's ability to discriminate and predict complex clinical states (such as sleep stages, cognitive load, psychological stress, and disease evolution trends).

[0055] (2) Lowering the data threshold and enhancing feasibility. The phased self-supervised learning strategy proposed in this invention no longer relies on large-scale synchronous multimodal data. The training process can be decomposed into: learning spatiotemporal features on large-scale asynchronous single-modal data and learning cross-modal mapping on small-scale synchronous data. This design significantly reduces the difficulty of data collection and the application threshold, making cross-modal physiological representations more feasible and practical in real-world environments.

[0056] (3) Building cross-scale analysis capabilities. Through the sliding scale segmentation mechanism, the model can simultaneously learn local dynamics and long-term fluctuation trends across multiple time scales and establish correlations between different scales. This enables the model to achieve more robust cross-scale physiological state modeling by utilizing the complementary relationship between temporal and spatial information in the case of long-term prediction and data gaps.

[0057] (4) Improved training efficiency and model performance. The phased training method of this invention enables the model to fully utilize massive single-modal data to learn stable spatiotemporal features in the first stage, and improve cross-modal semantic consistency and downstream task performance based on prior alignment. In the second stage, it quickly converges to the intermodal coupling relationship based on existing representations. This process improves both training efficiency and the cross-modal representation accuracy and robustness of the final model. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of a spatiotemporal cross-scale dimensionality reduction characterization method for multimodal physiological signals.

[0059] Figure 2 A schematic diagram of a spatiotemporal cross-scale dimensionality reduction characterization system, device, and its effects for multimodal physiological signals. Detailed Implementation

[0060] The specific implementation schemes of the present invention will be described in detail below with reference to the accompanying drawings and several embodiments. The following embodiments include implementation details for specific modalities (e.g., PPG / ECG / EEG / respiration) as well as alternative implementation strategies for general multimodal approaches. It should be understood that the following embodiments are merely implementation methods of the present invention and do not limit the scope of protection of the present invention in any way.

[0061] Example 1

[0062] This invention provides a spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals. First, the multimodal signals are temporally aligned, and the single-modal spatiotemporal physiological signals are segmented by sliding scale on the time scale. Then, single-modal spatiotemporal features are pre-trained and cross-modal coupling alignment is performed. Finally, the encoders corresponding to one or more modalities after cross-modal coupling alignment training are applied to downstream tasks. Based on the input of a single modal signal or multiple modal signals with fewer modalities than those used during training, a comprehensive representation of the overall physiological state is generated. In the pre-training process of single-modal spatiotemporal features, based on the single-modal dataset, the model is trained to learn the temporal dependencies and spatial correlations within each modality by performing temporal mask prediction tasks and spatial mask prediction tasks. For each single modality, a corresponding encoder is trained. The encoder takes as input the spatiotemporal physiological signal sequence of the corresponding modality after sliding scale segmentation and outputs spatiotemporal physiological features. Information from different scales is input to the encoder through splicing, stacking, or parallel multi-branching. In the cross-modal coupling alignment process, based on a temporally synchronized multimodal dataset, discrete clinical features are incorporated. By performing modality mask prediction tasks, the model is trained to learn the coupling relationships between different physiological signal modalities. During training, the spatiotemporal physiological features output by the encoder corresponding to the source modality are transformed by the cross-modal predictor and used for target feature prediction tasks of target modalities different from the source modality.

[0063] Figure 1 This example illustrates a detailed flowchart of a spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals, such as... Figure 1 As shown, it includes:

[0064] a) Multimodal signal timing alignment: Dynamic time warping (DTW), event anchor registration and other methods are used to achieve synchronization of multimodal signals;

[0065] b) Sliding scale segmentation: By dividing time windows of different sizes, modular extraction of information with different time histories and multidimensional granularity can be achieved;

[0066] c) Signal enhancement: Apply a set of enhancement operators to the segments before or after segmentation. The enhanced segments and the original segments can be used as model inputs together.

[0067] d) Single-modal spatiotemporal feature pre-training: Using asynchronous single-modal datasets, the model is trained to learn the temporal dependencies and spatial correlations within each modality by performing temporal mask prediction tasks and spatial mask prediction tasks.

[0068] e) Cross-modal coupling alignment: Utilizing a time-synchronized multimodal dataset and loading discrete clinical features, the model is trained to learn the coupling relationships between different physiological signal modalities by performing a modal mask prediction task;

[0069] f) Model transfer application: Apply the trained model to downstream tasks to generate a comprehensive representation of the overall physiological state based on the input low-dimensional modal signals.

[0070] In specific implementations, the multimodal signals include, but are not limited to: electroencephalography (EEG), respiration (such as breath sounds, breath waveforms, respiratory rate, and respiratory impedance RIP), multi-lead / single-lead electrocardiography (ECG), heart sounds, photoplethysmography (PPG), electromyography (EMG), electrodermal absorption (EDA / GSR), blood oxygen saturation (SpO2), blood pressure (BP), body temperature / skin temperature, cerebral blood flow (fNIRS), inertial measurement unit (IMU, including accelerometers and gyroscopes), and various electrical, optical, acoustic, mechanical, biochemical, ultrasonic, electromagnetic wave, and imaging signals. During cross-modal coupling alignment, the source mode and target mode can be a single mode or a combination of multiple modes (parallel or cascaded), and can be replaced or extended as needed in different implementations to adapt to specific application scenarios.

[0071] The multimodal signal timing alignment step is designed for small-scale synchronous datasets used for cross-modal coupling alignment. For example, this embodiment aligns the original signals of the input model through event anchoring mapping. Specifically, based on the reference signal x(t), the time axis... And the signal to be aligned, y(t), with time axis u, and the event anchor points of both. Solving the time mapping function t = g(u) using spline regression:

[0072]

[0073] in, For the timestamp of the i-th event in the reference modality, Let Z be the timestamp of the i-th event in the modality to be aligned, Z be the total number of observable anchor pairs, and i be the index of the event pair. For a sufficiently smooth function space, w i Let be the importance weight of the i-th anchor pair, and λ be the coefficient of the smoothing regularization term, controlling the smoothness of the time transformation function. The aligned signal is:

[0074] y aligned (t)=y(g -1 (t))

[0075] The sliding scaling step performs multi-scale segmentation on all data (single-modal and synchronous multimodal), enabling the model to simultaneously learn short-term rapid dynamics and long-term slowly changing feature patterns, and capture cross-scale dependencies. A scale set is defined. and the corresponding window length and step length Given the signal S = {s1, s2, ..., s...} N At each scale d k The upper segment is divided into a series of fragments:

[0076]

[0077] in, Representing scale d k The i-th segment, where N is the total length of the current signal, and the step size is defined as... γ∈(0,1). The window set can be a fixed set (e.g., {0.5s, 1s, 5s, 30s, 120s}) or dynamically selected based on the signal change rate or event density. This multi-scale segmentation can be adapted to Transformer-based models, implemented through parallel multi-scale branching or pyramid feature pooling (e.g., Temporal Pyramid Pooling), and input into a unified time-series encoder, compatible with CNN-Transformer, pure Transformer, BiLSTM, and other structures.

[0078] The signal enhancement step applies various enhancement methods to the segmented fragments to increase sample diversity and improve model robustness. The set of enhancement operators is defined as follows: The enhancement form of fragment p can be:

[0079]

[0080] The enhancement operators include, but are not limited to: (1) additive noise perturbation, p′=p+∈, Where σ is the noise amplitude coefficient; (2) Amplitude scaling, Where τ α To scale the expected value, σ α (3) Time trimming and splicing: randomly trim a certain length of subsequence from the fragment and splice it; (4) Time warping: use interpolation to map the original time axis to; (5) Frequency domain filtering: apply a bandpass filter H(f; f) in the Fourier domain. l ,f h ), where the passband range [f l ,f h ] is random sampling; (6) random occlusion, randomly set to zero in the time domain or channel dimension.

[0081] The single-modal spatiotemporal feature pre-training step divides the input signal into local spatiotemporal segments at multiple scales after sliding scaling and signal enhancement. These segments are then mapped to a shared semantic space through linear embedding and a unified encoder, forming multiple sets of token representations with a unified dimension. Subsequently, the tokens of all scale segments can be input into the Transformer through concatenation, stacking, or multi-branch encoding. The encoder and prediction head are then self-supervised and trained on a large-scale single-modal dataset. The decoder is used to extract latent representations from the encoder and mask labels M. tok The masked signal or features are reconstructed. In some embodiments, a CNN-Transformer hybrid structure, an RNN / BiLSTM connecting layer, a GNN / GAT extension, or other generative complementary model structures may also be used.

[0082] For example, based on the Transformer architecture, self-supervised tasks include:

[0083] (1) Temporal mask prediction: where the temporal reconstruction loss l temporal Defined as:

[0084]

[0085] Among them, M temporal Let p represent the set of signal segments that are masked in the time dimension, where p is M. temporal A real signal segment, For the model's reconstruction result of fragment p, |M temporal |For M temporal The total number of elements.

[0086] (2) Spatial mask prediction: where the spatial reconstruction loss L spatial Defined as:

[0087]

[0088] Among them, C masked Let c represent the set of signal segments that are masked in spatial dimensions, where c is C masked A real signal segment, The model reconstructs fragment c, |C masked |For C masked The total number of elements.

[0089] (3) Prior feature prediction: where the prior feature prediction loss L prior Defined as:

[0090]

[0091] Where M is the total number of prior features. For the model to predict priors, f (m) These are the true prior features obtained through deterministic operators. is the standard deviation of the m-th prior feature on the training set, used for normalization. Prior features include EEG band power and spectral entropy, ECG or PPG derived heart rate (HR), variability index SDNN, RMSSD, and pulse conduction time (PTT), etc.

[0092] In summary, the total loss for single-modal pre-training can be:

[0093] L intra =λ t L temporal +λ s L spatial +λ p L prior

[0094] λ t , λ s , λ p These are the weighting coefficients.

[0095] Other auxiliary losses and objectives, such as contrastive learning, cyclic consistency testing, and knowledge distillation, can also be introduced to improve semantic consistency and downstream performance.

[0096] The cross-modal coupling alignment step loads the pre-trained weights of each single-modal encoder onto a small-scale synchronous dataset to construct a joint framework. For each synchronous sample, multi-scale spatiotemporal feature representations are obtained through the source modal encoder and the target modal encoder, respectively.

[0097] Z source =E source (S source ),Z target =Etarget (S target )

[0098] Among them, E source and E target For the encoder that has been pre-trained on a single modality in the preceding steps, S source and S target These are the spatiotemporal signal sequences of the source mode and the target mode after slip-scale segmentation and alignment, respectively. source and Z target For the multi-scale, spatiotemporal embedded feature sequences (high-dimensional representations) output by their respective encoders.

[0099] Constructing a cross-modal predictor Ψ s→t , with source feature mode Z source As input, output a prediction of the target features. In a specific embodiment, Ψ s→t This can be an encoder or a nonlinear mapping module (possibly composed of multi-layer MLPs, Transformer blocks, etc.), whose purpose is to learn a transformation function to predict the structured representation of the target modality from source modal features. The target features are regressed point-by-point across multiple scales and time locations; the feature regression loss can be expressed as:

[0100]

[0101] in, Let be a two-dimensional set of time location u and scale index d. This represents the true feature vector extracted at time position u and scale layer d in the target modality. This represents the target mode features predicted from the feature vectors extracted from the source mode (e.g., from the source mode Z). source The estimated value at the same position (u,d) (mapped from).

[0102] For predictive features Apply target modal prior prediction head R t (·), and the true prior calculated on the synchronous data. Alignment, specifically, can be represented as:

[0103]

[0104] Among them, R t (·) represents the target modality prior predictor head, [·] m This indicates taking the m-th dimension from the prediction results. Let m be the m-th prior feature in the target mode.

[0105] To improve discriminative power, InfoNCE contrastive learning can be used to predict target features. Compared with the true target feature Z target They approach each other within the same projection space, while maintaining distance from the target features of other samples:

[0106]

[0107] Where φ(·) is the feature projection normalization operator, which projects the features into the embedding space of the contrastive learning, and τ is a temperature coefficient used to control the sharpness of the contrastive loss distribution. source Z target )∈ For a batch of positive samples paired together, <·,·> denote the dot product (inner product) between two vectors, Z′ target These are negative sample target modal features. Includes negative samples from the same batch.

[0108] Combining the above terms, the final alignment objective function can be:

[0109]

[0110] Weighting coefficient μ f μ p μ c It is a non-negative real number.

[0111] In addition, auxiliary alignment loss methods such as contrast loss, cyclic consistency, and knowledge distillation can be introduced to enhance the robustness and semantic consistency of alignment.

[0112] At this point, the encoder E, which has undergone single-modal pre-training and cross-modal alignment, is... joint The spatiotemporal coupling relationships between different modalities have been learned. Optionally, the model transfer and application steps focus on downstream tasks, combining the joint encoder with the classification head and using cross-entropy loss for fine-tuning:

[0113]

[0114] H = E joint (S′ source ;θ E )

[0115] Among them, E joint The term "joint encoder" refers to the fusion of multiple modal encoders and a cross-modal predictor Ψ, which have been trained through spatiotemporal feature extraction and cross-modal coupling alignment. s→t The structure and parameters enable implicit semantic mapping and feature fusion between modalities, and simultaneously model cross-scale and cross-temporal feature relationships among multiple modalities, S′ source This can refer to a single-mode or few-mode input signal, θ E For the joint encoder Ejoint The parameter set, H = [h1, h2, ..., h T The output temporal feature sequence is represented by ], where T represents the length of the input sequence (e.g., number of frames or time steps), and z is the feature convergence vector. These are attention weights, where q is a learnable query vector used to weight the temporal features H, and W... cls and b cls For classification header parameters, L task The cross-entropy loss is denoted by C, which represents the number of target classes in the downstream task, i.e., the total number of classes that the model ultimately needs to predict.

[0116] By employing the sliding scale segmentation, signal enhancement, staged MAE pre-training combined with prior feature prediction loss, and cross-modal coupling alignment on small-scale synchronous data as described in this invention, it is possible to: perform information dimensionality enhancement and semantic representation of high-dimensional target modalities (such as multi-channel EEG, multi-lead ECG, EMG, etc.) from low-dimensional, wearable sources (such as PPG, single-lead ECG, IMU, etc.); decompose the dependence on "large-scale synchronous multimodal data" into "large-scale asynchronous single-modal data" and "small-scale synchronous multimodal data", thereby reducing the data acquisition threshold and enhancing practicality; and improve the physiological and semantic consistency of cross-modal mapping and the performance of downstream tasks through prior prediction and prior alignment.

[0117] Example 2

[0118] This invention provides a spatiotemporal cross-scale dimensionality reduction representation system for multimodal physiological signals, comprising a forward device and a backward device. During the training phase, the forward device receives multimodal synchronous data and performs single-modal feature pre-training and cross-modal coupling alignment operations to learn the spatiotemporal dependencies and feature mapping rules between different modalities, constructing a unified joint encoder. During the deployment phase, the backward device, based on low-dimensional input signals from a single or a few modalities, utilizes the trained joint encoder to generate a compact dimensionality-reduced representation vector fusing features from multiple modalities, supporting real-time downstream task inference, such as sleep staging, emotion recognition, and health assessment. Through the cross-modal mapping capability of the joint encoder, the system can still construct a unified physiological state representation with high semantics and multiple scales even when some modalities are missing, reducing sensor complexity and improving edge inference efficiency and robustness.

[0119] The forward device may include, but is not limited to, the following modules:

[0120] (1) A multi-channel signal acquisition unit is used for synchronous / near-synchronous acquisition of multimodal signals. Examples of acquired signals include EEG, ECG (single-lead or multi-lead), PPG, respiration (RIP or flow), EDA, EMG, BP, body temperature, fNIRS, SpO2, IMU, etc. The acquisition unit can be composed of independent measurement devices, synchronous acquisition platforms, or distributed sensor networks using hardware triggering / timestamp synchronization. It supports configurable sampling rate and resolution to adapt to high-frequency (e.g., ECG, EEG) and low-frequency (e.g., body temperature, activity) modalities.

[0121] (2) The timing alignment unit is the physical entity that can be carried in step a) of Example 1. It can be used to achieve synchronization of multimodal signals by adopting methods based on cross-correlation function, dynamic time warping (DTW), event anchor registration, weighted linear regression, spline fitting or neural network alignment.

[0122] (3) Data preprocessing unit, which is the physical entity that can be carried in steps b) to c) of Example 1, can perform operations such as sliding scale segmentation, normalization, filtering, and signal enhancement. Signal enhancement may include additive noise, amplitude scaling, time clipping, time warping, frequency domain filtering, random occlusion, and hybrid enhancement.

[0123] (4) The training support unit, which is the physical entity that can carry out training (d) to (e) in Example 1), can input aligned and fused data (including physiological signals and discrete clinical features acquired by the multi-channel acquisition unit) into the cross-modal learning model and support single-modal feature pre-training and cross-modal coupling alignment. Clinical features include, but are not limited to: electronic medical record summaries (past medical history, family history, medication history), age, gender, race, height, weight, BMI, smoking / drinking history, comorbidities (hypertension, diabetes, etc.), past surgical history, drug or food allergies, major laboratory indicators (fasting blood glucose, blood oxygen, blood lipids, liver and kidney function, complete blood count, etc.), imaging / ultrasound examination summaries, lifestyle and activity level assessments, sleep and cognitive assessment scales, etc. The learning model can be based on a Transformer-based masked autoencoder (MAE), or can be implemented using convolutional neural networks (CNN), recurrent neural networks (RNN), graph neural networks (GNN), generative adversarial networks (GAN), variational autoencoders (VAE), self-distillation models, or combinations thereof.

[0124] The backward device is used during the deployment phase to represent the high-dimensional physiological state of the physical entity in step f) of Example 1 using low-dimensional source modal signals. It may include:

[0125] The signal acquisition unit is used to acquire easily obtainable low-dimensional or few-modal continuous physiological signals during the deployment phase (including but not limited to acquiring one or more of the following: PPG, single or few-lead ECG, single or few-channel EEG, skin resistance / skin impedance, electromyography, heart sounds, body temperature, blood pressure, blood glucose, blood oxygen, voice, gait, etc., including electrical, optical, acoustic, mechanical, biochemical, ultrasound, electromagnetic wave, and imaging signals). It is also compatible with various sensor types, installation locations, and data interfaces to ensure compatibility with consumer-grade, clinical-grade, and non-contact monitoring devices. Specifically, the acquisition unit may include, but is not limited to: wristbands, chest straps, patches / skin patches, ear-worn / in-ear devices, clothing / e-textile embedded sensors, chest patch / pleural electrode arrays, adhesive continuous glucose sensor (CGM), minimally invasive or implantable sensors, mattress / pressure pad and seat embedded sensors, portable or bedside clinical monitoring instruments, handheld ultrasound / Doppler probes and portable stethoscopes, as well as sensors for non-contact measurement (e.g., millimeter-wave radar, FMCW / 77GHz radar, Wi-Fi / wireless channel status information (CSI) passive detection, remote optical volumetric pulse wave imaging (rPPG) and video analysis from standard RGB / depth / infrared / thermal imaging cameras, acoustic / microphone arrays and laser Doppler refraction, etc.).

[0126] The model execution and computation unit can be deployed locally on the device, on an edge computing node, or in the cloud to run a trained cross-modal encoder and generate a comprehensive representation vector containing high-dimensional modal features. The encoder can be adapted to different hardware environments through model compression, pruning, quantization, or knowledge distillation techniques.

[0127] The application interface unit is used to input the comprehensive representation vector into several downstream task models (classifiers, regressors, sequence predictors, or generative models). Through prior prediction and alignment constraints, it improves the physiological and semantic consistency of cross-modal mapping and supports the improvement of the discriminative power and generalization performance of downstream tasks. These downstream tasks include, but are not limited to: cognitive load assessment, sleep staging, emotion / feeling recognition, stress detection, fatigue / attention monitoring, early disease warning, arrhythmia / respiratory abnormality detection, and intensive care support. The task models can be classifiers, regressors, sequence predictors, or generative models.

[0128] During the training phase, small-scale multimodal synchronous data provided by the feedforward device is combined with large-scale asynchronous single-modal data to complete the phased self-supervised learning of the cross-modal encoder. In the single-modal stage, the model learns the spatiotemporal dependencies and prior features of each modality, and in the cross-modal stage, it learns the mapping and alignment relationships between different modalities.

[0129] During the deployment phase, the backward device only needs to collect low-dimensional source modal signals to generate a comprehensive representation containing high-dimensional target modal features through the embedded representation unit, thereby achieving a comprehensive and multi-dimensional physiological state representation on consumer-grade devices that is close to that of professional devices.

[0130] Example 3

[0131] This invention provides a computing device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the method in Embodiment 1. The program code for implementing the aforementioned method can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the steps of the method of the invention to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0132] It should be understood that the embodiments and descriptions above only illustrate the principles, main features, and advantages of the present invention. All the terms "optional," "may be," and "may be adopted" indicate that equivalent or alternative technical routes can be used during implementation. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals, characterized in that, Includes the following steps: Timing alignment of multimodal signals; Single-modal spatiotemporal physiological signals are segmented using a sliding scale over time. Single-modal spatiotemporal feature pre-training, which is based on a single-modal dataset, trains the model to learn the temporal dependencies and spatial correlations within each modality by performing temporal mask prediction tasks and spatial mask prediction tasks; For each single modality, a corresponding encoder is trained. The encoder takes into account the spatiotemporal physiological signal sequence of the corresponding modality after sliding scale segmentation and outputs spatiotemporal physiological features. Information at different scales is input into the encoder through splicing, stacking or parallel multi-branching. Cross-modal coupling alignment, which is based on a time-synchronized multimodal dataset, incorporates discrete clinical features, and trains the model to learn the coupling relationship between different physiological signal modalities by performing a modality mask prediction task; during training, the spatiotemporal physiological features output by the encoder corresponding to the source modality are transformed by the cross-modal predictor and used for the target feature prediction task of the target modality that is different from the source modality. The encoders corresponding to one or more modalities, which have been trained through cross-modal coupling alignment, are applied to downstream tasks to generate a comprehensive representation of the overall physiological state based on the input of a single modal signal or multiple modal signals with fewer than the number of modalities used during training. The comprehensive loss for the cross-modal coupling alignment considers feature regression loss and prior feature alignment loss, or further considers cross-modal contrastive loss, wherein: Feature Regression Loss Defined as: ； in, For time position and scale index A two-dimensional set, Indicating the target mode, at time position and scale layer The real feature vector extracted from it. This indicates that the target mode features predicted from the feature vectors extracted from the source mode are at the same position. The estimated value; Prior feature alignment loss Defined as: ； in, The total number of prior features. For the target modality prior prediction head, , To represent the feature representation extracted from the source modes, As a cross-modal predictor, its function is to map source modal features to the target modal feature space. This indicates taking the first element from the prediction results. One dimension, For the first in the target mode A priori feature; Cross-modal contrast loss Defined as: ； in, This is a feature projection normalization operator, whose function is to project features into the embedding space of contrastive learning. This is a temperature coefficient used to control the sharpness of the contrast loss distribution. A batch paired with positive samples. Represents the inner product between two vectors. These are negative sample target modal features. Includes negative samples from the same batch.

2. The spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals according to claim 1, characterized in that, Multimodal signal timing alignment via event anchoring mapping, including: based on reference signal Timeline and the signal to be aligned The timeline is and the event anchors of both. time mapping function Perform spline regression to solve: ； in, For the reference mode, the first The timestamp of each event For the corresponding first mode in the mode to be aligned The timestamp of each event This represents the total number of observable anchor pairs. For indexes of event pairs, For function space, For the first The importance weight of each anchor point pair To smooth the coefficients of the regularization term, the smoothness of the time transform function is controlled; the aligned signal is: ； in, This represents the inverse function of the time mapping function.

3. The spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals according to claim 1, characterized in that, The slip scaling is achieved by setting a scale set. and the corresponding window length and step length , will signal At every scale The upper segment is divided into a series of fragments: ； in, Representing scale The first A segment, The current total signal length is defined as follows: step size is defined as... .

4. The spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals according to claim 1, characterized in that, Before pre-training the single-modal spatiotemporal features, signal enhancement is also included, which applies a set of enhancement operators to the segments before or after segmentation; the set of enhancement operators includes one or more of additive noise perturbation, amplitude scaling, time clipping and splicing, time warping, frequency domain filtering, and random occlusion.

5. The spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals according to claim 1, characterized in that, The loss in the single-modal spatiotemporal feature pre-training stage comprehensively considers temporal reconstruction loss and spatial reconstruction loss, or further considers prior feature prediction loss, wherein: Temporal reconstruction loss Defined as: ； in, This represents the set of signal segments that are masked in the time dimension. for A real signal segment, For the model to fragment The reconstruction results for The total number of elements; Space reconstruction loss Defined as: ； in, This represents a set of signal segments that are masked in a spatial dimension. for A real signal segment, For the model to fragment The reconstruction results for The total number of elements; Prior feature prediction loss Defined as: ； in, For model prediction priors, For true prior features, For the first The standard deviation of each prior feature on the training set is used for normalization.

6. The spatiotemporal cross-scale dimensionality reduction representation method for multimodal physiological signals according to claim 1, characterized in that, When applying encoders corresponding to one or more modalities, trained through cross-modal coupling and alignment, to downstream tasks, the joint encoder obtained through cross-modal coupling and alignment is used. Combined with the classification head, cross-entropy loss is used for fine-tuning to achieve cross-scale and cross-temporal feature mapping and prediction between modalities in downstream tasks: ； ； ； Among them, the joint encoder It integrates the structures and parameters of multiple modal encoders and cross-modal predictors trained through cross-modal coupling alignment. Indicates a single-mode or multi-mode input signal. For the joint encoder The set of parameters, The output is a time-series feature sequence, where Indicates the length of the input sequence. For feature convergence vectors, It is attention weight. It is a learnable query vector. and For classification header parameters, For cross-entropy loss, This represents the number of target categories in the downstream tasks.

7. A spatiotemporal cross-scale dimensionality reduction characterization system for multimodal physiological signals, used to implement the method according to any one of claims 1-6, characterized in that, Including forward and / or backward devices, The forward device is used to receive multimodal synchronous data during the training phase, and to perform single-modal feature pre-training and cross-modal coupling alignment operations to learn the spatiotemporal dependencies and feature mapping rules between different modalities and to construct a unified joint encoder. The backward device is used during the deployment phase to generate a dimensionality-reduced representation vector that fuses features from multiple modalities based on a single or fewer modal input signal used during training, using a trained joint encoder to support real-time downstream task inference.

8. The system according to claim 7, characterized in that, The forward device includes: a multi-channel signal acquisition unit for acquiring one or more multimodal physiological signals; a temporal alignment unit for temporally aligning the multimodal signals; a data preprocessing unit for sliding scale segmentation, or sliding scale segmentation combined with signal enhancement; and a training support unit for providing aligned multimodal data and clinical features to the cross-modal self-supervised learning model. The backward device includes: a signal acquisition unit for acquiring continuous physiological signals; a model running and computing unit for running a trained encoder model locally, at the edge, or in the cloud to generate a comprehensive representation vector containing cross-modal spatiotemporal features; and an application interface unit for applying the comprehensive representation vector to downstream tasks.

9. A computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

Images