A cross-modal generation method and system based on multi-modal biological signals

Abstract

This invention relates to the field of biosignal processing technology and discloses a method and system for cross-modal generation based on multimodal biosignals. The method includes: acquiring multimodal biosignals and corresponding heterogeneous metadata; preprocessing the multimodal biosignals and inputting them into a feature encoder to extract initial latent features for each modality; mapping each initial latent feature to a Gaussian latent space to obtain latent feature distribution parameters corresponding to each modality; performing feature completion on missing or masked modalities and fusing the completed latent features of each modality to obtain unified multimodal aligned latent features; embedding and encoding the heterogeneous metadata to obtain metadata latent features; inputting the multimodal aligned latent features and the metadata latent features into a heterogeneous information fusion module for cross-modal interaction to obtain fused latent features; and inputting the joint latent features into a decoder to generate a target modality biosignal, thereby achieving accurate cross-modal fusion generation of multimodal signals.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biological signal processing, and in particular to a cross-modal generation method and system based on multi-modal biological signals. BACKGROUND

[0002] Cardiovascular disease is one of the diseases with the highest mortality rate worldwide, and its early detection and dynamic monitoring are of great significance for reducing mortality and improving patient prognosis. Clinical practice shows that a single modality of biological signal (such as electrocardiogram ECG, photoplethysmogram PPG, or arterial blood pressure ABP) often fails to fully reflect the complex physiological state of the human body. In contrast, the joint analysis of multi-modal biological signals can provide more abundant and complementary cardiovascular function information, and has become an important development direction in the field of precision medicine and intelligent monitoring. With the popularity of wearable devices and intensive care systems, the collection of multi-modal biological signals has been scaled up, providing a data basis for subsequent intelligent analysis.

[0003] However, the existing technology still has several technical bottlenecks in the fusion and cross-modal generation of multi-modal biological signals, which limits its popularization and application in practical scenarios. First, different modalities of biological signals have different physical meanings and sampling characteristics, such as ECG reflecting cardiac electrical activity, PPG reflecting peripheral blood flow changes, and ABP reflecting blood vessel pressure changes. The feature mapping relationship between modalities is extremely complex, and there is a lack of a unified feature representation space, making it difficult to effectively align multi-modal information and fully exploit the complementary physiological information between modalities. Second, the analysis of biological signals often needs to combine individual heterogeneous metadata, such as age, gender, and circadian rhythm information. Such data has a large difference in dimension and distribution compared to waveform signals, and traditional fusion methods often use simple concatenation or weighting operations, which can easily cause the loss of key information or the introduction of noise, affecting the physiological authenticity and phenotype relevance of the generated signal. In addition, in actual clinical monitoring or home monitoring scenarios, factors such as poor sensor contact and human motion interference often lead to missing or degraded quality of some modal data, and existing models have poor adaptability to missing modalities, making it difficult to achieve a combination of modal inputs and robust generation.

[0004] In summary, how to achieve fine alignment of multi-modal biological signals in a unified Gaussian latent space, how to efficiently fuse heterogeneous metadata and signal features, and how to achieve a combination of cross-modal generation under the condition of missing modalities are the technical problems that need to be solved. SUMMARY

[0005] The purpose of this invention is to address the shortcomings of the existing technologies by providing a cross-modal generation method based on multimodal biological signals. This method enables refined alignment of multimodal biological signals, efficient fusion of biological signals and heterogeneous metadata, and composable generation under missing modalities. Ultimately, it improves the physiological authenticity, phenotypic correlation, and model robustness of cross-modal generated signals, providing highly reliable biological signal generation technology support for clinical monitoring and home health monitoring.

[0006] On the one hand, the present invention provides a cross-modal generation method based on multimodal biological signals, comprising the following steps: S1: Acquire multimodal biological signals and corresponding heterogeneous metadata, preprocess the multimodal biological signals, input the preprocessed multimodal biological signals into the feature encoder, and extract the initial latent features of each modality. S2: Based on the variational autoencoder architecture, the initial latent features are mapped to the Gaussian latent space to obtain the latent feature distribution parameters corresponding to each mode. A mask fusion strategy is adopted to complete the features of the missing or masked modes based on the latent features of the available modes, and the latent features of each mode after completion are fused to obtain unified multimodal aligned latent features. S3: Embed and encode the heterogeneous metadata to obtain metadata latent features; S4: Input the multimodal aligned latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction to obtain fused latent features; S5: Input the joint latent features into the decoder to generate the biological signal of the target modality.

[0007] Further, in step S1, inputting the preprocessed biomodal signals of each modality into the feature encoder includes: The preprocessed time-series biological signal is input into a one-dimensional convolutional layer. Local waveform features are extracted through convolution operations and then processed by batch normalization and ReLU activation function to obtain a shallow feature map. The shallow feature map is input into multiple serially connected residual blocks for multi-scale downsampling, gradually expanding the receptive field and extracting deep semantic features; The feature sequence output by the residual block is input into a bidirectional LSTM network, which captures global temporal dependency features by propagating temporal information in both forward and backward directions. The outputs of the bidirectional LSTM network at each time step are aggregated into a fixed-dimensional feature vector through an adaptive average pooling layer, which serves as the initial latent feature for this modality.

[0008] Further, in step S2, mapping each of the initial latent features to a Gaussian latent space includes: The initial latent features of each modality are input into a multilayer perceptron of a variational autoencoder. Through reparameterization, the mean vector of each modality's latent features in the Gaussian latent space is calculated. Sum of variance vectors ; wherein, the mean vector Sum of variance vectors Used to describe the distribution of latent features for each modality.

[0009] Further, in step S2, the unified multimodal alignment latent feature acquisition process includes: Random masking is performed on the latent features of each modality to simulate a modality missing scenario; The unmasked available modal latent features are input into the feature completion network, and the complementary information of the available modal features is used to complete the latent features of the masked region, thus obtaining the complete modal latent features. All the completed modal latent features are concatenated or weighted and summed to form a unified Gaussian latent feature, which is used as the multimodal aligned latent feature. During the model training phase, by minimizing the reconstruction loss KL divergence loss and comparative loss The joint loss function optimizes the alignment between feature encoding and latent space. The three loss functions are expressed as follows: , , , in, Indicates the first output by the decoder Reconstructed biosignals from individual samples Indicates the first The original input biological signal of each sample, The number of samples; For the first The latent feature vector of mode A of a sample. For the first The latent feature vector of mode B of each sample. It is an exponential function. These are scalar hyperparameters used to adjust the model's sensitivity to or focus on negative samples.

[0010] Preferably, in step S3, the embedding and encoding of the heterogeneous metadata includes: Retrieve heterogeneous metadata including gender, age, and timestamp; Encode according to data type, including: Binary embedding encoding is used for gender, numerical mapping encoding is used for age, and trigonometric function position embedding encoding is used for timestamps using sine and cosine functions to preserve their temporal continuity and periodicity. The encoded metadata features are concatenated to form a unified metadata latent feature.

[0011] Further, in step S4, inputting the multimodal alignment latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction includes: Perform a discrete Fourier transform on the multimodal aligned latent features to map them to the frequency domain; In the frequency domain, the transformed features are filtered by a learnable multi-head frequency filter to extract key features in the frequency domain. The filtered features are subjected to inverse Fourier transform and mapped back to the spatial domain to obtain frequency-enhanced biological signal features. A cross-attention mechanism is constructed, using the frequency-enhanced biological signal features as the query matrix and the metadata latent features as the key matrix and value matrix. Through attention calculation, the deep fusion of biological signal features and heterogeneous metadata features is achieved, and the fused latent features are output.

[0012] Further, in step S5, the biosignal for generating the target modality includes: The fused latent features are input into the expert product module, where the latent features of each available modality are regarded as an expert, and each expert corresponds to a Gaussian distribution; The joint Gaussian distribution of all available modes is calculated using the expert product algorithm to obtain the joint latent features. The calculation formula is as follows: , Where is the number of available modes. For the first The expert Gaussian distribution corresponding to each mode and Let be the mean and variance of the joint distribution, respectively. This represents the latent features after multimodal fusion. This means that the two sides of the equation differ only by a normalization constant; The joint latent features are input into a signal decoder composed of a fully convolutional network, and the temporal biological signal of the target modality is decoded and reconstructed according to the preset downstream task requirements.

[0013] Preferably, in step S5, the generation of the target modality's biosignal further includes model optimization, specifically including: The first stage is the pre-training stage, where the parameters of the signal decoder are frozen, and the parameters of the feature encoder and variational autoencoder modules are updated with signal reconstruction as the optimization goal. The second stage is the fine-tuning stage, in which the parameters of the thawing signal decoder and the heterogeneous information fusion module are optimized with cross-modal generation as the goal, and the parameters of the entire network model are updated, ultimately outputting the target modality biological signal that conforms to physiological characteristics.

[0014] Further, in step S1, the preprocessing includes: The quality of the acquired electrocardiograms, photoplethysmograms, and arterial blood pressure signals was assessed to filter out flat waves, baseline drift, and outlier segments. The signal is divided into fixed time windows, resampled to a uniform number of sampling points, and normalized to obtain a standardized input signal.

[0015] On the other hand, the present invention provides a cross-modal generation system based on multimodal biological signals, comprising: The signal acquisition and preprocessing module is used to acquire multimodal biological signals and corresponding heterogeneous metadata, preprocess the multimodal biological signals, and input the preprocessed biological signals of each modality into the feature encoder to extract the initial latent features of each modality. The latent space mapping module is used to map each of the initial latent features to the Gaussian latent space based on the variational autoencoder architecture, so as to obtain the latent feature distribution parameters corresponding to each mode. The mask fusion and metadata embedding module is used to employ a mask fusion strategy to complete the features of missing or masked modalities based on the latent features of available modalities, and to fuse the latent features of each completed modality to obtain unified multimodal aligned latent features. At the same time, the heterogeneous metadata is embedded and encoded to obtain metadata latent features. The heterogeneous information fusion module is used to input the multimodal aligned latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction to obtain fused latent features; The signal generation and decoding module is used to input the joint latent features into the decoder to generate the biological signal of the target modality.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention solves the problem of difficulty in aligning signals such as ECG, PPG, and ABP due to their different physical meanings by mapping the initial latent features of each modality to a Gaussian latent space, thus laying the foundation for subsequent fusion. This invention enables the model to complete the missing modalities based on the available modalities through a mask fusion strategy, adapting to real-world scenarios such as sensor detachment and data noise, thus improving the practicality of the method. This invention embeds and encodes metadata and interacts with signal features through a heterogeneous information fusion module, making the generated results more consistent with individual physiological characteristics and time series patterns. This invention provides a complete process from signal input to target signal generation, eliminating the need for manual design of mapping rules and offering a highly reliable signal generation solution for clinical monitoring and home surveillance. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a cross-modal generation method based on multimodal biological signals according to the present invention; Figure 2 This is a schematic diagram of the complete structure of a multimodal biosignal feature encoder according to the present invention; Figure 3 This is a schematic diagram of a multimodal Gaussian latent space alignment process based on mask fusion according to the present invention; Figure 4 This is a schematic diagram of a frequency domain cross-attention fusion module according to the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This invention is designed for high-precision monitoring scenarios within hospitals, requiring real-time acquisition of multimodal signals such as ECG, PPG, and ABP. However, sensor detachment or changes in patient positioning can easily lead to the loss of some modal data. Furthermore, clinical diagnosis requires comprehensive analysis combining patient age, gender, and circadian rhythms (such as nighttime blood pressure fluctuations), demanding extremely high physiological accuracy and phenotypic relevance in signal generation. The network environment is a local area network within the hospital, requiring the model to have low latency and high reliability. It can also be used for low-cost monitoring scenarios outside hospitals, primarily acquiring easily obtainable PPG signals through smart wearable devices (wristbands, watches), but struggling to acquire specialized signals such as ECG and ABP. Wearable devices are susceptible to human movement and environmental interference, resulting in high signal noise. The network environment is mobile internet, requiring a lightweight model with low computational load and noise resistance.

[0020] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. Example 1

[0021] Please see Figure 1The technical solution of the cross-modal generation method based on multimodal biological signals provided in this embodiment includes the following steps: S1: Acquire multimodal biological signals and corresponding heterogeneous metadata, preprocess the multimodal biological signals, input the preprocessed multimodal biological signals into the feature encoder, and extract the initial latent features of each modality.

[0022] The preprocessing includes: The quality of the acquired electrocardiograms, photoplethysmograms, and arterial blood pressure signals was assessed to filter out flat waves, baseline drift, and outlier segments. The signal is divided into fixed time windows, resampled to a uniform number of sampling points, and normalized to obtain a standardized input signal.

[0023] In this embodiment, we first collect ECG, PPG, and ABP multimodal biological signals, perform quality assessment on the signals (filtering out low-quality segments such as flattening, baseline drift, and outliers), segment the signals into 10-second time windows, resample to 512 sampling points, and perform normalization processing; at the same time, we collect heterogeneous metadata, including age (numerical), gender (binary), and signal acquisition timestamp (time series), forming a multimodal biological signal dataset and a metadata dataset to be processed.

[0024] Then, a feature encoder consisting of convolutional layers, residual blocks, and a bidirectional LSTM is constructed, with the complete structure as follows: Figure 2 As shown. Among them, Figure 2 The left side shows the LSTM time series feature extraction module, which uses a bidirectional LSTM (BiLSTM) structure, including a forward LSTM and a backward LSTM: the forward LSTM processes the time series signal from left to right. Output positive memory unit and hidden state Inverse LSTM processes timing signals from right to left. Output reverse memory unit and hidden state Meanwhile, the LSTM temporal feature extraction module demonstrates its internal gating mechanism (forget gate). Input gate Output gate (cell state updates), which is key to capturing long-term time dependencies. Figure 2The middle section describes the overall process of the Encoder, which aims to compress the input signal into multi-scale deep features. The input signal first undergoes initial one-dimensional convolution via Conv1d → BN → ReLU to complete feature mapping and normalization. Then, four Multi-scale ResNetBlocks (an improved module based on the standard ResNet residual network, designed to enhance the model's ability to extract multi-scale features) are stacked. In this embodiment, the Multi-scale ResNetBlocks perform Conv1d+BN in parallel with four different convolution kernels (1×1, 3×1, 7×1, 11×1) to extract multi-scale features. After concatenation, the features are processed through ReLU → Conv1d → BN → ReLU to obtain the final multi-scale feature map for the decoder. Then, a BiLSTM layer is added to fuse the local features extracted by convolution with global temporal dependencies. After that, Average Pooling and Max Pooling are performed in parallel to capture global average features and significant peak features, respectively. Finally, these features are fused by concatenation (C symbol). Figure 2 On the right is the Decoder, which is responsible for reconstructing latent embeddings into an output signal: First, the input latent embeddings are mapped to dimensions through a Linear layer; the Conv1d → BN → Multi-scale ResNetBlock structure is repeated 4 times to gradually upsample and restore details; finally, the final Output Signal is output through Conv1d → BN → ReLU → Conv1d, completing the signal reconstruction.

[0025] Based on this structure, the preprocessed biomodal signals are encoded separately: first, local spatial features are extracted by two layers of one-dimensional convolution + batch normalization + ReLU activation; then, multi-scale downsampling is performed by four residual blocks; next, global temporal dependent features are captured by bidirectional LSTM; and finally, the initial latent features of each modality are obtained by adaptive average pooling, realizing the mapping from the original temporal signal to high-dimensional features.

[0026] Specifically, the preprocessed biomodal signals are input into the feature encoder, including: The preprocessed time-series biological signal is input into a one-dimensional convolutional layer. Local waveform features are extracted through convolution operations and then processed by batch normalization and ReLU activation function to obtain a shallow feature map. The shallow feature map is input into multiple serially connected residual blocks for multi-scale downsampling, gradually expanding the receptive field and extracting deep semantic features; The feature sequence output by the residual block is input into a bidirectional LSTM network, which captures global temporal dependency features by propagating temporal information in both forward and backward directions. The outputs of the bidirectional LSTM network at each time step are aggregated into a fixed-dimensional feature vector through an adaptive average pooling layer, which serves as the initial latent feature for this modality.

[0027] S2: Based on the variational autoencoder architecture, the initial latent features are mapped to the Gaussian latent space to obtain the latent feature distribution parameters corresponding to each mode. A masking fusion strategy is adopted to complete the features of the missing or masked modes based on the latent features of the available modes, and the latent features of each mode after completion are fused to obtain unified multimodal aligned latent features.

[0028] In this step, firstly, mapping each of the initial latent features to a Gaussian latent space includes: The initial latent features of each modality are input into the multilayer perceptron (MLP) of the variational autoencoder (VAE). Through reparameterization, the mean vector of the latent features of each modality in the Gaussian latent space is calculated. Sum of variance vectors ; wherein, the mean vector Sum of variance vectors Used to describe the distribution of latent features for each modality.

[0029] Next, multimodal alignment is performed. The unified multimodal alignment latent feature acquisition process includes: Random masking is performed on the latent features of each modality to simulate a modality missing scenario; The unmasked available modal latent features are input into the feature completion network, and the complementary information of the available modal features is used to complete the latent features of the masked region, thus obtaining the complete modal latent features. All the completed modal latent features are concatenated or weighted and summed to form a unified Gaussian latent feature, which is used as the multimodal aligned latent feature. During the model training phase, by minimizing the reconstruction loss KL divergence loss and comparative loss The joint loss function optimizes the alignment between feature encoding and latent space. The three loss functions are expressed as follows: , , , in, Indicates the first output by the decoder Reconstructed biosignals from individual samples Indicates the first The original input biological signal of each sample, The number of samples; For the first The latent feature vector of mode A of a sample. For the first The latent feature vector of mode B of each sample. It is an exponential function. These are scalar hyperparameters used to adjust the model's sensitivity to or focus on negative samples.

[0030] See details Figure 3 The diagram illustrates the multimodal alignment process for mask fusion. The first step is the reparameterization of the input domain latent space: The input consists of multimodal biosignal features from the encoder, including ECG, PPG, and ABP modalities; each feature is passed through two linear layers, and the output is the mean. and variance Perform reparameterization techniques The features are mapped to a Gaussian latent space to obtain the latent features of each modality. , , (Corresponding to ECG, PPG, and ABP respectively). The loss formula for this step is the KL divergence loss. The constraint ensures that the distribution of latent features approximates the standard normal distribution, thus guaranteeing the regularity of the latent space. Figure 3 The next step is random masking and feature fusion: The latent features of each modality are randomly masked (the diagonal stripes in the figure are masked features, and the solid color blocks are unmasked features) to simulate the situation where some modalities are missing in real-world scenarios. The unmasked features are then concatenated / fused across modalities to obtain a complete sequence of latent features for subsequent decoding. Masking forces the model to learn complementary information between different modalities, thereby improving the robustness of cross-modal generation. Next is the decoder and reconstruction loss. : The fused latent features are input into the decoder to reconstruct the target modality signal. The corresponding loss function here is the reconstruction loss. This measures the difference between the original signal and the reconstructed signal to ensure the accuracy of feature decoding; Finally, compare the losses. : The latent features are segmented (Segment 1~4), and contrastive learning is performed in the feature space. The Push operation means that similar segments of features from different modalities are brought closer together (positive sample pairs), and dissimilar segments are moved away from each other (negative sample pairs).

[0031] S3: Embed and encode the heterogeneous metadata to obtain the metadata latent features.

[0032] Specifically, the embedding and encoding of the heterogeneous metadata includes: Retrieve heterogeneous metadata including gender, age, and timestamp; Encode according to data type, including: Binary embedding encoding is used for gender, numerical mapping encoding is used for age, and trigonometric function position embedding encoding is used for timestamps using sine and cosine functions to preserve their temporal continuity and periodicity. The encoded metadata features are concatenated to form a unified metadata latent feature.

[0033] In this embodiment, heterogeneous metadata is specifically embedded and encoded: gender is embedded using binary embedding, age is embedded using numerical embedding, and timestamps are embedded using sine / cosine trigonometric functions (preserving the continuity and periodicity of time sequence and solving the problem of timestamp value jump at night); the embedded metadata features are concatenated into unified metadata latent features to complete the featureization of heterogeneous metadata.

[0034] S4: Input the multimodal aligned latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction to obtain fused latent features.

[0035] In this step, inputting the multimodal aligned latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction includes: Perform a Discrete Fourier Transform (DFT) on the multimodal aligned latent features to map them to the frequency domain; In the frequency domain, the transformed features are filtered by a learnable multi-head frequency filter to extract key features in the frequency domain. The filtered features are subjected to inverse Fourier transform (IDFT) to map them back to the spatial domain, resulting in frequency-enhanced biological signal features. A cross-attention mechanism is constructed, using the frequency-enhanced biological signal features as the query matrix (Q) and the metadata latent features as the key matrix (K) and value matrix (V). Through attention calculation, the deep fusion of biological signal features and heterogeneous metadata features is achieved, and the fused latent features are output.

[0036] like Figure 4As shown, the core idea is to deeply integrate biological signal features with heterogeneous metadata in the frequency domain and through an attention mechanism, replacing traditional spatial domain self-attention with frequency domain operations. Figure 4 The module shown is disassembled as follows: The input includes: Biosignal Z (such as temporal features of ECG / PPG / ABP) from the encoder, which serves as the input for frequency domain attention; and Metadata C (such as age, gender, disease labels, etc.), which is first processed by the encoding module to encode binary features and numeric features respectively, and then sine / cosine positional encoding is added. Finally, the metadata features are output as the key (K) and value (V) for cross-attention. The system comprises two core modules: Multi-head FFT-attention and Multi-head cross-attention. Multi-head FFT-attention captures key patterns in biological signals, such as periodicity and rhythm (e.g., heart rate, respiratory rate), in the frequency domain while reducing computational complexity. This includes: Performing a Discrete Fourier Transform (DFT) on the biological signal feature Z maps the signal to the frequency domain; in the frequency domain, learnable parameters are then used to... and Matrix operations are performed to achieve attention weighting in the frequency domain; the enhanced signal features are then mapped back to the time domain via inverse discrete Fourier transform (IDFT). ; Among them, Multi-head cross-attention uses biological signal features enhanced by frequency domain attention as input. The query (Q), keys (K), and values ​​(V) from the encoded metadata features enable biosignal features to actively focus on task-related metadata information, achieving deep fusion of heterogeneous information. For example, ECG signal features can be associated with metadata such as "age" and "history of hypertension". Next With the original The residuals are summed to preserve the original signal features. Then, the cross-attention output is summed with the residuals of the above results to finally obtain Fused Embeddings, which are the final features that fuse the temporal features of biological signals and the semantic information of metadata.

[0037] S5: Input the joint latent features into the decoder to generate the biological signal of the target modality.

[0038] In this final step, the biosignal used to generate the target modality includes: The fused latent features are input into the expert product module (PoE), where the latent features of each available modality are regarded as an expert, and each expert corresponds to a Gaussian distribution; The joint Gaussian distribution of all available modes is calculated using the Expert Product Algorithm (PoE) (solving for the joint mean and variance), yielding the joint latent features. The calculation formula is as follows: , Where is the number of available modes. For the first The expert Gaussian distribution corresponding to each mode and Let be the mean and variance of the joint distribution, respectively. This represents the latent features after multimodal fusion. This means that the two sides of the equation differ only by a normalization constant; The joint latent features are input into a signal decoder composed of a fully convolutional network, and the temporal biological signal of the target modality is decoded and reconstructed according to the preset downstream task requirements.

[0039] In addition, this step also includes model optimization, specifically including: The first stage is the pre-training stage, where the parameters of the signal decoder are frozen, and the parameters of the feature encoder and variational autoencoder modules are updated with signal reconstruction as the optimization goal. The second stage is the fine-tuning stage. The parameters of the thaw signal decoder and the heterogeneous information fusion module are optimized with cross-modal generation as the goal, updating the parameters of the entire network model and ultimately outputting a target modality biological signal that conforms to physiological characteristics. This final output can be directly used for downstream tasks such as clinical phenotypic analysis and arrhythmia monitoring.

[0040] In summary, this invention revolves around five major steps: multimodal feature encoding, mask fusion modality alignment, heterogeneous metadata embedding, frequency domain cross-attention fusion, and PoE composable generation. The overall data flow is as follows: original multimodal biological signal + heterogeneous metadata → feature encoding → Gaussian latent space alignment → heterogeneous information fusion → composable latent feature generation → cross-modal signal decoding → output of generation results.

[0041] Based on the above method, the present invention also provides a cross-modal generation system based on multimodal biological signals, comprising: The signal acquisition and preprocessing module is used to acquire multimodal biological signals and corresponding heterogeneous metadata, preprocess the multimodal biological signals, and input the preprocessed biological signals of each modality into the feature encoder to extract the initial latent features of each modality. The latent space mapping module is used to map each of the initial latent features to the Gaussian latent space based on the variational autoencoder architecture, so as to obtain the latent feature distribution parameters corresponding to each mode. The mask fusion and metadata embedding module is used to employ a mask fusion strategy to complete the features of missing or masked modalities based on the latent features of available modalities, and to fuse the latent features of each completed modality to obtain unified multimodal aligned latent features. At the same time, the heterogeneous metadata is embedded and encoded to obtain metadata latent features. The heterogeneous information fusion module is used to input the multimodal aligned latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction to obtain fused latent features; The signal generation and decoding module is used to input the joint latent features into the decoder to generate the biological signal of the target modality.

[0042] It should be noted that the steps in the cross-modal generation method based on multimodal biological signals provided in this embodiment can be implemented based on the corresponding modules in the cross-modal generation system based on multimodal biological signals. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0043] Besides implementing the system and its various devices provided by this invention in purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the system and its various devices of this invention appear as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0044] Finally, it should be noted that the above description is only a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be pointed out that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

[0045] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A cross-modal generation method based on multimodal biological signals, characterized in that, Includes the following steps: S1: Acquire multimodal biological signals and corresponding heterogeneous metadata, preprocess the multimodal biological signals, input the preprocessed multimodal biological signals into the feature encoder, and extract the initial latent features of each modality. S2: Based on the variational autoencoder architecture, the initial latent features are mapped to the Gaussian latent space to obtain the latent feature distribution parameters corresponding to each mode. A mask fusion strategy is adopted to complete the features of the missing or masked modes based on the latent features of the available modes, and the latent features of each mode after completion are fused to obtain unified multimodal aligned latent features. S3: Embed and encode the heterogeneous metadata to obtain metadata latent features; S4: Input the multimodal aligned latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction to obtain fused latent features; S5: Input the joint latent features into the decoder to generate the biological signal of the target modality.

2. The cross-modal generation method based on multimodal biological signals according to claim 1, characterized in that, In step S1, the preprocessed bio-signals of each modality are input into the feature encoder, including: The preprocessed time-series biological signal is input into a one-dimensional convolutional layer. Local waveform features are extracted through convolution operations and then processed by batch normalization and ReLU activation function to obtain a shallow feature map. The shallow feature map is input into multiple serially connected residual blocks for multi-scale downsampling, gradually expanding the receptive field and extracting deep semantic features; The feature sequence output by the residual block is input into a bidirectional LSTM network, which captures global temporal dependency features by propagating temporal information in both forward and backward directions. The outputs of the bidirectional LSTM network at each time step are aggregated into a fixed-dimensional feature vector through an adaptive average pooling layer, which serves as the initial latent feature for this modality.

3. The cross-modal generation method based on multimodal biological signals according to claim 1, characterized in that, In step S2, mapping each of the initial latent features to a Gaussian latent space includes: The initial latent features of each modality are input into a multilayer perceptron of a variational autoencoder. Through reparameterization, the mean vector of each modality's latent features in the Gaussian latent space is calculated. Sum of variance vectors ; wherein, the mean vector Sum of variance vectors Used to describe the distribution of latent features for each modality.

4. The cross-modal generation method based on multimodal biological signals according to claim 3, characterized in that, In step S2, the unified multimodal alignment latent feature acquisition process includes: Random masking is performed on the latent features of each modality to simulate a modality missing scenario; The unmasked available modal latent features are input into the feature completion network, and the complementary information of the available modal features is used to complete the latent features of the masked region, thus obtaining the complete modal latent features. All the completed modal latent features are concatenated or weighted and summed to form a unified Gaussian latent feature, which is used as the multimodal aligned latent feature. During the model training phase, by minimizing the reconstruction loss KL divergence loss and comparative loss The joint loss function optimizes the alignment between feature encoding and latent space. The three loss functions are expressed as follows: ， ， ， in, Indicates the first output by the decoder Reconstructed biosignals from individual samples Indicates the first The original input biological signal of each sample, The number of samples; For the first The latent feature vector of mode A of a sample, For the first The latent feature vector of mode B of each sample. It is an exponential function. These are scalar hyperparameters used to adjust the model's sensitivity to or focus on negative samples.

5. The cross-modal generation method based on multimodal biological signals according to claim 1, characterized in that, In step S3, the embedding and encoding of the heterogeneous metadata includes: Retrieve heterogeneous metadata including gender, age, and timestamp; Encode according to data type, including: Binary embedding encoding is used for gender, numerical mapping encoding is used for age, and trigonometric function position embedding encoding is used for timestamps using sine and cosine functions to preserve their temporal continuity and periodicity. The encoded metadata features are concatenated to form a unified metadata latent feature.

6. The cross-modal generation method based on multimodal biological signals according to claim 1, characterized in that, In step S4, inputting the multimodal alignment latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction includes: Perform a discrete Fourier transform on the multimodal aligned latent features to map them to the frequency domain; In the frequency domain, the transformed features are filtered by a learnable multi-head frequency filter to extract key features in the frequency domain. The filtered features are subjected to inverse Fourier transform and mapped back to the spatial domain to obtain frequency-enhanced biological signal features. A cross-attention mechanism is constructed, using the frequency-enhanced biological signal features as the query matrix and the metadata latent features as the key matrix and value matrix. Through attention calculation, the deep fusion of biological signal features and heterogeneous metadata features is achieved, and the fused latent features are output.

7. The cross-modal generation method based on multimodal biological signals according to claim 4, characterized in that, In step S5, the biosignal for generating the target modality includes: The fused latent features are input into the expert product module, where the latent features of each available modality are regarded as an expert, and each expert corresponds to a Gaussian distribution; The joint Gaussian distribution of all available modes is calculated using the expert product algorithm to obtain the joint latent features. The calculation formula is as follows: ， Where is the number of available modes. For the first The expert Gaussian distribution corresponding to each mode and Let be the mean and variance of the joint distribution, respectively. This represents the latent features after multimodal fusion. This means that the two sides of the equation differ only by a normalization constant; The joint latent features are input into a signal decoder composed of a fully convolutional network, and the temporal biological signal of the target modality is decoded and reconstructed according to the preset downstream task requirements.

8. The cross-modal generation method based on multimodal biological signals according to claim 7, characterized in that, In step S5, the generation of the target modality's biosignal also includes model optimization, specifically including: The first stage is the pre-training stage, where the parameters of the signal decoder are frozen, and the parameters of the feature encoder and variational autoencoder modules are updated with signal reconstruction as the optimization goal. The second stage is the fine-tuning stage, in which the parameters of the thawing signal decoder and the heterogeneous information fusion module are optimized with cross-modal generation as the goal, and the parameters of the entire network model are updated, ultimately outputting the target modality biological signal that conforms to physiological characteristics.

9. The cross-modal generation method based on multimodal biological signals according to claim 1, characterized in that, In step S1, the preprocessing includes: The quality of the acquired electrocardiograms, photoplethysmograms, and arterial blood pressure signals was assessed to filter out flat waves, baseline drift, and outlier segments. The signal is divided into fixed time windows, resampled to a uniform number of sampling points, and normalized to obtain a standardized input signal.

10. A cross-modal generation system based on multimodal biological signals, characterized in that, include: The signal acquisition and preprocessing module is used to acquire multimodal biological signals and corresponding heterogeneous metadata, preprocess the multimodal biological signals, and input the preprocessed biological signals of each modality into the feature encoder to extract the initial latent features of each modality. The latent space mapping module is used to map each of the initial latent features to the Gaussian latent space based on the variational autoencoder architecture, so as to obtain the latent feature distribution parameters corresponding to each mode. The mask fusion and metadata embedding module is used to employ a mask fusion strategy to complete the features of missing or masked modalities based on the latent features of available modalities, and to fuse the latent features of each completed modality to obtain unified multimodal aligned latent features. At the same time, the heterogeneous metadata is embedded and encoded to obtain metadata latent features. The heterogeneous information fusion module is used to input the multimodal aligned latent features and the metadata latent features into the pre-built heterogeneous information fusion module for cross-modal interaction to obtain fused latent features; The signal generation and decoding module is used to input the joint latent features into the decoder to generate the biological signal of the target modality.

Images