Multi-modal physiological signal processing method, device, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a multi-modal physiological signal processing method, device, equipment and medium, comprising: acquiring and preprocessing multi-modal physiological signal data, generating a data segment, executing data enhancement, and generating a damaged data segment. Inputting each physiological signal into a modal encoder, generating a modal exclusive representation, aggregating a non-target modal representation, generating a positive sample aggregation representation, inputting a damaged data segment into a decoder, generating a reconstructed data segment, adjusting a representation similarity, optimizing the modal encoder and the decoder, and generating a target task result based on the updated modal encoder. According to the method, by enhancing the similarity between the target modality and the positive sample aggregation representation, reducing the similarity between the target modality and other data segments and optimizing the updating process of the modal encoder and decoder, the processing effect of the target task and the migration ability between different tasks are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal physiological signal processing method, device, equipment and storage medium. Background Art

[0002] In the healthcare business, the diagnosis and treatment of sleep disorders has always been a challenging task. Traditional sleep monitoring methods rely on manual analysis of multiple physiological signals, such as electroencephalogram (EEG), electrocardiogram (ECG), and respiratory signals. These signals provide important information about sleep status, but due to the complex correlation between different signals, single-modality analysis often has difficulty capturing subtle changes during sleep. In addition, traditional sleep data analysis relies on a large number of labeled samples. However, medical data labeling is expensive and subject to significant subjective differences, resulting in the quality and quantity of labeled samples often failing to meet the requirements of efficient model training.

[0003] Currently, existing technologies for sleep monitoring generally use supervised learning methods, which require large amounts of labeled data. However, in the healthcare field, especially in sleep data analysis, obtaining labeled data is not only time-consuming and labor-intensive, but also susceptible to subjective judgment by professionals, which affects the model's generalization and application scope. More importantly, traditional methods are difficult to transfer to other related tasks, such as apnea detection and sleep staging. Therefore, existing technologies suffer from high data labeling costs, poor model transferability, and insufficient generalization capabilities.

[0004] In the fintech sector, with the widespread adoption of smart health monitoring devices, financial institutions are beginning to incorporate health data into credit assessment and risk management systems. In particular, in businesses like health insurance, loans, and credit cards, a customer's sleep quality and health status are considered crucial factors influencing long-term health management and claims risk. However, current technical solutions mostly rely on single-signal analysis or traditional manual screening methods, which are not only inefficient but also fail to fully reflect a customer's health status. In the financial sector, leveraging multimodal physiological data for intelligent health risk assessment has become a pressing issue.

[0005] Existing models based on traditional supervised learning often fail to effectively integrate multiple physiological signals and lack cross-modal learning capabilities. This makes it difficult for the fintech industry to integrate data and suffer from poor model adaptability when using physiological signals for customer risk assessment. Furthermore, due to the high heterogeneity of health data across different customers, traditional models cannot be effectively transferred across multiple tasks and scenarios, limiting their application in the financial sector. Summary of the Invention

[0006] The main purpose of the present invention is to provide a multimodal physiological signal processing method, device, equipment and storage medium, aiming to solve the technical problems that the existing technology relies on a large amount of labeled data and has weak cross-modal joint representation capabilities, is sensitive to modality missing, and is difficult to migrate to different downstream tasks.

[0007] To achieve the above objectives, the present invention provides a multimodal physiological signal processing method, comprising:

[0008] Acquiring multimodal physiological signal data, and preprocessing the multimodal physiological signal data to obtain a plurality of preprocessed data segments;

[0009] Performing data enhancement processing on the preprocessed data segment to generate a damaged data segment;

[0010] Inputting each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation;

[0011] Aggregating the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation;

[0012] Inputting the damaged data segment into a decoder to generate a reconstructed data segment;

[0013] updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments;

[0014] The target task is performed based on the updated modal encoder to generate the target task result.

[0015] Furthermore, to achieve the above-mentioned object, the present invention provides a multimodal physiological signal processing device, comprising:

[0016] a data acquisition and preprocessing module, configured to acquire multimodal physiological signal data and preprocess the multimodal physiological signal data to obtain a plurality of preprocessed data segments;

[0017] A data enhancement module, configured to perform data enhancement processing on the preprocessed data segment to generate a damaged data segment;

[0018] a modality encoder module, configured to input each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation;

[0019] A modality aggregation module, configured to aggregate the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation;

[0020] a decoder module, configured to input the damaged data segment into a decoder to generate a reconstructed data segment;

[0021] an optimization and update module for updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments;

[0022] The task execution module is used to execute the target task based on the updated modal encoder and generate the target task result.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a multimodal physiological signal processing program stored in the memory and executable on the processor. When the multimodal physiological signal processing program is executed by the processor, the steps of the multimodal physiological signal processing method described above are implemented.

[0024] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a multimodal physiological signal processing program is stored. When the multimodal physiological signal processing program is executed by a processor, the steps of the multimodal physiological signal processing method described above are implemented.

[0025] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A multimodal physiological signal processing method, device, equipment and medium are disclosed, including: obtaining multimodal physiological signal data and preprocessing to generate multiple preprocessed data segments; performing data enhancement on the preprocessed data segments to generate damaged data segments; inputting each physiological signal into the corresponding modality encoder to generate a modality-specific representation; aggregating the modality-specific representation of non-target modalities to generate a positive sample aggregate representation; inputting the damaged data segment into the decoder to generate a reconstructed data segment; updating the modality encoder and decoder by increasing the similarity between the modality-specific representation of the target modality and the positive sample aggregate representation, reducing the similarity between the target modality and other data segments, reducing the difference between the reconstructed data segment and the preprocessed data segment; executing the target task based on the updated modality encoder to generate the target task result. The present invention optimizes the update process of the modality encoder and decoder by enhancing the similarity between the target modality and the positive sample aggregate representation and reducing the similarity between the target modality and other data segments, thereby improving the processing effect of the target task, especially significantly improving the migration ability and processing accuracy between different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0027] Figure 1 Schematic diagram of an application environment of a multimodal physiological signal processing method according to an embodiment of the present invention;

[0028] Figure 2 1 is a flow chart of an embodiment of a multimodal physiological signal processing method according to the present invention;

[0029] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a multimodal physiological signal processing device of the present invention;

[0030] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0031] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] The multimodal physiological signal processing method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can obtain multimodal physiological signal data through the user terminal and perform preprocessing to generate multiple preprocessed data segments; perform data enhancement on the preprocessed data segments to generate damaged data segments; input each physiological signal into the corresponding modality encoder to generate a modality-specific representation; aggregate the modality-specific representation of the non-target modality to generate a positive sample aggregate representation; input the damaged data segment into the decoder to generate a reconstructed data segment; by increasing the similarity between the modality-specific representation of the target modality and the positive sample aggregate representation, reducing the similarity between the target modality and other data segments, reducing the difference between the reconstructed data segment and the preprocessed data segment, and updating the modality encoder and decoder; perform the target task based on the updated modality encoder to generate the target task result. The present invention optimizes the update process of the modality encoder and decoder by enhancing the similarity between the target modality and the positive sample aggregate representation and reducing the similarity between the target modality and other data segments, thereby improving the processing effect of the target task, especially in terms of the migration ability and processing accuracy between different tasks. Among them, the user terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server side can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0034] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a multimodal physiological signal processing method provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0035] like Figure 2 As shown, the multimodal physiological signal processing method proposed in the present invention includes the following steps:

[0036] S10, acquiring multimodal physiological signal data, and preprocessing the multimodal physiological signal data to obtain a plurality of preprocessed data segments;

[0037] In this embodiment, multimodal physiological signal data is acquired and preprocessed to generate multiple preprocessed data segments. The multimodal physiological signal data includes electroencephalogram (EEG), electrocardiogram (ECG), respiratory signals, and snoring signals collected during the same time period. This process combines the acquisition and normalization of sleep-related physiological signals from multiple sources to provide a standardized and structured input data foundation for the subsequent unified encoding and joint modeling of multimodal data, thereby preventing differences in acquisition frequency, time synchronization, and signal amplitude range that could affect subsequent analysis.

[0038] Multimodal physiological signal data refers to multiple types of physiological signal information collected from the same subject and during the same time period by different physiological sensing devices. Electroencephalogram (EEG) signals are usually collected through electrodes on the surface of the scalp, reflecting the potential changes in brain nerve activity; electrocardiogram (ECG) signals are obtained through electrodes on the chest or limbs, reflecting the electrophysiological activity of the heart; respiratory signals can be obtained through nasal airflow sensors, chest and abdominal belts, or respiratory pressure sensors, indicating respiratory cycles and airflow changes; snoring signals are obtained through microphones or airway sound sensors, reflecting respiratory ventilation status and the degree of airway obstruction. Multimodal physiological signal data are usually time-synchronized to ensure that all signals correspond to the physiological state within the same observation period. Sources include, but are not limited to, polysomnography systems, portable multi-parameter physiological recorders, or smart wearable devices.

[0039] The acquired multimodal physiological signal data needs to undergo data preprocessing to meet the requirements of signal quality, structural uniformity and time sequence consistency. Data preprocessing operations include signal denoising, sampling rate unification, amplitude normalization and structural segmentation. Signal denoising can be combined with filtering algorithms, including low-pass filtering, high-pass filtering, band-pass filtering or adaptive filtering methods. The filtering parameters are set according to the characteristics of different physiological signals. For example, for EEG signals, a low-pass filter is usually selected, and the cutoff frequency is set to 30 Hz to 50 Hz to remove high-frequency artifact signals; for ECG signals, considering QRS wave detection and heart rhythm feature extraction, the low-pass filter parameters can be set below 40 Hz; the filtering parameters of respiratory signals and snoring signals are set according to the airflow period characteristics and the sound frequency band to ensure that effective signal information is retained.

[0040] Preprocessing also includes resampling data of different types of signals, with the aim of unifying all physiological signals to a consistent sampling frequency, commonly set to 128 Hz, 256 Hz or 512 Hz. The specific frequency is determined by the subsequent model input requirements or hardware performance. After unifying the sampling frequency, data segmentation is further performed for long-term continuous data. Usually, the signal sequence is divided into multiple continuous and non-overlapping data segments based on a fixed time window. The time window length can be set to 30 seconds, 60 seconds or 120 seconds according to actual needs to standardize the input data structure and ensure that subsequent multimodal data is input into the model for processing in a unified dimension. After the division is completed, each data segment is standardized separately to map the amplitude range of different signals to a unified numerical interval. Common methods include Z-score standardization, minimum-maximum scaling or logarithmic compression transformation to ensure that different signal data are comparable and stable.

[0041] In a specific embodiment, the acquisition of multimodal physiological signal data can be completed based on the polysomnography equipment in the sleep monitoring laboratory, or a portable physiological monitoring device can be used to collect EEG, ECG, respiratory signals and snoring signals through Bluetooth, Wi-Fi or local storage. In the preprocessing stage, the filtering parameters, resampling frequency and data segment length can be flexibly adjusted according to the application scenario. For scenarios that require precise identification of short-term sleep state changes, the data segment length can be shortened to less than 30 seconds; for scenarios facing long-term trend analysis, the data segment length can be extended to more than 120 seconds; the filter type and parameters are dynamically set according to the environmental noise level, signal quality and analysis focus. The resampling operation supports unified adjustment on the software side to ensure the standardization of the multimodal data input structure.

[0042] Example description: In the healthcare field, the multimodal physiological signal data obtained by this process, combined with a unified preprocessing process, can effectively improve the accuracy of sleep stage determination, sleep breathing disorder screening, and chronic disease risk prediction based on electroencephalograms, electrocardiograms, respiratory signals, and snoring signals. It is suitable for clinical sleep monitoring, portable home testing, and personalized health management scenarios.

[0043] In the fintech sector, multimodal physiological signal data can be used to monitor customer health, providing accurate health assessments and risk predictions, thereby supporting the design and pricing of personalized insurance products. By collecting and analyzing multimodal data, insurance companies can monitor health changes among policyholders in real time and promptly identify high-risk individuals, thereby optimizing claims management and reducing overall insurance risk. Furthermore, based on data analysis, fintech can help the insurance industry improve claims processing efficiency, prevent fraud, and provide customers with more precise insurance protection.

[0044] This embodiment solves the problems of inconsistent sampling frequencies, difficult time alignment, large differences in signal amplitude ranges, and inconsistent data structures among different signal sources by uniformly collecting and normalizing preprocessing multimodal physiological signal data. It improves the accuracy of multimodal data fusion and feature extraction, provides a stable and reliable data foundation for subsequent multimodal encoding, joint learning, and cross-modal representation optimization, and enhances the overall system's adaptability to multi-source physiological data and task generalization capabilities.

[0045] S20, performing data enhancement processing on the preprocessed data segment to generate a damaged data segment;

[0046] In this embodiment, preprocessed data segments refer to segmented, multi-channel data sets with stable physiological signal quality, obtained after acquiring multimodal physiological signal data and undergoing filtering, resampling, segmentation, and normalization. Data augmentation processing involves artificially designed interventions on these data segments to simulate signal loss, noise interference, and signal distortion that may occur in real-world application scenarios, thereby improving the generalization and robustness of the model. The purpose of data augmentation is to increase data diversity, expand the distribution of training samples, and reduce the risk of overfitting without changing the essential characteristics of the data.

[0047] Generating corrupted data segments involves constructing replicas of data with some missing information or reduced signal quality based on the results of data augmentation. This is typically used during the training phase to simulate incomplete or interfered input. Corrupted data segments can be achieved in a variety of ways, one of which is channel masking. This involves randomly selecting some signal channels (such as EEG, ECG, respiratory, or snoring channels) from a multimodal physiological signal and setting the corresponding channel signals to zero or to a specific mask value to simulate conditions such as signal loss, loose probes, or hardware failures. Another method is noise injection, which involves adding Gaussian white noise, impulse noise, or other types of interference signals to some signal channels while preserving the original data structure to enhance the model's adaptability to noisy environments. Data augmentation processing is not limited to the aforementioned methods and can also include temporal masking, amplitude perturbation, signal scrambling, and other methods. The specific operation method can be adjusted according to the actual application requirements.

[0048] In one implementation, signal channel analysis is first performed on each preprocessed data segment to identify the type and number of physiological signal channels contained within the data segment. Common channels include EEG, ECG, respiratory, and snoring channels. Based on this channel set information, a pseudo-random algorithm is used to select a subset of channels to form a selected masked channel set. For each selected channel, the corresponding signal data is reset to zero, forming a channel-masked data segment.

[0049] In another implementation, a Gaussian white noise signal with statistical properties is generated. Parameters such as the noise mean, standard deviation, and duration are set to ensure that the noise injection does not disrupt the overall data structure. This Gaussian white noise signal is then superimposed on a specific channel of the channel-masked data segment. The resulting noise-injected data segment then serves as the damaged data segment for subsequent training.

[0050] In different implementation environments, the channel selection ratio, noise intensity, and masking duration can be flexibly configured according to equipment accuracy and business needs to simulate data damage scenarios of varying severity.

[0051] Example: In healthcare, when applied to remote sleep monitoring systems, subjects wear multimodal physiological signal acquisition equipment for long-term sleep monitoring. Due to factors such as body position changes, loose leads, and environmental noise, localized channel loss or noise contamination often occurs in the actual signal. By constructing damaged data segments during the training phase, the model is able to handle signal loss and increased interference, ensuring high accuracy and stability for tasks such as sleep structure analysis and respiratory event detection even under conditions of fluctuating signal quality.

[0052] In the fintech sector, especially in the collection of health data related to the pricing of health and life insurance products, some users' uploaded physiological signals contain missing or noisy data. By applying data augmentation processing, the model can maintain relatively accurate health assessments even with incomplete health data. This helps insurance companies accurately price and control risks based on multimodal physiological data, mitigating the impact of undesirable data on business decisions.

[0053] This embodiment uses data augmentation processing to simulate channel signal loss, noise interference, and other conditions that may occur in real-world multimodal physiological signal processing. The construction of damaged data segments allows the model to be exposed to non-ideal input conditions early in the training phase, improving its robustness to incomplete and low-quality data and reducing its reliance on high-quality, perfect data. Compared to training based solely on raw data, models with data augmentation demonstrate greater stability and generalization capabilities when faced with issues such as signal loss and increased interference in complex, real-world environments. This significantly enhances the practical value of multimodal physiological signal processing systems in clinical, wearable, and remote monitoring scenarios.

[0054] S30, inputting each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation;

[0055] In this embodiment, the impaired data segment refers to a multi-channel physiological signal set formed after data enhancement processing. This data segment contains different types of physiological signals, such as EEG signals, ECG signals, respiratory signals, snoring signals, etc., and some signal channels may be subject to artificial masking, noise injection, or other forms of interference. Each physiological signal refers to an independent signal set within the multimodal physiological signal data structure, distinguished by signal source or type. EEG signals generally reflect brain activity, ECG signals reflect cardiac electrophysiological characteristics, respiratory signals are used to monitor respiratory rhythm and ventilation status, and snoring signals capture acoustic characteristics related to respiratory obstruction during sleep.

[0056] Modal encoders are feature extraction models designed specifically for different types of physiological signals, capable of extracting the structural features and temporal information within each signal. The architecture and parameter configuration of each modal encoder can be flexibly configured based on the spatiotemporal characteristics of the corresponding signal. These models typically include a combination of convolutional neural networks, transformer structures, temporal modeling units, and residual connection modules. These structures aim to recover or preserve key feature information from damaged signals, improving the accuracy and robustness of subsequent multimodal fusion and task processing.

[0057] Generating modality-specific representations means extracting features of different types of physiological signals in damaged data segments through modality encoders to form feature vectors or high-dimensional representations that represent the unique properties of each physiological signal. This representation not only reflects the temporal dynamics, spectral characteristics, and spatial distribution of the signal, but also appropriately captures the potential information of cross-modal associations, providing a basis for the integration and collaboration of multimodal data.

[0058] In one implementation, the damaged data segment undergoes a data splitting operation to extract the EEG, ECG, respiratory, and snoring signal components. Each signal is then fed into a modality encoder corresponding to its type. The EEG signal is fed into the EEG encoder, which uses a one-dimensional convolutional layer to extract local temporal features. Stacked convolutional blocks capture multi-scale temporal information, combined with a transformer structure to enhance long-range dependency modeling. Finally, a feature aggregation module outputs a modality-specific representation of the EEG.

[0059] Similarly, part of the electrocardiogram signal is input into the electrocardiogram encoder, part of the respiratory signal is input into the respiratory signal encoder, and part of the snoring signal is input into the snoring signal encoder. Each encoder flexibly designs the feature extraction level according to the signal characteristics, and uniformly outputs a modality-specific representation that corresponds one-to-one with the input signal.

[0060] In another implementation, the encoder structure can be adjusted based on business needs. For example, a stronger low-frequency feature extraction module can be used for breathing signals to improve the perception of slow rhythmic changes. For snoring signals, spectrogram conversion and two-dimensional convolution structures can be introduced to enhance the spatial representation of acoustic features. The parameters of different modality encoders can be trained independently or jointly optimized to ensure that the modality-specific representations generated by each retain the key characteristics of a single modality while also possessing cross-modal collaborative expression capabilities.

[0061] Example: In the healthcare field, for long-term sleep monitoring scenarios, the EEG, ECG, respiratory signals, and snoring signals collected by wearable multimodal devices are easily limited by bandwidth or storage limitations during data upload, and some signal channels have data missing or noise pollution. By inputting each type of damaged signal into an independently designed modal encoder, the modality-specific representation is extracted, the key internal features of the signal are effectively retained, and the accuracy of subsequent tasks such as sleep structure analysis and respiratory disorder detection is improved.

[0062] In the field of financial technology business, especially in the health insurance risk assessment link, when collecting users' multimodal physiological signals as health data input, there are quality fluctuations in different signal channels due to equipment differences or usage habits. By generating modality-specific representations through targeted modal encoders, reliable feature extraction can be achieved under incomplete and interfered health data, assisting insurance institutions in conducting health status assessments, personalized product pricing and risk control based on stable physiological signal characteristics.

[0063] By inputting each physiological signal in the damaged data segment into its corresponding modal encoder, this embodiment effectively isolates the feature extraction process for different signal types, avoiding signal interference and information confusion, and improving the accuracy and stability of single-modal feature representation. The generation of modality-specific representations enables the collaborative analysis of multimodal physiological signals based on a high-quality, clearly structured single modality, contributing to overall performance improvements in subsequent multimodal fusion, positive and negative sample construction, and task decision-making.

[0064] S40, aggregating the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation;

[0065] In this embodiment, the target modality refers to the specific signal type currently selected as a comparison reference or task focus within the multimodal physiological signal data, such as an electroencephalogram (EEG) signal, electrocardiogram (ECG) signal, respiratory signal, or snoring signal. This type of signal is typically determined dynamically based on specific task requirements or model training strategies. Non-target modalities refer to signal types other than the target modality within the same data segment. These signal types, in addition to the target modality, are still part of the multimodal physiological signal, possess independent information expression capabilities, and are synchronized with the target modality within the overall data structure.

[0066] Modality-specific representations are feature expressions independently generated by their respective modality encoders for different types of physiological signals. They reflect the structural properties, temporal variation characteristics, and potential high-dimensional representation of a single signal modality in the current data segment. The modality-specific representation of all non-target modalities is the set of remaining modality-specific representations for the same data segment after removing the modality-specific representation corresponding to the target modality. This set retains other signal features that coexist with the target modality and share interactive information.

[0067] Aggregation operation refers to the integration of modality-specific representations of all non-target modalities through specific mathematical functions or network structures to generate a unified representation that expresses the overall multimodal correlation characteristics. Common aggregation methods include arithmetic averaging, weighted averaging, splicing fusion, attention weighting, graph structure information integration, etc., which can be flexibly configured according to different application scenarios and data characteristics. Generating positive sample aggregate representation is the result of the aggregation operation, which aims to form a feature expression that is closely related to the current target modality information and derived from the non-target modality signal. This representation is used as a positive sample in contrastive learning to narrow the feature distance with the target modality representation, thereby improving multimodal collaborative expression and joint modeling capabilities.

[0068] In one implementation, the target modality for the current data segment is first determined. For example, the EEG signal is selected as the target modality. The entire set of modality-specific representations is extracted, and the EEG modality-specific representation is removed, retaining the modality-specific representations for the ECG, respiratory signal, and snoring signal. The modality-specific representations of the three retained non-target modalities are then fed into an aggregation module, where an arithmetic mean operation is performed to calculate the mean of the three representations along the corresponding dimensions to form an aggregated representation of the positive sample.

[0069] Another implementation utilizes a multi-layer perceptron (MLP) architecture to perform aggregation. The modality-specific representations of the non-target modalities are concatenated and fused, then fed into the MLP to extract nonlinear combined features and output an aggregated representation of the positive sample. Alternatively, an attention mechanism can be introduced to dynamically assign weights to the representations of different non-target modalities during the aggregation process, enhancing the representation of key modal information in the aggregated representation of the positive sample.

[0070] It is also possible to design a graph structure aggregation method based on the data characteristics, construct the modality-specific representation of the non-target modality as the node of the graph structure, extract the cross-modal relationship based on the graph convolution or graph attention mechanism, and generate a positive sample aggregation representation that integrates multimodal structural information.

[0071] Example description: In the medical and health field, for sleep monitoring tasks, EEG, ECG, respiratory signals, and snoring signals are collected synchronously. The EEG signal is selected as the target modality, and the modality-specific representation of the EEG is removed. The modality-specific representations of the ECG, respiratory signals, and snoring signals are aggregated to form a positive sample aggregate representation. This improves the multimodal feature fusion effect, assists in sleep structure identification and anomaly detection tasks, and enhances the ability to utilize multimodal collaborative information.

[0072] In the field of financial technology, during the pricing or risk assessment of health insurance, multimodal physiological signals are used to monitor the health status of users. Electrocardiogram signals are selected as the target modality, and modality-specific representations of electroencephalograms, respiratory signals, and snoring signals are aggregated to generate positive sample aggregate representations. This improves the accuracy and stability of health risk prediction, assists in personalized health management and insurance product configuration, and enhances data-driven decision-making capabilities.

[0073] This embodiment aggregates the modality-specific representations of all non-target modalities to generate a positive sample aggregate representation. This fully integrates multiple physiological signal features other than the target modality to form a high-quality, structurally complete multimodal fusion expression, enhancing the complementarity and overall expressiveness of multimodal information. Within the contrastive learning framework, the improved similarity between the positive sample aggregate representation and the target modality representation helps guide the model to capture multimodal synergy, strengthen the intrinsic connections between different modalities, reduce the impact of missing or interference from a single modality on system performance, and improve the expression robustness and task adaptability of the multimodal system.

[0074] S50, inputting the damaged data segment into a decoder to generate a reconstructed data segment;

[0075] In this embodiment, a damaged data segment refers to a collection of physiological signal data generated through data augmentation. Compared to the original preprocessed data segment, the damaged data segment has some information missing, signal interference, or structural perturbations. Common forms of impairment include channel masking, signal zeroing, noise injection, and amplitude perturbations. These processes are intended to simulate signal loss, interference, or abnormal scenarios in real applications, improving the robustness and generalization capabilities of the model. The damaged data segment generated by data augmentation still retains the temporal order, modal correspondence, and basic information characteristics of the overall data structure, facilitating subsequent input into the reconstruction module for recovery.

[0076] A decoder is a neural network structure capable of both structural restoration and information reconstruction. It typically comprises multiple layers of decoding units, incorporating modules such as convolution, upsampling, residual connections, attention mechanisms, and transformation structures. It receives damaged data segments as input and, through layer-by-layer information restoration and feature integration, outputs reconstructed data segments that are consistent with the original data structure. The reconstructed data segments represent the structured physiological signal data output by the decoder. Theoretically, they should be highly similar to the preprocessed data segments in terms of overall information expression, structural dimensionality, and temporal order, reflecting the model's reconstruction capabilities and information repair effectiveness.

[0077] In one specific approach, the damaged data segment is first fed into the decoder's feature transformation module, which extracts deep global features based on a multi-layer Transformer architecture, preserving the collaborative expressiveness of multimodal information. The output of the transformation module is then fed into the convolutional restoration module, which, through multiple layers of convolution and deconvolution, gradually restores the signal's local structural information and time series features. The convolutional module output is fed through a residual connection layer, improving information flow efficiency and alleviating the vanishing gradient problem during deep network training.

[0078] The data then passes through an upsampling module, using bilinear interpolation or deconvolution to restore the feature map size to the same time dimension and signal resolution as the preprocessed data segment. Finally, a convolution operation in the output layer adjusts the channel dimension and signal format to generate a structurally complete reconstructed data segment. The reconstructed data segment is consistent with the preprocessed data segment in terms of time length, signal type, number of modes, and data format, facilitating comparison with the original data, error assessment, and optimization updates.

[0079] It is also possible to design more complex decoder structures based on different task requirements. For example, by introducing a skip connection mechanism, low-level features extracted from the encoder can be directly transferred to the alignment layer of the decoder, enhancing the representation of high-frequency information and improving the detail restoration of the reconstructed data segments. Multi-scale feature fusion mechanisms can also be combined to integrate structural information at different scales, further improving reconstruction quality.

[0080] Example description: In the healthcare field, for sleep monitoring tasks, channel masking and noise injection are used to form damaged data segments. After input into the decoder, the structure and information content of the electroencephalogram, electrocardiogram, respiratory signal, and snoring signal are restored, and reconstructed data segments are generated that are consistent with the structure of the preprocessed data segments. This improves the system's sleep structure analysis and anomaly detection capabilities in signal loss and interference scenarios.

[0081] In the field of financial technology, in the dynamic health assessment of health insurance, damaged data segments are generated based on data enhancement to simulate signal interruptions or anomalies that occur when the user wears the device. After input into the decoder, reconstructed data segments are generated to achieve data quality recovery and structure correction, improve the accuracy and stability of health risk assessment, and enhance the ability to configure personalized insurance products supported by data.

[0082] This embodiment generates reconstructed data segments by inputting damaged data segments into a decoder. This effectively restores information lost or interfered with during the data enhancement process, improving the structural integrity and accuracy of multimodal physiological signal data. The output of the reconstructed data segments provides a foundation for contrastive learning and loss function calculation, facilitating the model's ability to capture information defects such as missing signals and structural damage during the reconstruction process. This strengthens the model's ability to repair information, optimizes the overall quality of multimodal data representation, and enhances the robustness, stability, and generalization capabilities of the system.

[0083] S60, updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive sample, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segment and the preprocessed data segment;

[0084] In this embodiment, the modality-specific representation of the target modality refers to the representation information generated by the corresponding modality encoder based on the target physiological signal selected in the damaged data segment. This representation includes the temporal characteristics, structural characteristics, and cross-modal related features of the target physiological signal, and has strong modality differentiation and information expression capabilities. The positive sample aggregate representation refers to the joint feature information generated by aggregating the modality-specific representations of all non-target modalities except the target modality. It reflects the information expression in multimodal data that is collaboratively related to the target modality and has cross-modal integration characteristics and auxiliary expression capabilities.

[0085] Similarity is usually measured by mathematical metrics to measure the degree of similarity between two feature representations. Cosine similarity, Euclidean distance, or other distance metrics in high-dimensional space are often used. Cosine similarity measures directional similarity. A larger value indicates more consistent feature expressions.

[0086] The modality-specific representation of other preprocessed data segments refers to the representation information generated by the corresponding modality encoder of the same modality in a data segment different from the data segment to which the current target modality belongs. It represents the same-modality feature expression across data segments and participates in the comparison calculation as a negative sample.

[0087] The reconstructed data segment refers to the structurally complete data set generated by reconstructing the damaged data segment through the decoder. The preprocessed data segment refers to the multi-channel structured data set formed after the original multimodal physiological signal data undergoes preprocessing operations such as filtering, resampling, segmentation, and standardization.

[0088] By increasing the similarity between the modality-specific representation of the target modality and the aggregate representation of positive samples, the consistency of the collaborative information expression of the target modality and other modalities is improved, and the cross-modal joint representation capability is enhanced. By reducing the similarity between the modality-specific representation of the target modality and the modality-specific representation of other data segments, the feature overlap and expression interference between different data segments are reduced, the ability to distinguish data segments is strengthened, and the discriminability and robustness of feature expression are improved. By reducing the difference between the reconstructed data segments and the preprocessed data segments, the information recovery capability is optimized, and the integrity of the data structure and the accuracy of information expression are improved.

[0089] Updating the modal encoder and decoder means constructing a joint optimization objective function based on the above-mentioned similarity and difference calculations, using optimization algorithms such as backpropagation and gradient descent, and dynamically adjusting the trainable parameters of the modal encoder and decoder to improve the overall system's representation learning ability, information integration ability, and data reconstruction ability.

[0090] In one specific implementation, the similarity between the modality-specific representation of the target modality and the aggregated representation of positive samples is first calculated based on cosine similarity. By maximizing this similarity, the model is encouraged to learn collaborative information between multiple modalities and enhance the ability to express joint features. Subsequently, modality-specific representations of the same modality in other data segments are selected and their cosine similarity with the modality-specific representation of the current target modality is calculated. This negative sample set is then constructed. By minimizing the average similarity of this set, the similarity of same-modality representations between different data segments is suppressed, thereby improving cross-segment discrimination capabilities.

[0091] At the same time, short-time Fourier transform is used to analyze the difference in frequency domain expression between the reconstructed data segment and the preprocessed data segment. By calculating the spectral difference or the time domain error based on the L1 and L2 norms, a reconstruction error index is formed to minimize the error and improve the reconstruction accuracy and information recovery ability of the decoder.

[0092] A joint loss function is constructed as a whole, integrating positive sample alignment, negative sample separation and reconstruction error constraints. Through the back-propagation algorithm, all trainable parameters in the modal encoder and decoder are jointly updated to improve the model's cross-modal information integration capabilities, cross-data segment differentiation capabilities and damaged data repair capabilities.

[0093] In the specific implementation, an adaptive temperature coefficient can be used to adjust the similarity distribution of positive and negative samples to enhance the stability and convergence efficiency of the optimization process. It can also be combined with a dynamic learning rate strategy to optimize the parameter update rate for different training stages to further improve training efficiency and expression ability.

[0094] Example description: In the healthcare field, for multimodal sleep monitoring systems, the above update operations optimize the encoders and decoders of each modality, improve the cross-modal expression consistency of electroencephalogram (EEG), electrocardiogram (ECG), respiratory signals, and snoring signals, enhance the distinguishability of feature expressions of different sleep stages and the integrity of data reconstruction, and significantly improve the accuracy and stability of tasks such as sleep staging and anomaly detection.

[0095] In the fintech business field, based on the dynamic risk assessment scenario of health insurance, the above-mentioned update operations will improve the expression accuracy and information repair capabilities of multimodal physiological signal data, enhance the assessment stability and risk prediction reliability in the case of signal loss and interference from user-worn devices, and support more accurate health management strategies and differentiated insurance product configuration plans.

[0096] Through the aforementioned update operations, this embodiment significantly enhances the joint representation learning capabilities of the modal encoder and decoder, improving the collaborative expression quality of multimodal physiological signal data and the cross-modal information fusion capabilities. The optimized modal encoder has stronger feature differentiation, cross-segment discrimination, and anti-interference capabilities, while the optimized decoder has higher data structure recovery capabilities and information restoration accuracy. This overall improves the robustness, generalization, and expression accuracy of the multimodal system in complex application scenarios, meeting the needs of subsequent diversified tasks.

[0097] S70, executing the target task based on the updated modal encoder to generate a target task result.

[0098] In this embodiment, in the process of executing the target task and generating the target task result based on the updated modal encoder, the updated modal encoder refers to a set of encoders that have undergone a pre-order parameter optimization process, combined with the joint expression structure of multimodal physiological signals, and adjusted the internal parameter configuration. This set includes an electroencephalogram encoder, an electrocardiogram encoder, a respiratory signal encoder, and a snoring signal encoder, which perform feature extraction operations on different types of physiological signal data. After the update is completed, the above-mentioned encoder has a more targeted cross-modal collaborative expression capability, which can accurately extract feature information from multimodal physiological signals and perform joint expression.

[0099] Executing the target task based on the updated modality encoder includes acquiring multimodal physiological signal data that meets the input structure requirements, inputting the EEG signal in the data into the updated EEG encoder to generate an EEG modality-specific feature expression; inputting the ECG signal into the updated ECG encoder to generate an ECG modality-specific feature expression; inputting the respiratory signal into the updated respiratory signal encoder to generate a respiratory signal modality-specific feature expression; and inputting the snoring signal into the updated snoring signal encoder to generate a snoring signal modality-specific feature expression. The above modality-specific feature expressions are jointly expressed through an internal fusion mechanism to form a multimodal fusion joint feature.

[0100] The joint features are input into a functional module designed for a specific application task. The functional module can be a classification structure, a regression structure, or other structure, and the specific structure is selected based on the task type. For example, for a classification task, after the joint features are input into the classification module, hierarchical discrimination and probability output are used to generate classification results. The classification results may include physiological state labels, sleep stage labels, or other label information.

[0101] The classification results or other output information are determined as the target task results. The form of the target task results is determined according to the application requirements, covering different types such as physiological state judgment, behavioral state recognition, and health indicator generation. The result information directly reflects the physiological state, health status or task-related indicators corresponding to the input data.

[0102] During this process, the internal parameters of the updated modal encoders have been specifically optimized, improving joint expression capabilities and enhancing the fusion of multimodal information. When inputting multimodal physiological signals, feature extraction becomes more accurate, stable, and complete. The output, leveraging the synergistic effect of multimodal information, more accurately reflects the inherent relationships and state changes between physiological signals. The multimodal signal data processing pipeline maintains structural continuity and logical integrity, ensuring consistent and repeatable task execution.

[0103] In different application environments, the updated modality encoder can be adapted to different application requirements by adjusting the data source, signal structure, and technical configuration of the functional modules of the task input. For example, in an implementation optimized for a single task, the acquired data only includes EEG and ECG signals. The updated modality encoder only includes an EEG encoder and an ECG encoder. Feature extraction is performed on the input EEG and ECG signals respectively to form corresponding modality-specific feature expressions. After fusion, these features are input to the classification module to output the sleep state category label.

[0104] In a multi-task collaborative environment, the acquired data includes EEG signals, ECG signals, respiratory signals and snoring signals. The updated modality encoder includes encoders corresponding to all modalities. After forming a joint expression, they are input into different functional modules respectively to generate sleep state recognition results, apnea risk prediction results and snoring intensity assessment results. Multiple results jointly constitute the target task result output to meet the needs of complex tasks.

[0105] In some real-time monitoring scenarios, the input data segment structure can also be adjusted, and continuous data segments of multimodal physiological signals can be obtained through the sliding window mechanism. The updated modal encoder is input in real time, and the corresponding task results are output in batches to achieve dynamic tracking of multimodal physiological signals and real-time status feedback.

[0106] Example description: In the healthcare field, when performing sleep state recognition tasks based on the updated modality encoder, the first step is to generate multimodal preprocessed data segments based on a unified preprocessing process for the acquired multimodal physiological signal data, including EEG signals, ECG signals, respiratory signals, and snoring signals. The updated modality encoder extracts modality-specific expressions for the signal data segments of different modalities, forms a joint expression, and directly inputs it into the sleep state discrimination module. The output target task result is the sleep state category within the corresponding time period. The updated modality encoder's multimodal information integration capability improves the accuracy and stability of sleep state recognition, meeting the needs of clinical real-time monitoring and auxiliary diagnosis.

[0107] In the field of fintech business, health management systems need to generate risk levels or health assessment results based on user physiological data. The acquired multimodal physiological signal data is preprocessed to form a unified data segment structure. The updated modal encoder jointly encodes the signal information of different modalities and inputs it into the risk assessment module. The output target task result is the user's health risk level or status classification information. In this process, the updated modal encoder enhances the comprehensive expression effect of physiological status information through optimized joint expression capabilities, improves the analysis accuracy of health data, meets the requirements of insurance product health assessment, risk pricing and other tasks, and reduces business deviations caused by incomplete information or inaccurate feature expression.

[0108] By applying the updated modal encoder to the target task, this embodiment effectively utilizes the optimized multimodal information expression structure, improving the accuracy and stability of physiological signal feature extraction, enhancing the joint expression of multimodal collaborative information, and reducing the impact of missing single modal signals or noise interference on task execution results. Multimodal joint expression provides a more complete overall representation of physiological state information. Combined with the specific discriminative capabilities of functional modules, it can output more accurate and stable target task results, meeting the diverse needs of different application scenarios and enabling the efficient application of multimodal physiological signals in complex tasks.

[0109] The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A multimodal physiological signal processing method, device, equipment and medium are disclosed, including: obtaining multimodal physiological signal data and preprocessing it to generate multiple preprocessed data segments; performing data enhancement on the preprocessed data segments to generate damaged data segments; inputting each physiological signal into a corresponding modality encoder to generate a modality-specific representation; aggregating the modality-specific representations of non-target modalities to generate a positive sample aggregate representation; inputting the damaged data segments into a decoder to generate a reconstructed data segment; updating the modality encoder and decoder by increasing the similarity between the modality-specific representation of the target modality and the positive sample aggregate representation, reducing the similarity between the target modality and other data segments, and reducing the difference between the reconstructed data segment and the preprocessed data segment; performing a target task based on the updated modality encoder to generate a target task result. The present invention optimizes the update process of the modality encoder and decoder by enhancing the similarity between the target modality and the positive sample aggregate representation and reducing the similarity between the target modality and other data segments, thereby improving the processing effect of the target task, especially significantly improving the migration ability and processing accuracy between different tasks.

[0110] In one embodiment, the above step S10 includes:

[0111] S101, collecting multimodal physiological signal data;

[0112] S102, performing low-pass filtering on the multimodal physiological signal data to obtain filtered physiological signal data;

[0113] S103, resampling the filtered physiological signal data to a uniform frequency to obtain resampled physiological signal data;

[0114] S104, dividing the resampled physiological signal data into a plurality of data segments of preset duration;

[0115] S105 , performing standardization processing on the plurality of data segments of preset duration to generate the plurality of pre-processed data segments.

[0116] In this embodiment, acquiring multimodal physiological signal data refers to using multiple physiological monitoring devices to synchronously collect different types of physiological signal data from the same subject. Specifically, these include electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, respiratory signals, and snoring signals. EEG signals reflect the brain's electrophysiological activity, ECG signals reflect changes in the heart's electrical activity, respiratory signals characterize changes in respiratory airflow or chest and abdominal fluctuations, and snoring signals reflect respiratory tract vibrations and obstructions. Acquisition of multimodal physiological signal data is typically based on a multi-channel physiological acquisition system, ensuring that each signal data is aligned on the same timeline to meet the requirements of subsequent joint processing.

[0117] Low-pass filtering of multimodal physiological signal data involves using a filter with a set cutoff frequency to suppress high-frequency noise in the raw physiological signal data. Common filter types include finite impulse response (FIR) and infinite impulse response (IIR) filters. This process effectively reduces high-frequency components introduced by environmental interference, hardware noise, or non-physiological factors, preserving the effective frequency information in the physiological signal and improving signal data quality.

[0118] Resampling filtered physiological signal data to a uniform frequency addresses sampling rate inconsistencies caused by different sources or sampling devices. This involves reconstructing various physiological signals to a uniform time scale through interpolation, downsampling, or upsampling. This ensures that multimodal signals have uniform temporal resolution, facilitating subsequent fusion processing. The resampling process must ensure that signal information is not distorted. Common methods include linear interpolation, spline interpolation, or resampling with anti-aliasing filtering.

[0119] Segmenting the resampled physiological signal data into multiple segments of preset duration refers to dividing a continuous physiological signal sequence into fixed-length data segments based on business needs or model input constraints. The preset duration is typically set based on the specific application scenario, such as 30 seconds, 60 seconds, or 90 seconds. The segmented data segments have a unified structure, facilitating batch input to downstream processing modules.

[0120] Normalization of multiple data segments of preset durations involves adjusting the numerical range of signal data from different modalities, channels, or time segments based on statistical characteristics. Standardization methods may include mean-variance normalization, maximum-minimum scaling, or quantile mapping. This operation eliminates dimensional differences and amplitude inconsistencies in signal data, enhancing comparability across different data segments and reducing the impact of numerical drift during model training or inference.

[0121] Through the above operations, this embodiment can achieve unified processing of raw physiological signal data from different devices and different modalities, improve the timing alignment, signal clarity and data consistency of multimodal physiological signal data, and generate preprocessed data segments with a unified time scale, unified amplitude range, and unified structural format, meeting the needs of multimodal joint expression, model input standardization and subsequent efficient information fusion, and improving the accuracy and robustness of multimodal physiological signal analysis.

[0122] In one embodiment, the above step S20 includes:

[0123] S201, performing a signal channel analysis operation on the preprocessed data segment to generate a signal channel set;

[0124] S202, performing a random masking channel selection operation based on the signal channel set to generate a selected masking channel set;

[0125] S203, performing a channel zeroing operation on the preprocessed data segment according to the selected masked channel set to generate a channel masked data segment;

[0126] S204, generating a Gaussian white noise signal;

[0127] S205 : Inject the Gaussian white noise signal into the channel-masked data segment to generate a noise-injected data segment, and mark the noise-injected data segment as a damaged data segment.

[0128] In this embodiment, data enhancement processing is performed on the preprocessed data segment, which means that while keeping the basic structure of the original multimodal physiological signal unchanged, some signals in the data segment are artificially intervened and disturbed to simulate the signal loss, noise interference or equipment failure that may occur in the actual environment, so as to improve the adaptability and robustness of the model to incomplete data or low-quality data.

[0129] Signal channel analysis on preprocessed data segments involves identifying and analyzing the presence, data integrity, and validity of independent data channels corresponding to different physiological signal types within the data segment. This generates a set of signal channels, specifically EEG, ECG, respiratory, and snoring channels, each with a unique identifier and data sequence. This channel analysis can be combined with signal amplitude range, temporal continuity, or noise level to aid in decision-making and ensure the effectiveness of subsequent masking operations.

[0130] Performing a random masking channel selection operation based on a signal channel set means randomly selecting one or more channels from the signal channel set to form a selected masking channel set by using a random number generation mechanism, a pseudo-random algorithm, or a sampling method based on a preset probability distribution. The number of channels and the selection strategy in the selected masking channel set can be dynamically adjusted according to specific business needs, data enhancement intensity, or model training stage, aiming to build diverse damaged data scenarios.

[0131] The channel zeroing operation is performed on the preprocessed data segment according to the selected masked channel set. This means that for the corresponding physiological signal channel in the selected masked channel set, all signal values ​​are set to zero at the corresponding position in the data segment to form a channel masked data segment. The channel zeroing operation simulates the channel loss, signal disconnection or equipment failure in the actual monitoring process. The data after zeroing maintains the original data format and time structure to avoid affecting the subsequent data processing process.

[0132] Generating a Gaussian white noise signal involves constructing a white noise signal sequence with zero mean and adjustable variance based on a Gaussian distributed random number generation method. Gaussian white noise signals exhibit statistical uniformity and spectral flatness, making them widely used in signal interference simulation and data augmentation. The duration and sampling rate of the noise signal are consistent with the channel masking data segment to ensure data structure matching.

[0133] A Gaussian white noise signal is injected into the channel-masked data segment to generate a noise-injected data segment. Specifically, the Gaussian white noise signal is fused into the channel-masked data segment through data superposition, numerical substitution, or amplitude perturbation at corresponding locations, increasing the complexity and diversity of data augmentation operations. The noise-injected data segment is marked as a damaged data segment. Damaged data segments exhibit the dual characteristics of missing information and noise interference, simulating the incompleteness and low-quality characteristics of physiological signal data in real environments, making them suitable for improving robustness during model training.

[0134] Through the above operations, this embodiment can introduce various interference forms such as missing and noise into multimodal physiological signal data based on a structured data enhancement mechanism without destroying the overall structure and temporal characteristics of the data, thereby constructing rich, realistic, and complex damaged data scenarios, improving the robustness and generalization ability of the multimodal physiological signal analysis system in the face of signal missing, equipment abnormalities, or environmental interference, and enhancing the practicality and reliability of the model in multi-source heterogeneous and complex scenarios.

[0135] In one embodiment, the above step S30 includes:

[0136] S301, separating an electroencephalogram signal portion, an electrocardiogram signal portion, a respiratory signal portion, and a snoring signal portion from the damaged data segment;

[0137] S302, inputting the electroencephalogram signal portion into an electroencephalogram encoder for processing to generate an electroencephalogram modality-specific representation;

[0138] S303, inputting the electrocardiogram signal portion into an electrocardiogram encoder for processing to generate an electrocardiogram modality-specific representation;

[0139] S304, inputting the respiratory signal portion into a respiratory signal encoder for processing to generate a respiratory signal modality-specific representation;

[0140] S305: Input the snoring signal portion into a snoring encoder for processing to generate a modality-specific representation of the snoring signal.

[0141] In this embodiment, each physiological signal in the damaged data segment is input into the corresponding modality encoder to generate a modality-specific representation. Signal separation is first performed on the different types of physiological signals in the damaged data segment. Signal separation involves distinguishing the EEG signal component, the ECG signal component, the respiratory signal component, and the snoring signal component from the overall data structure based on a multi-channel structure, signal labels, or data arrangement rules. The EEG signal component refers to the data channel derived from the EEG sensor, typically recording brain electrophysiological activity in a time series format, with frequency bands covering common EEG spectrum information such as delta waves, theta waves, alpha waves, and beta waves. The ECG signal component refers to the time series signal reflecting cardiac electrical activity, containing typical ECG features such as P waves, QRS complexes, and T waves, and is derived from precordial leads or limb leads. The respiratory signal component refers to time series data related to respiratory airflow and chest movement, obtained through a chest belt, flow meter, or pressure sensor, reflecting physiological states such as respiratory rhythm and respiratory rate. The snoring signal part refers to the sound signal related to airway stenosis and respiratory obstruction captured by an acoustic sensor or a vibration sensor, which is usually a non-stationary, non-periodic broadband signal.

[0142] The separated EEG signal is fed into an EEG encoder. The EEG encoder, which can be built using a one-dimensional convolutional neural network, a recurrent neural network, a time transformer, or a self-attention mechanism, extracts high-level features reflecting brain activity patterns from the EEG signal and outputs a modality-specific representation of the EEG. This modality-specific representation preserves the rhythmic information, temporal dependencies, and spatial distribution of the EEG signal, making it suitable for EEG analysis in downstream tasks.

[0143] Part of the ECG signal is input into the ECG encoder. The ECG encoder captures the morphological characteristics, waveform structure and timing changes of the ECG signal through a convolutional structure, feature extraction module or end-to-end network architecture, and generates an ECG modality-specific representation. The ECG modality-specific representation reflects the heart's functional status, rhythm changes and ECG characteristics, which is used for subsequent multimodal joint analysis.

[0144] The respiratory signal is partially input into the respiratory signal encoder, which extracts the frequency, amplitude and rhythm change characteristics of the respiratory signal through a time series analysis module, a respiratory rhythm recognition network or a multi-layer neural structure to generate a modal-specific representation of the respiratory signal. The modal-specific representation of the respiratory signal reflects the respiratory pattern, periodic characteristics and physiological state information.

[0145] Part of the snoring signal is input into the snoring encoder. The snoring encoder can use a convolutional neural network, an acoustic feature extraction network or a spectrum analysis module to identify relevant features such as airway stenosis and dyspnea from the snoring signal, and generate a modal-specific representation of the snoring signal. The modal-specific representation of the snoring signal retains the spectral characteristics, time domain changes and abnormal patterns of the sound signal, thereby improving the system's ability to identify sleep-related breathing disorders.

[0146] This embodiment can fully tap the unique information and deep features of various signals by inputting various physiological signals in the damaged data segment into corresponding structurally optimized and functionally independent modal encoders. While retaining the personalized characteristic expression of the signals, it can improve the overall information integration and expression capabilities of the multimodal system, provide high-quality, clearly structured modality-specific representations for subsequent multimodal fusion, joint modeling and task optimization, and enhance the system's adaptability and information utilization efficiency in multi-source complex data environments.

[0147] In one embodiment, the above step S40 includes:

[0148] S401, determining the current target mode;

[0149] S402, removing the modality-specific representation corresponding to the target modality from the set of all modality-specific representations to obtain the remaining modality-specific representations;

[0150] S403, performing an aggregation operation on the remaining modality-specific representations to generate an intermediate aggregated representation;

[0151] S404: Mark the intermediate aggregate representation as a positive sample aggregate representation corresponding to the target modality.

[0152] In this embodiment, the process of aggregating the modality-specific representations of all non-target modalities, excluding the target modality, to generate an aggregated representation of positive samples first requires determining the current target modality. The target modality refers to a single physiological signal type currently designated as the focus of contrastive learning or feature optimization in multimodal physiological signal processing. This can be an EEG modality, an ECG modality, a respiratory signal modality, or a snoring signal modality, and is dynamically assigned based on task requirements. By setting the target modality, it is possible to selectively focus on a specific signal type during multimodal joint analysis, improving the effectiveness of targeted feature optimization.

[0153] The modality-specific representation corresponding to the target modality is removed from the set of all modality-specific representations. The set of all modality-specific representations refers to the collection of all modality-specific representations generated by the corresponding modality encoder for each physiological signal in the current data segment or training sample. This collection includes modality-specific representations for EEG, ECG, respiratory signals, and snoring signals. The structure of the set can be a list, dictionary, or tensor matrix, designed based on the encoder output structure. The removal operation refers to removing the corresponding modality-specific representation from the set based on the specific type of the target modality, ensuring that the remaining set only retains the modality-specific representations of the non-target modality, eliminating redundant information interference, and ensuring the logical accuracy of the aggregation operation.

[0154] Aggregation operations are performed on the remaining modality-specific representations. Aggregation operations refer to integrating the modality-specific representations of multiple non-target modalities into a single expression form through methods such as arithmetic averaging, weighted averaging, feature splicing, or multi-layer perceptron fusion to form an intermediate aggregate representation. During the aggregation process, the fusion strategy can be dynamically adjusted according to the signal quality, task relevance, or preset weights of each modality to improve the expression efficiency and semantic consistency of the aggregation results, ensure that the intermediate aggregate representation has rich information integration capabilities, and can effectively capture the collaborative features between non-target modalities.

[0155] The intermediate aggregate representation is labeled as the positive sample aggregate representation corresponding to the target modality. A positive sample aggregate representation is a feature expression used to improve the correlation between the target modality and the overall multimodal information in contrastive learning or feature alignment tasks, and has a positive guiding effect. This labeling operation clearly associates the intermediate aggregate representation with the current target modality, providing standardized input for subsequent similarity calculations, feature optimization, and model updates, ensuring feature consistency and expression coordination in the system in a multimodal environment.

[0156] This embodiment removes the modality-specific representation corresponding to the target modality, retaining only the multi-source information of the non-target modality and performing aggregation. This effectively integrates deep collaborative features between different signal types, improves the overall structural and semantic consistency of the non-target modality information expression, and generates a positive sample aggregate representation with high information density. Associating the positive sample aggregate representation with the target modality can enhance the feature alignment effect between the target modality and the overall multimodal system during the multimodal comparison optimization process, improve feature learning efficiency and expression quality, enhance the system's joint modeling and task adaptation capabilities in complex physiological signal environments, and further optimize the overall system performance.

[0157] In one embodiment, the above step S50 includes:

[0158] S501, inputting the damaged data segment into a Transformer block sequence of a decoder for processing to generate transform decoding features;

[0159] S502, inputting the transform decoding feature into a convolution block sequence of a decoder for processing to generate a convolution decoding feature;

[0160] S503, performing a residual connection operation on the convolutional decoding feature to generate a residual connection feature;

[0161] S504, performing an upsampling operation on the residual connection feature to generate an upsampled feature;

[0162] S505: Process the upsampled features through an output convolutional layer to generate a reconstructed data segment.

[0163] In this embodiment, a damaged data segment is input into a decoder to generate a reconstructed data segment. This first involves inputting the damaged data segment into a sequence of Transformer blocks in the decoder for processing. A damaged data segment refers to a multimodal physiological signal data set that has been subjected to data augmentation operations to introduce masking, noise, or missing information. This includes the overall expression of electroencephalogram (EEG), electrocardiogram (ECG), respiratory signals, and snoring signals under controlled interference, and possesses incomplete and noisy characteristics. The decoder's sequence of Transformer blocks is a feature reconstruction module based on a multi-head self-attention mechanism and a feedforward network structure, capable of capturing global dependencies across time and modality. After the damaged data segment is input into this sequence, the self-attention operation dynamically captures high-order dependency features within and between signals. The feedforward network then enhances the nonlinear expression, ultimately generating transform decoding features. The transform decoding features are high-dimensional expression matrices that have undergone structural reshaping, information fusion, and feature correction, preserving the key information structure and reconstruction potential of the damaged data.

[0164] The transform decoded features are fed into the decoder's convolutional block sequence for processing. This convolutional block sequence, comprised of multiple standard convolutional layers, activation functions, and normalization modules, is capable of extracting local features, restoring spatial structure, and refining representations. Through layer-by-layer convolution, the multi-scale information in the transform decoded features is effectively extracted and reconstructed, generating convolutional decoded features. These features incorporate details of the local spatiotemporal structure, significantly improving the integrity and accuracy of the data's reconstructed representation.

[0165] A residual connection operation is performed on the convolutional decoded features. Residual connections are a structural optimization technique in deep neural networks. By element-by-element summation of the input features and the convolution output, they alleviate the vanishing gradient problem in information transfer and enhance the network's expressive depth and stability. This residual connection ensures that key information in the convolutional decoded features is not lost during transfer, generating residual connection features that preserve the original structural information while further improving the continuity and consistency of the reconstruction effect.

[0166] Upsampling is performed on the residual connection features. This operation uses interpolation, deconvolution, or nearest neighbor methods to gradually restore the low-resolution feature space to a high-resolution structure close to the original input, reconstructing the spatial position and distribution of the missing information to generate upsampled features. The upsampled features have the ability to express information close to the true signal resolution, ensuring that the reconstruction results closely match the original data in both time and space.

[0167] The upsampled features are processed through the output convolutional layer, which typically uses a structure with strictly defined kernel sizes and step sizes. This further fine-tunes the feature representation, corrects for local deviations and reconstruction errors, and ultimately generates a reconstructed data segment. The reconstructed data segment is a high-fidelity signal representation that closely resembles the structure and content of the preprocessed data segment, exhibiting integrity, continuity, and multimodal coordination to meet the requirements of subsequent tasks.

[0168] This embodiment effectively recovers multimodal physiological signal information from damaged data segments through global dependency feature extraction and correction based on a Transformer block sequence, combined with local structure reconstruction using a convolutional block sequence, information preservation through residual connections, spatial restoration through upsampling, and fine-tuning of output convolutions. This generates reconstructed data segments with complete structure, rich features, and high restoration. This process enhances the system's ability to recover damaged information, improves the continuity and accuracy of multimodal data expression, optimizes subsequent feature extraction and task execution based on the reconstructed data, and significantly improves the performance and stability of the overall system in multimodal physiological signal reconstruction and analysis.

[0169] In one embodiment, the above step S60 includes:

[0170] S601, determining the cosine similarity between the modality-specific representation of the target modality and the aggregated representation of the positive sample as the positive sample similarity;

[0171] S602, determining other pre-processed data segments relative to the data segment to which the target modality belongs;

[0172] S603, extracting a modality-specific representation of the same modality from the other preprocessed data segments as a negative sample representation for comparison;

[0173] S604, determining the cosine similarity between the modality-specific representation of the target modality and the comparison negative sample representation as a negative sample similarity set;

[0174] S605, determining a contrast loss based on the positive sample similarity and the negative sample similarity set;

[0175] S606, determining a short-time Fourier transform difference between the reconstructed data segment and the corresponding original pre-processed data segment;

[0176] S607, determining a reconstruction loss based on the short-time Fourier transform difference;

[0177] S608, combining the contrast loss and the reconstruction loss to form a joint loss;

[0178] S609 , minimizing the joint loss through back propagation, updating the trainable parameters of the modality encoder and decoder, and generating updated modality encoder and decoder.

[0179] In this embodiment, the modality encoder and decoder are updated by increasing the similarity between the modality-specific representation of the target modality and the positive sample aggregate representation, reducing the similarity between the modality-specific representation of the target modality and the modality-specific representation of other preprocessed data segments, and reducing the difference between the reconstructed data segments and the preprocessed data segments. The method first includes determining the cosine similarity between the modality-specific representation of the target modality and the positive sample aggregate representation as the positive sample similarity. The modality-specific representation of the target modality is a high-dimensional feature expression generated by the corresponding modality encoder based on the target modality signal in the damaged data segment. The positive sample aggregate representation is generated by aggregating the modality-specific representations of all non-target modalities except the target modality. The two together reflect the fusion information expression between different modalities. Cosine similarity measures the feature consistency between the two based on the angle between spatial vectors. The value range is -1 to 1. The larger the value, the closer the representation. This similarity is determined as the positive sample similarity and is used to quantify the effectiveness of multimodal information fusion.

[0180] Determine other preprocessed data segments relative to the data segment belonging to the target modality. The data segment belonging to the target modality is the complete multimodal physiological signal segment currently being processed. Other preprocessed data segments refer to homologous data sets that are different from the data segment belonging to the target modality. They maintain the consistency of the multimodal structure but come from different time segments or sample groups. They are used to construct a negative sample comparison group to enhance the system's discriminability.

[0181] Modality-specific representations of the same modality are extracted from other preprocessed data segments as comparative negative sample representations. The same modality is limited to physiological signals consistent with the target modality type (such as one of the electroencephalogram, electrocardiogram, respiratory signal, and snoring signal). The feature expressions obtained by the corresponding modality encoders form a negative sample feature set with unified structure and different sources.

[0182] The cosine similarity between the modality-specific representation of the target modality and the contrasting negative sample representation is determined as the negative sample similarity set. Based on the cosine similarity calculation results, multiple sets of similarity indicators are formed to reflect the distance relationship between the target modality representation and different negative sample representations. The overall set is used to characterize the negative sample distribution pattern and the discriminability of the target information.

[0183] The contrastive loss is determined based on the similarity sets of positive samples and negative samples. The contrastive loss is constructed through a mathematical function, usually using the cross entropy or NT-Xent loss under the contrastive learning framework. It encourages the target modality to be close to the positive sample features, while pushing the distance to the negative sample features away, thereby optimizing the discriminability and fusion expression capabilities of the multimodal representation.

[0184] Determine the short-time Fourier transform difference between the reconstructed data segment and the corresponding original preprocessed data segment. The short-time Fourier transform difference is based on frequency domain analysis, which measures the deviation between the reconstructed data segment and the undamaged original data segment in terms of frequency structure, energy distribution, etc., reflects the fidelity and continuity of the decoder reconstructed signal, and ensures that the multimodal information reconstruction result is true and valid.

[0185] The reconstruction loss is determined based on the short-time Fourier transform difference. The reconstruction loss quantifies the frequency domain difference through the L1 norm, L2 norm or other error functions, optimizes the decoder structure, and improves the overall restoration effect and local detail retention ability of the reconstructed data segment.

[0186] The contrast loss and reconstruction loss are combined to form a joint loss. The joint loss incorporates multimodal discriminative optimization and signal reconstruction quality into the optimization objectives through weighted summation or other fusion strategies, thereby enhancing the overall model training effect and ensuring the coordinated consistency of multimodal information fusion and restoration processes.

[0187] Backpropagation minimizes the joint loss, updates the trainable parameters of the modal encoder and decoder, and generates updated modal encoders and decoders. Backpropagation optimizes the neural network structure layer by layer based on gradient calculation and parameter adjustment, resulting in simultaneous improvements in the modal encoder's ability to distinguish and express multimodal information and the decoder's ability to reconstruct signals. This results in an updated model with optimized structure and enhanced performance, providing high-quality, multimodal information expression support for subsequent tasks.

[0188] Example: In the healthcare field, multimodal sleep monitoring data is processed to collect multimodal physiological signal data, including EEG, ECG, respiratory, and snoring signals, ensuring that each type of signal is recorded synchronously within the same time period. The collected data is first low-pass filtered to remove high-frequency interference and environmental noise, then uniformly resampled to a set frequency to ensure consistency across all modal data. The data is then divided into 30-second segments, and signal normalization is performed on each segment to generate preprocessed data segments suitable for subsequent processing.

[0189] Data augmentation is performed on preprocessed multimodal data segments. First, the signal channel status of each data segment is analyzed to establish a set of signal channels for the current data segment. Within this set of signal channels, a random selection of signal channels is used as masked channels. These selected channels are then zeroed, completely eliminating valid data from the zeroed channels within the signal segment, thereby simulating a signal-absence scenario. Simultaneously, a Gaussian white noise signal is generated based on predefined parameters and injected into the masked signal segment, resulting in a structurally intact, but impaired, data segment containing both signal damage and noise contamination.

[0190] The EEG, ECG, respiratory, and snoring signals in the damaged data segment are fed into the EEG, ECG, respiratory, and snoring encoders, respectively, to extract modality-specific representations for each physiological signal, ensuring that each modality encoder independently models different signal structures. For the currently selected target modality, the modality-specific representation corresponding to that modality is removed, while the modality-specific representations of the other non-target modalities are retained and aggregated. An intermediate aggregate representation is generated using arithmetic averaging or adaptive weighting, and this intermediate aggregate representation is labeled as the positive sample aggregate representation corresponding to the current target modality.

[0191] The damaged data segment is fed into the decoder, where it passes through a sequence of Transformer blocks to extract global temporal features. This is followed by a sequence of convolutional blocks to extract local spatial features. Residual connections are used during the convolution process to enhance the flow of deep information. The convolutional features are then upsampled to restore the signal to its original time dimension, and then passed through the output convolutional layer to generate the reconstructed data segment.

[0192] For the target modality currently being processed, the cosine similarity between the modality-specific representation of the target modality and the aggregate representation of the positive samples is calculated as the positive sample similarity. At the same time, other preprocessed data segments that are different from the data segments belonging to the target modality are selected, and the modality-specific representation of the same modality is extracted from them as the negative sample representation for comparison. The cosine similarity between the modality-specific representation of the target modality and the negative sample representation for comparison is calculated to form a negative sample similarity set. Based on the positive and negative sample similarities, the contrast loss is calculated to improve the similarity between the target modality and the aggregate representation of the positive samples and reduce the similarity between the target modality and the negative samples. The reconstruction loss is calculated by analyzing the frequency domain differences between the reconstructed data segment and the corresponding original preprocessed data segment through short-time Fourier transform. The contrast loss and the reconstruction loss are combined into a joint loss, and the trainable parameters of the modality encoder and decoder are optimized through backpropagation to generate an updated model with optimized structure.

[0193] During the target task execution phase, multimodal physiological signal data segments used for sleep staging are input. The same signal preprocessing methods as in the training phase are applied, including filtering, resampling, data segmentation, and normalization. The preprocessed data segments are input into the updated modal encoder to generate task feature representations, which are then input into the trained support vector machine classification module. The sleep stage classification results are output, ultimately generating a complete sleep structure classification result that includes light sleep, deep sleep, rapid eye movement, and wakefulness. The above processing flow effectively supports the complete processing of multimodal physiological signals, abnormal signal recovery, and sleep stage classification tasks in sleep monitoring scenarios, and is suitable for data processing and analysis systems of multi-channel sleep monitoring equipment.

[0194] In the field of financial technology, for customer health data processing scenarios in health insurance product design and health management services, multimodal physiological signal data, including electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, respiratory signals, and snoring signals, is collected. All physiological signals are collected and stored synchronously during the same time period to ensure time alignment between different signal sources. The collected multimodal physiological signal data is first low-pass filtered to eliminate high-frequency environmental interference and device noise generated by wearable devices during outdoor use. Subsequently, the physiological signals of different modalities are uniformly resampled to the set frequency to resolve the problem of inconsistent sampling rates between devices. The signals are then split into equal-length data segments through time window partitioning, and standardized processing is performed to eliminate individual differences, forming structurally standardized pre-processed data segments.

[0195] Data augmentation is used to construct model training data for multiple batches of physiological monitoring data submitted by customers in fintech scenarios. First, signal channel analysis is performed to analyze the signal channel conditions uploaded by the customer's wearable device. Some signal channels are randomly selected from the available channels as masking targets. Channel zeroing is performed on the selected channels to create a signal-absence environment. Gaussian white noise is dynamically generated to simulate device jitter or transmission noise. This Gaussian white noise is then injected into the signal segments, ultimately generating corrupted data segments containing both signal loss and noise interference. This improves the robustness of models in fintech scenarios to data quality fluctuations.

[0196] The EEG, ECG, respiratory, and snoring signals in the damaged data segment are fed into independently designed modality encoders to extract modality-specific representations for each physiological signal. For the currently selected target modality, the modality-specific representations of the target modality are discarded, and the modality-specific representations of non-target modalities are aggregated. This aggregation operation generates an aggregated representation of positive samples, ensuring the joint representation capability across different signal modalities.

[0197] The damaged data segment is input into the decoder structure, and the Transformer block is used to process the multimodal time series data to extract the global dependencies across time. Then, the local signal pattern is extracted through the convolution block, and the residual connection structure is used to optimize the gradient transfer in the decoding path. The signal time dimension is restored through the upsampling operation, and the reconstructed data segment is generated through the output convolution layer to reconstruct the data after signal loss and noise injection.

[0198] For the current target modality, the cosine similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples is calculated to determine the positive sample similarity. Other health monitoring data segments that are different from the data segment belonging to the target modality are selected, and the modality-specific representation of the same modality is extracted as the negative sample representation for comparison. The cosine similarity between the modality-specific representation of the target modality and the negative sample representation is calculated to form a negative sample similarity set. Based on the positive and negative sample similarities, a contrastive loss is calculated. Through contrastive learning, the similarity between the target modality and the positive samples is improved, while the similarity with the negative samples is reduced. Simultaneously, the difference in the short-time Fourier transform between the reconstructed data segment and the corresponding original data segment is calculated to generate a reconstruction loss. The contrastive loss and the reconstruction loss are combined to form a joint loss. The trainable parameters of the modality encoder and decoder are optimized through backpropagation to obtain a trained and updated model.

[0199] In financial technology applications, the real-time health monitoring data uploaded by insurance customers is processed based on the updated modal encoder, and data preprocessing consistent with the training process is performed, including filtering, resampling, data segmentation and standardization. The preprocessed data is input into the updated modal encoder to generate a characteristic representation of the customer's health status. The customer's current physiological state is identified through the support vector machine classification module, and sleep staging results, breathing abnormality detection results and physiological state change trends are generated. This provides real-time data support for insurance product health point calculation, dynamic premium adjustment and health management services. At the same time, it can be combined with the customer's historical health monitoring data for long-term risk assessment.

[0200] This embodiment combines comparative learning of positive and negative sample similarities with joint optimization of frequency domain reconstruction errors to enhance the modal encoder's ability to fuse and differentiate multimodal features. It also enhances the decoder's reconstruction quality for damaged multimodal physiological signals, ensuring the accuracy of multimodal information expression, the integrity of the reconstruction results, and the stability of the system's overall performance. This optimization process improves the system's adaptability to complex multimodal physiological signal data, enhances the consistency of data expression, and the effectiveness of downstream tasks, meeting the requirements of high-reliability, multimodal collaborative information processing.

[0201] In one embodiment, a multimodal physiological signal processing device is provided, which corresponds one-to-one to the multimodal physiological signal processing method in the above embodiment. Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multimodal physiological signal processing device of the present invention. It includes a data acquisition and preprocessing module 10, a data enhancement module 20, a modality encoder module 30, a modality aggregation module 40, a decoder module 50, an optimization and update module 60, and a task execution module 70. Each functional module is described in detail below:

[0202] The data acquisition and preprocessing module 10 is used to acquire multimodal physiological signal data and preprocess the multimodal physiological signal data to obtain a plurality of preprocessed data segments;

[0203] A data enhancement module 20 is configured to perform data enhancement processing on the preprocessed data segment to generate a damaged data segment;

[0204] a modality encoder module 30 for inputting each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation;

[0205] A modality aggregation module 40 is configured to aggregate the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation;

[0206] A decoder module 50 is configured to input the damaged data segment into a decoder to generate a reconstructed data segment;

[0207] an optimization and updating module 60 for updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments;

[0208] The task execution module 70 is used to execute the target task based on the updated modal encoder and generate the target task result.

[0209] In one embodiment, the data acquisition and preprocessing module 10 is specifically configured to:

[0210] Collect multimodal physiological signal data;

[0211] performing low-pass filtering on the multimodal physiological signal data to obtain filtered physiological signal data;

[0212] resampling the filtered physiological signal data to a uniform frequency to obtain resampled physiological signal data;

[0213] Dividing the resampled physiological signal data into a plurality of data segments of preset duration;

[0214] Standardization processing is performed on the multiple data segments of preset time lengths to generate the multiple pre-processed data segments.

[0215] In one embodiment, the data enhancement module 20 is specifically configured to:

[0216] Performing a signal channel analysis operation on the preprocessed data segment to generate a signal channel set;

[0217] performing a random masking channel selection operation based on the signal channel set to generate a selected masking channel set;

[0218] performing a channel zeroing operation on the preprocessed data segment according to the selected masked channel set to generate a channel masked data segment;

[0219] Generate a Gaussian white noise signal;

[0220] The Gaussian white noise signal is injected into the channel masked data segment to generate a noise injected data segment, and the noise injected data segment is marked as a damaged data segment.

[0221] In one embodiment, the modal encoder module 30 is specifically configured to:

[0222] separating an electroencephalogram signal portion, an electrocardiogram signal portion, a respiratory signal portion, and a snoring signal portion from the damaged data segment;

[0223] inputting the electroencephalogram signal portion into an electroencephalogram encoder for processing to generate an electroencephalogram modality-specific representation;

[0224] inputting the electrocardiogram signal portion into an electrocardiogram encoder for processing to generate an electrocardiogram modality-specific representation;

[0225] Inputting the respiratory signal portion into a respiratory signal encoder for processing to generate a respiratory signal modality-specific representation;

[0226] The snoring signal portion is input into a snoring encoder for processing to generate a modality-specific representation of the snoring signal.

[0227] In one embodiment, the modality aggregation module 40 is specifically configured to:

[0228] Determine the current target modality;

[0229] Removing the modality-specific representation corresponding to the target modality from the set of all modality-specific representations to obtain the remaining modality-specific representations;

[0230] performing an aggregation operation on the remaining modality-specific representations to generate an intermediate aggregated representation;

[0231] The intermediate aggregate representation is marked as a positive sample aggregate representation corresponding to the target modality.

[0232] In one embodiment, the decoder module 50 is specifically configured to:

[0233] Inputting the damaged data segment into a decoder's Transformer block sequence for processing to generate transform decoding features;

[0234] Inputting the transform decoding features into a convolution block sequence of a decoder for processing to generate convolution decoding features;

[0235] Performing a residual connection operation on the convolutional decoding features to generate residual connection features;

[0236] Performing an upsampling operation on the residual connection feature to generate an upsampling feature;

[0237] The upsampled features are processed through an output convolutional layer to generate reconstructed data segments.

[0238] In one embodiment, the optimization and update module 60 is specifically configured to:

[0239] Determine the cosine similarity between the modality-specific representation of the target modality and the aggregated representation of the positive sample as the positive sample similarity;

[0240] Determining other preprocessed data segments relative to the data segment to which the target modality belongs;

[0241] Extracting modality-specific representations of the same modality from the other preprocessed data segments as negative sample representations for comparison;

[0242] Determine the cosine similarity between the modality-specific representation of the target modality and the comparison negative sample representation as a negative sample similarity set;

[0243] Determining a contrast loss based on the positive sample similarity and the negative sample similarity set;

[0244] determining a short-time Fourier transform difference between the reconstructed data segment and a corresponding original preprocessed data segment;

[0245] determining a reconstruction loss based on the short-time Fourier transform difference;

[0246] Combining the contrast loss and the reconstruction loss to form a joint loss;

[0247] The joint loss is minimized by back-propagation, and the trainable parameters of the modality encoder and decoder are updated to generate updated modality encoder and decoder.

[0248] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a multimodal physiological signal processing method.

[0249] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a multimodal physiological signal processing method.

[0250] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0251] Acquiring multimodal physiological signal data, and preprocessing the multimodal physiological signal data to obtain a plurality of preprocessed data segments;

[0252] Performing data enhancement processing on the preprocessed data segment to generate a damaged data segment;

[0253] Inputting each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation;

[0254] Aggregating the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation;

[0255] Inputting the damaged data segment into a decoder to generate a reconstructed data segment;

[0256] updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments;

[0257] The target task is performed based on the updated modal encoder to generate the target task result.

[0258] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0259] Acquiring multimodal physiological signal data, and preprocessing the multimodal physiological signal data to obtain a plurality of preprocessed data segments;

[0260] Performing data enhancement processing on the preprocessed data segment to generate a damaged data segment;

[0261] Inputting each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation;

[0262] Aggregating the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation;

[0263] Inputting the damaged data segment into a decoder to generate a reconstructed data segment;

[0264] updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments;

[0265] The target task is performed based on the updated modal encoder to generate the target task result.

[0266] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0267] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0268] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0269] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A multimodal physiological signal processing method, characterized in that: The following steps are involved: Acquiring multimodal physiological signal data, and preprocessing the multimodal physiological signal data to obtain a plurality of preprocessed data segments; Performing data enhancement processing on the preprocessed data segment to generate a damaged data segment; Inputting each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation; Aggregating the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation; Inputting the damaged data segment into a decoder to generate a reconstructed data segment; updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments; The target task is performed based on the updated modal encoder to generate the target task result.

2. The multimodal physiological signal processing method according to claim 1, wherein: Acquiring multimodal physiological signal data and preprocessing the multimodal physiological signal data to obtain a plurality of preprocessed data segments, including: Collect multimodal physiological signal data; performing low-pass filtering on the multimodal physiological signal data to obtain filtered physiological signal data; resampling the filtered physiological signal data to a uniform frequency to obtain resampled physiological signal data; Dividing the resampled physiological signal data into a plurality of data segments of preset duration; Standardization processing is performed on the multiple data segments of preset time lengths to generate the multiple pre-processed data segments.

3. The multimodal physiological signal processing method according to claim 1, wherein: Performing data enhancement processing on the preprocessed data segment to generate a damaged data segment includes: Performing a signal channel analysis operation on the preprocessed data segment to generate a signal channel set; performing a random masking channel selection operation based on the signal channel set to generate a selected masking channel set; performing a channel zeroing operation on the preprocessed data segment according to the selected masked channel set to generate a channel masked data segment; Generate a Gaussian white noise signal; The Gaussian white noise signal is injected into the channel masked data segment to generate a noise injected data segment, and the noise injected data segment is marked as a damaged data segment.

4. The multimodal physiological signal processing method according to claim 1, wherein: Inputting each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation, including: separating an electroencephalogram signal portion, an electrocardiogram signal portion, a respiratory signal portion, and a snoring signal portion from the damaged data segment; inputting the electroencephalogram signal portion into an electroencephalogram encoder for processing to generate an electroencephalogram modality-specific representation; inputting the electrocardiogram signal portion into an electrocardiogram encoder for processing to generate an electrocardiogram modality-specific representation; Inputting the respiratory signal portion into a respiratory signal encoder for processing to generate a respiratory signal modality-specific representation; The snoring signal portion is input into a snoring encoder for processing to generate a modality-specific representation of the snoring signal.

5. The multimodal physiological signal processing method according to claim 1, wherein: Aggregate the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation, including: Determine the current target modality; Removing the modality-specific representation corresponding to the target modality from the set of all modality-specific representations to obtain the remaining modality-specific representations; performing an aggregation operation on the remaining modality-specific representations to generate an intermediate aggregated representation; The intermediate aggregate representation is marked as a positive sample aggregate representation corresponding to the target modality.

6. The multimodal physiological signal processing method according to claim 1, wherein: The damaged data segment is input into a decoder to generate a reconstructed data segment, including: Inputting the damaged data segment into a decoder's Transformer block sequence for processing to generate transform decoding features; Inputting the transform decoding features into a convolution block sequence of a decoder for processing to generate convolution decoding features; Performing a residual connection operation on the convolutional decoding features to generate residual connection features; Performing an upsampling operation on the residual connection feature to generate an upsampling feature; The upsampled features are processed through an output convolutional layer to generate reconstructed data segments.

7. The multimodal physiological signal processing method according to claim 1, wherein: Updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregate representation of the positive samples, and decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments, including: Determine the cosine similarity between the modality-specific representation of the target modality and the aggregated representation of the positive sample as the positive sample similarity; Determining other preprocessed data segments relative to the data segment to which the target modality belongs; Extracting modality-specific representations of the same modality from the other preprocessed data segments as negative sample representations for comparison; Determine the cosine similarity between the modality-specific representation of the target modality and the comparison negative sample representation as a negative sample similarity set; Determining a contrast loss based on the positive sample similarity and the negative sample similarity set; determining a short-time Fourier transform difference between the reconstructed data segment and a corresponding original preprocessed data segment; determining a reconstruction loss based on the short-time Fourier transform difference; Combining the contrast loss and the reconstruction loss to form a joint loss; The joint loss is minimized by back-propagation, and the trainable parameters of the modality encoder and decoder are updated to generate updated modality encoder and decoder.

8. A multimodal physiological signal processing device, characterized in that: The multimodal physiological signal processing device comprises: a data acquisition and preprocessing module, configured to acquire multimodal physiological signal data and preprocess the multimodal physiological signal data to obtain a plurality of preprocessed data segments; A data enhancement module, configured to perform data enhancement processing on the preprocessed data segment to generate a damaged data segment; a modality encoder module, configured to input each physiological signal in the damaged data segment into a corresponding modality encoder to generate a modality-specific representation; A modality aggregation module, configured to aggregate the modality-specific representations of all non-target modalities except the target modality to generate a positive sample aggregate representation; a decoder module, configured to input the damaged data segment into a decoder to generate a reconstructed data segment; an optimization and update module for updating the modality encoder and the decoder by increasing the similarity between the modality-specific representation of the target modality and the aggregated representation of the positive samples, decreasing the similarity between the modality-specific representation of the target modality and the modality-specific representations of other preprocessed data segments, and decreasing the difference between the reconstructed data segments and the preprocessed data segments; The task execution module is used to execute the target task based on the updated modal encoder and generate the target task result.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a multimodal physiological signal processing program stored in the memory and capable of running on the processor. When the multimodal physiological signal processing program is executed by the processor, the steps of the multimodal physiological signal processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a multimodal physiological signal processing program, which, when executed by a processor, implements the steps of the multimodal physiological signal processing method according to any one of claims 1 to 7.

Images