Self-supervised learning and multi-modal biological medical signal processing method, system and terminal based on same

Abstract

The application discloses a method and system for processing multi-modal biological medical signals based on self-supervised learning and a terminal, and belongs to the fields of biomedical engineering and information technology, and comprises the following steps: extracting feature representations of multi-modal data by using an unsupervised learning autoencoder; taking the feature representations of source domain data as input and the feature representations of target domain data as output, and training a large language model based on a Transformer architecture based on maximum likelihood estimation. The correlation and common features between multi-modal data are learned by the autoencoder, and the feature representations extracted by the autoencoder are applied to the pre-training of the large language model, so that the large language model learns more abstract and advanced feature representations, thereby improving the processing capacity and generalization performance of the multi-modal data; meanwhile, since the unsupervised learning method is adopted, the large language model can be pre-trained by using a large amount of unlabeled data, so that the data resources can be better utilized.

Description

Technical Field

[0001] This invention relates to the fields of biomedical engineering and information technology, and in particular to a system and terminal for biomedical signal processing and application based on self-supervised learning and multimodal biomedical signals. Background Technology

[0002] Traditional medical monitoring methods are typically limited to the collection and analysis of single or a limited number of physiological signals, making it difficult to comprehensively understand an individual's physiological state and changes, thus limiting the ability to understand and intervene in a patient's overall health status. Modern medicine, however, tends to require comprehensive monitoring of multiple physiological parameters and the ability to respond promptly to changes in different parameters in order to provide personalized services.

[0003] Traditional biomedical signal processing primarily relies on manually designed feature extraction methods, which limits the system's adaptability to individual differences and physiological variations. However, since single-modal data often fails to provide sufficient information, research focus has gradually shifted to multimodal data fusion for motion image recognition. Multimodal data encompasses different data types, such as images, text, and speech. Multimodal data acquisition techniques can simultaneously acquire multiple types of data, such as motion data, physiological data, and image data, thereby providing more comprehensive and accurate information for various assessments. Although multimodal data holds potential in biomedical engineering and information technology, traditional neural network models suffer from insufficient representation of multimodal data features, failing to fully exploit the relevant information between different modalities. Furthermore, feature learning in various fields mainly utilizes supervised or semi-supervised multimodal data, requiring labeling of the multimodal data, making it difficult to directly train models using unlabeled data. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of the prior art and provide a method, system and terminal based on self-supervised learning and multimodal biomedical signal processing.

[0005] The objective of this invention is achieved through the following technical solution: a method based on self-supervised learning and multimodal biomedical signal processing, which includes a training step:

[0006] An autoencoder using unsupervised learning is used to extract feature representations of multimodal data, which includes patient physiological data as source domain data and motor imagery data as target domain data.

[0007] Using the feature representation of source domain data as input and the feature representation of target domain data as output, the large language model based on the Transformer architecture is trained based on maximum likelihood estimation, thereby optimizing the parameters of the large language model and completing the training of the large language model.

[0008] In one example, the feature representation extracted by the autoencoder from multimodal data includes:

[0009] The encoder in the autoencoder is used to obtain the vector representations of the source domain data and the target source data in the high-dimensional feature space, and then input them into the decoder in the autoencoder.

[0010] The decoder processes the input data step by step and generates a hidden state vector at each time step. Using the hidden state vector of the decoder as input, the decoder's autoregressive mechanism generates feature information of the corresponding source domain data or target domain data step by step. At each time step, the decoder uses the current hidden state vector to predict the feature value of the next time step, and repeats this prediction step until a complete data feature sequence is generated.

[0011] In one example, when training a large language model, the cross-entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true probability distribution; the expression for the cross-entropy loss function is:

[0012]

[0013] Among them, L MLE The maximum likelihood estimation loss of the large language model is represented by N; the number of training samples is represented by P; and the conditional probability of the input sequence is represented by T. i Let x represent the sequence length of the i-th sample. i，t x represents the t-th data point in the i-th sample; i＜t Let represent the first t-1 data points in the i-th sample; α represents the learning rate.

[0014] In one example, the method further includes a step of generating adversarial alignment:

[0015] A conditional generative adversarial network (GAN) is constructed, using a trained large language model as the generator and a convolutional neural network as the discriminator. Generative adversarial learning is then performed based on the GAN, with the conditional information consisting of the features generated by the generator and the task label. The discriminator determines whether the generated target domain data and source domain data are authentic, and the cycle consistency loss L in the generative adversarial learning process is calculated. cyc Combating Losses L adv ;

[0016] The alignment loss L is measured by comparing the feature distribution of the data generated by the alignment generator with that of the real data using batch normalization and maximum mean normalization. MMD ;

[0017] Combining cycle consistency loss L cyc Combating Losses L adv Alignment loss L MMDA dynamic weighting mechanism is introduced to determine the final loss function L. total Calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator, and transfer matrix.

[0018] In one example, the alignment loss L MMD The expression is:

[0019]

[0020]

[0021] in, F represents the source domain feature representation; t represents the feature representation of the target domain; N represents the number of samples; k represents the Gaussian kernel function, used to calculate the similarity between samples; i and j are both feature representation indices.

[0022] In one example, the method further includes a multimodal data fusion step:

[0023] The size of the aligned feature vectors is standardized using bilinear interpolation.

[0024] Bilinear pooling layers are used to fuse the features of the two modalities to obtain the fused feature representation;

[0025] The cross-modal attention mechanism is used to calculate the weights between features of different modalities, and the learned attention weights are used to perform weighted fusion of features after bilinear pooling to obtain the modal data fusion result.

[0026] In one example, the method further includes classifier training and feature training steps:

[0027] The classifier was trained using the multimodal fusion dataset;

[0028] The multimodal data is identified based on the trained classifier, and the functional module device is controlled to work based on the identification results, such as controlling the rehabilitation robot to carry out rehabilitation training.

[0029] It should be further noted that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.

[0030] The present invention also includes a self-supervised learning and multimodal biomedical signal processing system, which has the same inventive concept as any or a combination of the above examples. The system includes a training module for extracting feature representations of multimodal data using an unsupervised learning autoencoder, taking the feature representations of source domain data as input and the feature representations of target domain data as output, training a large language model based on the Transformer architecture based on maximum likelihood estimation, thereby optimizing the parameters of the large language model and completing the training of the large language model; the multimodal data includes patient physiological data as source domain data and motor imagery data as target domain data.

[0031] In one example, the system further includes:

[0032] The Generative Adversarial Alignment (GAP) module constructs a Conditional Generative Adversarial Network (CGRAN) using a trained large language model as the generator and a convolutional neural network as the discriminator. GAPAN then performs generative adversarial learning, with the conditional information consisting of features generated by the generator and task labels. The discriminator determines whether the generated target domain data and source domain data are authentic and calculates the cycle consistency loss L in the GAP learning process. cyc Combating Losses L adv The alignment loss L is measured by comparing the data features generated by the alignment generator with the feature distribution of the real data using batch normalization and maximum mean normalization. MMD Combining cross-entropy loss and cycle consistency loss L cyc Combating Losses L adv Alignment loss L MMD A dynamic weighting mechanism is introduced to determine the final loss function L. total Calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator, and transfer matrix.

[0033] The multimodal data fusion module is used to unify the size of the aligned feature vectors using bilinear interpolation; it uses a bilinear pooling layer to fuse the features of two modalities to obtain the fused feature representation; then it uses a cross-modal attention mechanism to calculate the weights between the features of different modalities, and uses the learned attention weights to perform weighted fusion on the bilinear pooled features to obtain the modal data fusion result.

[0034] The classification training module is used to train the classifier using the multimodal fusion dataset;

[0035] The functional training control module identifies multimodal data based on the completed classifier and controls the functional module device to work based on the identification results, such as controlling the rehabilitation robotic arm to carry out rehabilitation training.

[0036] It should be further noted that the technical features corresponding to the above system examples can be combined or replaced to form new technical solutions.

[0037] The present invention also includes a storage medium storing computer instructions that, when executed, perform the steps of the self-supervised learning and multimodal biomedical signal processing method formed by any or more of the above examples.

[0038] The present invention also includes a terminal comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the steps of the self-supervised learning and multimodal biomedical signal processing method formed by any or more of the above examples when executing the computer instructions.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] 1. In one example, an autoencoder learns the correlations and common features between multimodal data, and the feature representations extracted by the autoencoder are applied to the pre-training of a large language model. This transforms different types of data into the same feature representation, achieving unified processing of different data types. Consequently, the large language model learns more abstract and advanced feature representations, thereby improving its ability to process multimodal data and its generalization performance. Furthermore, the large language model uses maximum likelihood estimation and incorporates regularization parameters during prediction to optimize its fitting and generalization abilities, further enhancing its performance and effectiveness in generation tasks. Simultaneously, because it employs an unsupervised learning method, it can utilize a large amount of unlabeled data for pre-training, thus better utilizing data resources and making it suitable for fields with scarce data samples, such as personalized medicine, disease prevention, and rehabilitation assistance.

[0041] 2. In one example, a conditional generative adversarial network is used to generate feature representations that transfer between the source and target domains, thereby achieving feature alignment between different domains or data distributions. Then, the learned multimodal characteristics are used to construct supervised information for self-supervised learning and fine-tuning of the pre-trained model (generator, discriminator), thereby reducing the dependence on labeled data during training. It can also avoid the problem caused by the deviation between strong prior knowledge and the true data distribution in unsupervised learning, and allow the model to fine-tune the pre-trained model for the target task to obtain a task-specific personalized model, thereby improving the overall performance of the target task.

[0042] 3. In one example, multimodal feature fusion combines feature information from different data sources, thereby improving the flexibility and expressive power of feature fusion. Simultaneously, by learning the correlation weights between features from different modalities through a cross-modal attention mechanism, the correlations between different modalities can be better explored, thus improving the accuracy and stability of feature representation. Attached Figure Description

[0043] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, which are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0044] Figure 1 This is a flowchart of a method in an example of the present invention;

[0045] Figure 2 This is a flowchart of a preferred embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of a multi-input fusion network (MIC-Net) in an example of the present invention. Detailed Implementation

[0047] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the use of ordinal numbers (e.g., "first and second," "first to fourth," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0049] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0050] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] In one example, such as Figure 1 As shown, a method based on self-supervised learning and multimodal biomedical signal processing includes a training step:

[0052] S11: An unsupervised learning autoencoder is used to extract feature representations from multimodal data. The multimodal data includes patient physiological data as source domain data, such as electroencephalogram (EEG) data, electromyogram (EMG) data, blood oxygen saturation data, and heart rate data, as well as motor imagery data as target domain data. The physiological data and motor imagery data are stored in the form of current, voltage, frequency, etc., and used in the training steps of this invention. Preferably, the original multimodal data is segmented to form time-series data blocks, and then an autoencoder is used for feature extraction, converting the feature vectors into specific representations.

[0053] S12: Using the feature representation of the source domain data as input and the feature representation of the target domain data as output, train the large language model based on the Transformer architecture using maximum likelihood estimation, thereby optimizing the parameters of the large language model and completing the training of the large language model. Preferably, the large language model is the GPT model, but it can also be a ChatGLM, BERT, LLM, T5 model, etc.

[0054] This invention utilizes feature representations extracted from autoencoders for pre-training the GPT model. This transforms different types of data into the same feature representation, enabling unified processing of various data types and improving the model's generalization ability. Autoencoders learn low-level features of data, while the GPT model learns high-level semantic representations. Using the feature representations extracted from autoencoders for pre-training the GPT model combines low-level features with high-level semantic representations, resulting in more abstract and advanced feature representations. This leads to a backbone network architecture with strong multimodal signal feature representation capabilities. Furthermore, since an unsupervised learning method is employed, a large amount of unlabeled data can be used for pre-training the GPT model, allowing for better utilization of data resources. Then, the model is fine-tuned using limited labeled data in the target application for transfer learning. Because sufficient pre-training ensures the model parameters are at a good initial point, and the number of parameters requiring updates is small, fine-tuning with only a small number of labeled samples in the downstream task allows the model to converge quickly.

[0055] In one example, a data preprocessing step is included before the training step. Specifically, the source domain samples and target domain samples are first standardized and denoised, and the features of the two-channel EEG data of the target domain are defined as X. egg The source domain electromyography data features are X emg The blood oxygen and heart rate data are X oxy The sum is X hr ;

[0056] S01: During the data acquisition process, the acquired signal is first subjected to baseline drift removal, power frequency interference removal and artifact interference removal preprocessing operations to remove noise and retain useful signals;

[0057] S02: Data is normalized to a uniform scale and time-series aligned to ensure that different types of data can correspond and match, exhibiting consistent temporal characteristics. Motor imagery (EEG) data primarily collects EEG data from channels C3 and C4; electromyography (EMG) data is collected by placing electrodes on both sides of the forearm. The collected data are all time-series data, arranged in two dimensions. Heart rate (Hr) and blood oxygen (Oxy) data are one-dimensional.

[0058] In one example, the autoencoder consists of an encoder E and a decoder D. Input data x is mapped to a vector representation z in a high-dimensional feature space by the encoder, and then mapped back to the original data x' by the decoder. Both encoder E and decoder D are stacks of multiple Transformer layers. Each Transformer layer contains a multi-head self-attention mechanism, a feedforward neural network, etc., so the encoder can also act as a feature extractor. In this case, the feature representation extracted by the autoencoder from multimodal data includes:

[0059] S111: Use the encoder in the autoencoder to obtain the vector representations of the source domain data and the target source data in the high-dimensional feature space, and input them into the decoder in the autoencoder;

[0060] S112: The decoder progressively processes the input data (source domain data such as blood oxygen saturation, heart rate, and dual-channel electromyography data, and target source data such as dual-channel electroencephalography data), generating a hidden state vector at each time step. These hidden state vectors can be considered as encoded representations of the input data. Using the decoder's hidden state vectors as input, the decoder's autoregressive mechanism progressively generates feature information corresponding to the source or target domain data. At each time step, the decoder uses the current hidden state vector to predict the feature value for the next time step, repeating this prediction step until a complete data feature sequence is generated. The data feature sequence generated during the decoding process can be considered a reconstruction of the original domain data. By comparing the differences between the reconstructed data and the original domain data, the performance and accuracy of the encoding and decoding process can be evaluated.

[0061] The encoding layer computation expression is: z = E(x); the input data x is processed by the i-th Transformer layer to obtain the intermediate representation z. i :z i =Transformer i The final encoded representation z is formed by stacking the outputs of all Transformer layers: z = [z1, z2, ..., z...]. n Furthermore, the encoder's output z serves as the decoder's input: x′ = z. The input data x′ is processed by the i-th Transformer layer to obtain an intermediate representation: z. i ′=z i Transformer i (x′), the final decoded result x′ is composed of the stacked outputs of all Transformer layers: x′=[z1′,z2′,...,z′ n ].

[0062] In one example, training a large language model is achieved through maximum likelihood estimation, with the goal of maximizing the likelihood probability of the observed data. The loss function is expressed as:

[0063]

[0064] Among them, L MLE The maximum likelihood estimation loss of the large language model is represented by ; N represents the number of training samples; P represents the conditional probability of the input sequence; x iInput data sequence; θ represents the model parameters. Maximum likelihood estimation is the process of maximizing the likelihood function of the data by adjusting the model parameters, given the observed data. In the GPT model, the likelihood function can be expressed as the conditional probability of the input sequence, i.e.:

[0065]

[0066] To maximize the likelihood function, this example uses the cross-entropy loss function to measure the difference between the probability distribution predicted by the model and the true probability distribution. The expression for the cross-entropy loss function is:

[0067]

[0068] Among them, T i Let x represent the sequence length of the i-th sample. i,t x represents the t-th data point in the i-th sample; i＜t Let represent the first t-1 data points in the i-th sample; α represents the learning rate. By minimizing the cross-entropy loss function and adjusting the model parameters, the likelihood function is maximized, learning the distribution and structure of the data, enabling GPT to accurately represent the classification probability mapping between the source and target domains. The loss function can be the cross-entropy loss function to measure the difference between the generated and real data. During training, optimization algorithms such as stochastic gradient descent can be used to update the model parameters, thereby training the GPT model.

[0069] In one example, the method also includes a step to generate adversarial alignment, implemented by constructing a CAN-BDA network (Cycle-consistent Adversarial-Balanced Distribution Adaptation Network), which includes:

[0070] S21: Construct a conditional generative adversarial network (GAN) using a trained large language model as the generator and a convolutional neural network as the discriminator. Perform generative adversarial learning based on the GAN. The conditional information consists of the feature information generated by the generator and the task label. The discriminator determines whether the generated target domain data and source domain data are authentic, and calculates the cycle consistency loss L in the generative adversarial learning process. cyc Combating Losses L adv ;

[0071] S22: The data features generated by the alignment generator and the feature distribution of the real data are compared using batch normalization and maximum mean diffraction to measure the alignment loss L. MMD ;

[0072] S23: Combining cross-entropy loss and cycle consistency loss L cycCombating Losses L adv Alignment loss L MMD A dynamic weighting mechanism is introduced to determine the final loss function L. total Calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator, and transfer matrix.

[0073] Specifically, before executing step S21, prepare multiple source domain datasets and one target domain dataset. Each dataset contains input signals and corresponding labels.

[0074] Two generators and two discriminators are then introduced to process the source and target domains, respectively. One generator converts source domain data into target domain data, and the other converts target domain data back into source domain data. The two discriminators are used to determine the authenticity of the generated target and source domain data. Specifically, generator G1 maps source domain data (such as blood oxygen, heart rate, and electromyography data) to the target domain (electroencephalography data), while generator G2 maps target domain data back to the source domain. Discriminator D1 evaluates the authenticity of the source and target domain data, and discriminator D2 evaluates the authenticity of the generated data.

[0075] To ensure that the generated data retains the characteristics of the original data, a cycle consistency loss is introduced. Specifically, for a source domain data x, if it is transformed into target domain data y, and then y is transformed back into source domain data x′, the cycle consistency loss is:

[0076]

[0077] Where ||||1 represents the L1 norm.

[0078] The cycle consistency loss ensures that data is consistent during circular transformations between domains, and it consists of two parts:

[0079] The loop loss from the source domain to the target domain and back to the source domain:

[0080] L cyc1 =||x source -G2(G1(x target )||1

[0081] The loop loss from the target domain to the source domain and back to the target domain:

[0082] L cyc2 =||x target -G1(G2(x target )||1.

[0083] Furthermore, through adversarial training between the generator and the discriminator, the generator learns to generate more realistic data. To ensure that the generated data maintains consistency between the source and target domains, CAN introduces a cycle consistency loss. This requires that when data is transferred from the source domain to the target domain and back to the source domain, the data should be as similar as possible to the original source domain data. At this point, the adversarial loss expressions for each generator in the chain are as follows:

[0084] L adv (G1)=-log(D1(G1(x source )))

[0085] L adv (G2)=-log(D1(G2(x target )))

[0086] The difference between generated and real data is measured using a binary cross-entropy loss function. The adversarial loss expression for the two discriminators is as follows:

[0087] L adv (D1)=-log(D1(x source ))-log(1-D1(G1(x target )))

[0088] L adv (D2)=-log(D2(x source ))-log(1-D2(G2(x target ))).

[0089] Furthermore, in step S22, batch normalization is first performed on the feature representations of the source and target domains: X' s =BatchNorm(X s ), X′ t =Batch Norm(X) t Then, the samples from the source and target domains are mapped to a shared feature space. The source domain features are represented as follows: The feature representation of the target domain is F t =f(X) t Furthermore, the large mean-based difference metric is the alignment loss, L. MMD The expression is:

[0090]

[0091]

[0092] Where k represents the Gaussian kernel function, used to calculate the similarity between samples; i and j are feature representation numbers.

[0093] In the feature alignment step, the goal is to learn a transformation matrix that maps the feature space of the source domain to the feature space of the target domain, thereby achieving feature alignment between the two domains. Combined with L... MLE Loss function, L cyc Loss function, L adv Loss function and L MMD The loss function is used to determine the optimal mapping relationship by minimizing the loss function. Multiple rounds of training are performed to continuously improve the model's performance by optimizing the generator, discriminator, and transfer matrix.

[0094] In step S23, it is assumed that each source domain dataset has its own weight w. i This is used to adjust the contribution of the corresponding feature alignment loss function, comprehensively considering the overall differences between various domains, and adjusting the importance of different domains through a balancing factor. The final loss function expression for fine-tuning the generator and learning the mapping matrix is:

[0095] L total =w1*L adv (G1)+w2*L adv (G2)+w3*L cyc1 +w4*L cyc2 +w5L MMD

[0096] Therefore, the training process for CAN-BDA involves updating the weights of the generator and discriminator, as well as the weights of the alignment mapping matrix, under these losses to maximize the realism and consistency of the generated data, and improve domain adaptation and transfer capabilities. This helps generate pseudo-data similar to the target domain data, making it suitable for generative adversarial learning and transfer learning of multi-source and target domain data.

[0097] Further, calculate L total The gradient of the transfer matrix. Specifically, this is achieved by calculating the partial derivative of the loss function with respect to each parameter in the transfer matrix W. For a parameter Wi... j , where i represents the index of the feature dimension of the target domain, j represents the index of the feature dimension of the source domain, and the gradient can be calculated as: Update the weight parameters based on the calculated gradient: α represents the learning rate, used to control the step size of parameter updates. Through backpropagation and parameter updates, adaptive domain mixing weight learning is achieved, thereby completing the training of the generator and discriminator.

[0098] This invention leverages the generative capabilities of generative adversarial networks (GANs) combined with the knowledge transfer mechanism of transfer learning to achieve feature alignment across data distributions in different domains. Its advantage lies in enabling information sharing and transformation between different domains, thus better handling the generation and transfer tasks of cross-domain data distributions. Compared to traditional transfer learning methods, CAN-BDA introduces a generator and discriminator from a GAN, utilizing adversarial training to optimize feature representations and achieve better data distribution alignment. Traditional transfer learning methods often struggle to accurately capture the complex relationships between data distributions when dealing with cross-domain data, especially in highly heterogeneous domain data. CAN-BDA can better capture these complex relationships, achieving more accurate feature alignment and transfer learning, making it particularly suitable for handling data distribution alignment problems in complex domains such as medical data. In summary, this application combines conditional generative adversarial networks with alignment networks to address issues such as cross-domain generation, domain adaptation, training stability, and robustness, demonstrating broad application prospects in multiple fields and contributing to improving the performance and adaptability of generative models across different domains and tasks.

[0099] Furthermore, research shows that adding additional learning tasks can enhance the feature learning ability of deep neural networks. This invention, while utilizing labeled target task data for supervised loss, incorporates self-supervised loss based on the target task data. In the alignment loss, it adds generative adversarial loss and cycle consistency loss. Through multi-task target learning, the model can better adapt to the characteristics of the target task data, improving target task performance while preventing catastrophic forgetting.

[0100] In one example, the method also includes a multimodal data fusion step:

[0101] S31: Use bilinear interpolation to unify the size of the aligned feature vectors; specifically, use a branch structure to process each aligned feature vector separately through branches, and then merge the branches.

[0102] S32: Use bilinear pooling layers to fuse the features of the two modalities to obtain the fused feature representation; in order to unify the feature vectors of different sizes, this step uses bilinear interpolation to interpolate the smaller feature vector to the same size as the largest size.

[0103] S33: Use a cross-modal attention mechanism to calculate the weights between features of different modalities, and use the learned attention weights to perform weighted fusion of features after bilinear pooling to obtain the modal data fusion result.

[0104] Specifically, in step S32, it is assumed that the aligned feature vectors are x iThis requires extensive fusion. When combining bilinear pooling and cross-modal attention mechanisms, bilinear pooling is typically used first to fuse the two features, resulting in a fused feature representation. Then, this fused feature representation is combined with the attention mechanism, learning attention weights to weight the fused features. The specific operation is as follows:

[0105] Bilinear pooling layers are used to fuse the features of two modalities, resulting in a fused feature representation. First, bilinear pooling is performed on the features of each pair of modalities to obtain the fused features. The fused features are represented as follows: Among them, W ij It is a learnable weight matrix; B ij It is the result of bilinear pooling, used to capture the correlation information between different modes.

[0106] Specifically, in step S33, a cross-modal attention mechanism is introduced to perform weighted fusion of the fused features. This example specifically uses a multilayer perceptron (MLP) to calculate the attention weights. For each modality's features, the corresponding attention weights are calculated. First, the attention score calculation function is defined: a multilayer perceptron (MLP) can be used to calculate the attention score for each modality. Using a two-layer MLP, a function `attention_score()` can be defined as follows:

[0107] a i =attention_score(x i = softmax(W) i *h i )+b i

[0108] Where, x i Let h represent the eigenvector of the i-th mode. i W represents the hidden layer output of an MLP. i and b i It refers to the weights and biases of the MLP. Attention score a i It is obtained by performing softmax normalization on the output of the attention_score() function. The purpose of normalization is to ensure that the sum of the attention weights is 1.

[0109] Finally, the learned attention weights are used to perform a weighted fusion of the bilinear pooled features. The importance of the fused features is determined based on the attention scores. Ultimately, for features from different modalities, the following weighted fusion operation can be performed:

[0110]

[0111] For each modality's feature vector x i The attention score a is obtained by MLP calculation. i Then, the softmax function is used to convert it into attention weights W. i For each modality's eigenvector x i The final feature vector is obtained by weighted fusion.

[0112] The above equation illustrates the process of large-scale fusion using bilinear pooling and cross-modal attention mechanisms in the case of four sets of bimodal features. Bilinear pooling fuses the features; the attention mechanism dynamically adjusts the importance of each modality, ultimately yielding the fused feature representation. This fusion result combines the characteristics of bilinear pooling and cross-modal attention mechanisms, while considering the correlation between modalities and the importance of different modalities.

[0113] This invention constructs a Multi-Input Fusion Network (MIC-Net) to unify the size and expressive power of different feature vectors. It also combines feature information from different modalities to fuse feature information from different data sources, thereby capturing a more comprehensive feature representation of the data, reducing redundant information, and improving the flexibility and expressive power of feature fusion. Compared with existing technologies, this invention proposes a flexible and efficient multimodal feature fusion method that can be applied to different fields and tasks.

[0114] In one example, the method of the present invention further includes classifier training and function training steps:

[0115] S41: The classifier is trained using the multimodal fusion dataset. This example specifically uses a Support Vector Machine (SVM) for classification. The goal of SVM is to find a hyperplane that separates samples of different classes and maximizes the distance from the hyperplane to the nearest sample point. Of course, a neural network model, such as the KNN classifier, can also be used instead of SVM for classification.

[0116] S42: The multimodal data is identified using the trained classifier, and the functional module device is controlled to perform actions based on the identification results. The functional module device is an auxiliary device for rehabilitation training; in this example, it is a smart glove worn on the patient's hand. Based on the results obtained from the classifier, the smart glove is controlled to perform corresponding operations, such as issuing prompts to imagine left and right hand movements. Upon receiving these prompts, the patient performs imagined left and right hand movements, thus undergoing rehabilitation training through the smart glove.

[0117] This example integrates a trained classifier with the design of a rehabilitation glove, enabling the glove to receive and process multimodal sensor data. A rehabilitation training plan is designed based on the classifier's output, and real-time feedback helps patients correctly perform rehabilitation movements and monitor their progress.

[0118] Combining the above examples yields preferred examples of the present invention, such as... Figure 3 As shown, the method includes the following steps:

[0119] S0': Denoising, normalizing, and time-series alignment are performed on the collected source domain data and target domain data;

[0120] S11': An autoencoder using unsupervised learning is used to extract feature representations of multimodal data;

[0121] S12': Using the feature representation of the source domain data as input and the feature representation of the target domain data as output, train a large language model based on the Transformer architecture based on maximum likelihood estimation;

[0122] S21': Construct a conditional generative adversarial network with a large language model that has been trained as the generator and a convolutional neural network as the discriminator, and perform generative adversarial learning based on the conditional generative adversarial network;

[0123] S22': The data features generated by the alignment generator and the feature distribution of the real data are compared using batch normalization and maximum mean diffraction to measure the alignment loss L. MMD ;

[0124] S23': Combining cross-entropy loss and cycle consistency loss L cyc Combating Losses L adv Alignment loss L MMD A dynamic weighting mechanism is introduced to determine the final loss function L. total Calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator, and transfer matrix.

[0125] S31': The size of the feature vector after uniform alignment using bilinear interpolation;

[0126] S32': Use bilinear pooling layers to fuse the features of the two modalities to obtain the fused feature representation;

[0127] S33': Use a cross-modal attention mechanism to calculate the weights between features of different modalities, and use the learned attention weights to perform weighted fusion of features after bilinear pooling to obtain the modal data fusion result.

[0128] S41': The classifier is trained using the multimodal fusion dataset;

[0129] S42': Recognize multimodal data based on the trained classifier, and control the functional module device to work based on the recognition results.

[0130] This invention employs a method based on self-supervised learning and multimodal signal processing to accurately classify multimodal signals, enabling personalized health monitoring and services covering prevention, rehabilitation, and guidance, with broad application prospects. It involves multimodal data fusion, self-supervised learning, domain adaptation, and signal processing. By collecting unlabeled data from the source and target domains, appropriate self-supervised tasks and loss functions are selected. A GPT network model is used for encoding and self-supervised learning training of unlabeled data from both the source and target domains. An encoder is created for each data type to learn the data distribution and structure, obtaining features and extensions for different data types. Features extracted from the GPT model are used as input to CAN-BDA, and the CAN-BDA network is used to align these features, calculating distance metrics and optimizing the loss function to achieve feature alignment. The aligned features are then fused and classified using a classifier to ensure the classifier's classification performance.

[0131] This invention applies brain-computer interface technology to health technology fields such as medical monitoring, personalized medicine, disease prevention, and rehabilitation assistance, realizing a complete feedback system. Through active training, it stimulates the patient's neuroplasticity, reconstructs neural pathways, and promotes the reconnection of damaged brain regions with injured limbs, thereby restoring function. This solution brings new technological breakthroughs to the fields of biomedical engineering and information technology, providing more accurate and personalized medical monitoring and services. In summary, this invention fuses data from different modalities and improves the processing and classification capabilities of cross-domain data through alignment and feature fusion methods, providing new solutions for research and practical applications in related fields.

[0132] This invention also includes a self-supervised learning and multimodal biomedical signal processing system, which includes a training module for:

[0133] An unsupervised learning autoencoder is used to extract feature representations of multimodal data. The feature representations of source domain data are used as input and the feature representations of target domain data are used as output. The large language model based on the Transformer architecture is trained based on maximum likelihood estimation, and then the parameters of the large language model are optimized to complete the training of the large language model. The multimodal data includes patient physiological data as source domain data and motor imagery data as target domain data.

[0134] In one example, the system also includes a generative adversarial alignment module, a multimodal data fusion module, a classification training module, and a functional training control module. The generative adversarial alignment module includes a CAN-BDA network, the multimodal data fusion module includes a multi-input fusion network MIC-Net, and the classification training module includes a classifier.

[0135] Specifically, the generative adversarial alignment module performs generative adversarial learning through a conditional generative adversarial network. The conditional information consists of the feature information generated by the generator and the task label. Based on the discriminator, it judges whether the generated target domain data and source domain data are authentic, and calculates the cycle consistency loss L in generative adversarial learning. cyc Combating Losses L adv The alignment loss L is measured by comparing the data features generated by the alignment generator with the feature distribution of the real data using batch normalization and maximum mean normalization. MMD Combining cross-entropy loss and cycle consistency loss L cyc Combating Losses L adv Alignment loss L MMD A dynamic weighting mechanism is introduced to determine the final loss function L. total Calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator, and transfer matrix.

[0136] like Figure 3 As shown, the multi-input fusion network MIC-Net in the multimodal data fusion module includes multiple branch structures. This example uses a two-branch structure, which consists of a bilinear interpolation module and a bilinear pooling layer connected sequentially. Both bilinear pooling layers are connected to a cross-modal attention mechanism, which in turn is connected to a weight fusion module. In this case, the bilinear interpolation module uses bilinear interpolation to unify the size of the aligned feature vectors; the bilinear pooling layer fuses the features from the two modalities to obtain the fused feature representation; then, the cross-modal attention mechanism calculates the weights between the features of different modalities, and uses the learned attention weights to perform weighted fusion on the bilinear pooled features to obtain the modal data fusion result.

[0137] In this example, the branching structure effectively handles multiple aligned feature vectors, processing them separately and then fusing them at the end. This ensures the consistency of the aligned feature vector size, better utilizes the information of each feature vector, and improves the representational power and classification performance of the aligned feature vectors. Simultaneously, the branching structure also enhances the model's interpretability and adjustability, making the model more flexible and applicable to different application scenarios.

[0138] The classification training module is used to train the classifier using a multimodal fusion dataset. This example specifically designs the module's functions by defining its input and output: based on the input data, the module's functionality is redesigned to make it more versatile, capable of handling multiple input data and producing corresponding output results. Taking the field of assisted rehabilitation as an example, the required input data (e.g., multimodal signal data) and output results (e.g., rehabilitation actions or instructions) are determined. The function training and control module is used to identify multimodal data based on the trained classifier and control the function module to perform actions based on the identification results. Addressing the data imbalance and generalization problems in existing rehabilitation equipment, this invention employs data augmentation and transfer learning methods. By increasing minority class samples and utilizing the knowledge of previously trained models, the classifier's generalization ability across imbalanced data and different users is improved. To meet the real-time requirements of the application, this invention optimizes the computational complexity of the classification algorithm and employs a large-scale model network structure and efficient signal processing techniques to reduce latency and improve control response speed. Furthermore, to address challenges in multimodal signal processing such as data heterogeneity, data correlation, data fusion, feature representation learning, and data annotation, this invention systematically proposes a comprehensive multimodal fusion method to effectively integrate information from multiple signal modes and improve classification performance.

[0139] By applying the aforementioned improvements, this multimodal physiological signal processing system can continuously learn from new data and optimize its model to adapt to individual physiological changes. The system can respond to changes in physiological parameters in real time, providing timely suggestions and adjustments to meet specific individual needs. Since single-modal data often cannot provide sufficient information, this invention, through the application of multimodal fusion technology, enables the system to better process and analyze information from multiple signal modalities, further improving classification performance.

[0140] This embodiment provides a storage medium that has the same inventive concept as the self-supervised learning and multimodal biomedical signal processing method formed by any or more of the above examples, and stores computer instructions thereon. When the computer instructions are executed, they perform the steps of the self-supervised learning and multimodal biomedical signal processing method formed by any or more of the above examples.

[0141] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] This application also includes a terminal having the same inventive concept as any or a combination of examples corresponding to the above-described self-supervised learning and multimodal biomedical signal processing method, comprising a memory and a processor. The memory stores computer instructions executable on the processor, which, when executing the computer instructions, performs the steps of the above-described self-supervised learning and multimodal biomedical signal processing method. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0143] In one example, the terminal, i.e., the electronic device, is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0144] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit can perform the above-described method based on self-supervised learning and multimodal biomedical signal processing.

[0145] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache storage unit, and may further include a read-only memory (ROM).

[0146] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0147] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0148] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0149] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored on a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method of the exemplary embodiment of this application.

[0150] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A method based on self-supervised learning and multimodal biomedical signal processing, characterized by: It includes the following training steps: An autoencoder using unsupervised learning is used to extract feature representations of multimodal data, which includes patient physiological data as source domain data and motor imagery data as target domain data; the patient physiological data includes at least one of electromyography data, blood oxygenation data, and heart rate data. Using the feature representation of source domain data as input and the feature representation of target domain data as output, the large language model based on the Transformer architecture is trained based on maximum likelihood estimation, thereby optimizing the parameters of the large language model and completing the training of the large language model. The method also includes a step of generating adversarial alignment: A conditional generative adversarial network (GAN) is constructed, using a trained large language model as the generator and a convolutional neural network as the discriminator. Generative adversarial learning is then performed based on the GAN, with the conditional information consisting of the features generated by the generator and the task labels. The discriminator determines whether the generated target domain data and source domain data are authentic, and the cycle consistency loss in the generative adversarial learning process is calculated. Combating losses ; The alignment loss is measured by comparing the feature distribution of the data generated by the alignment generator with that of the real data using batch normalization and maximum mean averaging. ; Combining cycle consistency loss Combating losses Alignment loss A dynamic weighting mechanism is introduced to determine the final loss function. ,calculate The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator, and transfer matrix.

2. The method for self-supervised learning and multimodal biomedical signal processing according to claim 1, characterized in that: The feature representation of the multimodal data extracted by the autoencoder includes: The encoder in the autoencoder is used to obtain the vector representations of the source domain data and the target source data in the high-dimensional feature space, and then input them into the decoder in the autoencoder. The decoder processes the input data step by step and generates a hidden state vector at each time step. Using the hidden state vector of the decoder as input, the decoder's autoregressive mechanism generates feature information of the corresponding source domain data or target domain data step by step. At each time step, the decoder uses the current hidden state vector to predict the feature value of the next time step, and repeats this prediction step until a complete data feature sequence is generated.

3. The method for self-supervised learning and multimodal biomedical signal processing according to claim 1, characterized in that: When training a large language model, the cross-entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true probability distribution; the expression for the cross-entropy loss function is: ； in, The maximum likelihood estimation loss of the large language model is represented by N; the number of training samples is represented by P; and the conditional probability of the input sequence is represented by P. This represents the sequence length of the i-th sample. This represents the t-th data point in the i-th sample; This represents the first t-1 data points in the i-th sample; This represents the learning rate.

4. The method for self-supervised learning and multimodal biomedical signal processing according to claim 1, characterized in that: The alignment loss The expression is: ； ； in, Represents the source domain features; The target domain feature representation is represented by N; the number of samples is N; and k represents the Gaussian kernel function, used to calculate the similarity between samples. , All are feature representation numbers.

5. The method for self-supervised learning and multimodal biomedical signal processing according to claim 1, characterized in that: The method also includes a multimodal data fusion step: The size of the aligned feature vectors is standardized using bilinear interpolation. Bilinear pooling layers are used to fuse the features of the two modalities to obtain the fused feature representation; The cross-modal attention mechanism is used to calculate the weights between features of different modalities, and the learned attention weights are used to perform weighted fusion of features after bilinear pooling to obtain the modal data fusion result.

6. The method for self-supervised learning and multimodal biomedical signal processing according to claim 5, characterized in that: The method also includes classifier training and function training steps: The classifier was trained using the multimodal fusion dataset; The multimodal data is identified using the trained classifier, and the control function module device is given relevant action commands based on the identification results.

7. A self-supervised learning and multimodal biomedical signal processing system, characterized in that: The system includes a training module for extracting feature representations of multimodal data using an unsupervised learning autoencoder. The feature representations of the source domain data are used as input, and the feature representations of the target domain data are used as output. The system trains a large language model based on the Transformer architecture based on maximum likelihood estimation, thereby optimizing the parameters of the large language model and completing the training of the large language model. The multimodal data includes patient physiological data as source domain data and motor imagery data as target domain data. Patient physiological data includes at least one of electromyography (EMG) data, blood oxygen saturation data, and heart rate data; The system also includes: The Generative Adversarial Alignment (GAP) module constructs a Conditional Generative Adversarial Network (CGN) using a trained large language model as the generator and a convolutional neural network as the discriminator. GAP learning is then performed based on this CGN, with the conditional information consisting of features generated by the generator and task labels. The discriminator determines whether the generated target domain data and source domain data are authentic and calculates the cycle consistency loss in the GAP learning process. Combating losses The alignment loss is measured by comparing the feature distribution of the data generated by the alignment generator with that of the real data using batch normalization and maximum mean averaging. Combining cross-entropy loss and cycle consistency loss Combating losses Alignment loss A dynamic weighting mechanism is introduced to determine the final loss function. ,calculate The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator, and transfer matrix. The multimodal data fusion module is used to unify the size of the aligned feature vectors using bilinear interpolation; it uses a bilinear pooling layer to fuse the features of two modalities to obtain the fused feature representation; then it uses a cross-modal attention mechanism to calculate the weights between the features of different modalities, and uses the learned attention weights to perform weighted fusion on the bilinear pooled features to obtain the modal data fusion result. The classification training module is used to train the classifier using the multimodal fusion dataset; The functional training control module is used to identify multimodal data based on the completed classifier and control the functional module device to work based on the identification results.

8. A terminal, comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, characterized in that: When the processor executes the computer instructions, it performs the steps of the self-supervised learning and multimodal biomedical signal processing method according to any one of claims 1-6.

Images