Self-supervised learning and multi-mode biomedical signal processing method, system and terminal

Abstract

The invention discloses a self-supervised learning and multi-modal biomedical signal processing method, system and terminal, and belongs to the technical field of biomedical engineering and information, and the method comprises the steps: employing an unsupervised learning auto-encoder to extract the feature representation of multi-modal data; and taking the feature representation of the source domain data as input, taking the feature representation of the target domain data as output, and training the large language model based on the Transform architecture based on maximum likelihood estimation. According to the method, relevance and common features among multi-modal data are learned through an auto-encoder, and feature representation extracted by the auto-encoder is applied to pre-training of a large language model, so that the large language model learns more abstract and advanced feature representation, and the processing capacity and generalization performance of the multi-modal data are improved; meanwhile, due to the fact that an unsupervised learning method is adopted, a large amount of unmarked data can be used for pre-training the large language model, and therefore data resources can be better utilized.

Description

Technical Field

[0001] The present invention relates to the fields of biomedical engineering and information technology, and in particular to a system and terminal for processing and applying biomedical signals based on self-supervised learning and multimodal biomedical signals. Background Art

[0002] Traditional medical monitoring methods are usually limited to the collection and analysis of a single or limited number of physiological signals, making it difficult to fully understand the individual's physiological state and changes, limiting the ability to understand and intervene in the patient's overall health status. Modern medical needs tend to comprehensively monitor multiple physiological parameters of patients and be able to respond to changes in different parameters in a timely manner to provide personalized services.

[0003] Traditional biomedical signal processing mainly relies on manually designed feature extraction methods, which limits the system's adaptability to individual differences and physiological changes. However, since single-modal data usually cannot provide enough information, the research focus has gradually shifted to multimodal data fusion for motor imagery recognition. Multimodal data contains different data types, such as images, text, and voice. Multimodal data acquisition technology can simultaneously acquire multiple types of data, such as motion data, physiological data, image data, etc., thereby providing more comprehensive and accurate information for various assessments. Although multimodal data has potential in the fields of biomedical engineering and information technology, traditional neural network models have the problem of insufficient representation of multimodal data features and cannot fully mine the relevant information between different modalities. In addition, supervised data or semi-supervised multimodal data are mainly used for feature learning in various fields, and multimodal data needs to be labeled, which makes it difficult to directly use unlabeled data for model training. Summary of the invention

[0004] The purpose of the present invention is to overcome the problems of the prior art and provide a method, system and terminal based on self-supervised learning and multimodal biomedical signal processing.

[0005] The object of the present invention is achieved through the following technical solution: Based on self-supervised learning and multimodal biomedical signal processing method, the method includes a training step:

[0006] An unsupervised autoencoder is used to extract feature representations of multimodal data, where the multimodal data includes patient physiological data as source domain data and motor imagery data as target domain data.

[0007] Taking the feature representation of source domain data as input and the feature representation of target domain data as output, the large language model based on Transformer architecture is trained based on maximum likelihood estimation, and then the parameters of the large language model are optimized to complete the training of the large language model.

[0008] In one example, the autoencoder extracts feature representation of multimodal data including:

[0009] The encoder in the autoencoder is used to obtain the vector representation of the source domain data and the target source data in the high-dimensional feature space, and input it into the decoder in the autoencoder;

[0010] The decoder will process the input data step by step and generate a hidden state vector at each time step; using the decoder's hidden state vector as input, the decoder's autoregressive mechanism will gradually generate feature information corresponding to the source domain data or the target domain data; at each time step, the decoder will use the current hidden state vector to predict the feature value of the next time step, and repeat the prediction step until a complete data feature sequence is generated.

[0011] In one example, when training a large language model, a cross entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true probability distribution; the expression of the cross entropy loss function is:

[0012]

[0013] Among them, L MLE represents the maximum likelihood estimation loss of the large language model; N represents the number of training samples; P represents the conditional probability of the input sequence; T i represents the sequence length of the i-th sample, x i，t represents the tth data in the ith sample; x i＜t represents the first t-1 data in the th sample; α represents the learning rate.

[0014] In one example, the method further comprises generating an adversarial alignment step:

[0015] Construct a conditional generative adversarial network with a large language model that has been trained as a generator and a convolutional neural network as a discriminator. Generative adversarial learning is performed based on the conditional generative adversarial network. The conditional information is the feature information generated by the generator and the task label. The discriminator determines whether the generated target domain data and source domain data are real, and calculates the cycle consistency loss L in generative adversarial learning. cyc , against the loss L adv ;

[0016] Align the data features generated by the generator with the feature distribution of the real data, and use batch normalization and maximum mean difference to measure the alignment loss L MMD ;

[0017] Combined with cycle consistency loss L cyc , against the loss L adv , alignment loss L MMD, and introduce a dynamic weight mechanism to determine the final loss function L total , calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator and transfer matrix.

[0018] In one example, the alignment loss L MMD The expression is:

[0019]

[0020]

[0021] in, represents the source domain feature representation; F t represents the feature representation of the target domain; N represents the number of samples; k represents the Gaussian kernel function, which is used to calculate the similarity between samples; i and j are both feature representation numbers.

[0022] In one example, the method further includes a multimodal data fusion step:

[0023] The bilinear interpolation method is used to unify the size of the aligned feature vectors;

[0024] Use the bilinear pooling layer to fuse the features of the two modalities to obtain the fused feature expression;

[0025] The cross-modal attention mechanism is used to calculate the weights between different modal features, and the learned attention weights are used to perform weighted fusion on the features after bilinear pooling to obtain the modal data fusion result.

[0026] In one example, the method further includes classifier training and function training steps:

[0027] The classifier is trained using the multimodal fusion dataset;

[0028] The multimodal data is identified according to the trained classifier, and the functional module device is controlled to work based on the identification result, such as controlling the rehabilitation manipulator to drive rehabilitation training.

[0029] It should be further explained that the technical features corresponding to the various examples of the above method can be combined or replaced with each other to form a new technical solution.

[0030] The present invention also includes a system based on self-supervised learning and multimodal biomedical signal processing, which has the same inventive concept as the method formed by any one of the above examples or a combination of multiple examples. The system includes a training module for extracting feature representations of multimodal data using an unsupervised learning autoencoder, taking the feature representations of source domain data as input and the feature representations of target domain data as output, training a large language model based on a Transfonmer architecture based on maximum likelihood estimation, and then optimizing the parameters of the large language model to complete the training of the large language model; the multimodal data includes patient physiological data as source domain data, and motor imagery data as target domain data.

[0031] In one example, the system further includes:

[0032] The generative adversarial alignment module is used to construct a conditional generative adversarial network with a large language model that has been trained as a generator and a convolutional neural network as a discriminator. Generative adversarial learning is then performed based on the conditional generative adversarial network. The conditional information is the feature information and task labels generated by the generator. The discriminator determines whether the generated target domain data and source domain data are real, and calculates the cycle consistency loss L in generative adversarial learning. cyc , against the loss L adv ; Align the data features generated by the generator with the feature distribution of the real data, and use batch normalization and maximum mean difference to measure the alignment loss L MMD ; Combined with cross entropy loss and cycle consistency loss L cyc , against the loss L adv , alignment loss L MMD , and introduce a dynamic weight mechanism to determine the final loss function L total , calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator and transfer matrix;

[0033] The multimodal data fusion module is used to unify the size of the aligned feature vector using the bilinear interpolation method; the bilinear pooling layer is used to fuse the features of the two modalities to obtain the fused feature expression; the cross-modal attention mechanism is then used to calculate the weights between the features of different modalities, and the learned attention weights are used to perform weighted fusion on the features after bilinear pooling to obtain the modal data fusion result;

[0034] A classification training module is used to train the classifier using the multimodal fusion data set;

[0035] The functional training control module identifies the multimodal data according to the trained classifier, and controls the functional module device to work based on the identification result, such as controlling the rehabilitation manipulator to drive rehabilitation training.

[0036] It should be further explained that the technical features corresponding to the above system examples can be combined or replaced with each other to form a new technical solution.

[0037] The present invention also includes a storage medium on which computer instructions are stored. When the computer instructions are executed, the steps of the method based on self-supervised learning and multimodal biomedical signal processing formed by any one or more of the above examples are executed.

[0038] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor runs the computer instructions, it executes the steps of the method based on self-supervised learning and multimodal biomedical signal processing formed by any one or more of the above examples.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. In one example, by learning the correlation and common features between multimodal data through an autoencoder, and applying the feature representation extracted by the autoencoder to the pre-training of a large language model, different types of data can be converted into the same feature representation, thereby achieving unified processing of different types of data, and then enabling the large language model to learn more abstract and advanced feature representations, thereby improving the processing capabilities and generalization performance of multimodal data; and the large language model uses maximum likelihood estimation and adds regularization parameters during prediction to optimize the model's fitting and generalization capabilities, further improving the performance and effect of the large language model in generation tasks. At the same time, because an unsupervised learning method is used, a large amount of unlabeled data can be used to pre-train the large language model, thereby making better use of data resources, which is suitable for personalized medicine, disease prevention, rehabilitation assistance and other fields where data samples are scarce.

[0041] 2. In one example, a conditional generative adversarial network is used to generate feature representations for migration between the source domain and the target domain, thereby achieving feature alignment between different domains or data distributions; the learned multimodal characteristics are then used to construct supervision information for self-supervised learning and fine-tune the pre-trained model (generator, discriminator), thereby reducing the dependence on labeled data during training, and avoiding the problems caused by the deviation between strong prior knowledge and the actual data distribution in unsupervised learning, and allowing the model to fine-tune the pre-trained model for the target task to obtain a task-specific personalized model, thereby improving the overall performance of the target task.

[0042] 3. In one example, through multimodal feature fusion, feature information from different data sources can be combined, thereby improving the flexibility and expressiveness of feature fusion. At the same time, by learning the association weights between different modal features through the cross-modal attention mechanism, the correlation between different modalities can be better mined, thereby improving the accuracy and stability of feature representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0044] Figure 1 A flowchart of a method in an example of the present invention;

[0045] Figure 2 A method flow chart in a preferred embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of a multi-input fusion network MIC-Net in an example of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the use of ordinal numbers (e.g., "first and second", "first to fourth", etc.) is to distinguish objects, and is not limited to this order, and cannot be understood as indicating or implying relative importance.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] In one example, if Figure 1 As shown, a method for processing multimodal biomedical signals based on self-supervised learning includes the following training steps:

[0052] S11: Use an unsupervised learning autoencoder to extract feature representations of multimodal data, where the multimodal data includes patient physiological data as source domain data, such as EEG data, EMG data, blood oxygen data, heart rate data, etc., and motor imagery data as target domain data. Physiological data and motor imagery data are stored in the form of current, voltage, frequency, etc., and are used in the training step of the present invention. Preferably, the original multimodal data is segmented to form time series data blocks, and then feature extraction is performed using an autoencoder to convert feature vectors into specific representations.

[0053] S12: Taking the feature representation of the source domain data as input and the feature representation of the target domain data as output, the large language model based on the Transformer architecture is trained based on the maximum likelihood estimation, and then the parameters of the large language model are optimized to complete the training of the large language model. Preferably, the large language model is a GPT model, and can also be a ChatGLM, BERT, LLM, T5 model, etc.

[0054] The present invention uses the feature representation extracted based on the autoencoder for pre-training of the GPT model, and can convert different types of data into the same feature representation, thereby realizing unified processing of different types of data and improving the generalization ability of the model. The autoencoder can learn the low-level features of the data, and the GPT model can learn the high-level semantic representation of the data. The feature representation extracted by the autoencoder is used for pre-training of the GPT model, and the low-level features and high-level semantic representations can be combined to obtain a more abstract and advanced feature representation, thereby obtaining a backbone network architecture with strong multimodal signal feature representation capabilities; at the same time, since an unsupervised learning method is adopted, a large amount of unlabeled data can be used to pre-train the GPT model, so that data resources can be better utilized. Then, limited labeled data is used to fine-tune the model for transfer learning in the target application. Since the model parameters are already at a good initial point due to sufficient pre-training, and the parameters that need to be updated for fine-tuning are also relatively small, fine-tuning using only a small number of labeled samples in downstream tasks can also allow the model to converge quickly.

[0055] In one example, a data preprocessing step is also included before the training step, specifically, the source domain samples and the target domain samples are first standardized and denoised, and the two-channel EEG data features of the target domain are defined as X egg , the source domain EMG data feature is X emg , the data of blood oxygen and heart rate are X oxy and X hr ;

[0056] S01: During the data acquisition process, the acquired signal is first pre-processed to remove baseline drift, power frequency interference and artifact interference to remove noise and retain useful signals;

[0057] S02: Normalize the data to a unified scale and perform time series alignment so that different types of data can match and have consistent time series characteristics. Motor imagery (EEG) data mainly collects EEG data from C3 and C4 channels; the EMG channel mainly collects data by placing electrodes on the front and back of the forearm; the collected data are all based on time, forming two-dimensional time series data. Heart rate data (Hr) and blood oxygen data (Oxy) are one-dimensional data.

[0058] In one example, the autoencoder includes an encoder E and a decoder D. The input data x is mapped to a vector representation z in a high-dimensional feature space by the encoder, and then mapped back to the original data x' by the decoder. Encoder E and decoder D are both stacks of multiple Transformer layers. Each Transformer layer contains a multi-head self-attention mechanism, a feedforward neural network, etc., so the encoder can also be used as a feature extractor. At this time, the feature representation of multimodal data extracted by the autoencoder includes:

[0059] S111: using the encoder in the autoencoder to obtain vector representations of source domain data and target source data in a high-dimensional feature space, and inputting the vector representations into the decoder in the autoencoder;

[0060] S112: The decoder will gradually process the input data (source domain data such as blood oxygen, heart rate, dual-channel electromyography data, and target source data such as dual-channel electroencephalogram data), and generate a hidden state vector at each time step. These hidden state vectors can be regarded as the encoded representation of the input data; using the decoder's hidden state vector as input, the decoder's autoregressive mechanism gradually generates feature information corresponding to the source domain data or target domain data; at each time step, the decoder will use the current hidden state vector to predict the feature value of the next time step, and repeat the prediction step until a complete data feature sequence is generated. The data feature sequence generated during the decoding process can be regarded as a reconstruction of the original domain data. By comparing the difference between the reconstructed data and the original domain data, the performance and accuracy of the encoding and decoding process can be evaluated.

[0061] The encoding layer calculation expression is: z = E (x); the input data x is calculated through the i-th Transformer layer to obtain the intermediate representation z i :z i =Transformer i ; The final encoding z is composed of the stacked outputs of all Transformer layers: z = [z 1 , z 2 , ..., z n ]; further, the encoder output z is used as the decoder input: x′=z, and the input data x′ is calculated by the i-th Transformer layer to obtain the intermediate representation: z i ′=z i ′*Transformer i (x′), the final decoding result x′ is the stacked output of all Transformer layers: x′ = [z 1 ′,z 2 ′, ..., z′ n ].

[0062] In one example, when training a large language model, it is achieved through maximum likelihood estimation, the goal is to maximize the likelihood probability of the observed data, and the loss function expression is:

[0063]

[0064] Among them, L MLErepresents the maximum likelihood estimation loss of the large language model; N represents the number of training samples; P represents the conditional probability of the input sequence; x i Input data sequence; θ represents model parameters. Maximum likelihood estimation is to maximize the likelihood function of the data by adjusting the model parameters given the observed data. In the GPT model, the likelihood function can be expressed as the conditional probability of the input sequence, that is:

[0065]

[0066] In order to maximize the likelihood function, this example uses the cross entropy loss function to measure the difference between the probability distribution predicted by the model and the true probability distribution. The expression of the cross entropy loss function is:

[0067]

[0068] Among them, T i represents the sequence length of the i-th sample, x i,t represents the tth data in the ith sample; x i＜t Represents the first t-1 data in the th sample; α represents the learning rate. By minimizing the cross entropy loss function, adjusting the model parameters, and maximizing the likelihood function, the distribution and structure of the learning data are enabled to accurately represent the classification probability mapping of the source domain and the target domain data. The loss function can use the cross entropy loss function to measure the difference between the generated and real data. During the training process, the model parameters can be updated using optimization algorithms such as stochastic gradient descent to achieve the training of the GPT model.

[0069] In one example, the method further includes a generation adversarial alignment step, which is implemented by constructing a CAN-BDA network (Cycle-consistent Adversarial-Balanced Distribution Adaptation Network), including:

[0070] S21: Construct a conditional generative adversarial network with a large language model that has been trained as a generator and a convolutional neural network as a discriminator. Perform generative adversarial learning based on the conditional generative adversarial network. The conditional information is the feature information generated by the generator and the task label. The discriminator determines whether the generated target domain data and source domain data are real, and calculates the cycle consistency loss L in generative adversarial learning. cyc , against the loss L adv ;

[0071] S22: Align the data features generated by the generator with the feature distribution of the real data, and use batch normalization and maximum mean difference to measure the alignment loss L MMD ;

[0072] S23: Combining cross entropy loss and cycle consistency loss L cyc , against the loss L adv , alignment loss L MMD , and introduce a dynamic weight mechanism to determine the final loss function L total , calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator and transfer matrix.

[0073] Specifically, before executing step S21, a plurality of source domain data sets and a target domain data set are prepared, each data set including an input signal and a corresponding label.

[0074] Two generators and two discriminators are introduced to process the source domain and target domain respectively. Among them, one generator converts the source domain data into the target domain data, and the other generator converts the target domain data into the source domain data. The two discriminators are used to judge whether the generated target domain data and source domain data are authentic. Specifically, generator G1 is used to map the source domain data (such as blood oxygen, heart rate, electromyography data) to the target domain (EEG data), while generator G2 is used to map the target domain data back to the source domain. Discriminator D1 is used to evaluate the authenticity of the source domain and target domain data, and discriminator D2 is used to evaluate the authenticity of the generated data.

[0075] In order to ensure that the generated data can maintain the characteristics of the original data, the cycle consistency loss is introduced. Specifically, for a source domain data x, it is converted to the target domain data y, and then y is converted back to the source domain data x′, then the cycle consistency loss is:

[0076]

[0077] Among them, |||| 1 represents the L1 norm.

[0078] The cycle consistency loss ensures that the cycle transformation of data between domains is consistent. It consists of two parts:

[0079] The cycle loss from the source domain to the target domain and then back to the source domain is:

[0080] L cyc1 =||x source -G2(G1(x target )|| 1

[0081] The cycle loss from the target domain to the source domain and then back to the target domain is:

[0082] L cyc2 =||x target -G1(G2(xtarget )|| 1 .

[0083] Furthermore, through adversarial training between the generator and the discriminator, the generator learns to generate more realistic data. In order to ensure that the generated data can maintain consistency between the source domain and the target domain, CAN introduces cycle consistency loss. This requires that when the data is transferred from the source domain to the target domain and then back to the source domain, the data should be as similar as possible to the original source domain data. At this time, the adversarial loss expressions of each generator in the chain are:

[0084] L adv (G1) = -log(D1(G1(x source )))

[0085] L adv (G2) = -log(D1(G2(x target )))

[0086] The difference between the generated data and the real data is measured by the binary cross entropy loss function. The adversarial loss expression of the two discriminators is:

[0087] L adv (D1) = -log(D1(x source ))-log(1-D1(G1(x target )))

[0088] L adv (D2) = -log(D2(x source ))-log(1-D2(G2(x target ))).

[0089] Furthermore, in step S22, the source domain and target domain feature representations are first batch normalized: X' s =BatchNorm(X s ), X′ t =Batch Norm(X t ), and then map the samples of the source domain and the target domain to the shared feature space. The source domain feature is represented as The feature representation of the target domain is F t =f(X t ). Furthermore, the large mean difference metric alignment loss, alignment loss L MMD The expression is:

[0090]

[0091]

[0092] Among them, k represents the Gaussian kernel function, which is used to calculate the similarity between samples; i and j are both feature representation numbers.

[0093] In the feature alignment step, the goal is to learn a transformation matrix that maps the feature space of the source domain to the feature space of the target domain, thereby achieving feature alignment between the two domains. MLE Loss function, L cyc Loss function, L adv Loss Function and L MMD The loss function is used to determine the optimal mapping relationship by minimizing the loss function. Multiple rounds of training are performed to continuously improve the performance of the model by optimizing the generator, discriminator, and transfer matrix.

[0094] In step S23, it is assumed that each source domain dataset has its own weight w i , which is used to adjust the contribution of the corresponding feature alignment loss function, comprehensively considers the overall differences between the domains, and adjusts the importance of different domains through the balance factor. The loss function expression for fine-tuning the generator and learning the mapping matrix is:

[0095] L total =w 1 *L adv (G1)+w 2 *L adv (G2)+w 3 *L cyc1 +w 4 *L cyc2 +w 5 L MMD

[0096] Therefore, the training process for CAN-BDA includes updating the weights of the generator and discriminator and the weights of the alignment mapping matrix under these losses to maximize the fidelity and consistency of the generated data and improve domain adaptation and migration capabilities. This helps to generate pseudo data similar to the target domain data and is suitable for generative adversarial learning and migration learning of multi-source and target domain data.

[0097] Furthermore, we calculate L total The gradient of the transfer matrix. Specifically, it is calculated by calculating the partial derivative of the loss function with respect to each parameter in the transfer matrix W. For a parameter Wi j , where i represents the index of the target domain feature dimension and j represents the index of the source domain feature dimension. The gradient can be calculated as: Update the weight parameters according to the calculated gradient: α represents the learning rate, which is used to control the step size of parameter update. Through the process of back propagation and parameter update, the weight learning of adaptive domain mixing is realized, thereby completing the training of the generator and discriminator.

[0098] The present invention utilizes the generation capability of generative adversarial networks, combined with the knowledge transfer mechanism of transfer learning, to achieve feature alignment between data distributions in different fields. Its advantage is that information sharing and transformation can be achieved between different fields, so that the generation and migration tasks of cross-domain data distribution can be better handled. Compared with traditional transfer learning methods, CAN-BDA introduces the generator and discriminator of generative adversarial networks, and optimizes feature representation by adversarial training to achieve better data distribution alignment. Traditional transfer learning methods are usually difficult to accurately capture the complex relationship between data distributions when dealing with cross-domain data distribution, especially in highly heterogeneous domain data. CAN-BDA can better capture the complex relationship between data distributions, achieve more accurate feature alignment and transfer learning, and is particularly suitable for processing data distribution alignment problems in complex fields such as medical data. In summary, this application combines conditional generative adversarial networks with alignment networks to solve problems such as cross-domain generation, domain adaptation, training stability and robustness, and can have broad application prospects in multiple application fields, which helps to improve the performance and adaptability of generative models in different fields and tasks.

[0099] In addition, studies have shown that adding additional learning tasks can improve the learning ability of deep neural networks for features. While using the target task data annotation for supervision loss, the present invention adds the target task data self-supervision loss, and adds the generative adversarial loss and cycle consistency loss to the alignment loss. Through multi-task target learning, the model can better adapt to the characteristics of the target task data, improve the target task performance, and prevent catastrophic forgetting.

[0100] In one example, the method further includes a multimodal data fusion step:

[0101] S31: bilinear interpolation is used to unify the size of the aligned feature vectors; a branch structure is specifically used to process each aligned feature vector separately through branches, and then the branches are merged.

[0102] S32: Use a bilinear pooling layer to fuse the features of the two modalities to obtain a fused feature expression; in order to unify feature vectors of different sizes, this step uses bilinear interpolation to interpolate smaller feature vectors to the same size as the maximum size.

[0103] S33: Use the cross-modal attention mechanism to calculate the weights between different modal features, and use the learned attention weights to perform weighted fusion on the features after bilinear pooling to obtain the modal data fusion result.

[0104] Specifically, in step S32, it is assumed that the aligned feature vectors are x i, a large fusion is needed. When bilinear pooling and cross-modal attention mechanism are used together, the two features are usually fused using bilinear pooling operation to obtain a fused feature representation. The fused feature representation is then combined with the attention mechanism, and the fused features are weighted by learning the attention weights. The specific operations are as follows:

[0105] The bilinear pooling layer is used to fuse the features of the two modalities to obtain the fused feature expression. First, a bilinear pooling operation is performed on the features of each two modalities to obtain the fused features. The fused feature expression is: Among them, W ij is a learnable weight matrix; B ij It is the result of bilinear pooling and is used to capture the correlation information between different modalities.

[0106] Specifically, in step S33, a cross-modal attention mechanism is introduced to perform weighted fusion on the fused features. This example specifically uses a multi-layer perceptron (MLP) to calculate the attention weights. For the features of each modality, the corresponding attention weights are calculated. First, define the attention score calculation function: a multi-layer perceptron (MLP) can be used to calculate the attention score of the modality. Using a two-layer MLP for calculation, a function attention_score() can be defined as follows:

[0107] a i =attention_score(x i )=softmax(W i *h i )+b i

[0108] Among them, x i represents the eigenvector of the i-th mode, h i represents the hidden layer output of MLP, W i and b i is the weight and bias of the MLP. The attention score a i It is obtained by performing softmax normalization on the output of the attention_score() function. The purpose of normalization is to ensure that the sum of the attention weights is 1.

[0109] Finally, the learned attention weights are used to perform weighted fusion on the features after bilinear pooling. The importance of fusion features is determined based on the attention scores. Finally, for features of different modalities, the following weighted fusion operations can be performed:

[0110]

[0111] For each mode’s eigenvector x i , and obtain its attention score a through MLP calculation i , and then use the softmax function to convert it into attention weight W i : For each mode's eigenvector x i Perform weighted fusion to obtain the final feature vector

[0112] The above formula represents the process of using bilinear pooling and cross-modal attention mechanism to perform large fusion in the case of four sets of bimodal features. Through bilinear pooling, the features are fused; through the attention mechanism, the importance of each modality is dynamically adjusted, and finally the fused feature representation fusion result is obtained. This fusion result combines the characteristics of bilinear pooling and cross-modal attention mechanism, while considering the correlation between modalities and the importance of different modalities.

[0113] The present invention unifies the size and expressiveness of different feature vectors by constructing a multi-input fusion network (MIC-Net), and can combine feature information of different modes to achieve the fusion of feature information from different data sources, thereby more comprehensively capturing the feature representation of data, reducing the existence of redundant information, and improving the flexibility and expressiveness of feature fusion. Compared with the prior art, a flexible and efficient multimodal feature fusion method is proposed, which can be applied to different fields and tasks.

[0114] In one example, the method of the present invention further includes classifier training and function training steps:

[0115] S41: Use the multimodal fusion data set to train the classifier; this example specifically uses the support vector machine (SVM) for classification. The goal of the SVM is to find a hyperplane that separates samples of different categories and maximizes the distance from the sample point closest to the hyperplane to the hyperplane. Of course, a neural network model can also be used to replace the SVM for classification, such as the KNN classifier.

[0116] S42: Identify the multimodal data according to the trained classifier, and control the function module device to perform actions based on the identification results. The function module device is an auxiliary device for rehabilitation training. In this example, it is specifically a smart glove worn on the patient's hand. According to the results obtained by the classifier, the smart glove is controlled to perform corresponding operations, such as issuing a prompt to imagine left and right hand movements. After receiving the prompt to imagine (left) right hand movement, the patient performs (left) right hand imagined movement, that is, rehabilitation training is performed through the smart glove.

[0117] This example integrates the trained classifier with the design of rehabilitation gloves to enable the gloves to receive and process multimodal sensor data. A rehabilitation training plan is designed based on the classifier output results, and real-time feedback is used to help patients correctly perform rehabilitation movements and monitor rehabilitation progress.

[0118] Combining the above examples, we can obtain a preferred example of the present invention, such as Figure 3 As shown, the method now includes the following steps:

[0119] S0': denoising, normalizing, and time-series alignment of the collected source domain data and target domain data;

[0120] S11': Extracting feature representations of multimodal data using unsupervised learning autoencoders;

[0121] S12': Using the feature representation of the source domain data as input and the feature representation of the target domain data as output, a large language model based on the Transformer architecture is trained based on maximum likelihood estimation;

[0122] S21': Construct a conditional generative adversarial network with a trained large language model as the generator and a convolutional neural network as the discriminator, and perform generative adversarial learning based on the conditional generative adversarial network;

[0123] S22': Align the data features generated by the generator with the feature distribution of the real data, and use batch normalization and maximum mean difference to measure the alignment loss L MMD ;

[0124] S23': Combining cross entropy loss and cycle consistency loss L cyc , against the loss L adv , alignment loss L MMD , and introduce a dynamic weight mechanism to determine the final loss function L total , calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator and transfer matrix.

[0125] S31': The size of the feature vector after unification using bilinear interpolation;

[0126] S32': Use a bilinear pooling layer to fuse the features of the two modalities to obtain a fused feature expression;

[0127] S33': Use the cross-modal attention mechanism to calculate the weights between different modal features, and use the learned attention weights to perform weighted fusion on the features after bilinear pooling to obtain the modal data fusion result.

[0128] S41': Use the multimodal fusion dataset to train the classifier;

[0129] S42': Identify the multimodal data according to the trained classifier, and control the function module device to work based on the identification result.

[0130] The present invention adopts a method based on self-supervised learning and multimodal signal processing to accurately classify multimodal signals, realizes personalized health monitoring and services, covers multiple aspects such as prevention, rehabilitation, guidance, etc., and has broad application prospects. It involves the fusion of multimodal data, self-supervised learning, domain adaptation and signal processing. By collecting unlabeled data in the source field and the target field, appropriate self-supervised tasks and loss functions are selected. The GPT network model is used for encoding and self-supervised learning to train the unlabeled data in the source field and the unlabeled data in the target field, an encoder is created for each data, the distribution and structure of the data are learned, and the characteristics and extensions of different data are obtained. The features extracted from the GPT model are used as the input of CAN-BDA, and the CAN-BDA network is used to align the features extracted from the GPT model, calculate the distance metric and optimize the loss function to align the features. The aligned features are fused and classified using a classifier to ensure the classification performance of the classifier.

[0131] The present invention applies brain-computer interface technology to health technology fields such as medical monitoring, personalized medicine, disease prevention, and rehabilitation assistance, and realizes a complete feedback system. Through active training, the patient's neural plasticity is stimulated, the neural pathways are rebuilt, and the reconnection of damaged brain areas with damaged limbs is promoted, thereby restoring function. This solution brings new technological breakthroughs to the fields of biomedical engineering and information technology, providing more accurate and personalized medical monitoring and services. In summary, the present invention fuses data from different modalities, and improves the processing and classification capabilities of cross-domain data through alignment and feature fusion methods, providing new solutions for research and practical applications in related fields.

[0132] The present invention also includes a self-supervised learning and multimodal biomedical signal processing system, the system including a training module for:

[0133] An unsupervised learning autoencoder is used to extract feature representations of multimodal data. The feature representations of source domain data are used as input and the feature representations of target domain data are used as output. The large language model based on the Transformer architecture is trained based on maximum likelihood estimation, and then the parameters of the large language model are optimized to complete the training of the large language model. The multimodal data include patient physiological data as source domain data and motor imagery data as target domain data.

[0134] In one example, the system further includes a generative adversarial alignment module, a multimodal data fusion module, a classification training module, and a functional training control module. The generative adversarial alignment module includes a CAN-BDA network, the multimodal data fusion module includes a multi-input fusion network MIC-Net, and the classification training module includes a classifier.

[0135] Specifically, the generative adversarial alignment module performs generative adversarial learning through a conditional generative adversarial network. The conditional information is the feature information and task label generated by the generator. The discriminator determines whether the generated target domain data and source domain data are real, and calculates the cycle consistency loss L in generative adversarial learning. cyc , against the loss L adv ; Align the data features generated by the generator with the feature distribution of the real data, and use batch normalization and maximum mean difference to measure the alignment loss L MMD ; Combined with cross entropy loss and cycle consistency loss L cyc , against the loss L adv , alignment loss L MMD , and introduce a dynamic weight mechanism to determine the final loss function L total , calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator and transfer matrix.

[0136] like Figure 3 As shown in the figure, the multi-input fusion network MIC-Net in the multimodal data fusion module includes multiple branch structures. This example takes two branch structures as an example. The two branch structures include bilinear interpolation modules and bilinear pooling layers connected in sequence. Both bilinear pooling layers are connected to the cross-modal attention mechanism, and the cross-modal attention mechanism is connected to a weight fusion module. At this time, the bilinear interpolation module uses bilinear interpolation to unify the size of the feature vector after alignment; the bilinear pooling layer fuses the features of the two modalities to obtain the fused feature expression; then the cross-modal attention mechanism is used to calculate the weights between the features of different modalities, and the learned attention weights are used to weightedly fuse the features after bilinear pooling to obtain the modal data fusion result.

[0137] In this example, the branch structure can effectively process multiple aligned feature vectors, process them separately and finally fuse them, thereby ensuring the consistency of the size of the aligned feature vectors, making better use of the information of each feature vector, and improving the representation ability and classification performance of the aligned feature vectors. At the same time, the branch structure can also improve the interpretability and adjustability of the model, making the model more flexible and applicable to different application scenarios.

[0138] The classification training module is used to train the classifier using the data set after multimodal fusion. This example specifically designs the module function by defining the input and output of the module: according to the input data, the module function is redesigned to make the module more versatile, able to process a variety of input data and generate corresponding output results. Taking the auxiliary rehabilitation field as an example, the input data (such as multimodal signal data) and output results (such as rehabilitation actions or instructions) required by the module are determined. The functional training control module is used to identify the multimodal data according to the trained classifier, and control the function module to act based on the recognition results. In view of the data imbalance and generalization problems in the use of existing rehabilitation equipment, the system of the present invention adopts data enhancement and transfer learning methods to improve the generalization ability of the classifier to unbalanced data and different users by increasing the minority category samples and using the knowledge of the previously trained model. In order to meet the real-time requirements of the application, the system of the present invention optimizes the computational complexity of the classification algorithm, and adopts a large-scale model network structure and efficient signal processing technology to reduce delays and improve control response speed. In addition, in order to solve the problems of data heterogeneity, data correlation, data fusion, feature representation learning and data labeling in multimodal signal processing, the present invention systematically proposes a comprehensive multimodal fusion method to effectively integrate the information of multiple signal modalities and improve the classification performance.

[0139] By applying the above-mentioned improvement measures, the multimodal physiological signal processing system can continuously learn from new data and continuously optimize the model to adapt to the physiological changes of individuals. The system can respond to changes in physiological parameters in real time and provide timely suggestions and adjustments to meet the specific needs of individuals. As single modality data often cannot provide sufficient information, the present invention can enable the system to better process and analyze information from multiple signal modalities by applying multimodal fusion technology, further improving classification performance.

[0140] This embodiment provides a storage medium, which has the same inventive concept as the method for processing multimodal biomedical signals based on self-supervised learning and formed by any one of the above examples or a combination of multiple examples, and stores computer instructions thereon, which, when executed, execute the steps of the method for processing multimodal biomedical signals based on self-supervised learning and formed by any one of the above examples or a combination of multiple examples.

[0141] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0142] The present application also includes a terminal, which has the same inventive concept as any example or combination of examples corresponding to the above-mentioned method for processing self-supervised learning and multimodal biomedical signals, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the above-mentioned method for processing self-supervised learning and multimodal biomedical signals when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0143] In one example, the terminal, i.e., the electronic device, is presented in the form of a general-purpose computing device, and the components of the electronic device may include but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0144] The storage unit stores a program code, which can be executed by the processing unit, so that the processing unit performs the steps of various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit can perform the above-mentioned self-supervised learning and multimodal biomedical signal processing method.

[0145] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0146] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0147] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0148] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0149] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method of the exemplary embodiment of the present application.

[0150] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. Based on self-supervised learning and multimodal biomedical signal processing method, the characteristics are: It includes the training steps: An unsupervised autoencoder is used to extract feature representations of multimodal data, where the multimodal data includes patient physiological data as source domain data and motor imagery data as target domain data. Taking the feature representation of source domain data as input and the feature representation of target domain data as output, the large language model based on Transformer architecture is trained based on maximum likelihood estimation, and then the parameters of the large language model are optimized to complete the training of the large language model.

2. The method for processing multimodal biomedical signals based on self-supervised learning according to claim 1, characterized in that: The autoencoder extracts feature representation of multimodal data including: The encoder in the autoencoder is used to obtain the vector representation of the source domain data and the target source data in the high-dimensional feature space, and input it into the decoder in the autoencoder; The decoder will process the input data step by step and generate a hidden state vector at each time step; using the decoder's hidden state vector as input, the decoder's autoregressive mechanism will gradually generate feature information corresponding to the source domain data or the target domain data; at each time step, the decoder will use the current hidden state vector to predict the feature value of the next time step, and repeat the prediction step until a complete data feature sequence is generated.

3. The method for processing multimodal biomedical signals based on self-supervised learning according to claim 1, characterized in that: When training a large language model, the cross entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true probability distribution; the expression of the cross entropy loss function is: Among them, L MLE represents the maximum likelihood estimation loss of the large language model; N represents the number of training samples; P represents the conditional probability of the input sequence; T i represents the sequence length of the i-th sample, x i,t represents the tth data in the ith sample; x i＜t represents the first t-1 data in the th sample; α represents the learning rate.

4. The method for processing multimodal biomedical signals based on self-supervised learning according to claim 1, characterized in that: The method further comprises the step of generating an adversarial alignment: Construct a conditional generative adversarial network with a large language model that has been trained as a generator and a convolutional neural network as a discriminator. Generative adversarial learning is performed based on the conditional generative adversarial network. The conditional information is the feature information generated by the generator and the task label. The discriminator determines whether the generated target domain data and source domain data are real, and calculates the cycle consistency loss L in generative adversarial learning. cyc , against the loss L adv ; Align the data features generated by the generator with the feature distribution of the real data, and use batch normalization and maximum mean difference to measure the alignment loss L MMD ; Combined with cycle consistency loss L cyc , against the loss L adv , alignment loss L MMD , and introduce a dynamic weight mechanism to determine the final loss function L total , calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator and transfer matrix.

5. The method for processing multimodal biomedical signals based on self-supervised learning according to claim 1, characterized in that: The alignment loss L MMD The expression is: in, represents the source domain feature representation; F t represents the target domain feature representation; N represents the number of samples; k represents the Gaussian kernel function, which is used to calculate the similarity between samples; i and j are both feature representation numbers.

6. The method for processing multimodal biomedical signals based on self-supervised learning according to claim 4, characterized in that: The method also includes a multimodal data fusion step: The bilinear interpolation method is used to unify the size of the aligned feature vectors; Use the bilinear pooling layer to fuse the features of the two modalities to obtain the fused feature expression; The cross-modal attention mechanism is used to calculate the weights between different modal features, and the learned attention weights are used to perform weighted fusion on the features after bilinear pooling to obtain the modal data fusion result.

7. The method for processing multimodal biomedical signals based on self-supervised learning according to claim 6, characterized in that: The method also includes classifier training and function training steps: The classifier is trained using the multimodal fusion dataset; The multimodal data is identified according to the trained classifier, and the functional module device is controlled to perform relevant action instructions based on the identification result.

8. Based on self-supervised learning and multimodal biomedical signal processing system, the characteristics are: The system includes a training module, which is used to extract feature representations of multimodal data using an unsupervised learning autoencoder, take feature representations of source domain data as input and feature representations of target domain data as output, train a large language model based on a Transformer architecture based on maximum likelihood estimation, and then optimize the parameters of the large language model to complete the training of the large language model; the multimodal data includes patient physiological data as source domain data, and motor imagery data as target domain data.

9. The self-supervised learning and multimodal biomedical signal processing system according to claim 8, characterized in that: The system further comprises: The generative adversarial alignment module is used to construct a conditional generative adversarial network with a large language model that has been trained as a generator and a convolutional neural network as a discriminator. Generative adversarial learning is then performed based on the conditional generative adversarial network. The conditional information is the feature information and task labels generated by the generator. The discriminator determines whether the generated target domain data and source domain data are real, and calculates the cycle consistency loss L in generative adversarial learning. cyc , against the loss L adv ; Align the data features generated by the generator with the feature distribution of the real data, and use batch normalization and maximum mean difference to measure the alignment loss L MMD ; Combined with cross entropy loss and cycle consistency loss L cyc , against the loss L adv , alignment loss L MMD , and introduce a dynamic weight mechanism to determine the final loss function L total , calculate L total The gradient of the transfer matrix is ​​used to update the weight parameters, thereby optimizing the generator, discriminator and transfer matrix; The multimodal data fusion module is used to unify the size of the aligned feature vector using the bilinear interpolation method; the bilinear pooling layer is used to fuse the features of the two modalities to obtain the fused feature expression; the cross-modal attention mechanism is then used to calculate the weights between the features of different modalities, and the learned attention weights are used to perform weighted fusion on the features after bilinear pooling to obtain the modal data fusion result; A classification training module is used to train the classifier using the multimodal fusion data set; The function training control module is used to identify the multimodal data according to the trained classifier and control the function module device to work based on the identification result.

10. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, wherein: When the processor runs the computer instructions, it performs the steps of the method based on self-supervised learning and multimodal biomedical signal processing as described in any one of claims 1-7.

Images