A training method and system of an emotion classification model for multi-modal physiological signals

Abstract

The application provides a training method and system of an emotion classification model for multi-modal physiological signals, relates to the technical field of cross between artificial intelligence and biomedical signal processing, and the method comprises the following steps: collecting multi-modal physiological signals of a plurality of sample users; processing the multi-modal physiological signals of the plurality of sample users to obtain multi-modal physiological vectors of the plurality of sample users; using the multi-modal physiological vectors of the plurality of sample users to perform autoregressive pre-training on a large language model; using the multi-modal physiological vectors of the plurality of sample users and text vectors corresponding to emotion classification prompt texts to fine-tune the large language model subjected to the autoregressive pre-training, and obtaining an emotion classification model. In the process of emotion prediction, the problems of multi-modal signal mode loss, inconsistent sampling rates and non-uniform channel numbers can be effectively overcome, and the accuracy of emotion prediction is improved.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of artificial intelligence and biomedical signal processing, and in particular to a training method and system for an emotion classification model for multimodal physiological signals. Background Technology

[0002] With the development of wearable devices, bedside monitoring, and mobile medical technologies, physiological signals such as EEG (Electroencephalogram), ECG (Electrocardiogram), EMG (Electromyography), and EOG (Electrooculogram) are widely, long-term, and low-cost collected, and are widely used in various tasks such as sleep staging, anomaly detection, emotion assessment, disease screening, and rehabilitation assessment. Physiological signals are characterized by strong temporal sequence, large inter-individual variability, complex noise, and high annotation costs. Therefore, traditional methods relying on manual features and small-scale supervised learning are difficult to achieve stable generalization in scenarios involving cross-datasets, cross-devices, and cross-populations.

[0003] To improve data utilization efficiency and versatility, "base model / pre-trained model" methods for physiological signals have emerged in recent years. These methods typically involve self-supervised pre-training on large amounts of unlabeled or weakly labeled physiological data to learn transferable feature representations, which are then adapted for specific downstream tasks. Common pre-training methods include mask reconstruction based on Transformers or convolutional encoders, mask value prediction, contrastive learning, and adaptive representation learning.

[0004] Furthermore, multimodal physiological analysis is gaining increasing attention. In practical applications, multiple physiological signal modalities (such as EEG, ECG, EMG, EOG, etc.) often coexist. These modalities complement each other in terms of physiological mechanisms and noise structures, helping to improve the ability to identify complex states. However, existing multimodal methods often rely on customized fusion structures for specific tasks or datasets and assume strict synchronization or alignment between modalities. In practical applications, factors such as missing modalities, inconsistent sampling rates, and inconsistent channel numbers make it difficult to construct general, multimodal representations.

[0005] Meanwhile, general-purpose LLM (Large Language Model) and instruction learning have made significant progress in multi-task modeling. However, LLM usually relies on discrete token sequences for autoregressive modeling, while physiological signals are continuous time-series data with strong heterogeneity between modalities, making it difficult to align with text tokens in the same sequence modeling framework.

[0006] Therefore, there is an urgent need for a training method for emotion classification models oriented towards multimodal physiological signals. Summary of the Invention

[0007] In view of the above problems, embodiments of this application provide a training method and system for an emotion classification model oriented towards multimodal physiological signals, so as to overcome the above problems or at least partially solve the above problems.

[0008] In a first aspect, this application provides a method for training an emotion classification model based on multimodal physiological signals, the method comprising:

[0009] Multimodal physiological signals were collected from multiple sample users, including at least two of the following physiological signals: EEG physiological signals, ECG physiological signals, EMG physiological signals, and EOG physiological signals.

[0010] The first phase of training is performed by processing the multimodal physiological signals of the multiple sample users to obtain the multimodal physiological vectors of the multiple sample users.

[0011] The second phase of training is performed by using the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model. The large language model, after autoregressive pre-training, learns the changing patterns of the multimodal physiological signals over time.

[0012] The third stage of training is performed by using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts to fine-tune the large language model that has been pre-trained by autoregression, thereby obtaining the sentiment classification model.

[0013] Optionally, the multimodal physiological signals include physiological signals of M modalities; performing the first stage of training involves processing the multimodal physiological signals of the multiple sample users to obtain the multimodal physiological vectors of the multiple sample users, including:

[0014] For the m-th mode physiological signal in the multimodal physiological signal, the quantized input vector of the m-th mode is obtained by using a pre-trained encoding module;

[0015] Using the quantization module to be trained, perform the following steps:

[0016] Based on the private codebook of the m-th mode, the quantized input vector of the m-th mode is subjected to nearest neighbor quantization to obtain the private code vector of the m-th mode and the corresponding private quantization error.

[0017] Based on the shared codebook shared by modes 1 to M, the quantized input vector of mode m is subjected to nearest neighbor quantization to obtain the shared code vector of mode m, and the corresponding shared quantization error is obtained.

[0018] If the shared quantization error is less than the private quantization error, the shared code vector of the m-th mode is determined as the final quantization vector of the m-th mode;

[0019] If the shared quantization error is not less than the private quantization error, the private code vector of the m-th mode is determined as the final quantization vector of the m-th mode.

[0020] Optionally, the method further includes:

[0021] The shared code vector of the m-th mode is processed using the stopping gradient operator to obtain the shared stopping gradient processing result of the m-th mode;

[0022] The private code vector of the m-th mode is processed using the stopping gradient operator to obtain the private stopping gradient processing result of the m-th mode;

[0023] The shared quantization loss is determined based on the difference between the shared stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode.

[0024] The private quantization loss is determined based on the difference between the private stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode.

[0025] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain a trained quantization module. The trained quantization module is used to process the multimodal physiological signals of the target user for emotion classification.

[0026] Optionally, a second phase of training is performed, utilizing the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model, including:

[0027] Based on the private codebooks of each of the 1st to Mth modalities and the shared codebook shared by the 1st to Mth modalities, the text vocabulary of the large language model is expanded to obtain an expanded vocabulary.

[0028] For the m-th modality physiological signal in the multimodal physiological signal, the extended vocabulary is used to predict the final quantization vector of the m-th modality in the next time step based on the final quantization vector of the m-th modality in the historical time step using the large language model, so as to obtain the loss of the m-th modality.

[0029] The multimodal loss is obtained by using the loss and corresponding weight of each of the first to M modes;

[0030] For the first text corpus, using the expanded vocabulary, the text vector for the next time step is predicted by the large language model based on the text vector of the historical time step, so as to obtain the first text loss;

[0031] Based on the multimodal loss and the first text loss, the model parameters of the large language model are updated to obtain a large language model that has been pre-trained by autoregression.

[0032] Optionally, a third stage of training is performed, using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts, to fine-tune the autoregressive pre-trained large language model to obtain a sentiment classification model, including:

[0033] The multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts are input into the large language model that has been pre-trained by autoregression. The large language model that has been pre-trained by autoregression then uses the expanded vocabulary to generate the predicted answer.

[0034] The instruction loss is obtained based on the difference between the predicted answer and the correct answer corresponding to the multimodal physiological signals of the multiple sample users;

[0035] For the second text corpus, the expanded vocabulary is used to predict the text vector of the next time step based on the text vector of the historical time step through the large language model that has been pre-trained by autoregression, so as to obtain the second text loss.

[0036] Based on the instruction loss and the second text loss, the model parameters of the autoregressive pre-trained large language model are updated to obtain the sentiment classification model.

[0037] Optionally, the multimodal physiological signal includes EEG physiological signal and physiological signal of at least one of the following modalities: ECG physiological signal, EMG physiological signal, and EOG physiological signal;

[0038] The multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts are input into the autoregressive pre-trained large language model. The autoregressive pre-trained large language model then uses the expanded vocabulary to generate predicted answers, including:

[0039] The final quantization vectors of the EEG modality and other modalities are obtained by using the extended vocabulary through the large language model that has been pre-trained by autoregression.

[0040] Based on the final quantization vector of the EEG mode and the final quantization vectors of other modes, the gated fusion coefficients are determined through a gated fusion network.

[0041] Determine the corresponding gating bias based on the task identifier of the emotion classification task;

[0042] Based on the determined gating bias and the gating fusion coefficient, the corrected gating fusion coefficient is determined;

[0043] Based on the corrected gating fusion coefficients, the attention calculation results of the final quantization vectors of other modalities to the final quantization vector of the EEG modality are fused with the final quantization vector of the EEG modality to obtain the final quantization enhancement vector of the EEG modality.

[0044] Based on the corrected gating fusion coefficients, the attention calculation results of the final quantization vector of the EEG mode to the final quantization vector of other modes are fused with the final quantization vector of other modes to obtain the final quantization enhancement vector of other modes.

[0045] Based on the final quantization augmentation vector of the EEG modality and the final quantization augmentation vector of other modalities, the predicted answer is generated by the autoregressive pre-trained large language model.

[0046] Optionally, the method further includes:

[0047] Perform a real-valued FFT on the physiological signal of the m-th mode to obtain the frequency domain amplitude spectrum of the m-th mode;

[0048] The time-domain reconstruction and frequency-domain amplitude spectrum reconstruction of the final quantization vector of the m-th mode are performed using the pre-trained decoding module to obtain the time-domain reconstruction result and the frequency-domain amplitude spectrum reconstruction result of the m-th mode.

[0049] The temporal reconstruction loss is determined based on the difference between the temporal reconstruction result of the m-th mode and the physiological signal of the m-th mode.

[0050] The frequency domain amplitude spectrum reconstruction loss is determined based on the difference between the frequency domain amplitude spectrum reconstruction result of the m-th mode and the frequency domain amplitude spectrum of the m-th mode.

[0051] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including:

[0052] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0053] Optionally, the method further includes:

[0054] Using the final quantization vector of the m-th mode as K and V, and the learnable query vector of the current round of the first stage as Q, the attention calculation result of the m-th mode is determined. Using the final quantization vector of the n-th mode as K and V, and the learnable query vector of the current round of the first stage as Q, the attention calculation result of the n-th mode is determined.

[0055] Based on the attention calculation result of the m-th modality, and with the attention calculation result of the n-th modality as a reference, the emotion category prediction result of the m-th modality is determined; and based on the attention calculation result of the n-th modality, and with the attention calculation result of the m-th modality as a reference, the emotion category prediction result of the n-th modality is determined.

[0056] Based on the sentiment category labels of the multimodal physiological signals to which the m-th modality and the n-th modality physiological signals belong, as well as the sentiment category prediction results of the n-th modality and the sentiment category prediction results of the n-th modality, a cross-modal contrast loss is obtained;

[0057] Based on the cross-modal contrastive loss, the learnable query vector of the current round in the first stage is updated, and the updated learnable query vector is used for the next round of training in the first stage.

[0058] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including:

[0059] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the cross-modal contrast loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0060] Optionally, the method further includes:

[0061] The final quantization vector of the m-th mode is processed using a pre-trained gradient inversion layer to obtain the physiological domain processing result. The physiological domain processing result is then used by a pre-trained domain discriminator to perform domain identification and obtain the physiological domain loss.

[0062] The pre-trained gradient inversion layer is used to process the text vectors in the text vocabulary of the large language model to obtain the text domain processing result. The pre-trained domain discriminator is then used to perform domain identification on the text domain processing result to obtain the text domain loss.

[0063] Based on the physiological domain loss and the text domain loss, the physiological-text domain alignment loss is obtained;

[0064] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including:

[0065] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the physiological-text domain alignment loss, the cross-modal contrast loss, the temporal reconstruction loss, and the frequency domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0066] A second aspect of this application provides a training system for an emotion classification model based on multimodal physiological signals, the system comprising:

[0067] The acquisition module is used to acquire multimodal physiological signals from multiple sample users. The multimodal physiological signals include at least two of the following physiological signals: EEG physiological signals, ECG physiological signals, EMG physiological signals, and EOG physiological signals.

[0068] The first-stage training module is used to perform the first-stage training, which processes the multimodal physiological signals of the multiple sample users to obtain the multimodal physiological vectors of the multiple sample users.

[0069] The second-stage training module is used to perform the second-stage training. It uses the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model. The large language model, after autoregressive pre-training, learns the changing patterns of the multimodal physiological signals over time.

[0070] The third-stage training module is used to perform the third-stage training. It uses the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts to fine-tune the large language model that has been pre-trained by autoregression to obtain the sentiment classification model.

[0071] Optionally, the multimodal physiological signals include physiological signals of M modalities; performing a first-stage training process, the multimodal physiological signals of the multiple sample users are processed to obtain multimodal physiological vectors of the multiple sample users. The first-stage training module includes:

[0072] A pre-training submodule is used to obtain the quantized input vector of the m-th modality from the multimodal physiological signal using a pre-trained encoding module. The pre-training submodule includes:

[0073] The first nearest neighbor quantization submodule is used to perform nearest neighbor quantization on the quantized input vector of the m-th mode based on the private codebook of the m-th mode, to obtain the private code vector of the m-th mode, and to obtain the corresponding private quantization error.

[0074] The second nearest neighbor quantization submodule is used to perform nearest neighbor quantization on the quantization input vector of the m-th mode based on the shared codebook shared by the 1st to Mth modes, to obtain the shared code vector of the m-th mode, and to obtain the corresponding shared quantization error.

[0075] The first determining submodule is used to determine the shared code vector of the m-th mode as the final quantization vector of the m-th mode when the shared quantization error is less than the private quantization error.

[0076] The second determining submodule is used to determine the private code vector of the m-th mode as the final quantization vector of the m-th mode when the shared quantization error is not less than the private quantization error.

[0077] Optionally, the system further includes:

[0078] The first processing submodule is used to process the shared code vector of the m-th mode using the stopping gradient operator to obtain the shared stopping gradient processing result of the m-th mode;

[0079] The second processing submodule is used to process the private code vector of the m-th mode using the stopping gradient operator to obtain the private stopping gradient processing result of the m-th mode.

[0080] The third determination submodule is used to determine the shared quantization loss based on the difference between the shared stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode;

[0081] The fourth determination submodule is used to determine the private quantization loss based on the difference between the private stopping gradient processing result of the m-th mode and the quantization input vector of the m-th mode;

[0082] The first parameter update submodule is used to update the model parameters of the quantization module to be trained based at least on the shared quantization loss and the private quantization loss, so as to obtain the trained quantization module. The trained quantization module is used to process the multimodal physiological signals of the target user for emotion classification.

[0083] Optionally, a second stage of training is performed, utilizing the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model. The second stage training module includes:

[0084] An extension submodule is used to extend the text vocabulary of the large language model based on the private codebooks of each of the 1st to Mth modalities and the shared codebook shared by the 1st to Mth modalities, to obtain an extended vocabulary.

[0085] The first prediction submodule is used to predict the final quantization vector of the m-th modality in the next time step using the extended vocabulary and the final quantization vector of the m-th modality based on the historical time step of the large language model, so as to obtain the loss of the m-th modality.

[0086] The fifth determination submodule is used to obtain the multimodal loss based on the loss and corresponding weight of each of the first to M modes;

[0087] The second prediction submodule is used to predict the text vector of the next time step based on the text vector of the first text corpus using the expanded vocabulary and the large language model based on the text vector of the historical time step, so as to obtain the first text loss.

[0088] The second parameter update submodule is used to update the model parameters of the large language model based on the multimodal loss and the first text loss, so as to obtain a large language model that has been pre-trained by autoregression.

[0089] Optionally, a third stage of training is performed, using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts, to fine-tune the autoregressive pre-trained large language model to obtain a sentiment classification model. The third stage training module includes:

[0090] The prediction answer generation submodule is used to input the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts into the autoregressive pre-trained large language model, and generate the prediction answer through the extended vocabulary of the autoregressive pre-trained large language model.

[0091] The sixth determining submodule is used to obtain the instruction loss based on the difference between the predicted answer and the correct answer corresponding to the multimodal physiological signals of the multiple sample users;

[0092] The third prediction submodule is used to predict the text vector of the next time step based on the text vector of the historical time step using the expanded vocabulary and the large language model that has been pre-trained by autoregression, in order to obtain the second text loss.

[0093] The third parameter update submodule is used to update the model parameters of the autoregressive pre-trained large language model based on the instruction loss and the second text loss, so as to obtain the sentiment classification model.

[0094] Optionally, the multimodal physiological signal includes EEG physiological signal and physiological signal of at least one of the following modalities: ECG physiological signal, EMG physiological signal, and EOG physiological signal;

[0095] The multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts are input into the autoregressive pre-trained large language model. The autoregressive pre-trained large language model uses the expanded vocabulary to generate predicted answers. The predicted answer generation submodule includes:

[0096] The first determining subunit is used to obtain the final quantization vector of the EEG modality and the final quantization vector of other modalities by using the extended vocabulary through the large language model that has been pre-trained by autoregression.

[0097] The second determining subunit is used to determine the gated fusion coefficients through a gated fusion network based on the final quantization vector of the EEG mode and the final quantization vectors of other modes.

[0098] The third determining subunit is used to determine the corresponding gating bias based on the task identifier of the emotion classification task;

[0099] The fourth determining subunit is used to determine the corrected gating fusion coefficient based on the determined gating bias and the gating fusion coefficient;

[0100] The first fusion subunit is used to fuse the attention calculation result of the final quantization vector of other modalities to the final quantization vector of the EEG modal with the final quantization vector of the EEG modal according to the modified gating fusion coefficient, so as to obtain the final quantization enhancement vector of the EEG modal.

[0101] The second fusion subunit is used to fuse the attention calculation result of the final quantization vector of the EEG mode to the final quantization vector of other modes with the final quantization vector of other modes according to the modified gating fusion coefficient, so as to obtain the final quantization enhancement vector of other modes.

[0102] The prediction answer generation subunit is used to generate a prediction answer using the autoregressive pre-trained large language model, based on the final quantization augmentation vector of the EEG modality and the final quantization augmentation vector of other modalities.

[0103] Optionally, the system further includes:

[0104] The seventh determination submodule is used to perform real-valued FFT amplitude on the physiological signal of the m-th mode to obtain the frequency domain amplitude spectrum of the m-th mode;

[0105] The eighth determination submodule is used to perform time-domain reconstruction and frequency-domain amplitude spectrum reconstruction on the final quantization vector of the m-th mode using the pre-trained decoding module, so as to obtain the time-domain reconstruction result and the frequency-domain amplitude spectrum reconstruction result of the m-th mode;

[0106] The ninth determination submodule is used to determine the temporal reconstruction loss based on the difference between the temporal reconstruction result of the m-th mode and the physiological signal of the m-th mode;

[0107] The tenth determination submodule is used to determine the frequency domain amplitude spectrum reconstruction loss based on the difference between the frequency domain amplitude spectrum reconstruction result of the m-th mode and the frequency domain amplitude spectrum of the m-th mode.

[0108] The model parameters of the quantization module to be trained are updated based at least on the shared quantization loss and the private quantization loss to obtain the trained quantization module. The first parameter update sub-module includes:

[0109] The first parameter update subunit is used to update the model parameters of the quantization module to be trained based at least on the shared quantization loss, the private quantization loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0110] Optionally, the system further includes:

[0111] The attention calculation result determination submodule is used to determine the attention calculation result of the m-th mode using the final quantization vector of the m-th mode as K and V, and the learnable query vector of the current round of the first stage as Q; and to determine the attention calculation result of the n-th mode using the final quantization vector of the n-th mode as K and V, and the learnable query vector of the current round of the first stage as Q.

[0112] The emotion category prediction result determination submodule is used to determine the emotion category prediction result of the m-th modality based on the attention calculation result of the m-th modality and with reference to the attention calculation result of the n-th modality, and to determine the emotion category prediction result of the n-th modality based on the attention calculation result of the n-th modality and with reference to the attention calculation result of the m-th modality.

[0113] The cross-modal contrast loss determination submodule is used to obtain the cross-modal contrast loss based on the sentiment category labels of the multimodal physiological signals to which the physiological signals of the m-th modality and the n-th modality belong, as well as the sentiment category prediction results of the n-th modality and the sentiment category prediction results of the n-th modality.

[0114] The learnable query vector update submodule is used to update the learnable query vector of the current round of the first stage according to the cross-modal contrastive loss, and the updated learnable query vector is used for the next round of training in the first stage.

[0115] The model parameters of the quantization module to be trained are updated based at least on the shared quantization loss and the private quantization loss to obtain the trained quantization module. The first parameter update sub-module includes:

[0116] The second parameter update subunit is used to update the model parameters of the quantization module to be trained based on at least the shared quantization loss and the private quantization loss, the cross-modal contrast loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0117] Optionally, the system further includes:

[0118] The third processing submodule is used to process the final quantization vector of the m-th mode using a pre-trained gradient inversion layer to obtain the physiological domain processing result, and to perform domain identification on the physiological domain processing result using a pre-trained domain discriminator to obtain the physiological domain loss.

[0119] The fourth processing submodule is used to process the text vectors in the text vocabulary of the large language model using the pre-trained gradient inversion layer to obtain the text domain processing result, and to perform domain identification on the text domain processing result using the pre-trained domain discriminator to obtain the text domain loss.

[0120] The physiological-text domain alignment loss determination submodule is used to obtain the physiological-text domain alignment loss based on the physiological domain loss and the text domain loss;

[0121] The model parameters of the quantization module to be trained are updated based at least on the shared quantization loss and the private quantization loss to obtain the trained quantization module. The first parameter update sub-module includes:

[0122] The third parameter update subunit is used to update the model parameters of the quantization module to be trained based on at least the shared quantization loss and the private quantization loss, the physiological-text domain alignment loss, the cross-modal contrast loss, the temporal reconstruction loss, and the frequency domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0123] The beneficial effects of this application are:

[0124] This application provides a training method for an emotion classification model based on multimodal physiological signals. The method includes: collecting multimodal physiological signals from multiple sample users, wherein the multimodal physiological signals include at least two of the following modalities: EEG physiological signals, ECG physiological signals, EMG physiological signals, and EOG physiological signals; performing a first-stage training, processing the multimodal physiological signals from the multiple sample users to obtain multimodal physiological vectors from the multiple sample users; performing a second-stage training, using the multimodal physiological vectors from the multiple sample users to perform autoregressive pre-training on a large language model, wherein the autoregressive pre-trained large language model learns the changing patterns of the multimodal physiological signals over time; and performing a third-stage training, using the multimodal physiological vectors from the multiple sample users and the text vectors corresponding to the emotion classification prompt text to fine-tune the autoregressive pre-trained large language model to obtain the emotion classification model. This application obtains a sentiment classification model by processing the collected physiological signals of multiple modalities and performing multiple stages of autoregressive pre-training in conjunction with a big prediction model. It can effectively overcome the problems of missing modalities, inconsistent sampling rates, and inconsistent channel numbers in the process of sentiment prediction, thereby improving the accuracy of sentiment prediction. Attached Figure Description

[0125] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0126] Figure 1 This is a flowchart illustrating the steps of a training method for an emotion classification model based on multimodal physiological signals, as provided in an embodiment of this application.

[0127] Figure 2 This is a schematic diagram of the processing steps of a code vector based on a stopping gradient operator provided in an embodiment of this application;

[0128] Figure 3 This is a flowchart illustrating the steps of a method for autoregressive pre-training of a large language model based on multimodal physiological signals, as provided in an embodiment of this application.

[0129] Figure 4 This is a schematic flowchart illustrating the steps for performing the third stage of training according to an embodiment of this application;

[0130] Figure 5 This is a schematic diagram of a large-scale training architecture for multimodal physiological signals provided in an embodiment of this application;

[0131] Figure 6 This is a schematic diagram of a training system for an emotion classification model based on multimodal physiological signals provided in an embodiment of this application. Detailed Implementation

[0132] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0133] In a first aspect, this application provides a method for training an emotion classification model based on multimodal physiological signals, such as... Figure 1 As shown, the method includes:

[0134] S101, Collect multimodal physiological signals from multiple sample users, wherein the multimodal physiological signals include at least two of the following physiological signals: EEG physiological signal, ECG physiological signal, EMG physiological signal, and EOG physiological signal;

[0135] S102, Perform the first stage of training, process the multimodal physiological signals of the multiple sample users, and obtain the multimodal physiological vectors of the multiple sample users;

[0136] S103, perform the second stage of training, using the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model, and the large language model after autoregressive pre-training learns the change law of the multimodal physiological signals over time.

[0137] S104, Perform the third stage of training, using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts to fine-tune the large language model that has been pre-trained by autoregression, and obtain the sentiment classification model.

[0138] This application provides a training method for an emotion classification model based on multimodal physiological signals. Through a three-stage training process, it effectively combines multimodal physiological signals with a Large Language Model (LLM) to improve the accuracy and robustness of emotion classification. The specific steps are as follows:

[0139] S101, Acquiring multimodal physiological signals. In S101, multimodal physiological signals from multiple sample users are acquired. In this application, the multimodal physiological signals include signals from at least two modalities, such as EEG, ECG, EMG, EOG, etc.

[0140] S102, Perform the first stage of training: In the first stage of training, the collected multimodal physiological signals are preprocessed and feature extracted to generate corresponding multimodal physiological vectors. In this application, the preprocessing process may include operations such as denoising, resampling, and standardization of the signals to ensure that the signals of different modalities have a uniform format and scale. The core of this first stage is to extract effective physiological features from signals of different modalities to form a vector representation representing the physiological state of the sample user.

[0141] S103, Perform the second stage of training: The second stage of training aims to perform autoregressive pre-training on the large language model using the multimodal physiological vectors generated in S102. The goal of autoregressive pre-training is to enable the large language model to learn the patterns of multimodal physiological signal changes over time. Through autoregressive modeling, the model can capture the temporal characteristics of physiological signals and the inter-modal relationships, thus providing effective temporal features for subsequent emotion classification tasks. In this stage, the large language model not only learns the temporal patterns of multimodal signals but also possesses a stronger ability to integrate cross-modal information.

[0142] S104, Perform the third stage of training: In the third stage, based on the autoregressive pre-trained large language model, the model is fine-tuned by combining the multimodal physiological vectors of sample users and the text vectors of sentiment classification prompts. In this application, the fine-tuning stage enables the model to adapt to specific sentiment classification needs according to different sentiment classification tasks. The sentiment classification prompts provide contextual information related to sentiment classification, which helps the large language model accurately understand and predict the user's emotional state in sentiment classification tasks. Through the third stage of training, a fine-tuned sentiment classification model is finally obtained.

[0143] This application, through the above three-stage training, not only effectively integrates multimodal physiological signal information but also learns the patterns of signal changes over time through autoregressive pre-training, thereby improving the performance of sentiment classification. Furthermore, employing a large language model for autoregressive pre-training and fine-tuning effectively addresses the problems of insufficient signal feature extraction and poor model generalization ability in traditional sentiment classification methods. The fine-tuned sentiment classification model exhibits good cross-dataset and cross-task adaptability, enabling it to handle various sentiment classification tasks.

[0144] In one embodiment, the multimodal physiological signals include physiological signals of M modalities; performing a first phase of training involves processing the multimodal physiological signals of the multiple sample users to obtain multimodal physiological vectors of the multiple sample users, including:

[0145] For the m-th mode physiological signal in the multimodal physiological signal, the quantized input vector of the m-th mode is obtained by using a pre-trained encoding module;

[0146] Using the quantization module to be trained, perform the following steps:

[0147] Based on the private codebook of the m-th mode, the quantized input vector of the m-th mode is subjected to nearest neighbor quantization to obtain the private code vector of the m-th mode and the corresponding private quantization error.

[0148] Based on the shared codebook shared by modes 1 to M, the quantized input vector of mode m is subjected to nearest neighbor quantization to obtain the shared code vector of mode m, and the corresponding shared quantization error is obtained.

[0149] If the shared quantization error is less than the private quantization error, the shared code vector of the m-th mode is determined as the final quantization vector of the m-th mode;

[0150] If the shared quantization error is not less than the private quantization error, the private code vector of the m-th mode is determined as the final quantization vector of the m-th mode.

[0151] In this embodiment, the acquired multimodal physiological signals include physiological signals of M modalities. For the physiological signals of M modalities, the quantized input vector of the m-th modality is obtained using a pre-trained encoding module. The specific quantization process is as follows:

[0152] Using the quantization module to be trained, the quantized input vector is processed according to different quantization strategies. For the quantized input vector of each mode, nearest neighbor quantization is first performed based on the private codebook to obtain the private code vector of that mode, and the corresponding private quantization error is calculated. At the same time, nearest neighbor quantization is performed based on the shared codebook shared by all modes to obtain the shared code vector, and the shared quantization error is calculated.

[0153] Furthermore, the final quantization vector for this mode is determined by comparing the private quantization error and the shared quantization error. If the shared quantization error is less than the private quantization error, the shared code vector is selected as the final quantization vector for this mode; if the shared quantization error is not less than the private quantization error, the private code vector is selected as the final quantization vector for this mode.

[0154] The quantization process in this embodiment can flexibly select different quantization strategies (private quantization or shared quantization), thereby reducing the computational complexity of the model while ensuring quantization accuracy. By combining shared and private codebooks, the efficiency of information sharing between different modalities is improved, while ensuring the uniqueness of each modality.

[0155] Based on the above embodiments, in another embodiment of this application, it further includes, as follows: Figure 2 The following steps are shown:

[0156] S201, the shared code vector of the m-th mode is processed using the stopping gradient operator to obtain the shared stopping gradient processing result of the m-th mode.

[0157] In this step, the shared code vector for each modality, such as the shared code vector of the m-th modality, is processed using the stopping gradient operator. The stopping gradient operator is used to prevent gradient flow during backpropagation, thereby avoiding updates to the parameters of the shared code vector. In this embodiment, the shared code vector of the m-th modality is processed using the stopping gradient operator, ensuring that during training, the calculation of the quantization error of the modality features is independent of the parameter updates of the shared code vector itself, maintaining the stability of the shared code vector, while simultaneously allowing the calculation of the difference from the quantized input vector.

[0158] S202, the private code vector of the m-th mode is processed using the stopping gradient operator to obtain the private stopping gradient processing result of the m-th mode.

[0159] In this step, the stopping gradient operator is used to process the private code vector of the m-th mode to obtain the private stopping gradient processing result. This embodiment prevents the parameters of the private code vector from being updated during backpropagation by applying the stopping gradient operator to the private code vector of each mode; only the difference between the quantized input vector and the private code vector is calculated. The introduction of the stopping gradient operator ensures the stability of the training process and prevents the calculation of quantization error from interfering with the parameters of the private code vector.

[0160] S203. Based on the difference between the shared stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode, determine the shared quantization loss.

[0161] In this step, the shared quantization loss is determined by comparing the difference between the shared stopping gradient processing result of the m-th mode and its quantized input vector. The shared quantization loss characterizes the error between the shared code vector obtained through shared codebook quantization and the quantized input vector, measuring the deviation in the quantization process. This shared quantization loss is used to evaluate the accuracy of the shared code vector in representing the quantized input vector.

[0162] S204. Determine the private quantization loss based on the difference between the private stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode.

[0163] In this step, the private quantization loss is determined by comparing the difference between the private stopping gradient processing result of the m-th mode and its quantized input vector. The private quantization loss characterizes the error between the private code vector and the quantized input vector, representing the deviation generated during the quantization process of the private codebook.

[0164] S205, at least based on the shared quantization loss and the private quantization loss, the model parameters of the quantization module to be trained are updated to obtain a trained quantization module. The trained quantization module is used to process the multimodal physiological signals of the target user for emotion classification.

[0165] In this step, the model parameters of the quantization module to be trained are updated based on the shared quantization loss and the private quantization loss. The parameters of the quantization module are updated according to the backpropagation of the loss during each training process to reduce quantization error and thus optimize the effect of the quantization process. Through this update process, the quantization module gradually adjusts its parameters, making the processed multimodal physiological signals more accurate and stable.

[0166] This embodiment effectively reduces the propagation of quantization errors by incorporating a stopping gradient operator during the quantization process of multimodal physiological signals, making the quantization module more stable during training. By calculating shared and private quantization losses and updating the parameters of the quantization module based on these losses, the accuracy and robustness of the quantization module are further improved. Ultimately, the trained quantization module can process multimodal physiological signals and provide more accurate and efficient input representations for downstream tasks such as sentiment classification, thereby improving the model's performance in sentiment classification tasks.

[0167] In one embodiment, a second phase of training is performed, utilizing the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model, including the following steps:

[0168] S301, based on the private codebooks of each of the first to M modalities and the shared codebook shared by the first to M modalities, the text vocabulary of the large language model is expanded to obtain an expanded vocabulary.

[0169] S302, for the physiological signal of the m-th modality in the multimodal physiological signal, using the extended vocabulary, and through the final quantization vector of the m-th modality based on the historical time step of the large language model, predict the final quantization vector of the m-th modality in the next time step, so as to obtain the loss of the m-th modality;

[0170] S303, based on the loss and corresponding weight of each of the 1st to Mth modes, the multimodal loss is obtained;

[0171] S304, For the first text corpus, using the expanded vocabulary, the large language model predicts the text vector for the next time step based on the text vector of the historical time step, so as to obtain the first text loss;

[0172] S305, based on the multimodal loss and the first text loss, update the model parameters of the large language model to obtain a large language model pre-trained by autoregression.

[0173] This embodiment provides a method for autoregressive pre-training of a large language model based on multimodal physiological signals. By utilizing multimodal physiological signals and text corpora to jointly train the large language model, the reasoning ability and sentiment classification performance of the large language model under multimodal data are improved. Specific steps are as follows: Figure 3 As shown:

[0174] S301, Expanding the Text Vocabulary of the Large Language Model: In the second stage of training, the text vocabulary of the large language model is first expanded based on the private and shared codebooks of modalities 1 to M. The private and shared codebooks for each modality provide the necessary lexical representations for the quantized input, ensuring that the large language model can process physiological signal data from different modalities. Therefore, the expanded vocabulary not only includes regular text vocabulary but also contains quantized vector representations of multimodal physiological signals, enabling the large language model to process multimodal information during autoregressive pre-training.

[0175] S302, Predicting the Physiological Signal Quantization Vector for the Next Time Step Based on an Expanded Vocabulary: Using a large language model and an expanded vocabulary, the final quantization vector of the m-th modality at the next time step is predicted based on the final quantization vector of the m-th modality at previous time steps. This step employs autoregressive modeling, enabling the large language model to predict the future state of multimodal physiological signals based on previous time step information, thereby calculating the loss for the m-th modality. The loss value for the m-th modality reflects the error between the model's prediction and the actual signal, providing optimization direction for model training.

[0176] S303, Calculate the multimodal loss: After obtaining the loss for each modality, combine the weights of each modality to calculate the multimodal loss. This multimodal loss value comprehensively considers all modalities and evaluates the model's performance when processing multimodal data.

[0177] S304, Predicting the text vector for the next time step based on the text corpus: For the first text corpus, predict the text vector for the next time step based on the text vectors of the historical time steps using a large language model, and calculate the first text loss.

[0178] S305, Update the parameters of the large language model: Based on the multimodal loss and text loss calculated above, update the model parameters of the large language model. Through backpropagation, the model parameters are adjusted according to the training data of multimodal signals and text corpora, ultimately obtaining a large language model pre-trained by autoregression. This pre-trained model can effectively establish relationships between multimodal physiological signals and text data, thereby improving its capabilities in cross-modal reasoning and multi-task processing.

[0179] By combining multimodal physiological signals and text corpora for autoregressive pre-training, this embodiment enables the large language model to learn the temporal and semantic relationships between multimodal and textual data. Through this method, the large language model not only performs exceptionally well in predicting multimodal data but also effectively handles complex cross-modal tasks, improving the performance of sentiment classification and other multimodal tasks.

[0180] In one embodiment, a third stage of training is performed, using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts to fine-tune the autoregressive pre-trained large language model, thereby obtaining a sentiment classification model, such as... Figure 4 As shown, it includes:

[0181] S401, input the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts into the large language model that has been pre-trained by autoregression, and generate the predicted answer through the large language model that has been pre-trained by autoregression using the expanded vocabulary;

[0182] S402, based on the difference between the predicted answer and the correct answer corresponding to the multimodal physiological signals of the multiple sample users, the instruction loss is obtained;

[0183] S403, For the second text corpus, using the expanded vocabulary, the large language model pre-trained by autoregression predicts the text vector of the next time step based on the text vector of the historical time step, so as to obtain the second text loss;

[0184] S404, based on the instruction loss and the second text loss, update the model parameters of the autoregressive pre-trained large language model to obtain the sentiment classification model.

[0185] In step S401, the multimodal physiological vectors of multiple sample users, along with the text vectors of the corresponding sentiment classification prompts, are input into a pre-trained autoregressive language model. At this point, the pre-trained language model has grasped the temporal and semantic relationships between the multimodal signals and text data. Through the expanded vocabulary, the language model can simultaneously process physiological signals and textual information, integrating the contextual information of both types of data into the generated predicted answer.

[0186] In step S402, after inputting multimodal physiological signal vectors and sentiment classification prompt text, a large language model generates a predicted answer. Next, the instruction loss is calculated by comparing the predicted answer with the actual correct answer (i.e., the sentiment label or classification result corresponding to the multimodal physiological signals of multiple sample users). The instruction loss characterizes the error between the sentiment classification prediction generated by the large language model and the actual label, driving the model to optimize the accuracy of its sentiment classification.

[0187] S403, for the second text corpus, utilizes an expanded vocabulary and a large language model pre-trained with autoregression to predict the text vector for the next time step based on the text vectors at the previous time step, and calculates the second text loss. By comparing the difference between the predicted text vectors and the actual text data, the ability of the large language model to process text is optimized.

[0188] S404 updates the parameters of the large language model through backpropagation by combining instruction loss and second text loss. During this process, the large language model continuously adjusts its parameters based on the joint information from multimodal physiological signals and text data, thereby improving the accuracy and robustness of sentiment classification. After training, a fine-tuned large language model is obtained, which can be used for sentiment classification tasks.

[0189] Through fine-tuning in the third stage, this embodiment enables the autoregressive pre-trained large language model to further adapt to sentiment classification tasks. Under dual training with multimodal physiological signals and sentiment classification prompts, the large language model can not only accurately understand the emotional information in the physiological signals but also effectively guide the classification process through text prompts. This multimodal training method significantly improves the model's sentiment classification performance and enhances its application capabilities in complex sentiment recognition scenarios.

[0190] In one embodiment, the multimodal physiological signal includes an EEG physiological signal and physiological signals of at least one of the following modalities: ECG physiological signal, EMG physiological signal, and EOG physiological signal;

[0191] The multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts are input into the autoregressive pre-trained large language model. The autoregressive pre-trained large language model then uses the expanded vocabulary to generate predicted answers, including:

[0192] The final quantization vectors of the EEG modality and other modalities are obtained by using the extended vocabulary through the large language model that has been pre-trained by autoregression.

[0193] Based on the final quantization vector of the EEG mode and the final quantization vectors of other modes, the gated fusion coefficients are determined through a gated fusion network.

[0194] Determine the corresponding gating bias based on the task identifier of the emotion classification task;

[0195] Based on the determined gating bias and the gating fusion coefficient, the corrected gating fusion coefficient is determined;

[0196] Based on the corrected gating fusion coefficients, the attention calculation results of the final quantization vectors of other modalities to the final quantization vector of the EEG modality are fused with the final quantization vector of the EEG modality to obtain the final quantization enhancement vector of the EEG modality.

[0197] Based on the corrected gating fusion coefficients, the attention calculation results of the final quantization vector of the EEG mode to the final quantization vector of other modes are fused with the final quantization vector of other modes to obtain the final quantization enhancement vector of other modes.

[0198] Based on the final quantization augmentation vector of the EEG modality and the final quantization augmentation vector of other modalities, the predicted answer is generated by the autoregressive pre-trained large language model.

[0199] This embodiment improves the accuracy of sentiment classification by utilizing a large language model for autoregressive pre-training and combining it with a multimodal signal fusion mechanism. The specific steps are as follows:

[0200] Multimodal physiological signals from multiple sample users were collected, including EEG signals and at least one other modality of physiological signal, such as ECG, EMG, or EOG. The multimodal physiological vectors from multiple sample users and their corresponding text vectors from sentiment classification prompts were then input into a large language model pre-trained with autoregressive mechanisms. The large language model used an expanded vocabulary to process these inputs, thereby generating corresponding predicted answers.

[0201] During the generation of predicted answers, a large language model is used to obtain the final quantization vectors for the EEG modality and other modalities through an expanded vocabulary. The quantization vectors for each modality provide an effective representation of the physiological signal, and these vectors will be used for further sentiment classification tasks.

[0202] Furthermore, a gated fusion network is used to determine the gating fusion coefficients based on the final quantization vectors of the EEG modality and other modalities. These gating fusion coefficients control the information flow between different modalities, ensuring that the large language model can reasonably balance the importance of each modality during fusion. Then, based on the task identifier of the sentiment classification task, a corresponding gating bias is determined. This bias further adjusts the fusion process to adapt to the needs of different sentiment classification tasks.

[0203] Based on the determined gating bias and gating fusion coefficient, the corrected gating fusion coefficient is calculated.

[0204] Furthermore, using the corrected gating fusion coefficients, the attention calculation results between the final quantization vector of the EEG modality and the final quantization vector of other modalities are calculated, and then fused with the final quantization vector of the EEG modality to obtain the final quantization enhancement vector of the EEG modality. Simultaneously, the attention calculation results between the final quantization vector of the EEG modality and the final quantization vector of other modalities are calculated, and then fused with the final quantization vectors of other modalities to obtain the final quantization enhancement vectors of the other modalities.

[0205] Furthermore, based on the final quantization enhancement vectors of the obtained EEG modality and the final quantization enhancement vectors of other modalities, a predicted answer is generated through a large language model for the sentiment classification task.

[0206] By employing a gating fusion mechanism, this embodiment effectively combines multimodal physiological signals with the emotion classification task, enhancing the large language model's ability to process signals from various modalities. Through an attention mechanism, effective information fusion between different modalities is achieved, enabling the large language model to better capture the correlations between multimodal signals, thereby improving the accuracy and robustness of emotion classification.

[0207] In one embodiment, a real-valued FFT amplitude is performed on the physiological signal of the m-th mode to obtain the frequency domain amplitude spectrum of the m-th mode;

[0208] The time-domain reconstruction and frequency-domain amplitude spectrum reconstruction of the final quantization vector of the m-th mode are performed using the pre-trained decoding module to obtain the time-domain reconstruction result and the frequency-domain amplitude spectrum reconstruction result of the m-th mode.

[0209] The temporal reconstruction loss is determined based on the difference between the temporal reconstruction result of the m-th mode and the physiological signal of the m-th mode.

[0210] The frequency domain amplitude spectrum reconstruction loss is determined based on the difference between the frequency domain amplitude spectrum reconstruction result of the m-th mode and the frequency domain amplitude spectrum of the m-th mode.

[0211] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including:

[0212] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0213] In this embodiment, a real-number Fast Fourier Transform (FFT) is performed on the physiological signal of the m-th mode to obtain the frequency domain amplitude spectrum of the m-th mode. Through FFT processing, the original time-domain physiological signal can be transformed into a frequency-domain representation, capturing the frequency component information in the signal.

[0214] Furthermore, a pre-trained decoding module is used to perform time-domain reconstruction and frequency-domain amplitude spectrum reconstruction on the final quantization vector of the m-th mode. Time-domain reconstruction restores the time-domain signal from the quantization vector using the decoding module; frequency-domain amplitude spectrum reconstruction maps the quantization vector back to the frequency-domain amplitude spectrum, thereby recovering the characteristics of the frequency components. The decoding module improves the signal reconstruction accuracy by learning how to extract effective time-domain and frequency-domain information from the quantization vector.

[0215] The temporal reconstruction loss is calculated based on the difference between the temporal reconstruction result of the m-th modality and the actual physiological signal. This temporal reconstruction loss measures the deviation between the reconstructed signal and the real signal and provides feedback for optimizing the large language model, enabling it to better learn how to recover the signal in the temporal domain.

[0216] Similarly, the frequency domain amplitude spectrum reconstruction loss is calculated based on the difference between the reconstructed frequency domain amplitude spectrum of the m-th mode and the actual frequency domain amplitude spectrum. This frequency domain amplitude spectrum reconstruction loss measures the accuracy of frequency domain recovery and helps large language models achieve better fitting on frequency components.

[0217] In this embodiment, during training, the model parameters of the quantization module are updated by considering at least the shared quantization loss and the private quantization loss. The shared quantization loss and the private quantization loss are derived from the quantization errors of the shared codebook and the private codebook, respectively. These losses guide the model to optimize the quantization process, ensuring that the quantization result is as close as possible to the original signal.

[0218] In this embodiment, to further improve the performance of the quantization module, in addition to the shared quantization loss and the private quantization loss, the time-domain reconstruction loss and the frequency-domain amplitude spectrum reconstruction loss are also incorporated into the update process. By comprehensively considering these four losses (shared quantization loss, private quantization loss, time-domain reconstruction loss, and frequency-domain amplitude spectrum reconstruction loss), the model parameters of the quantization module are updated during each training iteration, thereby enabling the trained quantization module to better handle the multimodal physiological signals of the target user.

[0219] Based on the above embodiments, in another embodiment of this application, the final quantization vector of the m-th mode is used as K and V, and the learnable query vector of the current round of the first stage is used as Q to determine the attention calculation result of the m-th mode. The final quantization vector of the n-th mode is used as K and V, and the learnable query vector of the current round of the first stage is used as Q to determine the attention calculation result of the n-th mode.

[0220] Based on the attention calculation result of the m-th modality, and with the attention calculation result of the n-th modality as a reference, the emotion category prediction result of the m-th modality is determined; and based on the attention calculation result of the n-th modality, and with the attention calculation result of the m-th modality as a reference, the emotion category prediction result of the n-th modality is determined.

[0221] Based on the sentiment category labels of the multimodal physiological signals to which the m-th modality and the n-th modality physiological signals belong, as well as the sentiment category prediction results of the n-th modality and the sentiment category prediction results of the n-th modality, a cross-modal contrast loss is obtained;

[0222] Based on the cross-modal contrastive loss, the learnable query vector of the current round in the first stage is updated, and the updated learnable query vector is used for the next round of training in the first stage.

[0223] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including:

[0224] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the cross-modal contrast loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0225] This embodiment, based on the above embodiments, further introduces the calculation of cross-modal contrastive loss, further optimizing the emotion classification task of multimodal physiological signals. By combining attention mechanisms, cross-modal contrastive learning, and joint updates of quantization loss, the training effect of the quantization module is improved. The specific steps are as follows:

[0226] First, for the physiological signal of the m-th modality, its final quantized vector is used as the key (K) and value (V), and attention is calculated using the learnable query vector (Q) of the current round in the first stage. The purpose of this calculation is to obtain the attention calculation result of the m-th modality through the interaction of the query vector and the key-value pairs. The attention calculation process of the m-th modality is shown in the following formula:

[0227] (1)

[0228] in, This represents the attention calculation result for the m-th mode; This indicates attention calculation; Represents a learnable query vector; This represents the final quantization vector of the physiological signal in the m-th mode. As key (K) and value (V) This represents a matrix with B rows and d columns.

[0229] For the nth modality, a similar approach is used, employing the final quantized vector of the nth modality as the key and value, while simultaneously using the same query vector Q for attention calculation, yielding the attention calculation result for the nth modality. The core of the attention mechanism lies in extracting information from different modalities through a weighted method to determine the importance of each modality in the sentiment classification task. The attention calculation process for the mth modality is shown in the following formula:

[0230] (2)

[0231] in, This represents the attention calculation result for the nth mode; This represents the final quantization vector of the physiological signal in the nth mode. As key (K) and value (V).

[0232] Furthermore, based on the attention calculation result of the m-th modality and with reference to the attention calculation result of the n-th modality, the sentiment category prediction result of the m-th modality is determined; similarly, based on the attention calculation result of the n-th modality and with reference to the attention calculation result of the m-th modality, the sentiment category prediction result of the n-th modality is determined. Through complementary information across modalities, the sentiment prediction of each modality can refer to the features of other modalities, thereby better capturing the multi-dimensional information of sentiment classification.

[0233] The sentiment category prediction process for the m-th modality is shown in the following formula:

[0234] (3)

[0235] in, This represents the cross-modal feature similarity matrix between the m-th modality and the n-th modality in the current batch of samples, i.e., the sentiment category prediction result of the m-th modality; T represents the matrix transpose; This represents the temperature coefficient.

[0236] The sentiment category prediction process for the nth modality is shown in the following formula:

[0237] (4)

[0238] in, This represents the cross-modal feature similarity matrix between the nth modality and the mth modality in the current batch of samples, which is the sentiment category prediction result of the nth modality.

[0239] Building upon this, a cross-modal contrastive loss is calculated. This loss evaluates the model's ability to contrast across different modalities by comparing the sentiment category predictions of modal m and modal n with their respective true labels. Specifically, the loss function is optimized based on the deviations between the sentiment predictions of modal m and modal n and their true labels, ensuring effective collaboration between different modalities and further improving the accuracy of sentiment classification.

[0240] The cross-modal contrast loss is shown in the following formula:

[0241] (5)

[0242] in, Indicates cross-modal contrast loss. Represents the cross-entropy loss function; This indicates that the label is correctly matched diagonally, i.e., the actual label.

[0243] Based on the obtained cross-modal contrastive loss, the learnable query vector for the current round of the first stage is updated. This update enables the query vector to better capture the relationships between modalities, enhances the model's adaptability in cross-modal tasks, and prepares it to provide more accurate query vectors for the next round of training.

[0244] Furthermore, the quantization module to be trained is updated by combining shared quantization loss and private quantization loss. Shared quantization loss and private quantization loss originate from the quantization errors of the shared codebook and private codebook, respectively, and they play important roles in optimizing the quantization module. In addition, time-domain reconstruction loss and frequency-domain amplitude spectrum reconstruction loss are also considered. Through joint optimization of multiple losses, the quantization module is improved in multiple dimensions, ultimately resulting in the trained quantization module.

[0245] Based on the above embodiments, in another embodiment, a pre-trained gradient inversion layer is used to process the final quantization vector of the m-th mode to obtain the physiological domain processing result, and a pre-trained domain discriminator is used to perform domain identification on the physiological domain processing result to obtain the physiological domain loss.

[0246] The pre-trained gradient inversion layer is used to process the text vectors in the text vocabulary of the large language model to obtain the text domain processing result. The pre-trained domain discriminator is then used to perform domain identification on the text domain processing result to obtain the text domain loss.

[0247] Based on the physiological domain loss and the text domain loss, the physiological-text domain alignment loss is obtained;

[0248] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including:

[0249] The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the physiological-text domain alignment loss, the cross-modal contrast loss, the temporal reconstruction loss, and the frequency domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0250] This embodiment further optimizes the training process of the quantization module by combining a GRL (Gradient Reversal Layer) and a domain discriminator to enhance the alignment between the physiological signal domain and the text signal domain. In this way, physiological signal and text data can be better coordinated and shared within the same model, thereby improving the emotion classification performance of multimodal physiological signals. The specific steps are as follows:

[0251] The final quantization vector of the m-th modulus is processed using a pre-trained gradient inversion layer to obtain the physiological domain processing result. The gradient inversion layer is used to reverse the gradient during backpropagation, forcing the model to learn features with better generalization ability. The physiological domain processing result represents the physiological signal features processed by the gradient inversion layer; this feature will be used for alignment with the text domain.

[0252] Furthermore, a pre-trained domain discriminator is used to perform domain identification on the physiological domain processing results, resulting in a physiological domain loss. The role of the domain discriminator is to help the model learn more domain-adversarial features by distinguishing whether the model output comes from the physiological domain or the text domain.

[0253] The loss of the physiological domain is shown in the following formula:

[0254] (6)

[0255] in, This represents the physiological domain loss of the m-th mode; Representation domain judgment and processing; Indicates gradient reversal; This represents the final quantization vector of the m-th mode; This represents the resistance strength coefficient.

[0256] Furthermore, the same pre-trained gradient inversion layer is used to process the text vectors in the text vocabulary of the large language model to obtain the text domain processing result. A pre-trained domain discriminator is then used to perform domain identification on the text domain processing result, yielding the text domain loss. In this way, the model avoids over-reliance on text features during optimization and allows text and physiological signal features to be learned collaboratively in the same shared space.

[0257] The text domain loss is shown in the following formula:

[0258] (7)

[0259] in, Represents the text domain loss for the m-th modality; Represents a text vector.

[0260] Furthermore, based on the physiological domain loss and the text domain loss, the physiological-text domain alignment loss is calculated. The physiological-text domain alignment loss is shown in the following formula:

[0261] (8)

[0262] in, This represents the physiological-text domain alignment loss for the m-th modality; This represents the set of valid modes.

[0263] During training, in addition to shared and private quantization losses, physiological-text domain alignment loss, cross-modal contrast loss, temporal reconstruction loss, and frequency domain amplitude spectrum reconstruction loss are also used as training objectives to update the quantization module to be trained. These loss terms work together to ensure that the quantization module can maximize the preservation of effective features from different modalities when processing multimodal signals and can accurately perform sentiment classification tasks.

[0264] This embodiment achieves alignment between the physiological and textual signal domains by introducing a gradient inversion layer and a domain discriminator, enhancing the model's ability to integrate information from different modalities. Through joint optimization of cross-modal contrastive loss and multiple reconstruction losses, the model performs excellently in multimodal sentiment classification tasks, enhancing the synergistic effect of cross-modal signals and task adaptability. Finally, the trained quantization module demonstrates high accuracy and robustness when handling complex multimodal sentiment classification tasks.

[0265] In this application, multiple datasets, such as sleep stage classification datasets and anomaly recognition datasets, can be combined, and partial data can be extracted and labeled for fine-tuning with joint instructions. Experimental results show that the large language model can effectively handle multimodal data and multiple tasks simultaneously during multi-task training.

[0266] In one embodiment, this application provides a schematic diagram of a large-scale training architecture for multimodal physiological signals, such as... Figure 5 The following is stated:

[0267] First, input multimodal physiological signal data (such as EEG, ECG, EMG, EOG, etc.) and a text corpus. In practical applications, the multimodal physiological signal data needs to be processed through preprocessing steps and feature extraction is performed using an encoder.

[0268] Entering the first stage of training:

[0269] Preprocessing and segmentation: The multimodal physiological signal data is preprocessed and segmented to convert it into a format suitable for quantization.

[0270] Encoder quantization processing: The processed multimodal physiological signal data is input into the encoder to generate a quantized input vector. This application adopts a hybrid quantization mechanism of shared codebook and private codebook.

[0271] The quantized vectors will be combined with the text corpus data to support subsequent model training.

[0272] Lexical Expansion: The vocabulary is expanded to integrate quantized vectors of text data and multimodal physiological signals into a single lexical space. The expanded vocabulary includes text terms as well as quantized representations of physiological signals (such as EEG, ECG, EMG, EOG), ensuring that data from different modalities can share the same representation space.

[0273] Entering the second phase of training:

[0274] Multi-channel autoregressive pre-training utilizes an expanded vocabulary to determine multimodal loss.

[0275] To maintain causality between time steps and allow information exchange between different channels, block attention masks are used for training.

[0276] By using a hybrid plain text language modeling loss, we obtain the first text loss, ensuring that the model maintains its text generation capability while performing multimodal learning.

[0277] The total loss for the second stage is constructed from the multimodal loss and the first text loss, and the large language model is pre-trained using autoregressive methods based on the total loss for the second stage.

[0278] Entering the third stage of training:

[0279] The text vectors of multimodal physiological vectors and sentiment classification prompts are used as input to a large language model that has been pre-trained by autoregression, and the output is the predicted answer.

[0280] The instruction loss is derived from the differences between the predicted answer and the correct answer corresponding to the multimodal physiological signals of multiple sample users.

[0281] By utilizing an expanded vocabulary, a large language model pre-trained with autoregression predicts the text vector for the next time step based on the text vector at the previous time step, thus obtaining a second text loss.

[0282] The model parameters of the autoregressive pre-trained large language model are updated based on the instruction loss and the second text loss to obtain the sentiment classification model.

[0283] Based on the same inventive concept, a second aspect of this application provides a training system for an emotion classification model oriented towards multimodal physiological signals, such as... Figure 6 As shown, the system includes:

[0284] The acquisition module 501 is used to acquire multimodal physiological signals from multiple sample users. The multimodal physiological signals include at least two of the following physiological signals: EEG physiological signals, ECG physiological signals, EMG physiological signals, and EOG physiological signals.

[0285] The first-stage training module 502 is used to perform the first-stage training, process the multimodal physiological signals of the multiple sample users, and obtain the multimodal physiological vectors of the multiple sample users.

[0286] The second-stage training module 503 is used to perform the second-stage training. It uses the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model. The large language model, after autoregressive pre-training, learns the change law of the multimodal physiological signals over time.

[0287] The third-stage training module 504 is used to perform the third-stage training. It uses the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts to fine-tune the large language model that has been pre-trained by autoregression to obtain the sentiment classification model.

[0288] Optionally, the multimodal physiological signals include physiological signals of M modalities; performing a first-stage training process, the multimodal physiological signals of the multiple sample users are processed to obtain the multimodal physiological vectors of the multiple sample users. The first-stage training module 502 includes:

[0289] A pre-training submodule is used to obtain the quantized input vector of the m-th modality from the multimodal physiological signal using a pre-trained encoding module. The pre-training submodule includes:

[0290] The first nearest neighbor quantization submodule is used to perform nearest neighbor quantization on the quantized input vector of the m-th mode based on the private codebook of the m-th mode, to obtain the private code vector of the m-th mode, and to obtain the corresponding private quantization error.

[0291] The second nearest neighbor quantization submodule is used to perform nearest neighbor quantization on the quantization input vector of the m-th mode based on the shared codebook shared by the 1st to Mth modes, to obtain the shared code vector of the m-th mode, and to obtain the corresponding shared quantization error.

[0292] The first determining submodule is used to determine the shared code vector of the m-th mode as the final quantization vector of the m-th mode when the shared quantization error is less than the private quantization error.

[0293] The second determining submodule is used to determine the private code vector of the m-th mode as the final quantization vector of the m-th mode when the shared quantization error is not less than the private quantization error.

[0294] Optionally, the system further includes:

[0295] The first processing submodule is used to process the shared code vector of the m-th mode using the stopping gradient operator to obtain the shared stopping gradient processing result of the m-th mode;

[0296] The second processing submodule is used to process the private code vector of the m-th mode using the stopping gradient operator to obtain the private stopping gradient processing result of the m-th mode.

[0297] The third determination submodule is used to determine the shared quantization loss based on the difference between the shared stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode;

[0298] The fourth determination submodule is used to determine the private quantization loss based on the difference between the private stopping gradient processing result of the m-th mode and the quantization input vector of the m-th mode;

[0299] The first parameter update submodule is used to update the model parameters of the quantization module to be trained based at least on the shared quantization loss and the private quantization loss, so as to obtain the trained quantization module. The trained quantization module is used to process the multimodal physiological signals of the target user for emotion classification.

[0300] Optionally, a second stage of training is performed, utilizing the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model. The second stage training module 503 includes:

[0301] An extension submodule is used to extend the text vocabulary of the large language model based on the private codebooks of each of the 1st to Mth modalities and the shared codebook shared by the 1st to Mth modalities, to obtain an extended vocabulary.

[0302] The first prediction submodule is used to predict the final quantization vector of the m-th modality in the next time step using the extended vocabulary and the final quantization vector of the m-th modality based on the historical time step of the large language model, so as to obtain the loss of the m-th modality.

[0303] The fifth determination submodule is used to obtain the multimodal loss based on the loss and corresponding weight of each of the first to M modes;

[0304] The second prediction submodule is used to predict the text vector of the next time step based on the text vector of the first text corpus using the expanded vocabulary and the large language model based on the text vector of the historical time step, so as to obtain the first text loss.

[0305] The second parameter update submodule is used to update the model parameters of the large language model based on the multimodal loss and the first text loss, so as to obtain a large language model that has been pre-trained by autoregression.

[0306] Optionally, a third stage of training is performed, using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts, to fine-tune the large language model that has been pre-trained through autoregression, thereby obtaining a sentiment classification model. The third stage training module 504 includes:

[0307] The prediction answer generation submodule is used to input the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts into the autoregressive pre-trained large language model, and generate the prediction answer through the extended vocabulary of the autoregressive pre-trained large language model.

[0308] The sixth determining submodule is used to obtain the instruction loss based on the difference between the predicted answer and the correct answer corresponding to the multimodal physiological signals of the multiple sample users;

[0309] The third prediction submodule is used to predict the text vector of the next time step based on the text vector of the historical time step using the expanded vocabulary and the large language model that has been pre-trained by autoregression, in order to obtain the second text loss.

[0310] The third parameter update submodule is used to update the model parameters of the autoregressive pre-trained large language model based on the instruction loss and the second text loss, so as to obtain the sentiment classification model.

[0311] Optionally, the multimodal physiological signal includes EEG physiological signal and physiological signal of at least one of the following modalities: ECG physiological signal, EMG physiological signal, and EOG physiological signal;

[0312] The multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts are input into the autoregressive pre-trained large language model. The autoregressive pre-trained large language model uses the expanded vocabulary to generate predicted answers. The predicted answer generation submodule includes:

[0313] The first determining subunit is used to obtain the final quantization vector of the EEG modality and the final quantization vector of other modalities by using the extended vocabulary through the large language model that has been pre-trained by autoregression.

[0314] The second determining subunit is used to determine the gated fusion coefficients through a gated fusion network based on the final quantization vector of the EEG mode and the final quantization vectors of other modes.

[0315] The third determining subunit is used to determine the corresponding gating bias based on the task identifier of the emotion classification task;

[0316] The fourth determining subunit is used to determine the corrected gating fusion coefficient based on the determined gating bias and the gating fusion coefficient;

[0317] The first fusion subunit is used to fuse the attention calculation result of the final quantization vector of other modalities to the final quantization vector of the EEG modal with the final quantization vector of the EEG modal according to the modified gating fusion coefficient, so as to obtain the final quantization enhancement vector of the EEG modal.

[0318] The second fusion subunit is used to fuse the attention calculation result of the final quantization vector of the EEG mode to the final quantization vector of other modes with the final quantization vector of other modes according to the modified gating fusion coefficient, so as to obtain the final quantization enhancement vector of other modes.

[0319] The prediction answer generation subunit is used to generate a prediction answer using the autoregressive pre-trained large language model, based on the final quantization augmentation vector of the EEG modality and the final quantization augmentation vector of other modalities.

[0320] Optionally, the system further includes:

[0321] The seventh determination submodule is used to perform real-valued FFT amplitude on the physiological signal of the m-th mode to obtain the frequency domain amplitude spectrum of the m-th mode;

[0322] The eighth determination submodule is used to perform time-domain reconstruction and frequency-domain amplitude spectrum reconstruction on the final quantization vector of the m-th mode using the pre-trained decoding module, so as to obtain the time-domain reconstruction result and the frequency-domain amplitude spectrum reconstruction result of the m-th mode;

[0323] The ninth determination submodule is used to determine the temporal reconstruction loss based on the difference between the temporal reconstruction result of the m-th mode and the physiological signal of the m-th mode;

[0324] The tenth determination submodule is used to determine the frequency domain amplitude spectrum reconstruction loss based on the difference between the frequency domain amplitude spectrum reconstruction result of the m-th mode and the frequency domain amplitude spectrum of the m-th mode.

[0325] The model parameters of the quantization module to be trained are updated based at least on the shared quantization loss and the private quantization loss to obtain the trained quantization module. The first parameter update sub-module includes:

[0326] The first parameter update subunit is used to update the model parameters of the quantization module to be trained based at least on the shared quantization loss, the private quantization loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0327] Optionally, the system further includes:

[0328] The attention calculation result determination submodule is used to determine the attention calculation result of the m-th mode using the final quantization vector of the m-th mode as K and V, and the learnable query vector of the current round of the first stage as Q; and to determine the attention calculation result of the n-th mode using the final quantization vector of the n-th mode as K and V, and the learnable query vector of the current round of the first stage as Q.

[0329] The emotion category prediction result determination submodule is used to determine the emotion category prediction result of the m-th modality based on the attention calculation result of the m-th modality and with reference to the attention calculation result of the n-th modality, and to determine the emotion category prediction result of the n-th modality based on the attention calculation result of the n-th modality and with reference to the attention calculation result of the m-th modality.

[0330] The cross-modal contrast loss determination submodule is used to obtain the cross-modal contrast loss based on the sentiment category labels of the multimodal physiological signals to which the physiological signals of the m-th modality and the n-th modality belong, as well as the sentiment category prediction results of the n-th modality and the sentiment category prediction results of the n-th modality.

[0331] The learnable query vector update submodule is used to update the learnable query vector of the current round of the first stage according to the cross-modal contrastive loss, and the updated learnable query vector is used for the next round of training in the first stage.

[0332] The model parameters of the quantization module to be trained are updated based at least on the shared quantization loss and the private quantization loss to obtain the trained quantization module. The first parameter update sub-module includes:

[0333] The second parameter update subunit is used to update the model parameters of the quantization module to be trained based on at least the shared quantization loss and the private quantization loss, the cross-modal contrast loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0334] Optionally, the system further includes:

[0335] The third processing submodule is used to process the final quantization vector of the m-th mode using a pre-trained gradient inversion layer to obtain the physiological domain processing result, and to perform domain identification on the physiological domain processing result using a pre-trained domain discriminator to obtain the physiological domain loss.

[0336] The fourth processing submodule is used to process the text vectors in the text vocabulary of the large language model using the pre-trained gradient inversion layer to obtain the text domain processing result, and to perform domain identification on the text domain processing result using the pre-trained domain discriminator to obtain the text domain loss.

[0337] The physiological-text domain alignment loss determination submodule is used to obtain the physiological-text domain alignment loss based on the physiological domain loss and the text domain loss;

[0338] The model parameters of the quantization module to be trained are updated based at least on the shared quantization loss and the private quantization loss to obtain the trained quantization module. The first parameter update sub-module includes:

[0339] The third parameter update subunit is used to update the model parameters of the quantization module to be trained based on at least the shared quantization loss and the private quantization loss, the physiological-text domain alignment loss, the cross-modal contrast loss, the temporal reconstruction loss, and the frequency domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

[0340] Each embodiment in this specification focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other.

[0341] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0342] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0343] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0344] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0345] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0346] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0347] The training method and system for an emotion classification model based on multimodal physiological signals have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method for an emotion classification model based on multimodal physiological signals, characterized in that, The method includes: Multimodal physiological signals were collected from multiple sample users, including at least two of the following physiological signals: EEG physiological signals, ECG physiological signals, EMG physiological signals, and EOG physiological signals. The first phase of training is performed by processing the multimodal physiological signals of the multiple sample users to obtain the multimodal physiological vectors of the multiple sample users. The multimodal physiological signals include physiological signals of M modalities; the first stage of training is performed, processing the multimodal physiological signals of the multiple sample users to obtain the multimodal physiological vectors of the multiple sample users, including: For the m-th mode physiological signal in the multimodal physiological signal, the quantized input vector of the m-th mode is obtained by using a pre-trained encoding module; Using the quantization module to be trained, perform the following steps: Based on the private codebook of the m-th mode, the quantized input vector of the m-th mode is subjected to nearest neighbor quantization to obtain the private code vector of the m-th mode and the corresponding private quantization error. Based on the shared codebook shared by modes 1 to M, the quantized input vector of mode m is subjected to nearest neighbor quantization to obtain the shared code vector of mode m, and the corresponding shared quantization error is obtained. If the shared quantization error is less than the private quantization error, the shared code vector of the m-th mode is determined as the final quantization vector of the m-th mode; If the shared quantization error is not less than the private quantization error, the private code vector of the m-th mode is determined as the final quantization vector of the m-th mode. The second phase of training is performed by using the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model. The large language model, after autoregressive pre-training, learns the changing patterns of the multimodal physiological signals over time. The third stage of training is performed by using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts to fine-tune the large language model that has been pre-trained by autoregression, thereby obtaining the sentiment classification model.

2. The training method for the emotion classification model oriented towards multimodal physiological signals according to claim 1, characterized in that, The method further includes: The shared code vector of the m-th mode is processed using the stopping gradient operator to obtain the shared stopping gradient processing result of the m-th mode; The private code vector of the m-th mode is processed using the stopping gradient operator to obtain the private stopping gradient processing result of the m-th mode; The shared quantization loss is determined based on the difference between the shared stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode. The private quantization loss is determined based on the difference between the private stopping gradient processing result of the m-th mode and the quantized input vector of the m-th mode. The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain a trained quantization module. The trained quantization module is used to process the multimodal physiological signals of the target user for emotion classification.

3. The training method for the emotion classification model oriented towards multimodal physiological signals according to claim 1, characterized in that, The second phase of training is performed, utilizing the multimodal physiological vectors of the multiple sample users to conduct autoregressive pre-training on the large language model, including: Based on the private codebooks of each of the 1st to Mth modalities and the shared codebook shared by the 1st to Mth modalities, the text vocabulary of the large language model is expanded to obtain an expanded vocabulary. For the m-th modality physiological signal in the multimodal physiological signal, the extended vocabulary is used to predict the final quantization vector of the m-th modality in the next time step based on the final quantization vector of the m-th modality in the historical time step using the large language model, so as to obtain the loss of the m-th modality. The multimodal loss is obtained by using the loss and corresponding weight of each of the first to M modes; For the first text corpus, using the expanded vocabulary, the text vector for the next time step is predicted by the large language model based on the text vector of the historical time step, so as to obtain the first text loss; Based on the multimodal loss and the first text loss, the model parameters of the large language model are updated to obtain a large language model that has been pre-trained by autoregression.

4. The training method for the emotion classification model oriented towards multimodal physiological signals according to claim 3, characterized in that, The third stage of training involves fine-tuning the pre-trained autoregressive language model using the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts, to obtain the sentiment classification model, including: The multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts are input into the large language model that has been pre-trained by autoregression. The large language model that has been pre-trained by autoregression then uses the expanded vocabulary to generate the predicted answer. The instruction loss is obtained based on the difference between the predicted answer and the correct answer corresponding to the multimodal physiological signals of the multiple sample users; For the second text corpus, the expanded vocabulary is used to predict the text vector of the next time step based on the text vector of the historical time step through the large language model that has been pre-trained by autoregression, so as to obtain the second text loss. Based on the instruction loss and the second text loss, the model parameters of the autoregressive pre-trained large language model are updated to obtain the sentiment classification model.

5. The training method for the emotion classification model based on multimodal physiological signals according to claim 4, characterized in that, The multimodal physiological signals include EEG physiological signals and at least one of the following modalities: ECG physiological signals, EMG physiological signals, and EOG physiological signals; The multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts are input into the autoregressive pre-trained large language model. The autoregressive pre-trained large language model then uses the expanded vocabulary to generate predicted answers, including: The final quantization vectors of the EEG modality and other modalities are obtained by using the extended vocabulary through the large language model that has been pre-trained by autoregression. Based on the final quantization vector of the EEG mode and the final quantization vectors of other modes, the gated fusion coefficients are determined through a gated fusion network. Determine the corresponding gating bias based on the task identifier of the emotion classification task; Based on the determined gating bias and the gating fusion coefficient, the corrected gating fusion coefficient is determined; Based on the corrected gating fusion coefficients, the attention calculation results of the final quantization vectors of other modalities to the final quantization vector of the EEG modality are fused with the final quantization vector of the EEG modality to obtain the final quantization enhancement vector of the EEG modality. Based on the corrected gating fusion coefficients, the attention calculation results of the final quantization vector of the EEG mode to the final quantization vector of other modes are fused with the final quantization vector of other modes to obtain the final quantization enhancement vector of other modes. Based on the final quantization augmentation vector of the EEG modality and the final quantization augmentation vector of other modalities, the predicted answer is generated by the autoregressive pre-trained large language model.

6. The training method for the emotion classification model oriented towards multimodal physiological signals according to claim 2, characterized in that, The method further includes: Perform a real-valued FFT on the physiological signal of the m-th mode to obtain the frequency domain amplitude spectrum of the m-th mode; The time-domain reconstruction and frequency-domain amplitude spectrum reconstruction of the final quantization vector of the m-th mode are performed using the pre-trained decoding module to obtain the time-domain reconstruction result and the frequency-domain amplitude spectrum reconstruction result of the m-th mode. The temporal reconstruction loss is determined based on the difference between the temporal reconstruction result of the m-th mode and the physiological signal of the m-th mode. The frequency domain amplitude spectrum reconstruction loss is determined based on the difference between the frequency domain amplitude spectrum reconstruction result of the m-th mode and the frequency domain amplitude spectrum of the m-th mode. The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including: The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

7. The training method for the emotion classification model oriented towards multimodal physiological signals according to claim 6, characterized in that, The method further includes: Using the final quantization vector of the m-th mode as K and V, and the learnable query vector of the current round of the first stage as Q, the attention calculation result of the m-th mode is determined. Using the final quantization vector of the n-th mode as K and V, and the learnable query vector of the current round of the first stage as Q, the attention calculation result of the n-th mode is determined. Based on the attention calculation result of the m-th modality, and with the attention calculation result of the n-th modality as a reference, the emotion category prediction result of the m-th modality is determined; and based on the attention calculation result of the n-th modality, and with the attention calculation result of the m-th modality as a reference, the emotion category prediction result of the n-th modality is determined. Based on the sentiment category labels of the multimodal physiological signals to which the m-th modality and the n-th modality physiological signals belong, as well as the sentiment category prediction results of the n-th modality and the sentiment category prediction results of the n-th modality, a cross-modal contrast loss is obtained; Based on the cross-modal contrastive loss, the learnable query vector of the current round in the first stage is updated, and the updated learnable query vector is used for the next round of training in the first stage. The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including: The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the cross-modal contrast loss, the time-domain reconstruction loss, and the frequency-domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

8. The training method for the emotion classification model oriented towards multimodal physiological signals according to claim 7, characterized in that, The method further includes: The final quantization vector of the m-th mode is processed using a pre-trained gradient inversion layer to obtain the physiological domain processing result. The physiological domain processing result is then used by a pre-trained domain discriminator to perform domain identification and obtain the physiological domain loss. The pre-trained gradient inversion layer is used to process the text vectors in the text vocabulary of the large language model to obtain the text domain processing result. The pre-trained domain discriminator is then used to perform domain identification on the text domain processing result to obtain the text domain loss. Based on the physiological domain loss and the text domain loss, the physiological-text domain alignment loss is obtained; The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss and the private quantization loss to obtain the trained quantization module, including: The model parameters of the quantization module to be trained are updated based on at least the shared quantization loss, the private quantization loss, the physiological-text domain alignment loss, the cross-modal contrast loss, the temporal reconstruction loss, and the frequency domain amplitude spectrum reconstruction loss, so as to obtain the trained quantization module.

9. A training system for an emotion classification model oriented towards multimodal physiological signals, characterized in that, The system includes: The acquisition module is used to acquire multimodal physiological signals from multiple sample users. The multimodal physiological signals include at least two of the following physiological signals: EEG physiological signals, ECG physiological signals, EMG physiological signals, and EOG physiological signals. The first-stage training module is used to perform the first-stage training, which processes the multimodal physiological signals of the multiple sample users to obtain the multimodal physiological vectors of the multiple sample users. The multimodal physiological signals include physiological signals of M modalities; the first-stage training module includes: A pre-training submodule is used to obtain the quantized input vector of the m-th modality from the multimodal physiological signal using a pre-trained encoding module. The pre-training submodule includes: The first nearest neighbor quantization submodule is used to perform nearest neighbor quantization on the quantized input vector of the m-th mode based on the private codebook of the m-th mode, to obtain the private code vector of the m-th mode, and to obtain the corresponding private quantization error. The second nearest neighbor quantization submodule is used to perform nearest neighbor quantization on the quantization input vector of the m-th mode based on the shared codebook shared by the 1st to Mth modes, to obtain the shared code vector of the m-th mode, and to obtain the corresponding shared quantization error. The first determining submodule is used to determine the shared code vector of the m-th mode as the final quantization vector of the m-th mode when the shared quantization error is less than the private quantization error. The second determining submodule is used to determine the private code vector of the m-th mode as the final quantization vector of the m-th mode when the shared quantization error is not less than the private quantization error. The second-stage training module is used to perform the second-stage training. It uses the multimodal physiological vectors of the multiple sample users to perform autoregressive pre-training on the large language model. The large language model, after autoregressive pre-training, learns the changing patterns of the multimodal physiological signals over time. The third-stage training module is used to perform the third-stage training. It uses the multimodal physiological vectors of the multiple sample users and the text vectors corresponding to the sentiment classification prompts to fine-tune the large language model that has been pre-trained by autoregression to obtain the sentiment classification model.

Images