Classification model, training and classification method and device of multi-modal human physiological data

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a multi-modal human physiological data classification model, a training and classification method and equipment, aiming to solve the classification problem under the condition of multi-modal physiological data loss. The classification model comprises a multi-head self-attention module, a normalization module, a fusion expert system and a decision module. The fusion expert system comprises four expert subsystems of electroencephalogram, electrocardiogram, skin electricity and multi-modal synchronous fusion. The multi-head self-attention module is used for feature extraction of multi-modal synchronous data; the normalization module is used for generating normalized feature data according to the extracted features; the four expert subsystems of electroencephalogram, electrocardiogram, skin electricity and multi-modal synchronous fusion are respectively used for performing classification tasks according to the normalized feature data corresponding to the electroencephalogram, electrocardiogram, skin electricity and multi-modal synchronous data; and the decision module is used for calculating a final classification result according to the four classification results. The present application can improve the accuracy of the classification result under the condition of multi-modal data loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-modal human physiological data classification model, a training and classification method and equipment. BACKGROUND

[0002] In the field of artificial intelligence, people often want to use a neural network model to classify the current physiological or psychological state of a subject according to the physiological data of the human body (such as electroencephalogram, electrocardiogram, etc.), for example, to determine whether the subject is currently in a state of joy or anger.

[0003] Due to poor contact of sensors or electrodes or motion of the subject, interference of other devices, etc., the collected physiological data may be lost, which may cause the classification result of the model to deviate.

[0004] To solve this problem, the commonly used solutions are: (1) directly discarding the physiological data with missing data, which is simple and crude and not practical in the case of high sample collection cost; (2) replacing the missing values with the corresponding nearest valid values, which is only suitable for cases where there is little data loss; (3) establishing a high-dimensional model to solve the problem of data loss, which is effective for single-modal data, but this method will cause the computational load to grow explosively for multi-modal data. SUMMARY

[0005] To solve the above problems in the prior art, the present application proposes a multi-modal human physiological data classification model, a training and classification method and equipment, which solves the classification problem in the case of missing multi-modal physiological data.

[0006] In a first aspect of the present application, a multi-modal human physiological data classification model is proposed, which comprises a multi-headed self-attention (Multi-headed Self-attention) module, a normalization module, a fusion expert system and a decision module.

[0007] The fusion expert system comprises an electroencephalogram expert subsystem, an electrocardiogram expert subsystem, an electrodermal activity expert subsystem and a multi-modal synchronous fusion expert subsystem.

[0008] The multi-headed self-attention module is configured to extract features from the input multi-modal synchronous data and output to the normalization module; the multi-modal synchronous data comprises physiological indicators of the subject; the physiological indicators comprise electroencephalogram (EEG) data, electrocardiogram (ECG) data and electrodermal activity (EDA) data.

[0009] The normalization module is configured to generate normalized feature data according to the extracted feature data and output the normalized feature data to the fusion expert system.

[0010] The EEG expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the EEG data of the subject, to obtain a first classification result.

[0011] The ECG expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the ECG data of the subject, to obtain a second classification result.

[0012] The skin electricity expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the skin electricity data of the subject, to obtain a third classification result.

[0013] The multi-modal synchronous fusion expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the multi-modal synchronous data, to obtain a fourth classification result.

[0014] The decision module is configured to calculate a final classification result according to the first classification result, the second classification result, the third classification result, the fourth classification result, and a weight corresponding to each classification result.

[0015] Preferably, the step of "calculating a final classification result according to the first classification result, the second classification result, the third classification result, the fourth classification result, and a weight corresponding to each classification result" includes:

[0016] The final classification result is calculated according to the following formula:

[0017] G = W1*f1 + W2*f2 + W3*f3 + W4*f4

[0018] Wherein, G represents the final classification result, f1, f2, f3 and f4 represent the first classification result, the second classification result, the third classification result and the fourth classification result respectively, and W1, W2, W3 and W4 all represent weights.

[0019] In a second aspect, a training method of a multi-modal human physiological data classification model is provided, which is suitable for the multi-modal human physiological data classification model as described above. The training method comprises:

[0020] The EEG expert subsystem and the multi-head self-attention module are trained using an EEG training set, and the weights of the ECG expert subsystem, the skin electricity expert subsystem and the multi-modal synchronous fusion expert subsystem are frozen.

[0021] training set, and freezing the weights of the multi-head self-attention module, the EEG expert subsystem, the ECG expert subsystem, and the multi-modal synchronous fusion expert subsystem;

[0022] training set, and freezing the weights of the multi-head self-attention module, the EEG expert subsystem, the ECG expert subsystem, and the multi-modal synchronous fusion expert subsystem;

[0023] training set, and adjusting the weights of the multi-head self-attention module, the EEG expert subsystem, the ECG expert subsystem, the EDA expert subsystem, and the multi-modal synchronous fusion expert subsystem.

[0024] Preferably, the method further comprises:

[0025] According to the EEG signal of each volunteer and the corresponding electrode point position, EEG data containing time domain information, frequency domain information and spatial information are generated, and the EEG training set is obtained;

[0026] The ECG signal of each volunteer is denoised, and the time information is aligned with the EEG signal of the volunteer, and the ECG training set is generated;

[0027] The EDA signal of each volunteer is denoised, and the time information is aligned with the EEG signal of the volunteer, and the EDA training set is generated;

[0028] The EEG training set, the ECG training set and the EDA training set are randomly masked to simulate data missing, and the multi-modal random coding training set is generated.

[0029] Preferably, the step of "generating EEG data containing time domain information, frequency domain information and spatial information according to the EEG signal of each volunteer and the corresponding electrode point position, and obtaining the EEG training set" comprises:

[0030] The EEG signal of each volunteer is subjected to fast Fourier transform to extract frequency domain information;

[0031] Data of the frequency band of interest is selected from the frequency domain information;

[0032] Based on the data of the frequency band of interest and the electrode point position when the EEG signal is collected, a multi-spectral image-like image based on different leads is obtained;

[0033] The multi-spectral image-like image is subjected to block processing, and the EEG data of the volunteer is generated according to the block data.

[0034] Preferably, the step of "obtaining the multi-spectral-like images based on different leads according to the data of the frequency band of interest and the electrode point positions during the acquisition of the electroencephalogram signals" comprises:

[0035] projecting the electrode point positions during the acquisition of the electroencephalogram signals from a three-dimensional space to a two-dimensional surface to obtain two-dimensional position information of each of the electrode points;

[0036] calculating the distance between each of the electrode points and other surrounding electrode points according to the two-dimensional position information;

[0037] for the data of each lead in the data of the frequency band of interest, setting a corresponding weight for the data of other leads according to the distance, and then obtaining the multi-spectral-like images based on different leads.

[0038] Preferably, the step of "performing block processing on the multi-spectral-like images, and generating the electroencephalogram data of the volunteer according to the block data" comprises:

[0039] performing block operation on the multi-spectral-like images as pictures, and dividing them into N=H*W / P 2 small blocks of the same size; wherein H, W, C and P represent the height, width, channel number and size of the small blocks of the image, respectively;

[0040] flattening the image in each of the small blocks into a vector, obtaining Patch embedding through linear projection, and adding a [I_CLS] token as the position encoding of the small block, so as to obtain the block data corresponding to each of the small blocks;

[0041] generating the electroencephalogram data of the volunteer according to the block data.

[0042] Preferably, the step of "projecting the electrode point positions during the acquisition of the electroencephalogram signals from a three-dimensional space to a two-dimensional surface to obtain two-dimensional position information of each of the electrode points" comprises:

[0043] using azimuthal equidistant projection in the Cartesian coordinate system to convert the electrode point positions during the acquisition of the electroencephalogram signals from a three-dimensional space to a two-dimensional surface, so that the converted lead data have spatial topological relationship.

[0044] In a third aspect, the present application provides a multi-modal human physiological data classification method, which uses the multi-modal human physiological data classification model as described above to perform a classification task.

[0045] In a fourth aspect, the present application provides a computer-readable storage device storing a computer program capable of being loaded and executed by a processor to perform the method as described above.

[0046] The present application has the following beneficial effects:

[0047] The present application improves the original VLMo (Vision Language pretrained Model) to make the classification model more adaptive to the physiological data of the human body, which can not only make full use of the information of each modality (electroencephalogram data, electrocardiogram data and skin electricity data) but also realize the multi-modal fusion of data, so that the performance of the model depends not only on the quality of a certain modality data, thereby improving the robustness of the model.

[0048] In order to make the classification model still maintain good performance under the condition of data loss, the data after preprocessing is artificially masked in the model training stage to simulate the condition of data loss, so that the trained model can generalize the masked part, that is, the data loss part in the actual situation, so as to obtain the most reliable result.

[0049] Considering that the electroencephalogram signal has information in time domain, frequency domain and space, after the electroencephalogram signal is transformed into frequency domain information, the present application obtains a multi-spectral image based on different leads according to the positions of the electrode points when the electroencephalogram signal is collected, then the multi-spectral image is processed by block, and the electroencephalogram data as the input data of the model is generated according to the block data. This method fully excavates the information contained in the multi-dimensional space and improves the use efficiency of the data. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the main structure schematic diagram of the multi-modal human physiological data classification model embodiment of the present application;

[0051] Figure 2 is the main step schematic diagram of the training method embodiment of the multi-modal human physiological data classification model of the present application. DETAILED DESCRIPTION

[0052] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0053] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0054] It should be noted that in the description of the present application, the terms "first", "second" are only for the convenience of description, and do not indicate or imply the relative importance of the devices, elements or parameters, and therefore cannot be understood as a limitation of the present application. In addition, the term "and / or" in the present application is only a description of the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper, unless otherwise specified, generally represents an "or" relationship between the associated objects before and after it.

[0055] Microsoft proposes a unified vision language model VLMo (Vision Language pretrained Model), which is equivalent to a hybrid expert model. The FFN (Feed Forward Network) part of the VLMo has three modes, which are a vision mode (V-FFN) for image encoding, a language mode (L-FFN) for text encoding, and a vision language mode (VL-FFN) for image-text fusion.

[0056] Figure 1 is the main constituent schematic diagram of the multi-modal human physiological data classification model embodiment of the present application. As shown in Figure 1 the classification model of the present embodiment comprises: a multi-head self-attention module 10, a normalization module 20, a fusion expert system 30 and a decision module 40.

[0057] The fusion expert system 30 comprises: an electroencephalogram expert subsystem 31, an electrocardiogram expert subsystem 32, a skin electricity expert subsystem 33 and a multi-modal synchronous fusion expert subsystem 34. The multi-modal synchronous data comprises physiological indicators of a subject; the physiological indicators comprise electroencephalogram data, electrocardiogram data and skin electricity data, and the data are all collected within the same time period.

[0058] In the present embodiment, the multi-head self-attention module 10 is used for feature extraction of the input multi-modal synchronous data and output to the normalization module 20.

[0059] In the present embodiment, the normalization module 20 is used for normalizing the extracted feature data and outputting to the fusion expert system 30.

[0060] In this embodiment, the electroencephalogram expert subsystem 31 is configured to perform a classification task based on the normalized feature data corresponding to the electroencephalogram data of the subject to obtain a first classification result; the electrocardiogram expert subsystem 32 is configured to perform a classification task based on the normalized feature data corresponding to the electrocardiogram data of the subject to obtain a second classification result; the skin electricity expert subsystem 33 is configured to perform a classification task based on the normalized feature data corresponding to the skin electricity data of the subject to obtain a third classification result; and the multi-modal synchronous fusion expert subsystem 34 is configured to perform a classification task based on the normalized feature data corresponding to the multi-modal synchronous data to obtain a fourth classification result.

[0061] In this embodiment, the decision module 40 is configured to calculate a final classification result based on the first classification result, the second classification result, the third classification result, the fourth classification result, and the weight corresponding to each classification result.

[0062] Specifically, the final classification result is calculated according to the following formula (1):

[0063] G = W1*f1 + W2*f2 + W3*f3 + W4*f4 (1)

[0064] wherein G represents the final classification result, f1, f2, f3, and f4 represent the first classification result, the second classification result, the third classification result, and the fourth classification result respectively, and W1, W2, W3, and W4 represent the weight.

[0065] Figure 2 is the main step schematic diagram of the training method embodiment of the multi-modal human physiological data classification model of the present application. The training method of this embodiment is applicable to the classification model of the multi-modal human physiological data as described above, as shown in Figure 2 The training method of this embodiment includes steps S10-S40:

[0066] Step S10: The electroencephalogram expert subsystem and the multi-head self-attention module are trained using the electroencephalogram training set, and the weights of the electrocardiogram expert subsystem, the skin electricity expert subsystem, and the multi-modal synchronous fusion expert subsystem are frozen.

[0067] Step S20: The electrocardiogram expert subsystem is trained using the electrocardiogram training set, and the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the skin electricity expert subsystem, and the multi-modal synchronous fusion expert subsystem are frozen.

[0068] Step S30: The skin electricity expert subsystem is trained using the skin electricity training set, and the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the electrocardiogram expert subsystem, and the multi-modal synchronous fusion expert subsystem are frozen.

[0069] Step S40, the classification model is trained by using the multi-modal random coding training set, so as to adjust the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the electrocardiogram expert subsystem, the skin electricity expert subsystem and the multi-modal synchronous fusion expert subsystem.

[0070] In an optional embodiment, the method of training the classification model further comprises steps S1-S4 of preprocessing the data:

[0071] Step S1, according to the electroencephalogram signal of each volunteer and the corresponding electrode point position, the electroencephalogram data containing time domain information, frequency domain information and spatial information is generated, and then the electroencephalogram training set is obtained.

[0072] Step S2, the electrocardiogram signal of each volunteer is denoised, and the time information is aligned with the electroencephalogram signal of the volunteer, and then the electrocardiogram training set is generated.

[0073] Because for a certain volunteer, only the electroencephalogram signal, electrocardiogram signal and skin electricity signal collected at the same time have related significance, so these data need to be aligned according to time.

[0074] Step S3, the skin electricity signal of each volunteer is denoised, and the time information is aligned with the electroencephalogram signal of the volunteer, and then the skin electricity training set is generated.

[0075] Step S4, the electroencephalogram training set, the electrocardiogram training set and the skin electricity training set are randomly masked to simulate the data missing condition, and the multi-modal random coding training set is generated.

[0076] Preferably, the above step S1 can specifically include steps S11-S14:

[0077] Step S11, the electroencephalogram signal of each volunteer is subjected to fast Fourier transform to extract frequency domain information.

[0078] Step S12, the data of the frequency band of interest is selected from the frequency domain information.

[0079] For example, for cognitive load tasks, we can use the data of theta (4-7 Hz), alpha (8-13 Hz) and beta (13-30 Hz) frequency bands closely related to memory.

[0080] Step S13, according to the data of the frequency band of interest and the electrode point position when the electroencephalogram signal is collected, a multi-spectrum-like image based on different leads is obtained.

[0081] Step S14, the multi-spectrum-like image is subjected to block processing, and the electroencephalogram data of the volunteer is generated according to the block data.

[0082] The electroencephalogram signal collected by the electroencephalograph is time series data, and the conventional operation is to transform it into the frequency domain, so as to use the information in the time domain and the frequency domain, which ignores the spatial information contained in the electroencephalogram signal. In order to make full use of the information in various dimensions, the electroencephalogram signal is specially processed in the application, so as to obtain information in three dimensions of time domain, frequency domain and space, and improve the utilization rate of information.

[0083] In a preferred embodiment, the above step S11 can further include:

[0084] (1) Projecting the electrode point position when collecting the electroencephalogram signal from three-dimensional space to two-dimensional surface to obtain two-dimensional position information of each electrode point. Specifically, the azimuth equidistant projection in the Cartesian coordinate system can be used to convert the electrode point position when collecting the electroencephalogram signal from three-dimensional space to two-dimensional surface, so that the converted lead data have spatial topological relationship.

[0085] EEG signal is a time series collected by a head-mounted electroencephalogram cap. The spatial topological structure exists in the distribution of the electrode points of the electroencephalogram cap on the scalp, so the collected EEG signal has both time information and spatial information. In order to convert the spatial distribution activity map into a two-dimensional image, it is necessary to project the position of the electrode points from three-dimensional space to two-dimensional surface. For this purpose, we use the azimuth equidistant projection in the Cartesian coordinate system for conversion. Compared with other projection methods, this projection method not only converts the coordinate points from three-dimensional to two-dimensional, but also preserves the distance relationship between the three-dimensional points, so that the converted EEG data also have spatial topological relationship between the leads.

[0086] (2) Calculate the distance between each electrode point and the surrounding other electrode points according to the two-dimensional position information.

[0087] (3) For the data of each lead in the data of the frequency band of interest, set the appropriate weight for the other lead data according to the calculated distance, and then obtain the multi-spectral image based on different leads.

[0088] Optionally, the above step S14 can further include:

[0089] (1) Block the multi-spectral image as a picture into N=H*W / P 2 small blocks of the same size; wherein H, W, C and P represent the height, width, channel number and size of the small block of the image, respectively.

[0090] (2) Flatten the image in each patch into a vector, obtain the Patch embedding through linear projection, and add a [I CLS] token as the position encoding of the patch, so as to obtain the corresponding patch data of each patch. The position encoding is used to record the position of the patch in the image.

[0091] (3) Generate the EEG data of the volunteer according to the patch data.

[0092] In the present application, the multispectral image is treated as a picture, and the width and height of the picture represent the spatial distribution of the brain cortex activity. In the present embodiment, the EEG data is processed into a 32*32 grid structure (P=32).

[0093] Although the above embodiment describes the steps in the above order, those skilled in the art can understand that, in order to achieve the effect of the present embodiment, the steps do not have to be executed in such an order, and they can be executed simultaneously (in parallel) or in a reversed order, and these simple changes are within the protection scope of the present application.

[0094] Further, based on the above classification model, the present application further provides an embodiment of a classification method of multi-modal human physiological data. In the present embodiment, the classification task is performed by using the multi-modal human physiological data classification model as described above.

[0095] Before performing the classification task, the EEG signal, ECG signal and skin electricity signal of the subject need to be preprocessed by referring to the preprocessing steps S1-S3 in the above training method embodiment, that is, the EEG data containing time domain information, frequency domain information and spatial information is generated according to the EEG signal of the subject and the corresponding electrode point position, and the ECG signal and the skin electricity signal are denoised and aligned.

[0096] Further, the present application further provides an embodiment of a computer readable storage device. The storage device of the present embodiment stores a computer program capable of being loaded and executed by a processor to perform the method as described above.

[0097] The computer readable storage device can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various devices capable of storing program codes.

[0098] Those skilled in the art should be able to appreciate that the method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0099] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the drawings. However, those skilled in the art will readily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the relevant technical features without deviating from the principles of the present application, and the technical solutions after such changes or replacements will all fall within the protection scope of the present application.

Claims

1. A classification model for multi-modal human physiological data, characterized in that, The classification model comprises a multi-head self-attention module, a normalization module, a fusion expert system and a decision module. The fusion expert system comprises an electroencephalogram expert subsystem, an electrocardiogram expert subsystem, an electrodermal expert subsystem and a multi-modal synchronous fusion expert subsystem. The multi-head self-attention module is configured to perform feature extraction on input multi-modal synchronous data and output to the normalization module; the multi-modal synchronous data comprises physiological indexes of a subject; the physiological indexes comprise electroencephalogram data, electrocardiogram data and electrodermal data. The normalization module is configured to generate normalized feature data according to the extracted feature data and output to the fusion expert system. The electroencephalogram expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the electroencephalogram data of the subject to obtain a first classification result. The electrocardiogram expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the electrocardiogram data of the subject to obtain a second classification result. The electrodermal expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the electrodermal data of the subject to obtain a third classification result. The multi-modal synchronous fusion expert subsystem is configured to perform a classification task according to the normalized feature data corresponding to the multi-modal synchronous data to obtain a fourth classification result. The decision module is configured to calculate a final classification result according to the first classification result, the second classification result, the third classification result and the fourth classification result, and the weight corresponding to each classification result. The training method used by the classification model comprises: The electroencephalogram expert subsystem and the multi-head self-attention module are trained using an electroencephalogram training set, and the weights of the electrocardiogram expert subsystem, the electrodermal expert subsystem and the multi-modal synchronous fusion expert subsystem are frozen; the electrocardiogram expert subsystem is trained using an electrocardiogram training set, and the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the electrodermal expert subsystem and the multi-modal synchronous fusion expert subsystem are frozen; and the electrodermal expert subsystem is trained using an electrodermal training set, and the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the electrocardiogram expert subsystem and the multi-modal synchronous fusion expert subsystem are frozen. The multi-modal random coding training set is used to train the classification model, so as to adjust the weights of the multi-head self-attention module, the EEG expert subsystem, the ECG expert subsystem, the skin electricity expert subsystem and the multi-modal synchronous fusion expert subsystem; The method further comprises: generating EEG data containing time domain information, frequency domain information and spatial information according to the EEG signal of each volunteer and the corresponding electrode point position, and then obtaining the EEG training set; The ECG signal of each volunteer is denoised, and the time information is aligned with the EEG signal of the volunteer, and then the ECG training set is generated; The skin electricity signal of each volunteer is denoised, and the time information is aligned with the EEG signal of the volunteer, and then the skin electricity training set is generated; The EEG training set, the ECG training set and the skin electricity training set are randomly masked to simulate the case of data missing, and the multi-modal random coding training set is generated.

2. The classification model of multi-modal human physiological data according to claim 1, wherein, The step of "calculating a final classification result according to the first classification result, the second classification result, the third classification result, the fourth classification result and the weight corresponding to each classification result" comprises: The final classification result G is calculated according to the following formula: G = W1*f1 + W2*f2 + W3*f3 + W4*f4 Wherein, G represents the final classification result, f1, f2, f3 and f4 respectively represent the first classification result, the second classification result, the third classification result and the fourth classification result, and W1, W2, W3 and W4 all represent weights.

3. A method for training a multi-modal human physiological data classification model, the method comprising: The application discloses a classification model suitable for multi-modal human physiological data, and the classification model comprises a multi-head self-attention module, a normalization module, a fusion expert system and a decision module; the multi-head self-attention module is used for performing feature extraction on input multi-modal synchronous data and outputting to the normalization module; the multi-modal synchronous data comprises physiological indexes of a subject; the physiological indexes comprise electroencephalogram data, electrocardiogram data and skin electricity data; the normalization module is used for generating normalized feature data according to the extracted feature data and outputting to the fusion expert system; the fusion expert system comprises an electroencephalogram expert subsystem, an electrocardiogram expert subsystem, a skin electricity expert subsystem and a multi-modal synchronous fusion expert subsystem; the electroencephalogram expert subsystem is used for performing a classification task according to the normalized feature data corresponding to the electroencephalogram data of the subject to obtain a first classification result; the electrocardiogram expert subsystem is used for performing a classification task according to the normalized feature data corresponding to the electrocardiogram data of the subject to obtain a second classification result; the skin electricity expert subsystem is used for performing a classification task according to the normalized feature data corresponding to the skin electricity data of the subject to obtain a third classification result; the multi-modal synchronous fusion expert subsystem is used for performing a classification task according to the normalized feature data corresponding to the multi-modal synchronous data to obtain a fourth classification result; the decision module is used for calculating a final classification result according to the first classification result, the second classification result, the third classification result and the fourth classification result and weights corresponding to each classification result; and a training method comprises the following steps: training the electroencephalogram expert subsystem and the multi-head self-attention module by using an electroencephalogram training set and freezing the weights of the electrocardiogram expert subsystem, the skin electricity expert subsystem and the multi-modal synchronous fusion expert subsystem; training the electrocardiogram expert subsystem by using an electrocardiogram training set and freezing the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the skin electricity expert subsystem and the multi-modal synchronous fusion expert subsystem; training the skin electricity expert subsystem by using a skin electricity training set and freezing the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the electrocardiogram expert subsystem and the multi-modal synchronous fusion expert subsystem; training the classification model by using a multi-modal random code training set, so as to adjust the weights of the multi-head self-attention module, the electroencephalogram expert subsystem, the electrocardiogram expert subsystem, the skin electricity expert subsystem and the multi-modal synchronous fusion expert subsystem; generating electroencephalogram data comprising time domain information, frequency domain information and spatial information according to the electroencephalogram signals of each volunteer and corresponding electrode point positions, and then obtaining the electroencephalogram training set; performing denoising processing on the electrocardiogram signals of each volunteer and aligning the time information with the electroencephalogram signals of the volunteer, and then generating the electrocardiogram training set; The skin electrical signal of each of the volunteers is denoised, and is aligned with the brain electrical signal of the volunteer according to time information, and then the skin electrical training set is generated; The brain electrical training set, the electrocardiogram training set and the skin electrical training set are randomly masked to simulate data missing, and the multi-modal random code training set is generated. 4.The method of Claim 3, wherein, The step of "generating brain electrical data containing time domain information, frequency domain information and spatial information according to the brain electrical signal of each volunteer and the corresponding electrode point position, and then obtaining the brain electrical training set" comprises: The brain electrical signal of each volunteer is subjected to fast Fourier transform to extract frequency domain information; Data of a frequency band of interest is selected from the frequency domain information; Based on the data of the frequency band of interest and the electrode point position when the brain electrical signal is collected, a multi-spectral image-like image based on different leads is obtained; The multi-spectral image-like image is subjected to block processing, and brain electrical data of the volunteer is generated according to the block data. 5.The method of Claim 4, wherein, The step of "obtaining a multi-spectral image-like image based on different leads based on the data of the frequency band of interest and the electrode point position when the brain electrical signal is collected" comprises: The electrode point position when the brain electrical signal is collected is projected from a three-dimensional space to a two-dimensional surface to obtain two-dimensional position information of each of the electrode points; The distance between each of the electrode points and other surrounding electrode points is calculated according to the two-dimensional position information; For the data of each lead in the data of the frequency band of interest, the distance is used to set a corresponding weight for other lead data, and then a multi-spectral image-like image based on different leads is obtained. 6.The method of claim 4, wherein, The step of "subjecting the multi-spectral image-like image to block processing, and generating brain electrical data of the volunteer according to the block data" comprises: The multi-spectrum-like image is divided into N = H*W / P small blocks of the same size as pictures 2 ; wherein H, W, C and P represent the height, width, channel number of the image and the size of the small block, respectively. The image in each of the small blocks is flattened into a vector, a Patch embedding is obtained through linear projection, and a token of [I_CLS] is added as the position encoding of the small block, so as to obtain the block data corresponding to each of the small blocks; The brain electrical data of the volunteer is generated according to the block data. 7.The method of claim 5, wherein, The step of "projecting the electrode point position when the brain electrical signal is collected from a three-dimensional space to a two-dimensional surface to obtain two-dimensional position information of each of the electrode points" comprises: The electrode point position when the brain electrical signal is collected is converted from a three-dimensional space to a two-dimensional surface by azimuthal equidistant projection in a Cartesian coordinate system, so that the lead data after conversion has a spatial topological relationship.

8. A method of classifying multi-modal human physiological data, characterized by, The method utilizes the multi-modal human physiological data classification model of any one of claims 1-2 to perform a classification task.

9. A computer-readable storage device, comprising: The computer program capable of being loaded and executed by the processor to perform the method of any one of claims 3-8 is stored.

Images