Anesthesia depth dynamic identification method based on multi-source information fusion
Through multi-source information fusion and multi-level identification model, the problem of single source of anesthesia depth monitoring signal and low recognition accuracy is solved, and high-accurate depth monitoring of anesthesia in complex environments is achieved, which is suitable for out-of-hospital applications.
Patent Information
- Application Number
- CN202411115286.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-08
AI Technical Summary
The lack of relatively accurate and objective multimodal physiological parameters in the prior art to determine the depth of anesthesia, resulting in a single source of the anesthesia depth monitoring signal, low recognition accuracy, and limited application of EEG monitoring equipment outside the hospital, making it difficult to be real-time in complex environments. Adjust the depth of anesthesia to prevent adverse events.
Using anaesthesia-deep dynamic identification method based on multi-source information fusion, pre-processing is performed by obtaining user modal information sets, including age, gender, BMI value, electrocardiogram sequence, pulse sequence, blood pressure sequence, blood oxygen sequence and respiratory parameters. and identification processing, using a multi-level identification model set for training and recognition, calculate heart rate variability values, and combine models such as coding and quantization networks to construct user awake prediction values to characterize the depth of anesthesia.
It improves the accuracy and universality of the in-depth monitoring of anesthesia, ensures the effectiveness of the model training process, enhances the convergence and accuracy of the identification model, and is suitable for in-depth monitoring of anesthesia in complex environments.
Smart Images

Figure CN118902397A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to artificial intelligence technology and medical health fields, and specifically to a method for dynamic identification of anesthesia depth based on multi-source information fusion. Background Art
[0002] General anesthesia is a special and complex state, including sedation, hypnosis, amnesia, analgesia, stress suppression and muscle relaxation, and goes through the whole process of general anesthesia induction, maintenance and awakening. Too shallow or too deep anesthesia can cause physical or mental harm to patients. Among them, too shallow anesthesia can cause stress response, endocrine disorders, metabolic abnormalities, intraoperative awareness, increased oxygen consumption and other hazards; too deep anesthesia can cause low stress response, inhibition of vital centers, respiratory function, circulatory function, and delayed awakening. Clinically, anesthesia depth monitors based on EEG signals, such as the bispectral index (BIS), Narcotrend and entropy index monitoring, have been widely used in clinical anesthesia monitoring. Although EEG signals can reflect the functional changes of the central nervous system, various physiological and pathological factors such as age, race, gender, surgical method, disease type, etc. can affect EEG signals, resulting in certain limitations in assessing the depth of anesthesia based on EEG signals.
[0003] Although there are many physiological parameters in the current anesthesia monitoring equipment, there is no connection between these parameters and physiological status. Anesthesiologists are still unable to timely understand the changes in the patient's physiological state during anesthesia, and are unable to conduct trend analysis of physiological state changes, making it difficult for anesthesiologists to give full play to their subjective initiative. The depth of anesthesia is a comprehensive response to indicators such as sedation level, analgesia level, and stimulus response degree. The central parts of these indicators are different, so anesthesia depth monitoring must be the result of multi-indicator and multi-method comprehensive monitoring. Anesthesia identification through non-invasive physiological parameters can help doctors better understand the patient's physiological state and conduct real-time monitoring. Therefore, conventional non-invasive physiological parameters (such as electrocardiogram, respiration, blood pressure, pulse, blood oxygen, etc.) are used as an economical, convenient and reliable monitoring method. Through accurate and comprehensive monitoring methods containing physical signs information, the dosage and speed of medication can be accurately grasped and controlled, and are widely used in the emergency treatment of the wounded.
[0004] Research on anesthesia identification based on non-invasive physiological parameters has made certain progress at home and abroad. The main research methods currently include electroencephalogram (EEG), pulse wave variability (PPV), functional brain imaging (fMRI), etc. In China, many research institutions and scholars focus on exploring and verifying new physiological parameter indicators by combining the characteristics of domestic patients and clinical practice, and applying them in clinical practice. Internationally, many research teams are committed to developing and verifying new physiological parameter indicators to improve the accuracy of anesthesia identification. At the same time, research uses machine learning algorithms to analyze and compare multiple physiological parameters to build models to predict the effect of anesthesia. Clinical anesthesia depth monitoring methods based on EEG monitoring technology have emerged at home and abroad, which can fully support the development of hospital surgeries, but the following problems still exist in the application process outside the hospital:
[0005] (1) EEG monitoring equipment has great limitations in use
[0006] EEG monitoring technology can reflect the level of consciousness in the cerebral cortex. The effect of anesthesia on the body's harmful stimulation during surgery mainly occurs in the subcortical center, and it is a good indicator for monitoring the depth of sedation. However, the changes in EEG signals vary greatly from person to person, and are easily interfered by other electromagnetic signals such as electromyography, mains electricity, and electrosurgical field. In addition, the installed capacity of EEG monitoring equipment is small and the consumables are expensive. It has not yet been widely used in outpatient treatment in harsh and complex environments with strong electromagnetic noise. At the same time, EEG signals are difficult to obtain hardware support in situations such as public health emergencies, on the front line of the battlefield, or in remote areas; or the collected signals are single, and there is a lack of relatively accurate and objective multimodal physiological parameters to judge the degree of intraoperative harmful stimulation and the patient's sedation level, so as to accurately evaluate and adjust the patient's intraoperative anesthesia depth in real time and prevent the occurrence of adverse anesthesia events.
[0007] (2) The source of anesthesia depth monitoring signal is single
[0008] At present, there is a lack of relatively accurate and objective multimodal physiological parameters at home and abroad to judge the degree of intraoperative noxious stimulation and the patient's sedation level, so as to accurately evaluate and adjust the patient's intraoperative anesthesia depth in real time and prevent the occurrence of adverse anesthesia events. Therefore, it is necessary to seek a new anesthesia depth monitoring method to overcome the shortcomings of the mainstream anesthesia depth monitoring method based on EEG signals, so as to improve the accuracy of anesthesia depth monitoring.
[0009] (3) Low accuracy of anesthesia depth monitoring and recognition
[0010] Currently, anesthesia depth monitoring equipment lacks core algorithms, and there is still room for improvement in the recognition accuracy and the effectiveness of handling special situations.
[0011] Therefore, it is necessary to seek a new method for anesthesia depth monitoring to solve the problem of using conventional physiological parameters for anesthesia monitoring in the absence of EEG signals for anesthesia depth monitoring, thereby improving the accuracy and universality of anesthesia depth monitoring. Summary of the invention
[0012] The present invention mainly solves the problem of using conventional physiological parameters for anesthesia monitoring in the absence of EEG signals for anesthesia depth monitoring, thereby improving the accuracy and universality of anesthesia depth monitoring. The present invention discloses a method and device for dynamic identification of anesthesia depth based on multi-source information fusion.
[0013] In a first aspect of the embodiments of the present application, a method for dynamic identification of user modalities based on multi-source information fusion is disclosed, comprising:
[0014] S1, obtaining a user modal information set; the user modal information set includes modal information of several users; the modal information includes user age, gender, BMI value, ASA grade value, ECG sequence, pulse sequence, blood pressure sequence, blood oxygen sequence, respiratory parameters and user wakefulness value; the user wakefulness value is 1, indicating that the user is in a conscious state, and the user wakefulness value is 0, indicating that the user is in an unconscious state;
[0015] S2, preprocessing the user modality information set to obtain a preprocessed user modality information set;
[0016] S3, performing identification processing on the pre-processed user modal information set to obtain identification result information; the identification result information is used to characterize the user's degree of wakefulness; when the identification result information takes a value of 1, it indicates that the user is in a wakeful state, and the corresponding anesthesia depth is shallow anesthesia; when the identification result information takes a value of 0, it indicates that the user is in an unconscious state, and the corresponding anesthesia depth is deep anesthesia.
[0017] The preprocessing of the user modality information set to obtain a preprocessed user modality information set includes:
[0018] Performing data cleaning processing on each type of data in the user modal information set to obtain a first modal information set;
[0019] Performing acquisition time alignment processing on each type of data in the first modal information set to obtain a second modal information set;
[0020] performing high-pass filtering on the respiratory parameters in the second modal information set to obtain updated respiratory parameters;
[0021] Using the updated breathing parameters, replacing the breathing parameters in the second modal information set to obtain a third modal information set;
[0022] A heart rate variability value is calculated using the electrocardiogram sequence in the third modality information set, and the heart rate variability value is added to the third modality information set to obtain a preprocessed user modality information set.
[0023] The performing identification processing on the pre-processed user modal information set to obtain identification result information includes:
[0024] S31, using the pre-processed user modal information set, training a preset recognition model set to obtain a training recognition model set;
[0025] S32, using the training recognition model set, performing recognition processing on the pre-processed user modal information set to obtain recognition result information.
[0026] The identification model set includes a first identification model, a second identification model, a third identification model, a fourth identification model and a fifth identification model;
[0027] The first recognition model includes an encoding network, a quantization network and a decoding network;
[0028] The encoding network is connected to the quantization network and includes a first processing module, a second processing module, a third processing module and a fourth processing module, which are used to encode the input data to obtain feature image data;
[0029] The first processing module includes an input module, a first two-dimensional convolution module, a first activation module, a first residual module, and a first normalization module; the first port of the first processing module is the input end of the input module; the second port of the first processing module is the output end of the first normalization module; the first port of the first processing module is used as the input end of the encoding network to receive input data; the second port of the first processing module is connected to the first port of the second processing module; in the first processing module, the input module, the first two-dimensional convolution module, the first activation module, the first residual module, and the first normalization module are connected in sequence;
[0030] The second processing module includes a second two-dimensional convolution module, a second activation module, and a second normalization module; the first port of the second processing module is the input end of the second two-dimensional convolution module; the second port of the second processing module is the output end of the second normalization module; the second port of the second processing module is connected to the first port of the third processing module; in the second processing module, the second two-dimensional convolution module, the second activation module, and the second normalization module are connected in sequence;
[0031] The third processing module includes a third two-dimensional convolution module, a third activation module, and a third residual module; the first port of the third processing module is the input end of the third two-dimensional convolution module; the second port of the third processing module is the output end of the third residual module; the second port of the third processing module is connected to the first port of the fourth processing module; in the third processing module, the third two-dimensional convolution module, the third activation module, and the third residual module are connected in sequence;
[0032] The fourth processing module includes a fourth two-dimensional convolution module, a fourth activation module, and a fourth residual module; the first port of the fourth processing module is the input end of the fourth two-dimensional convolution module; the second port of the fourth processing module is the output end of the fourth residual module; the second port of the fourth processing module is used as the output end of the encoding network to output feature image data; in the fourth processing module, the fourth two-dimensional convolution module, the fourth activation module, and the fourth residual module are connected in sequence;
[0033] The quantization network is connected to the encoding network and the decoding network, and is used to perform quantization processing on the feature image data to obtain compressed data; the quantization network includes a ninth two-dimensional convolution module, a ninth activation module and a quantization module; in the quantization network, the ninth two-dimensional convolution module, the ninth activation module and the quantization module are connected in sequence;
[0034] The decoding network is connected to the quantization network, and includes a fifth processing module, a sixth processing module, a seventh processing module and an eighth processing module, which are used to decode the compressed data to obtain a user sobriety prediction value;
[0035] The fifth processing module includes an input module, a random arrangement module, a fifth two-dimensional convolution module, a fifth activation module, and a fifth residual module; the first port of the fifth processing module is the input end of the input module, and is used to receive the compressed data; the second port of the fifth processing module is the output end of the fifth residual module, and is connected to the first port of the sixth processing module; in the fifth processing module, the input module, the random arrangement module, the fifth two-dimensional convolution module, the fifth activation module, and the fifth residual module are connected in sequence;
[0036] The sixth processing module includes a sixth two-dimensional convolution module, a sixth activation module, and a sixth residual module; the first port of the sixth processing module is the input end of the sixth two-dimensional convolution module; the second port of the sixth processing module is the output end of the sixth residual module; the second port of the sixth processing module is connected to the first port of the seventh processing module; in the sixth processing module, the sixth two-dimensional convolution module, the sixth activation module, and the sixth residual module are connected in sequence;
[0037] The seventh processing module includes a seventh two-dimensional convolution module, a seventh activation module, and a seventh normalization module; the first port of the seventh processing module is the input end of the seventh two-dimensional convolution module; the second port of the seventh processing module is the output end of the seventh normalization module; the second port of the seventh processing module is connected to the first port of the eighth processing module; in the seventh processing module, the seventh two-dimensional convolution module, the seventh activation module, and the seventh normalization module are connected in sequence;
[0038] The eighth processing module includes an eighth two-dimensional convolution module, an eighth activation module, an eighth residual module, and an eighth normalization module; the first port of the eighth processing module is the input end of the eighth two-dimensional convolution module; the second port of the eighth processing module is the output end of the eighth normalization module; the second port of the eighth processing module is the output end of the decoding network, which is used to output the calculated user wakefulness prediction value; in the eighth processing module, the eighth two-dimensional convolution module, the eighth activation module, the eighth residual module, and the eighth normalization module are connected in sequence.
[0039] The second recognition model includes a first input module, a first convolution module, a depthwise separable convolution module, a first dimension-raising convolution module, a second dimension-raising convolution module, a third dimension-raising convolution module, a fourth dimension-raising convolution module, a second convolution module, a first pooling module, a third convolution module and a first fully connected module;
[0040] The input end of the first input module of the second recognition model is used to receive input data; the output end of the first input module of the second recognition model is connected to the input end of the first convolution module of the second recognition model; the output end of the first convolution module of the second recognition model is connected to the input end of the depthwise separable convolution module of the second recognition model; the output end of the depthwise separable convolution module of the second recognition model is connected to the input end of the first dimensionality-raising convolution module of the second recognition model; the output end of the first dimensionality-raising convolution module of the second recognition model is connected to the input end of the second dimensionality-raising convolution module of the second recognition model; the output end of the second dimensionality-raising convolution module of the second recognition model is connected to the input end of the second dimensionality-raising convolution module of the second recognition model The output end of the third dimensionality-raising convolution module of the second recognition model is connected to the input end of the fourth dimensionality-raising convolution module of the second recognition model; the output end of the fourth dimensionality-raising convolution module of the second recognition model is connected to the input end of the second convolution module of the second recognition model; the output end of the second convolution module of the second recognition model is connected to the input end of the first pooling module of the second recognition model; the output end of the first pooling module of the second recognition model is connected to the input end of the third convolution module of the second recognition model; the output end of the third convolution module of the second recognition model is connected to the input end of the first fully connected module of the second recognition model;
[0041] The output end of the first fully connected module of the second identification model is used to output a user sobriety prediction value.
[0042] The third recognition model includes a second input module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a second pooling module, a seventh convolution module and a second fully connected module;
[0043] The input end of the second input module of the third recognition model is used to receive input data; the output end of the second input module of the third recognition model is connected to the input end of the fourth convolution module of the third recognition model; the output end of the fourth convolution module of the third recognition model is connected to the input end of the fifth convolution module of the third recognition model; the output end of the fifth convolution module of the third recognition model is connected to the input end of the sixth convolution module of the third recognition model; the output end of the sixth convolution module of the third recognition model is connected to the input end of the second pooling module of the third recognition model; the output end of the second pooling module of the third recognition model is connected to the input end of the seventh convolution module of the third recognition model; the output end of the seventh convolution module of the third recognition model is connected to the input end of the second fully connected module of the third recognition model;
[0044] The output end of the second fully connected module of the third identification model is used to output the user wakefulness prediction value.
[0045] The fourth recognition model includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a first fusion module, a second fusion module, a third fusion module, a first normalization module, a second normalization module, and a first activation module; the input end of the first convolution module is used to receive input data, and the output end of the first convolution module is respectively connected to the input end of the first fusion module and the input end of the second fusion module; the output end of the first fusion module is connected to the input end of the third convolution module; the output end of the third convolution module is connected to the input end of the first normalization module; the output end of the first normalization module The first end is connected to the input end of the third fusion module; the input end of the second convolution module is used to receive input data, and the output end of the second convolution module is respectively connected to the input end of the second convolution module and the input end of the first fusion module; the output end of the second fusion module is connected to the input end of the fourth convolution module; the output end of the fourth convolution module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the third fusion module; the output end of the third fusion module is connected to the input end of the first activation module; the output end of the first activation module is used to output the user wakefulness prediction value.
[0046] The fifth recognition model includes a fifth convolution module, a sixth convolution module, a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a first upsampling module, a second upsampling module, a first multi-head attention module, a second multi-head attention module, a third multi-head attention module, a fourth multi-head attention module, a second activation module, a third activation module, a fourth activation module, a fifth activation module, a sixth activation module, a seventh activation module, a third normalization module, a fourth normalization module, a fifth normalization module, a fourth fusion module and a fifth fusion module;
[0047] The input end of the fifth convolution module is used to receive input data, and the output end of the fifth convolution module is connected to the input end of the sixth convolution module; the output end of the sixth convolution module is connected to the input end of the first multi-head attention module; the output end of the first multi-head attention module is connected to the input end of the second activation module; the output end of the second activation module is connected to the input end of the second multi-head attention module; the output end of the second multi-head attention module is respectively connected to the input end of the seventh convolution module and the input end of the third activation module; the output end of the seventh convolution module is connected to the third normalization module. The output end of the third normalization module is connected to the input end of the eighth convolution module and the input end of the fifth fusion module respectively; the output end of the eighth convolution module is connected to the input end of the tenth convolution module; the output end of the tenth convolution module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the eleventh convolution module and the input end of the fifth fusion module respectively; the output end of the eleventh convolution module is connected to the input end of the twelfth convolution module; the output end of the third activation module is connected to the input end of the third multi-head attention module ; The output end of the third multi-head attention module is respectively connected to the input end of the fourth fusion module and the input end of the fifth activation module; the output end of the fourth fusion module is connected to the output end of the ninth convolution module; the output end of the ninth convolution module is respectively connected to the input end of the second upsampling module and the input end of the tenth convolution module; the output end of the second upsampling module is connected to the input end of the seventh convolution module; the output end of the fifth activation module is connected to the input end of the first pooling module; the output end of the first pooling module is connected to the input end of the fourth multi-head attention module; the output end of the fourth multi-head attention module is connected to the input end of the sixth activation module; the output end of the sixth activation module is respectively connected to the input end of the first upsampling module and the input end of the twelfth convolution module; the output end of the first upsampling module is connected to the input end of the fourth fusion module; the output end of the twelfth convolution module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fusion module; the output end of the fifth fusion module is connected to the input end of the seventh activation module; the output end of the seventh activation module is used to output the user wakefulness prediction value.
[0048] The method of using the pre-processed user modal information set to train a preset recognition model set to obtain a training recognition model set includes:
[0049] The pre-processed user modal information set is divided and processed according to a preset ratio to obtain a training set and a test set; the training set includes a plurality of training data and corresponding label values; the training data includes the remaining data after the user wakefulness value is removed from the user's modal information; the label value corresponding to the training data is the user wakefulness value in the modal information corresponding to the training data;
[0050] Using the training set, each recognition model in the preset recognition model set is trained to obtain a trained recognition model;
[0051] By using all trained recognition models, a set of trained recognition models is constructed.
[0052] The training process of the recognition model includes:
[0053] Initialize the number of training times;
[0054] Acquire training data from the training set;
[0055] Inputting the training data into the recognition model to obtain a user sobriety prediction value;
[0056] Using a loss function to calculate the user sobriety prediction value and the label value corresponding to the training data to obtain a difference value;
[0057] Using the difference value to update the parameter value of the identification model;
[0058] Performing an accumulation operation on the training times;
[0059] Determine whether the training times value is greater than a preset training times threshold value to obtain a first judgment result; when the first judgment result is yes, determine that the identification model after the parameter value is updated is the trained identification model; when the first judgment result is no, trigger the execution of obtaining training data from the training set.
[0060] The beneficial effects of the present invention are:
[0061] The present invention mainly solves the problem of using conventional physiological parameters for anesthesia monitoring in the absence of EEG signals for anesthesia depth monitoring, thereby improving the accuracy and universality of anesthesia depth monitoring.
[0062] The present invention uses the updated breathing parameters to replace the breathing parameters in the second modal information set to obtain a third modal information set; uses the electrocardiogram sequence in the third modal information set to calculate the heart rate variability value, and adds the heart rate variability value to the third modal information set to obtain a preprocessed user modal information set. In this way, the diversity of the preprocessed user modal information set is effectively improved, ensuring the effectiveness of the subsequent model training process.
[0063] The identification model set established by the present invention includes a first identification model, a second identification model, a third identification model, a fourth identification model and a fifth identification model, and improves the accuracy of identification by means of five levels of identification models of different dimensions.
[0064] The present invention proposes an update process for the identification model, and the update process includes: θ←θ+v; Through this updating process, the parameter values of the identification model are accurately and efficiently updated by utilizing the difference value, thereby ensuring the convergence of the identification model. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 It is a flow chart for implementing the method of the present invention. DETAILED DESCRIPTION
[0066] In order to better understand the content of the present invention, an embodiment is given here.
[0067] Figure 1 It is a flow chart for implementing the method of the present invention.
[0068] Embodiment 1:
[0069] In a first aspect of the embodiments of the present application, a method for dynamic identification of user modalities based on multi-source information fusion is disclosed, comprising:
[0070] S1, obtaining a user modal information set; the user modal information set includes modal information of several users; the modal information includes user age, gender, BMI value, ASA grade value, ECG sequence, pulse sequence, blood pressure sequence, blood oxygen sequence, respiratory parameters and user wakefulness value; the user wakefulness value is 1, indicating that the user is in a conscious state, and the user wakefulness value is 0, indicating that the user is in an unconscious state; the respiratory parameters include inspiratory time, mean airway pressure, positive end-expiratory pressure, minute ventilation, inspiratory peak pressure, inspiratory platform pressure, respiratory rate, inspiratory pressure, tidal volume and other parameters.
[0071] S2, preprocessing the user modality information set to obtain a preprocessed user modality information set;
[0072] S3, performing anesthesia depth identification processing on the pre-processed user modal information set to obtain anesthesia depth identification result information; the anesthesia depth identification result information is used to characterize the anesthesia depth of the user; when the anesthesia depth identification result information has a value of 1, it indicates that the user is in shallow anesthesia; when the anesthesia depth identification result information has a value of 0, it indicates that the user is in deep anesthesia;
[0073] The preprocessing of the user modality information set to obtain a preprocessed user modality information set includes:
[0074] Performing data cleaning processing on each type of data in the user modal information set to obtain a first modal information set;
[0075] Performing acquisition time alignment processing on each type of data in the first modal information set to obtain a second modal information set;
[0076] performing high-pass filtering on the respiratory parameters in the second modal information set to obtain updated respiratory parameters;
[0077] Using the updated breathing parameters, replacing the breathing parameters in the second modal information set to obtain a third modal information set;
[0078] Using the electrocardiogram sequence in the third modal information set, a heart rate variability value is calculated, and the heart rate variability value is added to the third modal information set to obtain a preprocessed user modal information set;
[0079] The step of calculating the heart rate variability value by using the electrocardiogram sequence in the third modality information set includes:
[0080] Using the ECG sequence, the RR interval time of the user's sinus rhythm and the instantaneous heart rate data are calculated, and the variance value of the RR interval time of the user's sinus rhythm and the variance value of the instantaneous heart rate data are determined as the heart rate variability value;
[0081] The RR interval time and instantaneous heart rate data of the user's sinus rhythm can be realized by time domain calculation or frequency domain calculation.
[0082] The data cleaning process includes filling missing values, smoothing noise data, and smoothing or deleting outliers; the smoothed noise data is obtained by first determining the noise data, and then smoothing the noise data according to the data before and after the noise data; the noise data is a value whose value is less than the detection sensitivity of the sensor of the observed data, or greater than the measurement upper limit of the sensor of the observed data. The outlier point can be determined by the Kalman filter method. The filling value of the missing value can be determined by averaging the measured values within a certain sampling interval before and after the missing value.
[0083] The acquisition time alignment process comprises:
[0084] For each type of observation data, set a unified data collection time;
[0085] When the time interval for collecting a certain type of data is greater than the time interval for the unified data collection time, interpolation processing is performed on adjacent data to obtain the data value at the unified data collection time, which is used as the standard observation data;
[0086] When the collection time interval of a certain type of data is less than the time interval of the unified data collection time, the data is sampled to obtain data consistent with the unified data collection time, which is used as the standard observation data;
[0087] Using all types of normative observation data, the second modal information set is constructed;
[0088] The performing anesthesia depth identification processing on the pre-processed user modality information set to obtain anesthesia depth identification result information includes:
[0089] S31, using the pre-processed user modal information set, training a preset recognition model set to obtain a training recognition model set;
[0090] S32, using the training recognition model set, performing recognition processing on the pre-processed user modal information set to obtain recognition result information.
[0091] The identification model set includes a first identification model, a second identification model, a third identification model, a fourth identification model and a fifth identification model;
[0092] The first recognition model includes an encoding network, a quantization network and a decoding network;
[0093] The encoding network is connected to the quantization network and includes a first processing module, a second processing module, a third processing module and a fourth processing module, which are used to encode the input data to obtain feature image data;
[0094] The first processing module includes an input module, a first two-dimensional convolution module, a first activation module, a first residual module, and a first normalization module; the first port of the first processing module is the input end of the input module; the second port of the first processing module is the output end of the first normalization module; the first port of the first processing module is used as the input end of the encoding network to receive input data; the second port of the first processing module is connected to the first port of the second processing module; in the first processing module, the input module, the first two-dimensional convolution module, the first activation module, the first residual module, and the first normalization module are connected in sequence;
[0095] The second processing module includes a second two-dimensional convolution module, a second activation module, and a second normalization module; the first port of the second processing module is the input end of the second two-dimensional convolution module; the second port of the second processing module is the output end of the second normalization module; the second port of the second processing module is connected to the first port of the third processing module; in the second processing module, the second two-dimensional convolution module, the second activation module, and the second normalization module are connected in sequence;
[0096] The third processing module includes a third two-dimensional convolution module, a third activation module, and a third residual module; the first port of the third processing module is the input end of the third two-dimensional convolution module; the second port of the third processing module is the output end of the third residual module; the second port of the third processing module is connected to the first port of the fourth processing module; in the third processing module, the third two-dimensional convolution module, the third activation module, and the third residual module are connected in sequence;
[0097] The fourth processing module includes a fourth two-dimensional convolution module, a fourth activation module, and a fourth residual module; the first port of the fourth processing module is the input end of the fourth two-dimensional convolution module; the second port of the fourth processing module is the output end of the fourth residual module; the second port of the fourth processing module is used as the output end of the encoding network to output feature image data; in the fourth processing module, the fourth two-dimensional convolution module, the fourth activation module, and the fourth residual module are connected in sequence;
[0098] The quantization network is connected to the encoding network and the decoding network, and is used to perform quantization processing on the feature image data to obtain compressed data; the quantization network includes a ninth two-dimensional convolution module, a ninth activation module and a quantization module; in the quantization network, the ninth two-dimensional convolution module, the ninth activation module and the quantization module are connected in sequence;
[0099] The decoding network is connected to the quantization network, and includes a fifth processing module, a sixth processing module, a seventh processing module and an eighth processing module, which are used to decode the compressed data to obtain a user sobriety prediction value;
[0100] The fifth processing module includes an input module, a random arrangement module, a fifth two-dimensional convolution module, a fifth activation module, and a fifth residual module; the first port of the fifth processing module is the input end of the input module, and is used to receive the compressed data; the second port of the fifth processing module is the output end of the fifth residual module, and is connected to the first port of the sixth processing module; in the fifth processing module, the input module, the random arrangement module, the fifth two-dimensional convolution module, the fifth activation module, and the fifth residual module are connected in sequence;
[0101] The sixth processing module includes a sixth two-dimensional convolution module, a sixth activation module, and a sixth residual module; the first port of the sixth processing module is the input end of the sixth two-dimensional convolution module; the second port of the sixth processing module is the output end of the sixth residual module; the second port of the sixth processing module is connected to the first port of the seventh processing module; in the sixth processing module, the sixth two-dimensional convolution module, the sixth activation module, and the sixth residual module are connected in sequence;
[0102] The seventh processing module includes a seventh two-dimensional convolution module, a seventh activation module, and a seventh normalization module; the first port of the seventh processing module is the input end of the seventh two-dimensional convolution module; the second port of the seventh processing module is the output end of the seventh normalization module; the second port of the seventh processing module is connected to the first port of the eighth processing module; in the seventh processing module, the seventh two-dimensional convolution module, the seventh activation module, and the seventh normalization module are connected in sequence;
[0103] The eighth processing module includes an eighth two-dimensional convolution module, an eighth activation module, an eighth residual module, and an eighth normalization module; the first port of the eighth processing module is the input end of the eighth two-dimensional convolution module; the second port of the eighth processing module is the output end of the eighth normalization module; the second port of the eighth processing module is the output end of the decoding network, which is used to output the calculated user wakefulness prediction value; in the eighth processing module, the eighth two-dimensional convolution module, the eighth activation module, the eighth residual module, and the eighth normalization module are connected in sequence.
[0104] The activation module is used to implement an activation function; the activation function may be a sigmoid function;
[0105] The residual module can be implemented using a Resnet network;
[0106] The normalization module can be implemented by using the Quantile Normalization function in phthon.
[0107] The random permutation module is implemented using the shuffle function in Python.
[0108] The ninth activation module is implemented using a tanh function;
[0109] The calculation formula of the quantization module is:
[0110]
[0111] Wherein, X is the input of the quantization module.
[0112] The structures of the convolution module, the activation module, the residual module, and the normalization module in the eighth processing module are the same as those of the convolution module, the activation module, the residual module, and the normalization module in the first processing module;
[0113] The structures of the convolution module, the activation module, the residual module, and the normalization module in the seventh processing module are the same as the structures of the convolution module, the activation module, the residual module, and the normalization module in the second processing module;
[0114] The structures of the convolution module, the activation module, the residual module, and the normalization module in the sixth processing module are the same as the structures of the convolution module, the activation module, the residual module, and the normalization module in the third processing module;
[0115] The structures of the convolution module, the activation module, the residual module, and the normalization module in the fifth processing module are the same as the structures of the convolution module, the activation module, the residual module, and the normalization module in the fourth processing module;
[0116] The second recognition model includes a first input module, a first convolution module, a depthwise separable convolution module, a first dimension-raising convolution module, a second dimension-raising convolution module, a third dimension-raising convolution module, a fourth dimension-raising convolution module, a second convolution module, a first pooling module, a third convolution module and a first fully connected module;
[0117] The input end of the first input module of the second recognition model is used to receive input data; the output end of the first input module of the second recognition model is connected to the input end of the first convolution module of the second recognition model; the output end of the first convolution module of the second recognition model is connected to the input end of the depthwise separable convolution module of the second recognition model; the output end of the depthwise separable convolution module of the second recognition model is connected to the input end of the first dimensionality-raising convolution module of the second recognition model; the output end of the first dimensionality-raising convolution module of the second recognition model is connected to the input end of the second dimensionality-raising convolution module of the second recognition model; the output end of the second dimensionality-raising convolution module of the second recognition model is connected to the input end of the second dimensionality-raising convolution module of the second recognition model The output end of the third dimensionality-raising convolution module of the second recognition model is connected to the input end of the fourth dimensionality-raising convolution module of the second recognition model; the output end of the fourth dimensionality-raising convolution module of the second recognition model is connected to the input end of the second convolution module of the second recognition model; the output end of the second convolution module of the second recognition model is connected to the input end of the first pooling module of the second recognition model; the output end of the first pooling module of the second recognition model is connected to the input end of the third convolution module of the second recognition model; the output end of the third convolution module of the second recognition model is connected to the input end of the first fully connected module of the second recognition model;
[0118] The output end of the first fully connected module of the second identification model is used to output a user sobriety prediction value.
[0119] The third recognition model includes a second input module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a second pooling module, a seventh convolution module and a second fully connected module;
[0120] The input end of the second input module of the third recognition model is used to receive input data; the output end of the second input module of the third recognition model is connected to the input end of the fourth convolution module of the third recognition model; the output end of the fourth convolution module of the third recognition model is connected to the input end of the fifth convolution module of the third recognition model; the output end of the fifth convolution module of the third recognition model is connected to the input end of the sixth convolution module of the third recognition model; the output end of the sixth convolution module of the third recognition model is connected to the input end of the second pooling module of the third recognition model; the output end of the second pooling module of the third recognition model is connected to the input end of the seventh convolution module of the third recognition model; the output end of the seventh convolution module of the third recognition model is connected to the input end of the second fully connected module of the third recognition model.
[0121] The output end of the second fully connected module of the third identification model is used to output the user wakefulness prediction value.
[0122] The convolution module can be implemented by using 3D multi-channel convolution;
[0123] The depthwise separable convolution module can be implemented by connecting a channel splitting submodule and a single-channel convolution submodule. Specifically, it can be implemented by using the depthwise separable convolution module in the MobileNet network.
[0124] The dimension-raising convolution module can be implemented by using the dimension-raising convolution module in the ResNet network;
[0125] The pooling module can be implemented by using a maximum pooling operation.
[0126] Specifically, when the convolution module, depthwise separable convolution module, dimensionality-increasing convolution module and pooling module are implemented by using multiple three-dimensional filters in parallel, the data characteristics of the matrices of the corresponding three-dimensional filters are different. The data characteristics of the matrices include the mean, variance, eigenvalues, etc. of the matrices.
[0127] The first convolution module, the depthwise separable convolution module, the first dimension-raising convolution module, the second dimension-raising convolution module, the third dimension-raising convolution module, the fourth dimension-raising convolution module, the second convolution module, the first pooling module and the third convolution module of the second recognition model are all implemented by N parallel three-dimensional filters; the three-dimensional filters in the front and back adjacent modules in the second recognition model are connected according to the sequence numbers;
[0128] The fourth convolution module, the fifth convolution module, the sixth convolution module, the second pooling module and the seventh convolution module of the third recognition model are all implemented by using N parallel three-dimensional filters; the three-dimensional filters in the front and back adjacent modules in the third recognition model are connected according to the sequence number;
[0129] The three-dimensional filters in the front and rear adjacent modules in the first or third recognition model are connected according to the serial number, specifically, the first three-dimensional filter of the previous convolution module is connected to the first three-dimensional filter of the next convolution module, the second three-dimensional filter of the previous convolution module is connected to the second three-dimensional filter of the next convolution module, the third three-dimensional filter of the previous convolution module is connected to the third three-dimensional filter of the next convolution module, and specifically, the last three-dimensional filter of the previous convolution module is connected to the last three-dimensional filter of the next convolution module.
[0130] The fourth recognition model includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a first fusion module, a second fusion module, a third fusion module, a first normalization module, a second normalization module, and a first activation module; the input end of the first convolution module is used to receive input data, and the output end of the first convolution module is respectively connected to the input end of the first fusion module and the input end of the second fusion module; the output end of the first fusion module is connected to the input end of the third convolution module; the output end of the third convolution module is connected to the input end of the first normalization module; the output end of the first normalization module The first end is connected to the input end of the third fusion module; the input end of the second convolution module is used to receive input data, and the output end of the second convolution module is respectively connected to the input end of the second convolution module and the input end of the first fusion module; the output end of the second fusion module is connected to the input end of the fourth convolution module; the output end of the fourth convolution module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the third fusion module; the output end of the third fusion module is connected to the input end of the first activation module; the output end of the first activation module is used to output the user wakefulness prediction value.
[0131] The fusion module can be implemented using a fully connected network;
[0132] The fifth recognition model includes a fifth convolution module, a sixth convolution module, a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a first upsampling module, a second upsampling module, a first multi-head attention module, a second multi-head attention module, a third multi-head attention module, a fourth multi-head attention module, a second activation module, a third activation module, a fourth activation module, a fifth activation module, a sixth activation module, a seventh activation module, a third normalization module, a fourth normalization module, a fifth normalization module, a fourth fusion module and a fifth fusion module;
[0133] The input end of the fifth convolution module is used to receive input data, and the output end of the fifth convolution module is connected to the input end of the sixth convolution module; the output end of the sixth convolution module is connected to the input end of the first multi-head attention module; the output end of the first multi-head attention module is connected to the input end of the second activation module; the output end of the second activation module is connected to the input end of the second multi-head attention module; the output end of the second multi-head attention module is respectively connected to the input end of the seventh convolution module and the input end of the third activation module; the output end of the seventh convolution module is connected to the third normalization module. The output end of the third normalization module is connected to the input end of the eighth convolution module and the input end of the fifth fusion module respectively; the output end of the eighth convolution module is connected to the input end of the tenth convolution module; the output end of the tenth convolution module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the eleventh convolution module and the input end of the fifth fusion module respectively; the output end of the eleventh convolution module is connected to the input end of the twelfth convolution module; the output end of the third activation module is connected to the input end of the third multi-head attention module ; The output end of the third multi-head attention module is respectively connected to the input end of the fourth fusion module and the input end of the fifth activation module; the output end of the fourth fusion module is connected to the output end of the ninth convolution module; the output end of the ninth convolution module is respectively connected to the input end of the second upsampling module and the input end of the tenth convolution module; the output end of the second upsampling module is connected to the input end of the seventh convolution module; the output end of the fifth activation module is connected to the input end of the first pooling module; the output end of the first pooling module is connected to the input end of the fourth multi-head attention module; the output end of the fourth multi-head attention module is connected to the input end of the sixth activation module; the output end of the sixth activation module is respectively connected to the input end of the first upsampling module and the input end of the twelfth convolution module; the output end of the first upsampling module is connected to the input end of the fourth fusion module; the output end of the twelfth convolution module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fusion module; the output end of the fifth fusion module is connected to the input end of the seventh activation module; the output end of the seventh activation module is used to output the user wakefulness prediction value.
[0134] The method of using the pre-processed user modal information set to train a preset recognition model set to obtain a training recognition model set includes:
[0135] The pre-processed user modal information set is divided and processed according to a preset ratio to obtain a training set and a test set; the training set includes a plurality of training data and corresponding label values; the training data includes the remaining data after the user wakefulness value is removed from the user's modal information; the label value corresponding to the training data is the user wakefulness value in the modal information corresponding to the training data;
[0136] Using the training set, each recognition model in the preset recognition model set is trained to obtain a trained recognition model;
[0137] By using all trained recognition models, a set of trained recognition models is constructed.
[0138] The training process of the recognition model includes:
[0139] Initialize the number of training times;
[0140] Acquire training data from the training set;
[0141] Inputting the training data into the recognition model to obtain a user sobriety prediction value;
[0142] Using a loss function to calculate the user sobriety prediction value and the label value corresponding to the training data to obtain a difference value;
[0143] Using the difference value to update the parameter value of the identification model;
[0144] Performing an accumulation operation on the training times;
[0145] Determine whether the training number value is greater than a preset training number threshold value to obtain a first judgment result; when the first judgment result is yes, determine that the identification model after the parameter value is updated is the trained identification model; when the first judgment result is no, trigger the execution of obtaining training data from the training set;
[0146] The using the training recognition model set to perform recognition processing on the pre-processed user modal information set to obtain recognition result information includes:
[0147] S321, using each recognition model in the training recognition model set, processing data other than the user wakefulness value in the modal information of each user in the pre-processed user modal information set to obtain a user wakefulness prediction value of the user by the recognition model;
[0148] S322, calculating the occurrence probabilities of the user sobriety prediction values of all the identification models for the user; determining the user sobriety prediction value corresponding to the maximum occurrence probability as the identification result information of the user;
[0149] Repeat S321 to S322 for all users to obtain identification result information of all users;
[0150] The maximum occurrence probability refers to the calculated occurrence probability of all user awake prediction values, specifically including the occurrence probability of 0 value and the occurrence probability of 1 value, and the prediction value with the largest occurrence probability is determined as the identification result information of the user.
[0151] The parameter value of the identification model is updated by using the difference value, and the updating process includes:
[0152]
[0153] θ←θ+v;
[0154] In the formula, x (i) is the label information of the i-th training data in the training set, is the loss function, v is the parameter update value, θ is the parameter of the identification model, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤1, It means to find the partial derivative of variable θ, f(x (i) ; θ) represents the user sobriety prediction value calculated by the recognition model for the i-th training data of the training set, f(·) is the calculation function corresponding to the recognition model; exp represents the power operation of the constant e; η and α are preset values.
[0155] The loss function may adopt a cross entropy loss function.
[0156] Embodiment 2:
[0157] This embodiment discloses a user modality dynamic identification method based on multi-source information fusion, including:
[0158] The first part mainly combines the out-of-hospital medical environment, selects conventional non-invasive parameters that have lower acquisition costs than EEG, have a wide testing environment, and can deeply reflect the patient's clinical situation. It also collects patient data based on clinical needs and builds a clinical database, which includes the patient's age, gender, BMI, ASA grade, ECG, respiration, blood pressure, pulse, blood oxygen and ventilator parameters.
[0159] The second part filters the data in the data set, eliminates abnormal values and mixed noise signals that appear during the data acquisition process, and eliminates baseline drift, ensuring the comparability of characteristic parameters before and after, and providing a good data basis for the establishment of anesthesia depth identification model. The moment of patient state transition is determined by the electromyographic entropy threshold, and the denoising algorithm is selected. The block diagram of the signal preprocessing part is shown in the figure. Respiratory interference elimination. During anesthesia monitoring, patient breathing is inevitable, and the respiratory effect can be eliminated by
[0160] A rhythmic signal (usually 0Hz~0.8Hz) is added to the processing signal, and a high-pass filter with a cut-off frequency of 0.5Hz can be used to filter out this part of the interference.
[0161] The third part is to calculate the derived parameters: heart rate variability, an indirect indicator, can indirectly determine and quantify the activity of the sympathetic and parasympathetic nervous systems, and can reflect the anesthesia state of surgical patients. The present invention uses time domain, frequency domain, and time-frequency analysis methods to analyze the variability of heart rate, extract the RR interval or instantaneous heart rate data of sinus rhythm within a period of time, and calculate the corresponding standard deviation.
[0162] The fourth part realizes the dynamic parameter calculation of the mined characteristic parameters, unifies the parameters with different resolutions, different effective data volumes, and different sampling lengths in form, and establishes the corresponding machine learning model; it mines the physiological state of the injured hidden in the time series parameters from multiple angles, aspects, and levels.
[0163] The fifth part performs feature dimensionality reduction on the dynamic parameters mined in the previous step. Not all dynamic parameters have significant differences on the depth of anesthesia. Reducing invalid dynamic parameters can reduce the redundant features and noise error factors of parameters, reduce the degree of model overfitting, and select the correct features by deleting unnecessary feature lists from the data set, thereby improving the performance of the model, making the model loss converge faster, and reducing the computational cost.
[0164] The sixth part is a dynamic identification model of anesthesia depth based on multi-source information fusion. By reconstructing the structure of the data, the task of fitting the bispectral index (BIS) that can reflect the depth of anesthesia is established based on the dynamic characteristics of non-invasive parameters after dimensionality reduction. Select a variety of machine learning models, and compare the loss convergence efficiency, loss convergence limit, robustness and accuracy of the model in cross-validation between different models. Use the real-time measured bis in the database as the gold standard, and use K-fold cross-validation to optimize the model parameters and improve the model performance.
[0165] The seventh part is to establish a dynamic structure suitable for a variety of machine learning models. Reconstructing the original structured data structure and adding the concept of time make it easier for machine learning to analyze the changing relationship between the changes in physiological parameters and clinical prognosis in the patient flow data during the training process.
[0166] The eighth part fine-tunes the parameters of the dynamic identification model trained in the clinical database to achieve model migration. The anesthesia depth identification model is roughly adjusted using a large amount of patient data in the database, and the model parameters are fine-tuned without changing the model structure, making the model more adaptable to the local clinical environment, improving the model's robustness, and providing the possibility for the model's implementation.
[0167] This embodiment collects anesthesia depth monitoring data and other related intraoperative monitoring data of 1022 anesthetized patients to establish a clinical anesthesia depth database. The database contains BIS data, as well as patient age, gender, BMI, ASA classification, ECG, respiration, blood pressure, pulse, blood oxygen and ventilator parameters. Heart rate variability parameters are calculated from ECG waveforms, including: time domain parameters, frequency domain parameters, and nonlinear parameters.
[0168] Included: patients aged 18 ≤ ≤ 65 years, ASA grade I-III, undergoing elective surgery, with an estimated operation time of ≥ 2 hours, routine electrocardiogram, blood pressure, pulse, blood oxygen saturation, end-tidal carbon dioxide partial pressure and BIS monitoring, and anesthesia using total intravenous anesthesia; Excluded: patients undergoing cranial neurosurgery, transplantation, and establishing extracorporeal circulation.
[0169] On the one hand, transplant patients and patients with extracorporeal circulation do not have the ability to provide treatment during the emergency rescue process during anesthesia; on the other hand, transplant patients and patients with extracorporeal circulation often have large changes in hemodynamic parameters, so these patients are excluded.
[0170] Then, according to the different resolutions of the parameters in the database, different time windows are formulated to match the parameters at the corresponding BIS moment. The time series data needs to be cleaned before being applied to the model. The box group plot method is used to screen out outliers and the mean interpolation method is used to replace data points. For the gold standard BIS, the missing segments of BIS are first excluded; secondly, the data with SQI lower than 50 during BIS monitoring are excluded; finally, the extracted BIS signal is filtered by the Savitzky-Golay method of data stream smoothing and denoising, and the local polynomial least squares fitting filtering method is used in the time domain to remove noise while ensuring that the shape and width of the signal remain unchanged.
[0171] The machine learning model was constructed using the Lightgbm algorithm. Patients in the database were randomly sampled into 80% of the training set for model training, and 20% of the validation set was used for internal validation. The anesthesia status of the patients was Conscious and unconscious, and the corresponding ratio was C:U. For the specific application of anesthesia depth recognition, the structure of the Lightgbm algorithm was adjusted, and the model output results were Conscious and unconscious. In the case that the number of awakenings of patients during anesthesia surgery was too small and could not be solved by changing the distribution of the data set, the data imbalance problem was solved by changing the machine learning algorithm. The machine learning parameter class_weight was adjusted and set to the ratio of (0:1=C:U). The consistency of the model with the clinical identification process was verified by feature importance analysis and SHAP analysis, which increased the recognition of the method by medical staff to a certain extent. Finally, the anesthesia depth dynamic identification system based on multi-source information fusion outputs the anesthesia depth state.
[0172] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A user modality dynamic identification method based on multi-source information fusion, characterized in that: include: S1, obtain the user modal information set; The user modality information set includes modality information of several users; S2, preprocessing the user modality information set to obtain a preprocessed user modality information set; S3, performing identification processing on the pre-processed user modal information set to obtain identification result information; The identification result information is used to characterize the user's sobriety level.
2. The user modality dynamic identification method based on multi-source information fusion according to claim 1 is characterized in that: The preprocessing of the user modality information set to obtain a preprocessed user modality information set includes: Performing data cleaning processing on each type of data in the user modal information set to obtain a first modal information set; Performing acquisition time alignment processing on each type of data in the first modal information set to obtain a second modal information set; performing high-pass filtering on the respiratory parameters in the second modal information set to obtain updated respiratory parameters; Using the updated breathing parameters, replacing the breathing parameters in the second modal information set to obtain a third modal information set; A heart rate variability value is calculated using the electrocardiogram sequence in the third modality information set, and the heart rate variability value is added to the third modality information set to obtain a preprocessed user modality information set.
3. The user modality dynamic identification method based on multi-source information fusion according to claim 1, characterized in that: The performing identification processing on the pre-processed user modal information set to obtain identification result information includes: S31, using the pre-processed user modal information set, training a preset recognition model set to obtain a training recognition model set; S32, using the training recognition model set, performing recognition processing on the pre-processed user modal information set to obtain recognition result information.
4. The user modality dynamic identification method based on multi-source information fusion as claimed in claim 3, characterized in that: The identification model set includes a first identification model, a second identification model, a third identification model, a fourth identification model and a fifth identification model; The first recognition model includes an encoding network, a quantization network and a decoding network; The encoding network is connected to the quantization network and includes a first processing module, a second processing module, a third processing module and a fourth processing module, which are used to encode the input data to obtain feature image data; The first processing module includes an input module, a first two-dimensional convolution module, a first activation module, a first residual module, and a first normalization module; the first port of the first processing module is the input end of the input module; the second port of the first processing module is the output end of the first normalization module; the first port of the first processing module is used as the input end of the encoding network to receive input data; the second port of the first processing module is connected to the first port of the second processing module; in the first processing module, the input module, the first two-dimensional convolution module, the first activation module, the first residual module, and the first normalization module are connected in sequence; The second processing module includes a second two-dimensional convolution module, a second activation module, and a second normalization module; the first port of the second processing module is the input end of the second two-dimensional convolution module; the second port of the second processing module is the output end of the second normalization module; the second port of the second processing module is connected to the first port of the third processing module; in the second processing module, the second two-dimensional convolution module, the second activation module, and the second normalization module are connected in sequence; The third processing module includes a third two-dimensional convolution module, a third activation module, and a third residual module; the first port of the third processing module is the input end of the third two-dimensional convolution module; the second port of the third processing module is the output end of the third residual module; the second port of the third processing module is connected to the first port of the fourth processing module; in the third processing module, the third two-dimensional convolution module, the third activation module, and the third residual module are connected in sequence; The fourth processing module includes a fourth two-dimensional convolution module, a fourth activation module, and a fourth residual module; the first port of the fourth processing module is the input end of the fourth two-dimensional convolution module; the second port of the fourth processing module is the output end of the fourth residual module; the second port of the fourth processing module is used as the output end of the encoding network to output feature image data; In the fourth processing module, the fourth two-dimensional convolution module, the fourth activation module, and the fourth residual module are connected in sequence; The quantization network is connected to the encoding network and the decoding network, and is used to perform quantization processing on the feature image data to obtain compressed data; The quantization network includes a ninth two-dimensional convolution module, a ninth activation module and a quantization module; In the quantization network, the ninth two-dimensional convolution module, the ninth activation module and the quantization module are connected in sequence; The decoding network is connected to the quantization network, and includes a fifth processing module, a sixth processing module, a seventh processing module and an eighth processing module, which are used to decode the compressed data to obtain a user sobriety prediction value; The fifth processing module includes an input module, a random arrangement module, a fifth two-dimensional convolution module, a fifth activation module, and a fifth residual module; the first port of the fifth processing module is the input end of the input module, and is used to receive the compressed data; the second port of the fifth processing module is the output end of the fifth residual module, and is connected to the first port of the sixth processing module; in the fifth processing module, the input module, the random arrangement module, the fifth two-dimensional convolution module, the fifth activation module, and the fifth residual module are connected in sequence; The sixth processing module includes a sixth two-dimensional convolution module, a sixth activation module, and a sixth residual module; the first port of the sixth processing module is the input end of the sixth two-dimensional convolution module; the second port of the sixth processing module is the output end of the sixth residual module; the second port of the sixth processing module is connected to the first port of the seventh processing module; in the sixth processing module, the sixth two-dimensional convolution module, the sixth activation module, and the sixth residual module are connected in sequence; The seventh processing module includes a seventh two-dimensional convolution module, a seventh activation module, and a seventh normalization module; the first port of the seventh processing module is the input end of the seventh two-dimensional convolution module; the second port of the seventh processing module is the output end of the seventh normalization module; the second port of the seventh processing module is connected to the first port of the eighth processing module; in the seventh processing module, the seventh two-dimensional convolution module, the seventh activation module, and the seventh normalization module are connected in sequence; The eighth processing module includes an eighth two-dimensional convolution module, an eighth activation module, an eighth residual module, and an eighth normalization module; the first port of the eighth processing module is the input end of the eighth two-dimensional convolution module; the second port of the eighth processing module is the output end of the eighth normalization module; the second port of the eighth processing module is the output end of the decoding network, which is used to output the calculated user wakefulness prediction value; in the eighth processing module, the eighth two-dimensional convolution module, the eighth activation module, the eighth residual module, and the eighth normalization module are connected in sequence.
5. The user modality dynamic identification method based on multi-source information fusion as claimed in claim 3, characterized in that: The second recognition model includes a first input module, a first convolution module, a depthwise separable convolution module, a first dimension-raising convolution module, a second dimension-raising convolution module, a third dimension-raising convolution module, a fourth dimension-raising convolution module, a second convolution module, a first pooling module, a third convolution module and a first fully connected module; The input end of the first input module of the second identification model is used to receive input data; The output end of the first input module of the second recognition model is connected to the input end of the first convolution module of the second recognition model; the output end of the first convolution module of the second recognition model is connected to the input end of the depthwise separable convolution module of the second recognition model; the output end of the depthwise separable convolution module of the second recognition model is connected to the input end of the first dimensionally increased convolution module of the second recognition model; the output end of the first dimensionally increased convolution module of the second recognition model is connected to the input end of the second dimensionally increased convolution module of the second recognition model; the output end of the second dimensionally increased convolution module of the second recognition model is connected to the input end of the third dimensionally increased convolution module of the second recognition model. The output end of the third dimensionality-raising convolution module of the second recognition model is connected to the input end of the fourth dimensionality-raising convolution module of the second recognition model; the output end of the fourth dimensionality-raising convolution module of the second recognition model is connected to the input end of the second convolution module of the second recognition model; the output end of the second convolution module of the second recognition model is connected to the input end of the first pooling module of the second recognition model; the output end of the first pooling module of the second recognition model is connected to the input end of the third convolution module of the second recognition model; the output end of the third convolution module of the second recognition model is connected to the input end of the first fully-connected module of the second recognition model; The output end of the first fully connected module of the second identification model is used to output a user sobriety prediction value.
6. The user modality dynamic identification method based on multi-source information fusion as claimed in claim 3, characterized in that: The third recognition model includes a second input module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a second pooling module, a seventh convolution module and a second fully connected module; The input end of the second input module of the third recognition model is used to receive input data; the output end of the second input module of the third recognition model is connected to the input end of the fourth convolution module of the third recognition model; the output end of the fourth convolution module of the third recognition model is connected to the input end of the fifth convolution module of the third recognition model; the output end of the fifth convolution module of the third recognition model is connected to the input end of the sixth convolution module of the third recognition model; the output end of the sixth convolution module of the third recognition model is connected to the input end of the second pooling module of the third recognition model; the output end of the second pooling module of the third recognition model is connected to the input end of the seventh convolution module of the third recognition model; the output end of the seventh convolution module of the third recognition model is connected to the input end of the second fully connected module of the third recognition model; The output end of the second fully connected module of the third identification model is used to output the user wakefulness prediction value.
7. The user modality dynamic identification method based on multi-source information fusion as claimed in claim 3, characterized in that: The fourth recognition model includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a first fusion module, a second fusion module, a third fusion module, a first normalization module, a second normalization module, and a first activation module; the input end of the first convolution module is used to receive input data, and the output end of the first convolution module is respectively connected to the input end of the first fusion module and the input end of the second fusion module; the output end of the first fusion module is connected to the input end of the third convolution module; the output end of the third convolution module is connected to the input end of the first normalization module; the output end of the first normalization module The first end is connected to the input end of the third fusion module; the input end of the second convolution module is used to receive input data, and the output end of the second convolution module is respectively connected to the input end of the second convolution module and the input end of the first fusion module; the output end of the second fusion module is connected to the input end of the fourth convolution module; the output end of the fourth convolution module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the third fusion module; the output end of the third fusion module is connected to the input end of the first activation module; the output end of the first activation module is used to output the user wakefulness prediction value.
8. The user modality dynamic identification method based on multi-source information fusion as claimed in claim 3, characterized in that: The fifth recognition model includes a fifth convolution module, a sixth convolution module, a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a first upsampling module, a second upsampling module, a first multi-head attention module, a second multi-head attention module, a third multi-head attention module, a fourth multi-head attention module, a second activation module, a third activation module, a fourth activation module, a fifth activation module, a sixth activation module, a seventh activation module, a third normalization module, a fourth normalization module, a fifth normalization module, a fourth fusion module and a fifth fusion module; The input end of the fifth convolution module is used to receive input data, and the output end of the fifth convolution module is connected to the input end of the sixth convolution module; the output end of the sixth convolution module is connected to the input end of the first multi-head attention module; the output end of the first multi-head attention module is connected to the input end of the second activation module; the output end of the second activation module is connected to the input end of the second multi-head attention module; the output end of the second multi-head attention module is respectively connected to the input end of the seventh convolution module and the input end of the third activation module; the output end of the seventh convolution module is connected to the third normalization module. The output end of the third normalization module is connected to the input end of the eighth convolution module and the input end of the fifth fusion module respectively; the output end of the eighth convolution module is connected to the input end of the tenth convolution module; the output end of the tenth convolution module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the eleventh convolution module and the input end of the fifth fusion module respectively; the output end of the eleventh convolution module is connected to the input end of the twelfth convolution module; the output end of the third activation module is connected to the input end of the third multi-head attention module ; The output end of the third multi-head attention module is respectively connected to the input end of the fourth fusion module and the input end of the fifth activation module; the output end of the fourth fusion module is connected to the output end of the ninth convolution module; the output end of the ninth convolution module is respectively connected to the input end of the second upsampling module and the input end of the tenth convolution module; the output end of the second upsampling module is connected to the input end of the seventh convolution module; the output end of the fifth activation module is connected to the input end of the first pooling module; the output end of the first pooling module is connected to the input end of the fourth multi-head attention module; the output end of the fourth multi-head attention module is connected to the input end of the sixth activation module; the output end of the sixth activation module is respectively connected to the input end of the first upsampling module and the input end of the twelfth convolution module; the output end of the first upsampling module is connected to the input end of the fourth fusion module; the output end of the twelfth convolution module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fusion module; the output end of the fifth fusion module is connected to the input end of the seventh activation module; the output end of the seventh activation module is used to output the user wakefulness prediction value.
9. The user modality dynamic identification method based on multi-source information fusion as claimed in claim 3, characterized in that: The method of using the pre-processed user modal information set to train a preset recognition model set to obtain a training recognition model set includes: The pre-processed user modal information set is divided and processed according to a preset ratio to obtain a training set and a test set; the training set includes a plurality of training data and corresponding label values; the training data includes the remaining data after the user wakefulness value is removed from the user's modal information; the label value corresponding to the training data is the user wakefulness value in the modal information corresponding to the training data; Using the training set, each recognition model in the preset recognition model set is trained to obtain a trained recognition model; Using all trained recognition models, a set of trained recognition models is constructed.
10. The user modality dynamic identification method based on multi-source information fusion according to claim 9, characterized in that: The training process of the recognition model includes: Initialize the number of training times; Acquire training data from the training set; Inputting the training data into the recognition model to obtain a user sobriety prediction value; Using a loss function to calculate the user sobriety prediction value and the label value corresponding to the training data to obtain a difference value; Using the difference value to update the parameter value of the identification model; Performing an accumulation operation on the training times; Determine whether the training times value is greater than a preset training times threshold value to obtain a first judgment result; when the first judgment result is yes, determine that the identification model after the parameter value is updated is the trained identification model; when the first judgment result is no, trigger the execution of obtaining training data from the training set.
