Establishment system of typing recognition deep learning model based on electroencephalogram and electromyographic signals

Abstract

The construction system of the typing recognition deep learning model based on the electroencephalogram and electromyographic signals comprises a signal acquisition module, a labeling module, a signal reserving module, a signal preprocessing module, a signal fragment division module, a signal processing module, an optimal channel selection module, a feature extraction module, a sample construction module, a model construction module and a model recognition module. The modules are utilized to collect tremor, ankylosing myoelectricity and multi-channel electroencephalogram signals of an identifier in a resting state and tremor, ankylosing myoelectricity, gait and multi-channel electroencephalogram signals in a motion state; the method comprises the following steps: acquiring a plurality of signals, sequentially performing signal retention, signal preprocessing, signal segment division, signal processing, optimal channel selection and feature extraction on the signals to obtain corresponding accurate time-frequency domain features, training and verifying a constructed typing recognition deep learning model by using the accurate time-frequency domain features to obtain an optimal model, and recognizing a person to be recognized by using the optimal model to obtain a recognition result. Whether a person to be recognized has the Parkinson's disease or not is accurately recognized, and specific typing is carried out when the person has the Parkinson's disease.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) signals, electromyographic (EMG) signals and deep learning models, and in particular to a system for constructing a deep learning model for typing and recognition based on EEG signals and electromyographic (EMG) signals. Background Art

[0002] Parkinson's disease is a highly heterogeneous neurodegenerative disease and one of the fastest growing neurological diseases, mainly affecting the middle-aged and elderly population. Parkinson's disease has different subtypes, which vary in clinical manifestations, progression rate, treatment response, and imaging, and have a wide range of motor and non-motor symptoms. Among them, Parkinson's disease can be divided into five types based on motor symptoms, namely tremor-dominant type, rigid-hypokinetic type, gait disorder type, mixed type, and uncertain type.

[0003] Tremor-dominant type: The disease starts with tremor, mainly static tremor of one side of the limb, and the symptoms of muscle rigidity and overall bradykinesia are mild or not obvious.

[0004] Stiff-hypokinetic type: Patients show obvious symptoms of muscle stiffness, muscle rigidity and bradykinesia, with mild tremors or even no tremors for a long time. They are relatively prone to early non-motor symptoms such as depression, apathy, and decreased sense of smell, and cognitive decline progresses relatively quickly.

[0005] Gait disorder type: Patients may quickly experience freezing of gait, difficulty starting, or even unstable posture and falls.

[0006] Mixed type: patients have both limb tremor and muscle rigidity, that is, rigidity-tremor type or rigidity-tremor type, which accounts for the majority of Parkinson's disease.

[0007] Uncertain type: If the patient's symptoms do not belong to the above classifications, it is classified as uncertain type.

[0008] At present, the existing technology for identifying Parkinson's disease has the following defects: 1) The existing Parkinson's disease diagnosis method relies on the doctor's subjective judgment and analysis, which is subjective and requires a lot of time and cost; 2) Although there is a technology for automatically detecting Parkinson's disease, the biological signals currently used are relatively simple and cannot provide sufficient information, resulting in inaccurate final detection results; 3) The existing technology only performs simple preprocessing on the collected biological signals, and does not process the abnormal signal segments of the biological signals, resulting in inaccurate final detection results. Summary of the invention

[0009] In view of the problems and shortcomings of the prior art, the present invention provides a system for constructing a deep learning model for typing recognition based on brain and myoelectric signals.

[0010] The present invention solves the above technical problems through the following technical solutions:

[0011] The present invention provides a construction system of a deep learning model for classification and recognition based on brain and myoelectric signals, which is characterized in that it includes: a signal acquisition module, which is used to collect tremor signals, myoelectric signals and multi-channel EEG signals of multiple identifiers within a set time t in a resting state, and tremor signals, myoelectric signals, gait signals and multi-channel EEG signals within a set time t in a moving state at a preset acquisition frequency;

[0012] a labeling module for labeling each identifier, wherein the multiple identifiers include identifiers without Parkinson's disease labeled as healthy controls and identifiers with labels for each type, including tremor-dominant type, rigid-hypokinetic type, gait disorder type, mixed type, and uncertain type;

[0013] A signal retention module, for each signal collected from each identifier in a resting state and a moving state, removes the signal of the first set time period t1 and the second set time period t2, and only retains the signal of the time period t-t1-t2;

[0014] a signal preprocessing module, for performing signal preprocessing operations on the tremor signal, myoelectric signal and multi-channel EEG signal retained in the resting state of each identifier, and performing signal preprocessing operations on the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal retained in the moving state;

[0015] a signal segment division module, for performing a signal segment division operation on the pre-processed tremor signal, myoelectric signal and multi-channel EEG signal of each identifier in a resting state by using the signal of each third set time period t3 as a signal segment, and performing a signal segment division operation on the pre-processed tremor signal, myoelectric signal, gait signal and multi-channel EEG signal of each identifier in a moving state by using the signal of a complete gait as a signal segment, wherein the time period t-t1-t2 is a multiple of t3;

[0016] a signal processing module, for analyzing the tremor signal segments, myoelectric signal segments and multi-channel EEG signal segments of any third set time period t3 of each identifier, and when there is an abnormality in the tremor signal segment, myoelectric signal segment or any channel EEG signal segment, the signal segments of the third set time period t3 are eliminated, and the tremor signal segments left by each identifier are spliced ​​to form a normal tremor signal, the myoelectric signal segments left are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel left are spliced ​​to form a normal EEG signal of the channel;

[0017] Analyze the tremor signal segments, myoelectric signal segments, gait signal segments and multi-channel EEG signal segments of any complete gait of each identifier. When there is an abnormality in the tremor signal segment, myoelectric signal segment, gait signal segment or any channel EEG signal segment, eliminate the signal segments of the complete gait, splice the tremor signal segments left by each identifier to form a normal tremor signal, splice the myoelectric signal segments left to form a normal myoelectric signal, splice the gait signal segments left to form a normal gait signal, and splice the EEG signal segments of any channel left to form a normal EEG signal of the channel;

[0018] a feature extraction module, for extracting features of normal tremor signals, normal myoelectric signals and multi-channel normal EEG signals of each identifier in a resting state, respectively, to obtain tremor time-frequency features, myoelectric time-frequency features and multi-channel EEG time-frequency features in a resting state; for extracting features of normal tremor signals, normal myoelectric signals, normal gait signals and multi-channel normal EEG signals of each identifier in a moving state, respectively, to obtain tremor time-frequency features, myoelectric time-frequency features, gait time-frequency features and multi-channel EEG time-frequency features in a moving state;

[0019] A sample construction module is used to construct a feature set, including the tremor time-frequency features, myoelectric time-frequency features and multi-channel EEG time-frequency features of each identifier in a resting state, the tremor time-frequency features, myoelectric time-frequency features, gait time-frequency features and multi-channel EEG time-frequency features in a moving state, and the corresponding annotation information. The feature set is divided into a sample training set and a sample verification set in a certain proportion;

[0020] A model building module is used to use a convolutional neural network to build a deep learning model for classification and recognition, use a sample training set to train the deep learning model for classification and recognition, use a sample verification set to verify the deep learning model for classification and recognition, continuously adjust the model hyperparameters until the optimal hyperparameters are obtained, and use the optimal hyperparameters to build an optimal deep learning model for classification and recognition of Parkinson's disease.

[0021] The positive and progressive effects of the present invention are:

[0022] The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals designed by the present invention does not rely on the subjective judgment and analysis of doctors. It not only collects tremor signals, myoelectric signals and multi-channel EEG signals of the person being identified in a resting state, but also collects tremor signals, myoelectric signals, gait signals and multi-channel EEG signals of the person being identified in a moving state. The collected biological signals are sufficient to provide sufficient signal information, thereby improving the accuracy of the final typing recognition result. The present invention sequentially performs signal retention, signal preprocessing, signal segmentation, signal processing, optimal channel selection and feature extraction on the collected biological signals, thereby obtaining accurate time-frequency domain features of the biological signals, and uses the accurate time-frequency domain features of the biological signals to train and verify the constructed typing recognition deep learning model, thereby obtaining the optimal typing recognition deep learning model, and using the optimal typing recognition deep learning model to identify the person to be identified, thereby accurately identifying whether the person to be identified has Parkinson's disease and the specific typing of Parkinson's disease when the person has it. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a structural block diagram of a system for constructing a deep learning model for typing recognition based on brain and myoelectric signals according to a preferred embodiment of the present invention.

[0024] Figure 2 This is an architecture diagram of a deep learning model for typing recognition according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] like Figure 1 As shown, this embodiment provides a system for constructing a deep learning model for classification and identification based on brain and myoelectric signals, which is used for classification and identification of Parkinson's disease, including a signal acquisition module 1, a labeling module 2, a signal retention module 3, a signal preprocessing module 4, a signal segmentation module 5, a signal processing module 6, an optimal channel selection module 7, a feature extraction module 8, a sample construction module 9, a model construction module 10 and a model identification module 11.

[0027] The signal acquisition module 1 is used to collect tremor signals, myoelectric signals and multi-channel EEG signals of multiple identifiers in a resting state within a set time t, as well as tremor signals, myoelectric signals, gait signals and multi-channel EEG signals in a moving state within a set time t at a preset acquisition frequency.

[0028] In this embodiment, multiple signals of each identifier in a resting state and multiple signals in a moving state are collected. The large number of collected signals can provide sufficient signal information, which is a guarantee for the accuracy of the final recognition result.

[0029] The labeling module 2 is used to label each identifier, and the multiple identifiers include identifiers without Parkinson's disease who are labeled as healthy controls and identifiers with all types labeled, including tremor-dominant type, rigid-hypokinetic type, gait disorder type, mixed type and uncertain type. In other words, these identifiers include healthy identifiers, tremor-dominant Parkinson's disease identifiers, rigid-hypokinetic Parkinson's disease identifiers, gait disorder type Parkinson's disease identifiers, mixed type Parkinson's disease identifiers and uncertain type Parkinson's disease identifiers.

[0030] The signal retention module 3 is used to remove the signals of the first set time period t1 and the second set time period t2 for each signal collected by each identifier in the resting state and the moving state, and only retain the signals of the time period t-t1-t2.

[0031] For example, for each identifier, 180 seconds of signals (tremor signals, myoelectric signals and multi-channel EEG signals) are collected in a resting state, and the signals of the first 20 seconds and the last 20 seconds are removed, leaving only the signals of the middle 140 seconds.

[0032] The signal preprocessing module 4 is used to perform signal preprocessing operations on the tremor signal, myoelectric signal and multi-channel EEG signal retained by each identifier in the resting state. Specifically, each signal is subjected to bandpass filtering to retain the effective frequency band of the signal, notch filtering to remove power frequency interference, independent component analysis to remove impurity signals in the signal, and signal enhancement.

[0033] The signal preprocessing module 4 is also used to perform signal preprocessing operations on the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal retained in each identifier's motion state. Specifically, each signal is sequentially subjected to bandpass filtering to retain the effective frequency band of the signal, notch filtering to remove power frequency interference, independent component analysis to remove impurity signals in the signal, and signal enhancement.

[0034] The signal segmentation module 5 is used to use the signal of every third set time period t3 (such as every 20 seconds) as a signal segment to perform signal segmentation operations on the pre-processed tremor signal, myoelectric signal and multi-channel EEG signal of each identifier in a resting state; and use the signal of a complete gait as a signal segment to perform signal segmentation operations on the pre-processed tremor signal, myoelectric signal, gait signal and multi-channel EEG signal of each identifier in a moving state, wherein the time period t-t1-t2 is a multiple of t3.

[0035] In this embodiment, if the signals at the head end and the tail end of the gait signal do not constitute a complete gait, they are ignored, and the corresponding tremor signal, myoelectric signal and multi-channel EEG signal are also ignored.

[0036] The signal processing module 6 is used to analyze the tremor signal segment, myoelectric signal segment and multi-channel EEG signal segment of any third set time period t3 of each identifier. When there is no abnormality in the tremor signal segment, myoelectric signal segment or any channel EEG signal segment of the third set time period t3 (that is, all are signals without abnormalities), the signal segment of the third set time period t3 is retained, and the third set time period t3 is used as the retained time period; otherwise, the signal segment of the third set time period t3 is used as a pending signal segment, and the third set time period t3 is used as a pending time period, so as to analyze whether each third set time period t3 is a retained time period or a pending signal segment; the number of reserved time periods is counted, and when the number reaches the first set number, it indicates that the number of signal segments is sufficient, and the signal segments of all pending time periods are discarded, and the tremor signal segments in the reserved time period are spliced ​​according to the time sequence to form a normal tremor signal, the myoelectric signal segments are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel are spliced ​​to form a normal EEG signal of the channel.

[0037] When the number of these signal segments does not reach the first set number, i signal segments (any i of tremor signal segments, myoelectric signal segments or channel EEG signal segments) are selected from the signal segments of all pending time periods with abnormalities, and the i abnormal signal segments in each selected pending time period are corrected. After correction, the selected pending time period is used as the reserved time period, and it is determined whether the number of reserved time periods reaches the first set number. If so, the tremor signal segments in the reserved time period are spliced ​​in time sequence to form a normal tremor signal, the myoelectric signal segments are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel are spliced ​​to form a normal EEG signal of the channel. Otherwise, i=i+1 continues to be corrected, and i is sequentially taken from 1 to n+2, where n is the number of channels of the EEG signal, until the number of reserved time periods reaches the first set number.

[0038] Furthermore, average value interpolation correction is performed on i abnormal signal segments in each selected pending time period: for any abnormal signal segment, when the signal of a certain acquisition point is an abnormal signal, and the signals of the acquisition points before and after the acquisition point are all non-abnormal signals, the abnormal signal of the acquisition point is replaced by the average value of the non-abnormal signals of the acquisition points before and after the acquisition point; when the signals of multiple consecutive acquisition points are all abnormal signals, and the signals of the acquisition points before and after the multiple consecutive acquisition points are all non-abnormal signals, the abnormal signals of the multiple consecutive acquisition points are replaced by the average value of the non-abnormal signals of the acquisition points before and after the multiple consecutive acquisition points.

[0039] The signal processing module 6 is also used to analyze the tremor signal segment, myoelectric signal segment, gait signal segment and multi-channel EEG signal segment of any complete gait of each identifier. When there is no abnormality in the tremor signal segment, myoelectric signal segment, gait signal segment or any channel EEG signal segment of the complete gait (that is, all are signals without abnormalities), the signal segment of the complete gait is retained and the complete gait is used as the retained gait; otherwise, the signal segment of the complete gait is used as a pending signal segment and the complete gait is used as a pending gait, so as to analyze whether each complete gait is a retained gait or a pending gait; the number of retained gaits is counted, and when the number reaches a second set number, it indicates that the number of signal segments is sufficient, and the signal segments of all pending gaits are discarded, and the tremor signal segments in the retained gait are spliced ​​in time sequence to form a normal tremor signal, the myoelectric signal segments are spliced ​​to form a normal myoelectric signal, the gait signal segments are spliced ​​to form a normal gait signal, and the EEG signal segments of any channel are spliced ​​to form a normal EEG signal of the channel.

[0040] When the number of signal segments does not reach the second set number, i signal segments (any i of tremor signal segments, myoelectric signal segments, gait signal segments or channel EEG signal segments) are selected from the signal segments of all pending gaits with abnormalities, and the i abnormal signal segments in each selected pending gait are corrected. After correction, the selected pending gait is used as the reserved gait, and it is judged whether the number of reserved gaits reaches the second set number. If so, the tremor signal segments in the reserved gait are spliced ​​in time sequence to form a normal tremor signal, the myoelectric signal segments are spliced ​​to form a normal myoelectric signal, the gait signal segments are spliced ​​to form a normal gait signal, and the EEG signal segments of any channel are spliced ​​to form a normal EEG signal of the channel; otherwise, i=i+1 continues to be corrected, and i is sequentially taken from 1 to n+3, where n is the number of channels of the EEG signal, until the number of reserved gaits reaches the second set number.

[0041] Furthermore, average value interpolation correction is performed on i abnormal signal segments in each selected pending gait. For any abnormal signal segment, when the signal of a certain collection point is an abnormal signal, and the signals of the collection points before and after the collection point are all non-abnormal signals, the abnormal signal of the collection point is replaced by the average value of the non-abnormal signals of the collection points before and after the collection point. When the signals of multiple consecutive collection points are all abnormal signals, and the signals of the collection points before and after the multiple consecutive collection points are all non-abnormal signals, the abnormal signals of the multiple consecutive collection points are replaced by the average value of the non-abnormal signals of the collection points before and after the multiple consecutive collection points.

[0042] The optimal channel selection module 7 is used to select a normal EEG signal of an optimal channel from the normal EEG signals of multiple channels of each identifier in a resting state: calculate the correlation coefficient between the normal EEG signal of any channel and the normal EEG signals of other channels, calculate the average correlation coefficient corresponding to any channel, and select the normal EEG signal of the channel corresponding to the largest average correlation coefficient as the normal EEG signal of the optimal channel.

[0043] For example: there are 4 channels of EEG signals. The correlation coefficients between the normal EEG signal of the first channel and the normal EEG signals of the other three channels are calculated to obtain the three coefficients corresponding to the first channel; the correlation coefficients between the normal EEG signal of the second channel and the normal EEG signals of the other three channels are calculated to obtain the three coefficients corresponding to the second channel; the correlation coefficients between the normal EEG signal of the third channel and the normal EEG signals of the other three channels are calculated to obtain the three coefficients corresponding to the third channel; the correlation coefficients between the normal EEG signal of the fourth channel and the normal EEG signals of the other three channels are calculated to obtain the three coefficients corresponding to the fourth channel; the average value of the three correlation coefficients corresponding to the first channel is calculated, the average value of the three correlation coefficients corresponding to the second channel is calculated, the average value of the three correlation coefficients corresponding to the third channel is calculated, and the average value of the three correlation coefficients corresponding to the fourth channel is calculated, and the normal EEG signal of the channel corresponding to the largest average value of the correlation coefficient is selected as the normal EEG signal of the optimal channel.

[0044] The optimal channel selection module 7 is also used to select a normal EEG signal of an optimal channel from the normal EEG signals of multiple channels under each identifier's motion state: calculate the correlation coefficient between the normal EEG signal of any channel and the normal EEG signals of other channels, calculate the average correlation coefficient corresponding to any channel, and select the normal EEG signal of the channel corresponding to the largest average correlation coefficient as the normal EEG signal of the optimal channel.

[0045] The feature extraction module 8 is used to perform feature extraction on the normal tremor signal, normal myoelectric signal and normal EEG signal of the optimal channel in the resting state of each identifier, so as to obtain the tremor time-frequency characteristics, myoelectric time-frequency characteristics and EEG time-frequency characteristics in the resting state; and perform feature extraction on the normal tremor signal, normal myoelectric signal, normal gait signal and normal EEG signal of the optimal channel in the moving state, so as to obtain the tremor time-frequency characteristics, myoelectric time-frequency characteristics, gait time-frequency characteristics and EEG time-frequency characteristics in the resting state.

[0046] The sample construction module 9 is used to construct a feature set, including the tremor time-frequency features, myoelectric time-frequency features and multi-channel EEG time-frequency features of each identifier in a resting state, the tremor time-frequency features, myoelectric time-frequency features, gait time-frequency features and multi-channel EEG time-frequency features in a moving state, and the corresponding annotation information. The feature set is divided into a sample training set and a sample verification set in a certain ratio (such as 8:2).

[0047] The model construction module 10 is used to use a convolutional neural network to construct a typing recognition deep learning model, use a sample training set to train the typing recognition deep learning model, use a sample verification set to verify the typing recognition deep learning model, continuously adjust the model hyperparameters until the optimal hyperparameters are obtained, and use the optimal hyperparameters to construct an optimal typing recognition deep learning model. The typing recognition deep learning model corresponding to the optimal hyperparameters is the optimal typing recognition deep learning model.

[0048] Among them, see Figure 2 The deep learning model for typing recognition consists of multiple convolutional layers, a pooling layer and a fully connected layer. Each convolutional layer contains a batch normalization layer before and an activation function layer after it. The activation function of the activation function layer uses the ReLU function, the pooling layer uses the average pooling layer, and the fully connected layer uses the Softmax activation function to classify the input feature values. Each convolutional layer uses a 3*3 convolution kernel.

[0049] The model recognition module 11 is used to call the signal acquisition module 1, and the signal acquisition module 1 is used to collect the tremor signal, myoelectric signal and multi-channel EEG signal of the person to be identified within a set time t in a resting state, as well as the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal of the person to be identified within a set time t in a moving state at a preset acquisition frequency.

[0050] The signal retention module 3 is called, and the signal retention module 3 is used to remove the signals of the first set time period t1 and the second set time period t2 for each signal collected in the resting state and the moving state, and only retain the signals of the time period t-t1-t2.

[0051] The signal preprocessing module 4 is called, and the signal preprocessing module 4 is used to perform signal preprocessing operations on the tremor signal, myoelectric signal and multi-channel EEG signal retained in the resting state, and perform signal preprocessing operations on the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal retained in the moving state.

[0052] The signal segment division module 5 is called. The signal segment division module 5 is used to use the signal of each third set time period t3 as a signal segment to perform signal segment division operations on the tremor signal, myoelectric signal and multi-channel EEG signal preprocessed in the resting state of the identifier, and use the signal of a complete gait as a signal segment to perform signal segment division operations on the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal preprocessed in the moving state of the identifier.

[0053] Calling the signal processing module 6, the signal processing module 6 is used to analyze the tremor signal segments, myoelectric signal segments and multi-channel EEG signal segments of any third set time period t3. When there is an abnormality in the tremor signal segment, myoelectric signal segment or any channel EEG signal segment, the signal segments of the third set time period t3 are eliminated, and the tremor signal segments left by each identifier are spliced ​​to form a normal tremor signal, the myoelectric signal segments left are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel left are spliced ​​to form a normal EEG signal of the channel; for any The tremor signal fragments, myoelectric signal fragments, gait signal fragments and multi-channel EEG signal fragments of a complete gait are analyzed. When there are abnormalities in the tremor signal fragments, myoelectric signal fragments, gait signal fragments or any channel EEG signal fragments, the signal fragments of the complete gait are eliminated, and the tremor signal fragments left by each identifier are spliced ​​to form a normal tremor signal, the myoelectric signal fragments left are spliced ​​to form a normal myoelectric signal, the gait signal fragments left are spliced ​​to form a normal gait signal, and the EEG signal fragments of any channel left are spliced ​​to form a normal EEG signal of that channel.

[0054] The optimal channel selection module 7 is called, and the optimal channel selection module 7 is used to select a normal EEG signal of an optimal channel from the normal EEG signals of multiple channels in a resting state of the person to be identified, and to select a normal EEG signal of an optimal channel from the normal EEG signals of multiple channels in a moving state of the person to be identified.

[0055] The feature extraction module 8 is called, and the feature extraction module 8 is used to perform feature extraction on the normal tremor signal, normal myoelectric signal and multi-channel normal EEG signal of the person to be identified in the resting state, respectively, to obtain the tremor time-frequency features, myoelectric time-frequency features and EEG time-frequency features of the optimal channel in the resting state; the normal tremor signal, normal myoelectric signal, normal gait signal and multi-channel normal EEG signal of the person to be identified in the moving state are respectively extracted to obtain the tremor time-frequency features, myoelectric time-frequency features, gait time-frequency features and EEG time-frequency features of the optimal channel in the moving state, to form a feature set of the person to be identified.

[0056] The model identification module 11 is used to input the feature set of the person to be identified into the optimal classification identification deep learning model for learning and identification, so as to identify whether the person to be identified has Parkinson's disease and the specific classification of Parkinson's disease if he has it.

[0057] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that these are only examples, and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A system for constructing a deep learning model for typing recognition based on brain and myoelectric signals, characterized in that: include: a signal acquisition module, for acquiring tremor signals, myoelectric signals and multi-channel EEG signals of multiple identifiers in a resting state within a set time t, and tremor signals, myoelectric signals, gait signals and multi-channel EEG signals of multiple identifiers in a moving state within a set time t at a preset acquisition frequency; a labeling module for labeling each identifier, wherein the multiple identifiers include identifiers without Parkinson's disease labeled as healthy controls and identifiers with labels for each type, including tremor-dominant type, rigid-hypokinetic type, gait disorder type, mixed type, and uncertain type; A signal retention module, for each signal collected from each identifier in a resting state and a moving state, removes the signal of the first set time period t1 and the second set time period t2, and only retains the signal of the time period t-t1-t2; a signal preprocessing module, for performing signal preprocessing operations on the tremor signal, myoelectric signal and multi-channel EEG signal retained in the resting state of each identifier, and performing signal preprocessing operations on the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal retained in the moving state; a signal segment division module, for performing a signal segment division operation on the pre-processed tremor signal, myoelectric signal and multi-channel EEG signal of each identifier in a resting state by using the signal of each third set time period t3 as a signal segment, and performing a signal segment division operation on the pre-processed tremor signal, myoelectric signal, gait signal and multi-channel EEG signal of each identifier in a moving state by using the signal of a complete gait as a signal segment, wherein the time period t-t1-t2 is a multiple of t3; a signal processing module, for analyzing the tremor signal segments, myoelectric signal segments and multi-channel EEG signal segments of any third set time period t3 of each identifier, and when there is an abnormality in the tremor signal segment, myoelectric signal segment or any channel EEG signal segment, the signal segments of the third set time period t3 are eliminated, and the tremor signal segments left by each identifier are spliced ​​to form a normal tremor signal, the myoelectric signal segments left are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel left are spliced ​​to form a normal EEG signal of the channel; Analyze the tremor signal segments, myoelectric signal segments, gait signal segments and multi-channel EEG signal segments of any complete gait of each identifier. When there is an abnormality in the tremor signal segment, myoelectric signal segment, gait signal segment or any channel EEG signal segment, eliminate the signal segments of the complete gait, splice the tremor signal segments left by each identifier to form a normal tremor signal, splice the myoelectric signal segments left to form a normal myoelectric signal, splice the gait signal segments left to form a normal gait signal, and splice the EEG signal segments of any channel left to form a normal EEG signal of the channel; a feature extraction module, for extracting features of normal tremor signals, normal myoelectric signals and multi-channel normal EEG signals of each identifier in a resting state, respectively, to obtain time-frequency features of tremor, time-frequency features of myoelectric signals and multi-channel EEG time-frequency features in a resting state; The features of normal tremor signal, normal myoelectric signal, normal gait signal and multi-channel normal EEG signal of each identifier in motion are extracted respectively to obtain the time-frequency features of tremor, myoelectric signal, gait and multi-channel EEG in motion; A sample construction module is used to construct a feature set, including the tremor time-frequency features, myoelectric time-frequency features and multi-channel EEG time-frequency features of each identifier in a resting state, the tremor time-frequency features, myoelectric time-frequency features, gait time-frequency features and multi-channel EEG time-frequency features in a moving state, and the corresponding annotation information. The feature set is divided into a sample training set and a sample verification set in a certain proportion; A model building module is used to use a convolutional neural network to build a deep learning model for classification and recognition, use a sample training set to train the deep learning model for classification and recognition, use a sample verification set to verify the deep learning model for classification and recognition, continuously adjust the model hyperparameters until the optimal hyperparameters are obtained, and use the optimal hyperparameters to build an optimal deep learning model for classification and recognition of Parkinson's disease.

2. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as described in claim 1, characterized in that: The signal processing module is used to analyze the tremor signal segment, myoelectric signal segment and multi-channel EEG signal segment of any third set time period t3 of each identifier, and when there is no abnormality in the tremor signal segment, myoelectric signal segment or any channel EEG signal segment of the third set time period t3, retain the signal segment of the third set time period t3, and the third set time period t3 is used as the reserved time period; otherwise, the third set time period t3 is used as the pending time period, and the number of reserved time periods is counted, and when the number reaches the first set number, all signal segments of the pending time period are discarded, and the tremor signal segments in the reserved time period are spliced ​​to form a normal tremor signal, the myoelectric signal segments are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel are spliced ​​to form a normal EEG signal of the channel according to the time sequence; When the number of abnormalities does not reach the first set number, i signal segments with abnormalities are selected from the signal segments of all pending time periods, and the i abnormal signal segments in each selected pending time period are corrected. After correction, the selected pending time period is used as the reserved time period, and it is judged whether the number of reserved time periods reaches the first set number. If so, the tremor signal segments in the reserved time period are spliced ​​in time sequence to form a normal tremor signal, the myoelectric signal segments are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel are spliced ​​to form a normal EEG signal of the channel. Otherwise, i=i+1 continues to be corrected, i is successively 1 to n+2, n is the number of channels, until the number of reserved time periods reaches the first set number.

3. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as described in claim 2, characterized in that: The signal processing module is used to perform average value interpolation correction on i abnormal signal segments in each selected pending time period. For any abnormal signal segment, when the signal of a certain acquisition point is an abnormal signal, and the signals of the acquisition points before and after the acquisition point are all non-abnormal signals, the abnormal signal of the acquisition point is replaced by the average value of the non-abnormal signals of the acquisition points before and after the acquisition point. When the signals of multiple consecutive acquisition points are all abnormal signals, and the signals of the acquisition points before and after the multiple consecutive acquisition points are all non-abnormal signals, the abnormal signals of the multiple consecutive acquisition points are replaced by the average value of the non-abnormal signals of the acquisition points before and after the multiple consecutive acquisition points.

4. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as described in claim 1, characterized in that: The signal processing module is used to analyze the tremor signal segment, myoelectric signal segment, gait signal segment and multi-channel EEG signal segment of any complete gait of each identifier, and when there is no abnormality in the tremor signal segment, myoelectric signal segment, gait signal segment or any channel EEG signal segment of the complete gait, retain the signal segment of the complete gait and use the complete gait as the reserved gait; otherwise, the complete gait is used as the pending gait, and the number of reserved gaits is counted. When the number reaches a second set number, all signal segments of the pending gaits are discarded, and the tremor signal segment in the reserved gait is spliced ​​to form a normal tremor signal, the myoelectric signal segment is spliced ​​to form a normal myoelectric signal, the gait signal segment is spliced ​​to form a normal gait signal, and the EEG signal segment of any channel is spliced ​​to form a normal EEG signal of the channel according to the time sequence; When the number of signal segments of all pending gaits does not reach the second set number, i signal segments with abnormalities are selected from the signal segments of all pending gaits, and the i abnormal signal segments in each selected pending gait are corrected. After correction, the selected pending gait is used as the reserved gait, and it is judged whether the number of reserved gaits reaches the second set number. If so, the tremor signal segments in the reserved gait are spliced ​​in time sequence to form a normal tremor signal, the myoelectric signal segments are spliced ​​to form a normal myoelectric signal, the gait signal segments are spliced ​​to form a normal gait signal, and the EEG signal segments of any channel are spliced ​​to form a normal EEG signal of the channel; otherwise, i=i+1 continues to be corrected, i is successively 1 to n+3, n is the number of channels, until the number of reserved gaits reaches the second set number.

5. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as described in claim 4, characterized in that: The signal processing module is used to perform average value interpolation correction on i abnormal signal segments in each selected pending gait. For any abnormal signal segment, when the signal of a certain collection point is an abnormal signal, and the signals of the collection points before and after the collection point are all non-abnormal signals, the abnormal signal of the collection point is replaced by the average value of the non-abnormal signals of the collection points before and after the collection point. When the signals of multiple consecutive collection points are all abnormal signals, and the signals of the collection points before and after the multiple consecutive collection points are all non-abnormal signals, the abnormal signals of the multiple consecutive collection points are replaced by the average value of the non-abnormal signals of the collection points before and after the multiple consecutive collection points.

6. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as described in claim 1, characterized in that: The system also includes an optimal channel selection module; The optimal channel selection module is used to select a normal EEG signal of an optimal channel from the normal EEG signals of multiple channels in a resting state of each identifier, and to select a normal EEG signal of an optimal channel from the normal EEG signals of multiple channels in a moving state; The feature extraction module is used to extract features of normal tremor signals, normal myoelectric signals and normal EEG signals of the optimal channel of each identifier in a resting state, respectively, to obtain the time-frequency features of tremor, myoelectric time-frequency features and EEG time-frequency features in a resting state; Feature extraction was performed on normal tremor signals, normal myoelectric signals, normal gait signals and normal EEG signals of the optimal channel in motion state, respectively, to obtain the time-frequency features of tremor, myoelectric time-frequency features, gait time-frequency features and EEG time-frequency features in resting state.

7. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as described in claim 6, characterized in that: The optimal channel selection module is used to calculate the correlation coefficient between the normal EEG signal of any channel and the normal EEG signals of other channels for the normal EEG signals of multiple channels in a resting state, calculate the average value of the correlation coefficient corresponding to any channel, and select the normal EEG signal of the channel corresponding to the maximum average value of the correlation coefficient as the normal EEG signal of the optimal channel; For normal EEG signals of multiple channels in motion, the correlation coefficient between the normal EEG signal of any channel and the normal EEG signals of other channels is calculated, the average correlation coefficient corresponding to any channel is calculated, and the normal EEG signal of the channel with the largest average correlation coefficient is selected as the normal EEG signal of the optimal channel.

8. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as claimed in claim 1, characterized in that: The typing recognition deep learning model consists of multiple convolutional layers, a pooling layer and a fully connected layer. Each convolutional layer also contains a batch normalization layer before and an activation function layer after. The activation function of the activation function layer adopts the ReLU function, the pooling layer adopts the average pooling layer, and the fully connected layer adopts the Softmax activation function to classify the input feature values. Each convolutional layer adopts a 3*3 convolution kernel.

9. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as claimed in claim 1, characterized in that: The signal preprocessing module is used to perform bandpass filtering on the tremor signal, myoelectric signal and multi-channel EEG signal retained by each identifier in a resting state to retain the effective frequency band of the signal, notch filtering to remove power frequency interference, remove impurity signals in the signal by independent component analysis, and enhance the signal; For the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal retained in each identifier's motion state, each signal is band-pass filtered to retain the effective frequency band of the signal, notch filtered to remove power frequency interference, independent component analysis is used to remove impurity signals in the signal, and the signal is enhanced.

10. The system for constructing a deep learning model for typing recognition based on brain and myoelectric signals as claimed in claim 1, characterized in that: The system also includes a model identification module; The model recognition module is used to call the signal acquisition module, and the signal acquisition module is used to collect the tremor signal, myoelectric signal and multi-channel EEG signal of the person to be identified in a resting state within a set time t, and the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal of the person to be identified in a moving state within a set time t at a preset acquisition frequency; Calling a signal retention module, the signal retention module is used to remove the signals of the first set time period t1 and the second set time period t2 for each signal collected in the resting state and the moving state, and only retain the signals of the time period t-t1-t2; Calling a signal preprocessing module, the signal preprocessing module is used to perform signal preprocessing operations on the tremor signal, myoelectric signal and multi-channel EEG signal retained in the resting state, and perform signal preprocessing operations on the tremor signal, myoelectric signal, gait signal and multi-channel EEG signal retained in the moving state; Calling a signal segment division module, the signal segment division module is used to use the signal of each third set time period t3 as a signal segment to perform a signal segment division operation on the pre-processed tremor signal, myoelectric signal and multi-channel electroencephalogram signal of the subject in a resting state, and use the signal of a complete gait as a signal segment to perform a signal segment division operation on the pre-processed tremor signal, myoelectric signal, gait signal and multi-channel electroencephalogram signal of the subject in a moving state; Calling a signal processing module, the signal processing module is used to analyze the tremor signal segments, myoelectric signal segments and multi-channel EEG signal segments of any third set time period t3, when there is an abnormality in the tremor signal segment, myoelectric signal segment or any channel EEG signal segment, the signal segments of the third set time period t3 are eliminated, the tremor signal segments left by each identifier are spliced ​​to form a normal tremor signal, the myoelectric signal segments left are spliced ​​to form a normal myoelectric signal, and the EEG signal segments of any channel left are spliced ​​to form a normal EEG signal of the channel; Analyze the tremor signal segments, myoelectric signal segments, gait signal segments and multi-channel EEG signal segments of any complete gait. When there are abnormalities in the tremor signal segments, myoelectric signal segments, gait signal segments or any channel EEG signal segments, eliminate the signal segments of the complete gait, and splice the tremor signal segments left by each identifier to form a normal tremor signal, splice the myoelectric signal segments left to form a normal myoelectric signal, splice the gait signal segments left to form a normal gait signal, and splice the EEG signal segments of any channel left to form a normal EEG signal of the channel; Calling a feature extraction module, the feature extraction module is used to extract features of a normal tremor signal, a normal myoelectric signal and a multi-channel normal EEG signal of the subject in a resting state, respectively, to obtain a tremor time-frequency feature, a myoelectric time-frequency feature and a multi-channel EEG time-frequency feature in a resting state; The normal tremor signal, normal myoelectric signal, normal gait signal and multi-channel normal EEG signal of the subject to be identified in motion are subjected to feature extraction respectively, and the time-frequency features of tremor, myoelectric signal, gait and multi-channel EEG in motion are obtained to form a feature set of the subject to be identified; The model recognition module is used to input the feature set of the person to be identified into the optimal classification recognition deep learning model for learning and recognition, so as to identify whether the person to be identified has Parkinson's disease and the specific classification of Parkinson's disease if so.

Images