Control method and device, training method, electronic device, and storage medium

Abstract

The application discloses a control method and a training method based on an electroencephalogram signal, a control device, electronic equipment and a storage medium. The control method comprises the following steps: acquiring an electroencephalogram signal of a user controlling a target object, processing the electroencephalogram signal by using a neural network model to obtain control information, wherein the neural network model is trained by using an electroencephalogram training signal and an electromyogram training signal, and controlling the target object according to the control information. In this way, the performance of the overall brain-computer interface is improved, the artificial limb can flexibly and freely complete the instructed action of the brain, and the residual limb patient can also perform daily life communication and cooperation, and recover part of the self-care and work communication ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology and medicine, in particular to a control method based on electroencephalogram signals, a control device, a neural network model training method, an electronic device and a nonvolatile computer readable storage medium. BACKGROUND

[0002] Most disabled patients are caused by accidental traffic accidents, natural disasters, various diseases, etc. The paralysis or loss of limbs leads to a great impact on daily life. With the continuous progress of science and technology, using brain-computer interface to recognize limb movement and using it to assist the control of artificial limbs for patients with residual limbs has become a current research hotspot.

[0003] Brain-computer interface technology is to directly read human thinking activities through electroencephalogram signals and decode them into limb control information. However, due to the low signal-to-noise ratio of electroencephalogram signals, the complexity and difficulty of measurement and analysis are relatively high, which makes it too difficult to directly read human thinking activities through electroencephalogram and decode them into limb control information. SUMMARY

[0004] Therefore, the present application provides a control method based on electroencephalogram signals, a control device, a neural network model training method, a training device, an electronic device and a nonvolatile computer readable storage medium.

[0005] The control method provided by the embodiments of the present application comprises:

[0006] Obtaining an electroencephalogram signal of a user controlling a target object;

[0007] Processing the electroencephalogram signal using a neural network model to obtain control information, wherein the neural network model is trained by an electroencephalogram training signal and an electromyogram training signal;

[0008] Controlling the target object according to the control information.

[0009] In some embodiments, the processing the electroencephalogram signal using a neural network model to obtain control information comprises:

[0010] Filtering the electroencephalogram signal using a fourth-order Butterworth band-pass filter with a preset frequency range to obtain a preprocessed electroencephalogram signal;

[0011] Processing the preprocessed electroencephalogram signal using the neural network model to obtain the control information.

[0012] In some embodiments, the neural network model comprises a first sub-network and a second sub-network connected in series, and the processing the electroencephalogram signal using a neural network model to obtain control information comprises:

[0013] obtaining muscle activity representation by processing the electroencephalogram signal through the first sub-network, the first sub-network being trained by the electroencephalogram training signal and the electromyogram training signal;

[0014] obtaining the control information by processing the muscle activity representation through the second sub-network, the second sub-network being trained according to the electromyogram training signal.

[0015] In some embodiments, the neural network model comprises a third sub-network and a fourth sub-network, the third sub-network being trained by mutual learning with the fourth sub-network, the third sub-network being trained by mutual learning with the fourth sub-network according to the electroencephalogram training signal and the electromyogram training signal.

[0016] In some embodiments, the control method further comprises:

[0017] obtaining an electromyogram signal of a user controlling a target object;

[0018] the processing of the electroencephalogram signal by the neural network model to obtain the control information comprises:

[0019] processing the electroencephalogram signal and the electromyogram signal by the neural network to obtain the control information.

[0020] In some embodiments, the neural network comprises a third sub-network and a fourth sub-network, the processing of the electroencephalogram signal and the electromyogram signal by the neural network to obtain the control information comprises:

[0021] obtaining a first classification result by processing the electroencephalogram signal through the third sub-network;

[0022] obtaining a second classification result by processing the electromyogram signal through the fourth sub-network;

[0023] determining the control information according to the first classification result and the second classification result;

[0024] In some embodiments, the neural network comprises a third sub-network and a fourth sub-network, the processing of the electroencephalogram signal and the electromyogram signal by the neural network to obtain the control information comprises:

[0025] In some embodiments, the electroencephalogram signal, the electroencephalogram training signal and the electromyogram training signal have the same sampling frequency, and the electroencephalogram signal, the electroencephalogram training signal and the electromyogram training signal have the same collection duration.

[0026] The training method of the neural network model provided by the embodiments of the present application comprises:

[0027] obtaining an electroencephalogram (EEG) training signal and an electromyogram (EMG) training signal of a user controlling a target object;

[0028] determining a training muscle activity representation according to the EMG training signal;

[0029] training a first preset network using the EEG training signal and the training muscle activity representation to obtain a first sub-network;

[0030] training a second preset network using the training muscle activity representation to obtain a second sub-network;

[0031] concatenating the first sub-network and the second sub-network to obtain the neural network model.

[0032] In some embodiments, the determining a training muscle activity representation according to the EMG training signal comprises:

[0033] filtering the EMG training signal using a preset frequency trap filter to obtain a preprocessed EMG training signal;

[0034] performing dimensionality reduction mapping on the preprocessed EMG training signal using manifold learning to obtain the training muscle activity representation.

[0035] In some embodiments, the training a first preset network using the EEG training signal to obtain a first sub-network comprises:

[0036] filtering the EEG training signal using a fourth-order Butterworth band-pass filter with a preset frequency range to obtain a preprocessed EEG training signal;

[0037] training the first preset network using the preprocessed EEG training signal and the training muscle activity representation to obtain the first sub-network.

[0038] In some embodiments, the training a first preset network using the preprocessed EEG training signal to obtain a first sub-network comprises:

[0039] training the first preset network using the preprocessed EEG training signal to obtain a training output muscle activity representation;

[0040] calculating a loss value of the first preset network based on the training output muscle activity representation and the training muscle activity representation through a first loss function;

[0041] correcting parameters of the first preset network according to the loss value of the first preset network to obtain the first sub-network.

[0042] In some embodiments, the first preset network comprises a three-layer bidirectional LSTM, each layer of the bidirectional LSTM comprises 100 hidden layer units, dropout = 50%, and the learning rate is 0.005.

[0043] In some embodiments, the first loss function is represented as:

[0044]

[0045] wherein f(x i ) is a training output muscle activity representation output by the first preset network, T i is a training muscle activity representation.

[0046] In some embodiments, the training of the second preset network by using the training muscle activity representation obtains a second sub-network, comprising:

[0047] The second preset network is trained by using the training muscle activity representation to obtain training control information.

[0048] Based on the training control information, a loss value of the second preset network is calculated by a second loss function.

[0049] According to the loss value of the second preset network, parameters of the second preset network are corrected to obtain the second sub-network.

[0050] In some embodiments, the second preset network comprises a three-layer bidirectional LSTM, each layer of the bidirectional LSTM comprises 100 hidden layer units, dropout = 50%, and the learning rate is 0.005.

[0051] In some embodiments, the second loss function is represented as:

[0052]

[0053] wherein, p m (t i ) is a category probability value generated by the second preset network for a sample t i .

[0054] In some embodiments, the concatenation of the first sub-network and the second sub-network obtains the neural network model, comprising:

[0055] The first sub-network and the second sub-network are concatenated to obtain a concatenated network.

[0056] The concatenated network is trained by using the electroencephalogram training signal to obtain concatenated output control information.

[0057] The loss value of the series network is calculated by a third loss function based on the series output control information.

[0058] The parameters of the series network are corrected according to the loss value of the series network to obtain the neural network model.

[0059] In some embodiments, the third loss function is represented as:

[0060]

[0061] wherein, p m (t i ) is a category probability value generated by the third preset network. i

[0062] The training method of the neural network model provided by the embodiments of the present application comprises:

[0063] The electroencephalogram training signal and the electromyogram training signal of a user controlling a target object are obtained.

[0064] The training muscle activity representation is determined according to the electromyogram training signal.

[0065] The third sub-network and the fourth sub-network learn from each other according to the electroencephalogram training signal and the training muscle activity representation.

[0066] The neural network model is generated according to the trained third sub-network.

[0067] In some embodiments, the training muscle activity representation is determined according to the electromyogram training signal, which comprises:

[0068] The electromyogram training signal is filtered by using a trap wave filter of a preset frequency to obtain a preprocessed electromyogram training signal.

[0069] The preprocessed electromyogram training signal is mapped by using manifold learning to obtain the training muscle activity representation.

[0070] In some embodiments, the third sub-network and the fourth sub-network learn from each other according to the electroencephalogram training signal and the training muscle activity representation, which comprises:

[0071] The electroencephalogram training signal is processed by the third sub-network to obtain a first training classification result.

[0072] The training muscle activity representation is processed by the fourth sub-network to obtain a second training classification result.

[0073] ​The loss value of the third sub-network and / or the fourth sub-network is calculated by a fourth loss function based on the first training classification result and the second training classification result.

[0074] The parameters of the third sub-network are corrected according to the loss value of the third sub-network, and / or the parameters of the fourth sub-network are corrected according to the loss value of the fourth sub-network.

[0075] In some embodiments, the fourth loss function is represented as:

[0076]

[0077] wherein, k = 1, 2, 3, L, K,

[0078] In some embodiments, the generating the neural network model according to the trained third sub-network comprises:

[0079] The trained third sub-network is determined as the neural network model.

[0080] In some embodiments, the generating the neural network model according to the trained third sub-network comprises:

[0081] The neural network model is obtained by jointly training the trained third sub-network and the trained fourth sub-network.

[0082] The control device based on electroencephalogram signals provided by the embodiments of the present application comprises:

[0083] A first obtaining module is configured to obtain an electroencephalogram signal of a user controlling a target object;

[0084] A processing module is configured to process the electroencephalogram signal to obtain control information by using a neural network model trained by an electroencephalogram training signal and an electromyogram training signal;

[0085] A control module is configured to control the target object according to the control information.

[0086] The training device of the neural network model provided by the embodiments of the present application comprises:

[0087] A second obtaining module is configured to obtain an electroencephalogram training signal and an electromyogram training signal of a user controlling a target object;

[0088] A first determining module is configured to determine a training muscle activity representation according to the electromyogram training signal;

[0089] The first training module is configured to train the first preset network by using the EEG training signal and the training muscle activity representation to obtain a first sub-network.

[0090] The second training module is configured to train the second preset network by using the training muscle activity representation to obtain a second sub-network.

[0091] The series module is configured to connect the first sub-network and the second sub-network in series to obtain the neural network model.

[0092] The training device of the neural network model provided by the embodiment of the application comprises:

[0093] The third acquisition module is configured to acquire the EEG training signal and the EMG training signal of the user controlling the target object.

[0094] The second determination module is configured to determine the training muscle activity representation according to the EMG training signal.

[0095] The third training module is configured to train the third sub-network according to the EEG training signal and the fourth sub-network according to the training muscle activity representation.

[0096] The generation module is configured to generate the neural network model according to the trained third sub-network.

[0097] The electronic device provided by the embodiment of the application comprises a processor, a memory and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the control method based on the EEG signal and the training method of the neural network model.

[0098] The nonvolatile computer readable storage medium containing the computer program provided by the embodiment of the application, when the computer program is executed by the processor, makes the processor execute the control method based on the EEG signal or executes the training method of the neural network model.

[0099] In the control method, the control device, the training method, the electronic device and the readable storage medium of the embodiment of the application, the EEG signal of the control target object is acquired, and the EEG signal is processed by the neural network model trained by the EEG training signal and the EMG training signal, so that the EEG signal can be converted into the control information of the movement of the control target object, thus improving the performance of the overall brain-computer interface, enabling the prosthesis to flexibly and freely complete the instructed action of the brain, and enabling the residual limb patient to communicate and cooperate in daily life, to restore part of the self-care and work communication ability.

[0100] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. Attached Figure Description

[0101] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0102] Figure 1 This is a flowchart illustrating the control method of certain embodiments of this application;

[0103] Figure 2 This is a schematic diagram of the control device according to certain embodiments of this application;

[0104] Figures 3-6 This is a flowchart illustrating the control method of certain embodiments of this application;

[0105] Figure 7 This is a flowchart illustrating the training method of some embodiments of this application;

[0106] Figure 8 This is a schematic diagram of a training device according to certain embodiments of this application;

[0107] Figures 9-13 This is a flowchart illustrating the training method of some embodiments of this application;

[0108] Figure 14 This is a flowchart illustrating the training method of some embodiments of this application;

[0109] Figure 15 This is a schematic diagram of a training device according to certain embodiments of this application;

[0110] Figures 16-19 This is a flowchart illustrating the training method of some embodiments of this application. Detailed Implementation

[0111] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0112] Please see Figure 1 This application provides a control method based on electroencephalogram (EEG) signals, comprising the following steps:

[0113] 01. Acquire the brainwave signals of the target object controlled by the user;

[0114] 02. Control information is obtained by processing EEG signals using a neural network model, which is trained using EEG training signals and EMG training signals.

[0115] 03. Controlling the target object according to the control information.

[0116] Referring to Figure 2 The embodiment of the present application provides a control device 10 based on an electroencephalogram signal. The control device 10 comprises a first acquisition module 110, a processing module 120 and a control module 130. Wherein, 01 can be realized by the first acquisition module 110, 02 can be realized by the processing module 120, and 03 can be realized by the control module 130.

[0117] In other words, the first acquisition module 110 can be used for acquiring the electroencephalogram signal of a user controlling a target object. The processing module 120 can be used for processing the electroencephalogram signal by using a neural network model to obtain control information, wherein the neural network model is trained by an electroencephalogram training signal and an electromyogram training signal. The control module 130 can be used for controlling the target object according to the control information.

[0118] The embodiment of the present application further provides an electronic device, which comprises a memory, a processor and a computer program. The computer program is stored in the memory, and the computer program comprises a control method based on an electroencephalogram signal, which is executed by the processor, so that the processor realizes the following steps: acquiring an electroencephalogram signal of a user controlling a target object, processing the electroencephalogram signal by using a neural network model to obtain control information, wherein the neural network model is trained by an electroencephalogram training signal and an electromyogram training signal, and controlling the target object according to the control information.

[0119] In the control method based on an electroencephalogram signal, a control device and an electronic device provided by the present application, the electroencephalogram signal of a user controlling a target object is acquired, and the electroencephalogram signal is processed by using a neural network model trained by an electroencephalogram training signal and an electromyogram training signal, so that the electroencephalogram signal can be converted into control information for controlling the movement of the target object. In this way, the performance of the whole brain-computer interface is improved, the artificial limb can flexibly and freely complete the instructed action of the brain, and the residual limb patient can communicate and cooperate in daily life, and the ability of self-care and work communication is restored.

[0120] In some embodiments, the processing device 10 can be part of an electronic device. In other words, the electronic device comprises the processing device 10.

[0121] In some embodiments, the processing device 10 can be a discrete element assembled in a certain way to have the aforementioned functions, or a chip in the form of an integrated circuit having the aforementioned functions, or a computer software code segment that makes a computer have the aforementioned functions when running on the computer.

[0122] The neural network model can be generated after training by a recursive neural network (RNN) using the electroencephalogram training signal and the electromyogram training signal. The recursive neural network is an artificial neural network (ANN) with a tree-like hierarchical structure and network nodes recursively processing input information according to their connection order, and is one of deep learning algorithms.

[0123] It should be noted that the electroencephalogram training signal refers to an electroencephalogram signal used in training of the neural network model, and the electromyogram training signal refers to an electromyogram signal used in training of the neural network model.

[0124] It should be further noted that the target object can be a human body (for example, an arm, a finger, or a foot of a human body, etc.), a prosthesis, or other devices. The target object is not particularly limited. For example, in the present embodiment, the target object can be a prosthesis of a user, that is, in the present embodiment, the control of the prosthesis of the user is taken as an example to explain that the electroencephalogram signal of the user controlling the prosthesis is acquired, so that the control information for controlling the prosthesis of the user is generated according to the electroencephalogram signal. The electroencephalogram signal can be a signal of C3 and C4 leads.

[0125] The sampling frequency of the electroencephalogram signal, the electroencephalogram training signal, and the electromyogram training signal is the same, and the collection time length of the electroencephalogram signal, the electroencephalogram training signal, and the electromyogram training signal is the same. In the present embodiment, the sampling frequency of the electroencephalogram signal can be 500 Hz, and the collection time length can be 8 seconds. Of course, it can be understood that the sampling frequency and the collection time length of the electroencephalogram signal can also be other values, that is, the sampling frequency and the collection time length of the electroencephalogram signal are not particularly limited.

[0126] Please refer to Figure 3 In some embodiments, step 02 includes:

[0127] 021, filtering the electroencephalogram signal using a fourth-order Butterworth band-pass filter with a preset frequency range to obtain a preprocessed electroencephalogram signal;

[0128] 022, processing the preprocessed electroencephalogram signal using the neural network model to obtain control information.

[0129] Please further refer to Figure 2 In some embodiments, sub-steps 021-022 can be implemented by the processing module 120. In other words, the processing module 120 can be configured to filter the electroencephalogram signal using a fourth-order Butterworth band-pass filter with a preset frequency range to obtain a preprocessed electroencephalogram signal, and process the preprocessed electroencephalogram signal using the neural network model to obtain control information.

[0130] In some embodiments, the processor can be configured to filter the electroencephalogram signal using a 4th order Butterworth band-pass filter of a preset frequency range to obtain a preprocessed electroencephalogram signal, and process the preprocessed electroencephalogram signal using a neural network model to obtain the control information.

[0131] The preset frequency range can be 8-30 Hz, that is, in this application, the 4th order Butterworth band-pass filter of 8-30 Hz is used to denoise the electroencephalogram signal to obtain the preprocessed electroencephalogram signal.

[0132] In this way, the electroencephalogram signal is preprocessed by the 4th order Butterworth band-pass filter, the clutter interference is reduced, and the accuracy of the control information is improved.

[0133] Please refer to Figure 4 In some embodiments, the neural network model includes a first subnetwork and a second subnetwork connected in series, and step 02 includes:

[0134] 023, processing the electroencephalogram signal through the first subnetwork to obtain a muscle activity representation, the first subnetwork being trained by the electroencephalogram training signal and the electromyogram training signal;

[0135] 024, processing the muscle activity representation through the second subnetwork to obtain the control information, the second subnetwork being trained according to the electromyogram training signal.

[0136] Please further refer to Figure 2 In some embodiments, sub-steps 023-024 can be implemented by the processing module 120. Alternatively, the processing module 120 can be configured to process the electroencephalogram signal through the first subnetwork to obtain a muscle activity representation, the first subnetwork being trained by the electroencephalogram training signal and the electromyogram training signal, and process the muscle activity representation through the second subnetwork to obtain the control information, the second subnetwork being trained according to the electromyogram training signal.

[0137] In some embodiments, the processor can be configured to process the electroencephalogram signal through the first subnetwork to obtain a muscle activity representation, the first subnetwork being trained by the electroencephalogram training signal and the electromyogram training signal, and process the muscle activity representation through the second subnetwork to obtain the control information, the second subnetwork being trained according to the electromyogram training signal.

[0138] It should be noted that the first subnetwork can use a bidirectional LSTM to directly decode the electroencephalogram signal into a muscle activity representation, or use a time-frequency analysis method (such as short-time Fourier transform, wavelet transform, Hilbert-Huang transform, etc.) to convert the electroencephalogram signal into time-domain spectrum information, and then use a convolutional neural network (such as CNN, ResNet, FCNN, etc.) for classification to obtain the muscle activity representation. That is, the method used by the first subnetwork is not limited.

[0139] The second sub-network can employ a trained bidirectional LSTM, that is, the muscle activity representation is processed by the trained bidirectional LSTM to obtain the control information.

[0140] In this way, by decoding the electroencephalogram signal into the muscle activity representation by the first sub-network in the first stage, and decoding the muscle activity representation into the control information of the prosthesis movement by the second stage in the second stage, the accuracy and efficiency of the electroencephalogram control of the movement of the target object are improved.

[0141] Please refer to Figure 5 In some embodiments, the control method further comprises:

[0142] 04, obtaining the electromyogram signal of the user controlling the target object;

[0143] Step 02 further comprises a sub-step:

[0144] 025, processing the electroencephalogram signal and the electromyogram signal by the neural network model to obtain the control information.

[0145] In some embodiments, step 04 can be implemented by a first acquisition module 110, and sub-step 025 can be implemented by a processing module 120. Alternatively, the first acquisition module 110 can be configured to obtain the electromyogram signal of the user controlling the target object, and the processing module 120 can be configured to process the electroencephalogram signal and the electromyogram signal by the neural network to obtain the control information.

[0146] In some embodiments, the processor can be configured to obtain the electromyogram signal of the user controlling the target object, and process the electroencephalogram signal and the electromyogram signal by the neural network to obtain the control information.

[0147] The electromyogram signal can be the electromyogram signal at the anterior deltoid muscle, brachi-radialis muscle, flexor digitorum muscle, extensor digitorum communis muscle, and first dorsal interosseous muscle of the user's right arm.

[0148] Before processing the electroencephalogram signal and the electromyogram signal by the neural network model to obtain the control information, the electromyogram signal can be pre-processed, for example, the electromyogram signal can be filtered by a 50Hz notch filter.

[0149] In addition, it should be noted that in the present embodiment, the neural network model is based on deep mutual learning (Deep Mutual Learning) to process the electroencephalogram signal and the electromyogram signal to obtain the control information, thereby improving the accuracy and efficiency of the electroencephalogram control of the movement of the target object.

[0150] In some embodiments, the neural network model comprises a third sub-network, the third sub-network being configured to process the electroencephalogram signal to obtain the control information, the third sub-network being trained by mutual learning with the fourth sub-network, the third sub-network being trained by mutual learning with the fourth sub-network according to the electroencephalogram training signal and the fourth sub-network being trained by mutual learning with the third sub-network according to the electromyogram training signal.

[0151] In this way, the third sub-network trained by mutual learning can be used as the neural network model, and the control information can be obtained by processing the electroencephalogram signal, and the target object can be controlled by the control information, thereby improving the accuracy and efficiency of the electroencephalogram control target object.

[0152] Please refer to Figure 6 In some embodiments, the neural network model comprises a third sub-network and a fourth sub-network, and the sub-step 025 comprises:

[0153] 0251, processing the electroencephalogram signal by the third sub-network to obtain a first classification result;

[0154] 0252, processing the electromyogram signal by the fourth sub-network to obtain a second classification result;

[0155] 0253, determining the control information according to the first classification result and the second classification result.

[0156] In some embodiments, the sub-steps 0251-0253 can be implemented by the processing module 120. Alternatively, the processing module 120 can be configured to process the electroencephalogram signal by the third sub-network to obtain a first classification result, process the electromyogram signal by the fourth sub-network to obtain a second classification result, and determine the control information according to the first classification result and the second classification result.

[0157] In some embodiments, the processor can be configured to process the electroencephalogram signal by the third sub-network to obtain a first classification result, process the electromyogram signal by the fourth sub-network to obtain a second classification result, and determine the control information according to the first classification result and the second classification result.

[0158] It should be noted that the third sub-network is trained by mutual learning with the fourth sub-network, the third sub-network is trained by mutual learning with the fourth sub-network according to the electroencephalogram training signal, and the fourth sub-network is trained by mutual learning with the third sub-network according to the electromyogram training signal.

[0159] In some embodiments, the neural network model comprises a third sub-network, the third sub-network being trained by mutual learning with the fourth sub-network, the third sub-network being trained by mutual learning with the fourth sub-network according to the electroencephalogram training signal and the fourth sub-network being trained by mutual learning with the third sub-network according to the electromyogram training signal.

[0160] The third sub-network can include, but is not limited to, FCNN, CNN, RNN, ResNet, RCR-net, etc. For example, in the present application, the third sub-network can adopt ResNet18+ bidirectional LSTM, that is, the electroencephalogram signal can be processed by ResNet18+ bidirectional LSTM to obtain the first classification result.

[0161] The fourth sub-network can include, but is not limited to, FCNN, CNN, RNN, ResNet, RCR-net, etc. In the present application, the fourth sub-network can adopt ResNet18+ bidirectional LSTM, that is, the electromyogram signal can be processed by ResNet18+ bidirectional LSTM to obtain the second classification result.

[0162] In addition, in some other examples, the fourth sub-network can also reduce the dimension of the electromyogram signal based on manifold learning (such as PCA, Laplacian Eigenmap, local linear embedding, etc.), map the high-dimensional electromyogram signal to a low-dimensional space, obtain the muscle activity representation, and then input the muscle activity representation into traditional machine learning (such as support vector machine, random forest, etc.) or deep learning (such as bidirectional LSTM, RNN, etc.) processing to obtain the second classification result.

[0163] The electromyogram signal can be reduced in dimension by using Laplacian Eigenmap, and the steps are as follows:

[0164] The 8-second electromyogram signals obtained at the five muscles are directly connected as inputs: i = 1, 2, 3L k, and the output is set as: m = l, i = 1, 2, 3L k.

[0165] The connection graph is reconstructed: n neighboring points of x i are selected, and the connection graph is formed by connecting the points, and the weight value of the connection between the points is determined: if x i and x j are neighboring points, then W ij = 1, otherwise W ij = 0

[0166] Further feature mapping: calculate the eigenvalue λ and eigenvector f such that Lf = λDf, where D is the diagonal weight matrix, D ii = ∑ j W ji , and L = D-W is the Laplacian operator.

[0167] Finally, solve argmin Y T LYs.t.Y TDY=1, the Y obtained by retaining the non-zero value is the final output, that is, the muscle activity representation. In this way, after Laplace feature mapping, the electromyographic signal can be reduced to a 125-dimensional feature vector to obtain the muscle activity representation.

[0168] Further, after obtaining the first classification result and the second classification result through the third sub-network and the fourth sub-network, the control information can be determined according to the first classification result and the second classification result through deep mutual learning.

[0169] In this way, the control information is obtained by simultaneously using the electroencephalogram signal and the electromyographic signal based on mutual learning, thereby improving the accuracy and efficiency of the electroencephalogram control target object.

[0170] Please refer to Figure 7 The present application provides a neural network model training method, comprising the steps of:

[0171] 001, obtaining the electroencephalogram training signal and the electromyographic training signal of the user control target object;

[0172] 002, determining the training muscle activity representation according to the electromyographic training signal;

[0173] 003, training the first preset network using the electroencephalogram training signal and the training muscle activity representation to obtain the first sub-network;

[0174] 004, training the second preset network using the training muscle activity representation to obtain the second sub-network;

[0175] 005, concatenating the first sub-network and the second sub-network to obtain the neural network model.

[0176] Please refer to Figure 8 The present application provides a neural network model training device 20. The training device 20 comprises a second acquisition module 210, a determination module 220, a first training module 230, a second training module 240 and a concatenation module 250. Wherein, step 001 can be realized by the second acquisition module 210. Step 002 can be realized by the determination module 220, step 003 can be realized by the first training module 230. Step 004 can be realized by the second training module 240, and step 005 can be realized by the concatenation module 250.

[0177] Or, the second acquisition module 210 can be configured to acquire the EEG training signal and the EMG training signal of the user controlling the target object; the determination module 220 can be configured to determine the training muscle activity representation according to the EMG training signal; the first training module 230 can be configured to train the first preset network by using the EEG training signal and the training muscle activity representation to obtain the first sub-network; the second training module 240 can be configured to train the second preset network by using the training muscle activity representation to obtain the second sub-network; and the series connection module 250 can be configured to connect the first sub-network and the second sub-network in series to obtain the neural network model.

[0178] The electronic device provided in the embodiments of the present application includes a memory and a processor, and the memory stores a computer program. When the computer program is executed by the processor, the processor implements the following steps: acquiring an EEG training signal and an EMG training signal of a user controlling a target object; determining a training muscle activity representation according to the EMG training signal; training a first preset network by using the EEG training signal and the training muscle activity representation to obtain a first sub-network; training a second preset network by using the training muscle activity representation to obtain a second sub-network; and connecting the first sub-network and the second sub-network in series to obtain a neural network model.

[0179] In the training method, the training device 20 and the electronic device provided in the embodiments of the present application, the EEG training signal and the EMG training signal of the user controlling the target object are acquired, and the first preset network and the second preset network are trained according to the EEG training signal and the EMG training signal, to obtain the first sub-network and the second sub-network, and then the first sub-network and the second sub-network are connected in series to obtain the neural network model. Therefore, the EEG signal can be processed by using the neural network model in the future, to generate control information for controlling the target object. In this way, the performance of the overall brain-computer interface is improved, the prosthetic limb can flexibly and freely complete the instructions issued by the brain, and the amputee patient can communicate and cooperate in daily life, to restore the ability of self-care and work communication.

[0180] It should be noted that the user can be a disabled person or a healthy person, and the number of users can be one or more. For example, in some examples, the EEG signals and EMG signals collected from multiple subjects can be used as the EEG training signal and the EMG training signal. For another example, in some examples, for a single user, the daily use data (EEG signal and EMG signal) of the single user can be used as the training data set within one month.

[0181] The sampling frequency of the EEG training signal and the EMG training signal is the same, and the collection time of the EEG training signal and the EMG training signal is the same. In this embodiment, the sampling frequency of the EEG training signal and the EMG training signal can be 500 Hz, and the collection time can be 8 seconds. Of course, it can be understood that the sampling frequency and the collection time of the EEG training signal and the EMG training signal can also be other values, that is, the sampling frequency and the collection time of the EEG training signal and the EMG training signal are not limited in particular.

[0182] In addition, in order to ensure the accuracy of the EEG training signal and the EMG training signal, the selected users are the same handedness (for example, all left-handed or all right-handed).

[0183] The EEG training signal can be the EEG signal of the lead at the C3 and C4 channels. The EMG training signal can include the EMG signals at the anterior deltoid, brachi-radialis, flexor digitorum, extensor digitorum communis and first dorsal interosseous muscles, and the actions performed by the target object can include finger extension, finger flexion, forearm pronation, forearm supination and rest.

[0184] For example, in some examples, 100 right-handed users are selected, the EEG training signal of the C3 and C4 channels is collected, the sampling frequency is 500 Hz, the collection time of each action is 8 seconds, the EMG training signal of the right anterior deltoid, brachi-radialis, flexor digitorum, extensor digitorum communis and first dorsal interosseous muscles is collected synchronously, the sampling frequency is 500 Hz, the collection time of each action is 8 seconds, the actions include five kinds of finger extension, finger flexion, forearm pronation, forearm supination and rest, each action is repeated 120 times, and a total of 72,000 EEG training signals and EMG training signals are obtained.

[0185] In addition, in some embodiments, it can be difficult to implement prosthesis control through the neural network model due to the difficulty of collecting EMG signals for the disabled population. The movement signals of the left and right hands of a person are reflected in both the left and right brains, but the proportions are different, that is, although the left brain controls the movement of the right hand, the left brain also contains brain signals for controlling the movement of the left hand. Therefore, for part of the disabled population, such as people with lesions in the left brain, the EEG training signals collected from the left and right brains can be preprocessed to obtain more targeted EEG training signals for right arm movement, for example, the EEG training signals collected at the C3 and C4 channels are respectively converted into frequency domain images through wavelet transform, and then the two frequency domain images are superimposed to obtain preprocessed EEG training signals for inputting into the first preset network for training, thereby further improving the robustness of the neural network model.

[0186] It should be noted that the first preset network can adopt a bidirectional LSTM network, or can adopt a frequency analysis method (such as short-time Fourier transform, wavelet transform, Hilbert-Huang transform, etc.) and a convolutional neural network (such as CNN, ResNet, FCNN, etc.).

[0187] For example, in the present embodiment, the first preset network uses a 3-layer bidirectional LSTM, each layer has 100 hidden units, dropout = 50%, learning rate is 0.005, and uses the ADAM optimizer to minimize the mean squared error. That is, the first sub-network is obtained by training the 3-layer bidirectional LSTM using the EEG training signal and the training muscle activity representation.

[0188] The second preset network uses a bidirectional LSTM network, that is, the second sub-network is obtained by training the bidirectional LSTM network using the training muscle activity representation.

[0189] For example, in the present embodiment, the second preset network uses a 3-layer bidirectional LSTM, each layer has 100 hidden units, dropout = 50%, learning rate is 0.005, and uses the ADAM optimizer to minimize the cross-entropy loss function. The input is the training muscle activity representation, and the output is the control information of the prosthesis movement, including hand opening, hand closing, forearm internal rotation, forearm external rotation, and rest.

[0190] Please refer to Figure 9 In some embodiments, step 002 comprises:

[0191] 0021, filtering the electromyogram training signal using a preset frequency trap to obtain a preprocessed electromyogram training signal;

[0192] 0022, using manifold learning to reduce the dimension of the preprocessed electromyogram training signal to obtain a training muscle activity representation.

[0193] Please further refer to Figure 8 In some embodiments, sub-steps 0021-0022 can be implemented by the determination module 220. In other words, the determination module 220 can be configured to filter the electromyogram training signal using a preset frequency trap to obtain a preprocessed electromyogram training signal, and use manifold learning to reduce the dimension of the preprocessed electromyogram training signal to obtain a training muscle activity representation.

[0194] In some embodiments, the processor can be configured to filter the electromyogram training signal using a preset frequency trap to obtain a preprocessed electromyogram training signal, and use manifold learning to reduce the dimension of the preprocessed electromyogram training signal to obtain a training muscle activity representation.

[0195] The preset frequency can be 50Hz, that is, the electromyogram training signal is filtered using a 50Hz trap to obtain a preprocessed electromyogram training signal.

[0196] The manifold learning can include, but is not limited to, PCA (Principal Component Analysis), Laplacian Eigenmap, local linear embedding, etc. In the embodiment, the Laplacian Eigenmap is used to reduce the dimension of the preprocessed electromyogram training signal to obtain the training muscle activity representation, and the specific steps are as follows:

[0197] First, the 8-second electromyogram training signals obtained at the five muscles are directly connected as inputs: i = 1, 2, 3L k, and the output is set as: m = 1, i = 1, 2, 3L k

[0198] Then, a connection graph is constructed: select n adjacent points of x i , and connect them to form a connection graph, and determine the weight value of the connection between the points: if x i and x j are adjacent points, then W ij = 1, otherwise W ij = 0

[0199] Further, feature mapping is performed: calculate the eigenvalue λ and eigenvector f such that Lf = λDf, where D is the diagonal weight matrix, D ii = ∑ j W ji , and L = D-W is the Laplacian operator.

[0200] Finally, solve argmin Y T LYs.t.Y T DY = 1, and the non-zero value obtained by retaining Y is the final output, that is, the training muscle activity representation. In this way, after the Laplacian Eigenmap, the electromyogram training signal can be reduced to a 125-dimensional feature vector training muscle activity representation.

[0201] Referring to Figure 10 , in some embodiments, step 003 includes:

[0202] 0031, filtering the electroencephalogram training signal using a fourth-order Butterworth bandpass filter with a preset frequency range to obtain a preprocessed electroencephalogram training signal;

[0203] 0032, training the first preset network using the preprocessed electroencephalogram training signal and the training muscle activity representation to obtain a first sub-network.

[0204] Please further refer to Figure 8In some embodiments, the sub-steps 0031-0032 can be implemented by the first training module 230. In other words, the first training module 230 can be configured to filter the EEG training signal using a 4th order Butterworth band-pass filter with a preset frequency range to obtain a preprocessed EEG training signal, and train the first preset network using the preprocessed EEG training signal and the training muscle activity representation to obtain the first sub-network.

[0205] In some embodiments, the processor can be configured to filter the EEG training signal using a 4th order Butterworth band-pass filter with a preset frequency range to obtain a preprocessed EEG training signal, and train the first preset network using the preprocessed EEG training signal and the training muscle activity representation to obtain the first sub-network.

[0206] The preset frequency range can be 8-30 Hz, that is, in the present application, a 4th order Butterworth band-pass filter with a frequency range of 8-30 Hz is used to denoise the EEG signal to obtain a preprocessed EEG signal.

[0207] In this way, the EEG training signal is preprocessed by the 4th order Butterworth band-pass filter, which reduces the interference of noise and improves the accuracy of the control information.

[0208] Please refer to Figure 11 In some embodiments, step 0032 comprises:

[0209] 00321, training the first preset network using the preprocessed EEG training signal to obtain a training output muscle activity representation;

[0210] 00322, calculating a loss value of the first preset network by a first loss function based on the training output muscle activity representation and the training muscle activity representation;

[0211] 00323, correcting parameters of the first preset network according to the loss value of the first preset network to obtain the first sub-network.

[0212] Please further refer to Figure 8 In some embodiments, the sub-steps 0033-0035 can be implemented by the first training module 230. In other words, the first training module 230 can be configured to train the first preset network using the preprocessed EEG training signal to obtain a training output muscle activity representation, calculate a loss value of the first preset network by a first loss function based on the training output muscle activity representation and the training muscle activity representation, and correct parameters of the first preset network according to the loss value of the first preset network to obtain the first sub-network.

[0213] In some embodiments, the processor can be configured to train the first preset network using the preprocessed electroencephalogram training signals to obtain a training output muscle activity representation, calculate a loss value of the first preset network based on the training output muscle activity representation and the training muscle activity representation through a first loss function, and correct parameters of the first preset network according to the loss value of the first preset network to obtain the first subnetwork.

[0214] It should be noted that if N electroencephalogram training signals are given, they can be represented as The output corresponding to each electroencephalogram training signal (training output muscle activity representation) can be represented as The first loss function of the first preset network can be represented as:

[0215]

[0216] where f(x i ) is the training output muscle activity representation output by the first preset network, T i is the training muscle activity representation.

[0217] Please refer to Figure 12 In some embodiments, step 004 includes:

[0218] 0041, training the second preset network using the training muscle activity representation to obtain training control information;

[0219] 0042, calculating a loss value of the second preset network based on the training control information through a second loss function;

[0220] 0043, correcting parameters of the second preset network according to the loss value of the second preset network to obtain a second subnetwork.

[0221] Please further refer to Figure 2 In some embodiments, sub-steps 0041-0043 can be implemented by the second training module 240. In other words, the second training module 240 can be configured to train the second preset network using the training muscle activity representation to obtain training control information, calculate a loss value of the second preset network based on the training control information through a second loss function, and correct parameters of the second preset network according to the loss value of the second preset network to obtain a second subnetwork.

[0222] In some embodiments, the processor can be configured to train the second preset network using the training muscle activity representation to obtain training control information, calculate a loss value of the second preset network based on the training control information through a second loss function, and correct parameters of the second preset network according to the loss value of the second preset network to obtain a second subnetwork.

[0223] Specifically, if N samples of M categories are given, they are represented as The label corresponding to each sample can be represented as where y i ∈{1,2,L,M}.

[0224] The sample t i The calculation formula of the category probability value generated by the second preset network is:

[0225]

[0226] where z m is the output of the softmax layer.

[0227] The loss function of the second preset network can be represented as:

[0228]

[0229] where, p m (t i ) is the category probability value of the sample t i generated by the second preset network.

[0230] Please refer to Figure 13 In some embodiments, step 005 includes:

[0231] 0051, the first sub-network and the second sub-network are connected in series to obtain a series network;

[0232] 0052, the series network is trained by the EEG training signal to obtain series output control information;

[0233] 0053, based on the series output control information, a loss value of the series network is calculated by a third loss function;

[0234] 0054, the parameters of the series network are corrected according to the loss value of the series network to obtain a neural network model.

[0235] Please further refer to Figure 8 In some embodiments, sub-steps 0051-0054 can be implemented by the series module 250. In other words, the series module 250 can be used to connect the first sub-network and the second sub-network in series to obtain a series network, and train the series network by the EEG training signal to obtain series output control information. The series module 250 can also be used to calculate the loss value of the series network based on the series output control information by the third loss function, and correct the parameters of the series network according to the loss value of the series network to obtain a neural network model.

[0236] In some embodiments, the processor can be configured to concatenate the first sub-network and the second sub-network to obtain a concatenated network, train the concatenated network by the EEG training signal to obtain concatenated output control information, and train the parameters of the concatenated network based on the concatenated output control information by a third loss function to obtain the neural network model.

[0237] The third loss function is represented as:

[0238]

[0239] wherein, p m (t i ) is a sample t i category probability value generated by the third preset network.

[0240] Thus, the parameters of the concatenated network are fine-tuned by the third loss function to obtain the neural network model, so that the neural network model can be optimized, and the EEG signal after preprocessing can be directly input into the neural network model to obtain accurate control information.

[0241] Referring to Figure 14 , the present application provides a neural network model training method, comprising the steps of:

[0242] 11. obtaining EEG training signals and EMG training signals of a user controlling a target object;

[0243] 12. determining a training muscle activity representation according to the EMG training signal;

[0244] 13. training the third sub-network according to the EEG training signal and the fourth sub-network according to the training muscle activity representation;

[0245] 14. generating a neural network model according to the trained third sub-network.

[0246] Referring to Figure 15 , the present application provides a neural network model training device 30. The training device 30 comprises a third acquisition module 310, a second determination module 320, a third training module 330 and a generation module 340. Step 11 can be realized by the third acquisition module 310. Step 12 can be realized by the second determination module 320, and step 13 can be realized by the third training module 330. Step 14 can be realized by the generation module 340.

[0247] Or, the third obtaining module 310 can be configured to obtain the EEG training signal and the EMG training signal of the user controlling the target object; the second determining module 320 can be configured to determine the training muscle activity representation according to the EMG training signal; the third training module 330 can be configured to train the third sub-network according to the EEG training signal and the fourth sub-network according to the training muscle activity representation; and the generating module 340 can be configured to generate the neural network model according to the trained third sub-network.

[0248] The embodiment of the present application further provides an electronic device, which comprises a memory, a processor and a computer program, wherein the computer program is stored in the memory, the computer program comprises a training method of the neural network model, and the computer program causes the processor to realize the following when the computer program is executed by the processor: obtaining EEG training signal and EMG training signal of a user controlling a target object, determining a training muscle activity representation according to the EMG training signal, training a third sub-network according to the EEG training signal and a fourth sub-network according to the training muscle activity representation, and generating a neural network model according to the trained third sub-network.

[0249] In the training method of the neural network model, the training device 30 and the electronic device, the EEG training signal and the EMG training signal of the user controlling the target object are obtained, the training muscle activity representation is determined according to the EMG training signal, the third sub-network is trained according to the EEG training signal and the fourth sub-network is trained according to the training muscle activity representation, and finally the neural network model is generated according to the trained third sub-network. Therefore, the EEG signal can be processed by the neural network model in the future, and control information for controlling the target object is generated. In this way, the control information is obtained by simultaneously using the EEG signal and the EMG signal based on mutual learning, which improves the accuracy and efficiency of the EEG control target object, and is beneficial to the daily life communication and cooperation of the residual limb patient, and restores part of the life self-care and work communication ability.

[0250] In step 12, the EMG training signal is dimensionally reduced based on manifold learning (such as PCA, Laplace eigenmap, local linear embedding, etc.), the high-dimensional EMG training signal is mapped to a low-dimensional space, and the training muscle activity representation is obtained.

[0251] The training muscle activity representation can include but is not limited to fist extension, fist flexion, forearm internal rotation, forearm external rotation and rest.

[0252] It should be noted that the third sub-network can include but is not limited to FCNN, CNN, RNN, ResNet, RCR-net, etc., for example, in the present application, the third sub-network can adopt ResNet18+bidirectional LSTM, and the fourth sub-network can include but is not limited to FCNN, CNN, RNN, ResNet, RCR-net, etc., in the present application, the fourth sub-network can adopt ResNet18+bidirectional LSTM, that is, ResNet18+bidirectional LSTM according to the electroencephalogram signal and ResNet18+bidirectional LSTM according to the training muscle activity representation are mutually learned and trained.

[0253] Please refer to Figure 16 In some embodiments, step 12 comprises:

[0254] 121, filtering the electromyogram training signal using a trap filter with a preset frequency to obtain a pretreated electromyogram training signal;

[0255] 122, using manifold learning to perform dimensionality reduction mapping on the pretreated electromyogram training signal to obtain a training muscle activity representation.

[0256] Please further refer to Figure 2 In some embodiments, sub-steps 121-122 can be implemented by the second determination module 320. Alternatively, the second determination module 320 can be used to filter the electromyogram training signal using a trap filter with a preset frequency to obtain a pretreated electromyogram training signal, and use manifold learning to perform dimensionality reduction mapping on the pretreated electromyogram training signal to obtain a training muscle activity representation.

[0257] In some embodiments, the processor can be used to filter the electromyogram training signal using a trap filter with a preset frequency to obtain a pretreated electromyogram training signal, and use manifold learning to perform dimensionality reduction mapping on the pretreated electromyogram training signal to obtain a training muscle activity representation.

[0258] Specifically, the preset frequency can be 50 Hz, that is, the electromyogram training signal is filtered using a trap filter with a frequency of 50 Hz to obtain a pretreated electromyogram training signal.

[0259] Manifold learning can include but is not limited to PCA (Principal Component Analysis), Laplacian Eigenmap, local linear embedding, etc. In the present embodiment, Laplacian Eigenmap is used to perform dimensionality reduction mapping on the pretreated electromyogram training signal to obtain a training muscle activity representation, and the specific steps are as follows:

[0260] First, the 8-second electromyogram training signal obtained at the five muscles is directly connected as input: i = 1, 2, 3L k, and set the output as: m = 1, i = 1, 2, 3L k

[0261] Reconstruct the connection graph: select n adjacent points of x i , connect them to form a connection graph, and determine the weight value of the connection between points: if x i and x j are adjacent points, then W ij = 1, otherwise W ij = 0

[0262] Further feature mapping: calculate the eigenvalue λ and eigenvector f such that Lf = λDf, where D is the diagonal weight matrix, D ii = ∑ j W ji , and L = D-W is the Laplace operator.

[0263] Finally, solve arg min Y T LYs.t.Y T DY = 1, and the Y obtained by retaining the non-zero values is the final output, that is, the trained muscle activity representation. In this way, after Laplace feature mapping, the electromyographic training signal can be reduced to a 125-dimensional feature vector of the trained muscle activity representation.

[0264] Please refer to Figure 17 , in some embodiments, step 13 includes the following sub-steps:

[0265] 131, processing the electroencephalogram training signal through the third sub-network to obtain a first training classification result;

[0266] 132, processing the trained muscle activity representation through the fourth sub-network to obtain a second training classification result;

[0267] 133, based on the first training classification result and the second training classification result, calculating the loss value of the third sub-network and / or the fourth sub-network through the fourth loss function;

[0268] 134, correcting the parameters of the third sub-network according to the loss value of the third sub-network, and / or correcting the parameters of the fourth sub-network according to the loss value of the fourth sub-network.

[0269] Please further combine Figure 15In some embodiments, the sub-steps 131-134 can be implemented by the third training module 330. Alternatively, the third training module 330 can be configured to process the EEG training signals through the third sub-network to obtain a first training classification result, and process the training muscle activity representation through the fourth sub-network to obtain a second training classification result; the third training module 330 can also be configured to calculate a loss value of the third sub-network and / or the fourth sub-network based on the first training classification result and the second training classification result through the fourth loss function, and correct the parameters of the third sub-network according to the loss value of the third sub-network, and / or correct the parameters of the fourth sub-network according to the loss value of the fourth sub-network.

[0270] In some embodiments, the processor can be configured to process the EEG training signals through the third sub-network to obtain a first training classification result, and process the training muscle activity representation through the fourth sub-network to obtain a second training classification result; the processor can also be configured to calculate a loss value of the third sub-network and / or the fourth sub-network based on the first training classification result and the second training classification result through the fourth loss function, and correct the parameters of the third sub-network according to the loss value of the third sub-network, and / or correct the parameters of the fourth sub-network according to the loss value of the fourth sub-network.

[0271] For example, if there are N samples of M categories, which are represented as The label corresponding to each sample can be represented as where y i ∈{1,2,L,M}, the sample x i is processed by the kth neural network θ k The calculation formula of the category probability value generated by the kth neural network is:

[0272]

[0273] where: z m is the output of the softmax layer.

[0274] For the kth neural network, the fourth loss function is:

[0275]

[0276] where, k=1,2,3,L,K,

[0277] Here, six categories are set, including fist extension, fist closing, forearm internal rotation, forearm external rotation and rest, so M=6, two sub-networks, so K=2.

[0278] Please refer to Figure 18 In some embodiments, step 14 includes the following sub-steps:

[0279] 141, determining the trained third sub-network as the neural network model.

[0280] Please further combine Figure 15 In some embodiments, the sub-step 141 can be implemented by the generating module 340. In other words, the generating module 340 can be configured to determine the trained third sub-network as the neural network model.

[0281] In some embodiments, the processor can be configured to determine the trained third sub-network as the neural network model.

[0282] In this way, by determining the trained third sub-network as the neural network model, the control information can be obtained by subsequently processing the electroencephalogram signal through the neural network model.

[0283] Please refer to Figure 19 In some embodiments, the step 14 comprises a sub-step:

[0284] 142, obtaining the neural network model by combining the trained third sub-network and the trained fourth sub-network.

[0285] Please further combine Figure 2 In some embodiments, the sub-step 141 can be implemented by the generating module 340. In other words, the generating module 340 can be configured to obtain the neural network model by combining the trained third sub-network and the trained fourth sub-network.

[0286] In some embodiments, the processor can be configured to obtain the neural network model by combining the trained third sub-network and the trained fourth sub-network.

[0287] In this way, by determining the trained third sub-network and the fourth sub-network as the neural network model, the control information can be obtained by subsequently processing the electroencephalogram signal and the electromyogram signal.

[0288] The embodiments of the present application also provide a non-volatile computer readable storage medium, and the readable storage medium stores a computer program. When the computer program is executed by one or more processors, the processor executes the control method described above.

[0289] In the embodiments described above, all or some of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or some of the embodiments can be implemented in the form of a computer program product storing computer program instructions. When the computer program instructions are loaded into and executed by a computer, all or some of the procedures or functions described in the embodiments of the present application are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, or another programmable apparatus. The computer program instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer program instructions can be transmitted from a website, a computer, an electronic device or a data center to another website, computer, electronic device or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or data storage device, such as an electronic device, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0290] Those skilled in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0291] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0292] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each of the units can exist physically, or two or more units can be integrated in one unit.

[0293] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A control method based on electroencephalogram (EEG) signals, characterized in that, The control method includes: Acquire the electroencephalogram (EEG) signals of the target object controlled by the user; Control information is obtained by processing the electroencephalogram (EEG) signals using a neural network model. The neural network model is trained using EEG training signals and electromyography (EMG) training signals. The neural network model includes a first sub-network and a second sub-network connected in series. The process of obtaining control information by processing the EEG signals using the neural network model includes: processing the EEG signals through the first sub-network to obtain muscle activity representations, where the first sub-network is trained using the EEG training signals and EMG training signals; and processing the muscle activity representations through the second sub-network to obtain the control information, where the second sub-network is trained based on the EMG training signals. The target object is controlled according to the control information.

2. The control method according to claim 1, characterized in that, The process of using a neural network model to process the electroencephalogram (EEG) signals to obtain control information includes: The EEG signal is filtered using a 4th-order Butterworth bandpass filter with a preset frequency range to obtain a preprocessed EEG signal. The control information is obtained by processing the preprocessed EEG signal using the neural network model.

3. The control method according to claim 1, characterized in that, The EEG signal, the EEG training signal, and the EMG training signal have the same sampling frequency, and the acquisition duration of the EEG signal, the EEG training signal, and the EMG training signal is the same.

4. A method for training a neural network model, characterized in that, include: Acquire EEG and EMG training signals of the user-controlled target object; The electromyographic training signals are used to determine the muscle activity characteristics of the trained muscles. The first sub-network is obtained by training the first preset network using the EEG training signal and the training muscle activity representation; The second sub-network is obtained by training the second preset network using the aforementioned muscle activity representation; The neural network model is obtained by concatenating the first sub-network and the second sub-network.

5. The training method according to claim 4, characterized in that, The step of determining the muscle activity representation based on the electromyographic training signal includes: The electromyography training signal is filtered using a notch filter with a preset frequency to obtain a preprocessed electromyography training signal. The preprocessed electromyographic training signal is dimensionality-reduced and mapped using manifold learning to obtain the training muscle activity representation.

6. The training method according to claim 5, characterized in that, The step of training a first sub-network using the EEG training signal to train a first preset network includes: The EEG training signal is filtered using a 4th-order Butterworth bandpass filter with a preset frequency range to obtain a preprocessed EEG training signal. The first sub-network is obtained by training the first preset network using the preprocessed EEG training signal and the training muscle activity representation.

7. The training method according to claim 5, characterized in that, The step of training the first preset network using the preprocessed EEG training signal to obtain the first sub-network includes: The preprocessed EEG training signal is used to train the first preset network to obtain the training output muscle activity representation. Based on the training output muscle activity representation and the training muscle activity representation, the loss value of the first preset network is calculated using a first loss function; The parameters of the first preset network are corrected based on the loss value of the first preset network to obtain the first sub-network.

8. The training method according to claim 7, characterized in that, The first preset network includes a 3-layer bidirectional LSTM, each layer of which includes 100 hidden units, dropout=50%, and learning rate of 0.

005.

9. The training method according to claim 5, characterized in that, The step of training the second preset network using the muscle activity representation to obtain the second sub-network includes: The training control information is obtained by training the second preset network using the muscle activity representation described above. Based on the training control information, the loss value of the second preset network is calculated using the second loss function; The second sub-network is obtained by correcting the parameters of the second preset network based on the loss value of the second preset network.

10. The training method according to claim 9, characterized in that, The second preset network includes a 3-layer bidirectional LSTM, each layer of which includes 100 hidden units, dropout=50%, and a learning rate of 0.

005.

11. The training method according to claim 4, characterized in that, The step of concatenating the first sub-network and the second sub-network to obtain the neural network model includes: The first subnetwork and the second subnetwork are connected in series to obtain a series network; The serial network is trained using the EEG training signals to obtain serial output control information; Based on the series output control information, the loss value of the series network is calculated using a third loss function; The parameters of the concatenated network are corrected based on the loss value of the concatenated network to obtain the neural network model.

12. A control device based on electroencephalogram (EEG) signals, characterized in that, The control device includes: The first acquisition module is used to acquire the electroencephalogram (EEG) signals of the target object controlled by the user. A processing module is used to process the electroencephalogram (EEG) signals using a neural network model to obtain control information. The neural network model is trained using EEG training signals and electromyography (EMG) training signals. The neural network model includes a first sub-network and a second sub-network connected in series. Specifically, the processing module is used to: process the EEG signals through the first sub-network to obtain muscle activity representations, the first sub-network being trained using the EEG training signals and EMG training signals; and process the muscle activity representations through the second sub-network to obtain the control information, the second sub-network being trained based on the EMG training signals. The control module is used to control the target object according to the control information.

13. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the control method based on electroencephalogram (EEG) signals according to any one of claims 1-3 or the training method for a neural network model according to any one of claims 4-11.

14. A non-volatile computer-readable storage medium containing a computer program, characterized in that, When the computer program is executed by the processor, the processor performs the control method based on EEG signals as described in any one of claims 1-3 or the training method for the neural network model as described in any one of claims 4-11.

Images