Lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion

Abstract

The invention discloses a lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion, and the system comprises a data obtaining and preprocessing module which is used for obtaining an original EEG electroencephalogram signal data set, a visual recognition data set of an actual scene, and a gait data set of a lower limb exoskeleton rehabilitation robot, preprocessing the original EEG signal data set, and dividing the preprocessed EEG signal data set into a training set and a verification set; the data processing module is used for processing the preprocessed EEG signal data set, the visual identification data set of the actual scene and the gait data set of the lower limb exoskeleton rehabilitation robot, and constructing a weighted Bayesian fusion network to fuse an EEG classification result and a visual classification result to output a final classification result; and the execution module controls the motion state of the lower limb exoskeleton rehabilitation robot based on the final classification result. The classification instruction accuracy and the rehabilitation effect of the lower limb exoskeleton rehabilitation robot controlled by the stroke patient can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation robot control, and in particular to a lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion. Background Art

[0002] "Stroke", also known as "stroke", is an acute cerebrovascular disease. Early rehabilitation training is very helpful for the recovery of motor ability of hemiplegic patients. Therefore, there is a huge market demand for lower limb exoskeleton rehabilitation robots. Traditional passive lower limb exoskeleton rehabilitation robots cannot meet the real-time rehabilitation needs of patients, and the rehabilitation effect is not good. Lower limb exoskeleton rehabilitation robots based on EEG signal control are more ideal solutions. However, due to the instability of command classification caused by unstable EEG signals, a new classification method is needed to enhance the correctness of command output. Therefore, there is an urgent need for a method that can improve command classification and apply it to the control of lower limb exoskeleton rehabilitation robots.

[0003] Traditionally, brain-computer interfaces (BCIs) use brain signals to communicate directly with machines and have been applied in the medical field, such as motor rehabilitation, to help stroke patients recover their physical functions. Since EEG signals are easily interfered by noise and the control of lower limb exoskeleton rehabilitation robots used on stroke patients requires precision, the classification results of the EEGNet convolutional neural network used for EEG signal classification are still somewhat far from the requirements of practical applications in terms of accuracy and need to be further improved, especially for patients with poor EEG signals. This shows that building classification methods and systems for lower limb rehabilitation exoskeletons is still a challenging problem. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a lower limb exoskeleton rehabilitation robot control system based on multimodal fusion, which can effectively improve the accuracy of classification instructions and rehabilitation effect of stroke patients controlling lower limb exoskeleton rehabilitation robots. The present invention can identify motor imagery EEG brain electrical signals and visual signals and obtain a classification result for each, and use a weighted Bayesian fusion network to fuse the two classification results according to the weights to output a final instruction, so that the motion instructions provided to the lower limb exoskeleton rehabilitation robot are more accurate, and can make up for the shortcomings of using only machine vision classification results to cause the patient's subjective rehabilitation intention and using only EEG signals for result classification to cause poor classification results when the EEG signals of some patients are not good, resulting in poor rehabilitation effects for patients.

[0005] To achieve the above purpose, the technical solution provided by the present invention is: a lower limb exoskeleton rehabilitation robot control system based on multimodal fusion, comprising:

[0006] The data acquisition and preprocessing module is used to obtain the original EEG signal dataset, the visual recognition dataset of the actual scene and the gait dataset of the lower limb exoskeleton rehabilitation robot, and preprocess the collected original EEG signal dataset to obtain a clearer, less noisy and stable high-quality EEG signal dataset, and then divide the preprocessed EEG signal dataset into a training set and a validation set according to a set ratio;

[0007] The data processing module is used to process the preprocessed EEG signal dataset, the visual recognition dataset of the actual scene, and the gait dataset of the lower limb exoskeleton rehabilitation robot, and construct a weighted Bayesian fusion network to fuse the EEG classification results and the visual classification results to output the final classification results; in the processing of the EEG signal dataset, the training set is sent to the EEGNet convolutional neural network for training, and the training effect of the network is verified by the verification set to obtain the optimal network, and the optimal network is used for classification in the subsequent EEG classification tasks to obtain the EEG classification results; in the processing of the visual recognition dataset of the actual scene, the collected visual recognition dataset is used The convolutional neural network is input for training to obtain an optimal network, and the optimal network is used for classification in subsequent visual classification tasks to obtain visual classification results; in the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, data points are manually added to the collected gait data to make the discrete data smoother and the gait more standardized, and the processed gait data set is used to standardize the motion gait of the lower limb exoskeleton rehabilitation robot in subsequent motion execution tasks; the weighted Bayesian fusion network introduces a weight factor into the Bayesian fusion network to adjust the contribution of the input signal to the output result of the network, so as to obtain a final classification result that can comprehensively consider the EEG classification result and the visual classification result;

[0008] The execution module controls the motion state of the lower limb exoskeleton rehabilitation robot based on the final classification result output by the data processing module. The control is divided into two steps: the first step is to connect and communicate the devices used in the system, including electroencephalogram devices, visual devices and brain-computers, and set the states for the brain-computers and visual devices respectively in the program, and control the lower limb exoskeleton rehabilitation robot by changing the states of the two; the second step is to set a loop in the program to continuously monitor messages from the visual device, where the visual device message is its state, and different control logics are executed according to the different states monitored. When the initial state is changed, the execution process of the control logic of the changed state will be entered, and finally the instructions will be output to the lower limb exoskeleton rehabilitation robot to perform the corresponding actions and complete the rehabilitation exercise.

[0009] Furthermore, the EEG signal data set is acquired by using a 64-channel EEG device to collect data, and the selected channel list is: FC5, FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CP2, CP4 and CP6; the EEG signal data of each task are collected through the standard paradigm of motor imagery to form a data set, and the data set includes the subject's flat ground walking motor imagery data, stair climbing motor imagery data and stair descending motor imagery data, and each task The process must be repeated several times to obtain sufficient data for subsequent analysis, and the real-time display function should be turned on in the EEG acquisition software to view the EEG waveforms of each channel and preliminarily evaluate the signal quality. That is, a normal EEG signal should have smooth and regular fluctuations, and should not have obvious spikes or violent fluctuations. The preprocessing operation process includes: resampling, filtering, denoising, artifact removal, baseline calibration, data segmentation, feature extraction, feature selection and dimensionality reduction, classification and regression prediction, and result evaluation. The preprocessed EEG signal data set is divided into a training set and a validation set in a ratio of 8:2.

[0010] Furthermore, the visual recognition data set of the actual scene is obtained using ESP32 and OpenMV, wherein ESP32 and OpenMV communicate through a serial port connection, and OpenMV obtains visual data of the actual scene, including data of flat ground, going up stairs and going down stairs, and repeatedly shoots different angles of different terrains to obtain data and form a data set.

[0011] Furthermore, the gait data set of the lower limb exoskeleton rehabilitation robot is acquired using the active rehabilitation mode of the lower limb exoskeleton rehabilitation robot. In this mode, a customizable gait can be achieved by collecting the wearer's gait trajectory data. The wearer performs standardized flat ground walking, stair climbing and stair descending movements in the active rehabilitation mode. The lower limb exoskeleton rehabilitation robot will convert the curved movements during exercise into discrete data points and record them in the interface program.

[0012] Furthermore, in the processing of EEG brain wave signal data set, the EEGNet convolutional neural network has three convolution layers, the first layer is Convolution, that is, ordinary convolution layer operation, called ordinary convolution conv, the second layer is DepthwiseConvolution, that is, channel-by-channel convolution layer operation, called Depthwise conv, and the third layer is separableConvolution, that is, depth separation convolution layer operation, called separable conv, which is composed of a DepthwiseConvolution and a Pointwise Convolution, that is, point-by-point convolution layer operation; a convolution kernel of DepthwiseConvolution is responsible for one channel, that is, one channel is convolved by only one convolution kernel; the shape of the convolution kernel of PointwiseConvolution is 1×1×M×C, M is the number of channels in the previous layer, and C is the number of convolution kernels; the training set is input into the EEGNet convolutional neural network for training, and the training effect of the network is verified by the verification set to obtain the optimal network for classifying EEG brain wave signal data, and the classification result output by the optimal network will be used as one of the data input into the weighted Bayesian fusion network.

[0013] Furthermore, in the processing of visual recognition data sets of actual scenes, OpenMV is used to collect data of actual scenes, including flat ground, going up stairs and going down stairs, and the data is input into the convolutional neural network for training to obtain the optimal network for identifying actual scenes and outputting classification results. ESP32 is used to forward the classification results to the weighted Bayesian fusion network, that is, the classification results output by the optimal network will be used as one of the data input into the weighted Bayesian fusion network.

[0014] Further, in the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, the lower limb exoskeleton rehabilitation robot is selected as the active rehabilitation mode. In this mode, a customizable gait can be achieved by collecting the wearer's gait trajectory data. There are three rehabilitation modes set and classified: walking on flat ground is category "A", climbing stairs is category "B" and descending stairs is category "C"; the trajectory data of the wearer's three different movement gaits are collected through the data collection function of the lower limb exoskeleton rehabilitation robot, and the collected data are written into the interface of the active rehabilitation mode, and the corresponding calling conditions are set, that is, when receiving the control instruction of "A", the data of walking on flat ground is called; when receiving the control instruction of "B", the data of climbing stairs is called; when receiving the control instruction of "C", the data of descending stairs is called; then, the lower limb exoskeleton rehabilitation robot can call the corresponding data to perform the corresponding action by receiving the corresponding instruction; at the same time, a stop movement instruction "0" is also provided, that is, when the control instruction of "0" is received, the lower limb exoskeleton rehabilitation robot will stop moving; since the collected data are discrete data points, the discrete data points are made smoother on the whole by manually filling in points.

[0015] Furthermore, the classification results input into the weighted Bayesian fusion network include the EEG classification results output by the EEGNet convolutional neural network and the visual classification results output by the OpenMV. Before inputting into the weighted Bayesian fusion network, weights are added to the EEG classification results and the visual classification results, and weight factors are introduced to adjust the contribution of the two classification results to the weighted Bayesian fusion network, so as to obtain the final classification result that can comprehensively consider the EEG classification results and the visual classification results, wherein the EEGNet convolutional neural network is the EEG classifier, and OpenMV is the OpenMV visual classifier; the fusion criterion used in the weighted Bayesian fusion network is the maximum a posteriori probability criterion, and the meanings of the symbols are as follows:

[0016] Since the lower limb exoskeleton rehabilitation robot has three movement rehabilitation modes: walking on flat ground, climbing stairs and descending stairs, there are three corresponding classification results, walking on flat ground is classification A, climbing stairs is classification B, and descending stairs is classification C, so the EEG classifier and OpenMV visual classifier are both three classifiers, where the probability of the EEG classifier outputting A is P1, the probability of outputting B is P2, and the probability of outputting C is P3; the probability of the OpenMV visual classifier outputting A is P1′, the probability of outputting B is P2′, and the probability of outputting C is P3′; the EEG classifier accounts for ω1, and the OpenMV visual classifier accounts for ω2, where ω1+ω2=1, 0<ω1≤ω2<1;

[0017] S1 is the EEG classifier, hereinafter referred to as classifier 1; S2 is the OpenMV visual classifier, hereinafter referred to as classifier 2;

[0018] P(A|S1,S2) represents the probability of occurrence of category A given the observations of classifiers 1 and 2;

[0019] P(B|S1,S2) represents the probability of occurrence of category B given the observations of classifiers 1 and 2;

[0020] P(C|S1,S2) represents the probability of category C occurring given the observations of classifiers 1 and 2;

[0021] P(A) represents the prior probability of category A, that is, the probability of category A occurring in the absence of any observations;

[0022] P(B) represents the prior probability of category B, that is, the probability of category B occurring in the absence of any observations;

[0023] P(C) represents the prior probability of category C, that is, the probability of category C occurring in the absence of any observations;

[0024] P(S1,S2|A) represents the joint observation probability of classifier 1 and classifier 2 in the case of category A;

[0025] P(S1,S2|B) represents the joint observation probability of classifier 1 and classifier 2 in the case of category B;

[0026] P(S1,S2|C) represents the joint observation probability of classifier 1 and classifier 2 in the case of category C;

[0027] P(S1,S2) represents the normalization constant, which is used to ensure that the sum of the posterior probabilities is 1;

[0028] The specific steps of weighted Bayesian fusion network are as follows:

[0029] 1) Define prior probability:

[0030] The events set in the scene are a section of walking on flat ground, a section of climbing stairs, and a section of descending stairs. The probability of the three events occurring is the same, that is, the prior probabilities are equal, that is, P(A)=P(B)=P(C);

[0031] 2) Calculate the weighted joint probability:

[0032]

[0033] In the formula, The probability P1 of the EEG classifier output A is expressed as the power of the weight ω1 of the EEG classifier; The probability P2 of the EEG classifier output B is expressed as the power of the weight ω1 of the EEG classifier; The probability P3 of the EEG classifier output C is expressed as the power of the weight ω1 of the EEG classifier; The probability P1′ of the EEG classifier outputting A is expressed as the power of the weight ω2 of the EEG classifier; The probability P2′ of the EEG classifier outputting A is expressed as the power of the weight ω2 of the EEG classifier; The probability P3′ that the EEG classifier outputs A is expressed as the power of the weight ω2 of the EEG classifier.

[0034] The reason why the weighted Bayesian fusion uses the power function instead of directly multiplying the weights is that it can more reasonably reflect the importance of different classifiers. The use of the power function can achieve nonlinear adjustment, so that the classifier with a large weight has a greater impact on the final result. In addition, the power operation is mathematically reasonable and can better reflect the changes in probability distribution.

[0035] 3) Calculate the normalization constant:

[0036] P(S1,S2)=P(S1,S2|A)·P(A)+P(S1,S2|B)·P(B)+P(S1,S2|C)·P(C)

[0037] 4) Calculate the posterior probability:

[0038]

[0039] Finally, the weighted Bayesian fusion network compares the posterior probabilities of the three and outputs the category with the largest posterior probability.

[0040] Furthermore, the brain-computer, ESP32 and OpenMV, lower limb exoskeleton rehabilitation robot and EEG device used in the system are connected, including connecting the brain-computer to the WIFI of the lower limb exoskeleton rehabilitation robot, where the brain-computer is the current host that is currently running the program; then, configure the IP address and subnet mask of the current host so that the two are in the same local area network, that is, the current host and the lower limb exoskeleton rehabilitation robot achieve communication conditions; then set the configuration file, which includes the IP address of the EEG device, port number port, sampling rate Sampling Rate, EEG electrode location information Chanlocs and ESP32's IP address and port number port, etc., to facilitate smooth communication between the current host, ESP32 and OpenMV and the lower limb exoskeleton rehabilitation robot; initialize a DataServerThread thread in the program, which is responsible for establishing a TCP / IP connection with the EEG device and receiving data, and calls the connect() method to try to establish a TCP connection with the EEG device to ensure that EEG signal data can be received; then, the current host and OpenMV try to establish a connection. Since the two cannot communicate directly wirelessly, ESP32 is used as the main control board to act as a router to connect to the current host, where OpenMV and ESP32 use serial port connection for serial communication; ESP32 establishes a TCP / IP connection with the current host, and then the host sends a status message "connect_server" to the OpenMV device through the socket, informing that the machine is ready to receive data; followed by receiving the response , waiting for the response of the OpenMV device. Only when the returned information is in the "ready" state, it indicates that the connection is successful. When the connection is successful, the program will enable a new thread data_handler to process data and logic by calling the function "handle_data". The parameters that this function needs to accept include: OpenMV's TCP socket, DataServerThread thread, EEG device information and EEGNet convolutional neural network for EEG classification. In the TCP / IP connection established between the lower limb exoskeleton rehabilitation robot and the brain-computer computer, the lower limb exoskeleton rehabilitation robot acts as a server to receive control instructions from the brain-computer computer. The physical gimbal equipped with OpenMV includes a large shell, a large shell cover, a small shell and a small shell cover. The large shell is hollow, and ESP32 and OpenMV are placed in the small shell. The large shell and the large shell cover are fixed by screws, and the small shell and the small shell cover are fixed by screws. The large shell and the small shell are connected by an adjustable angle bracket. The angle of the camera can be adjusted by adjusting the bracket to accurately identify the landmark.The gimbal is fixed in front of the waist and abdomen through the waist belt, and the large shell is in contact with the waist and abdomen to alleviate the displacement of the entire gimbal caused by the body during the action, which causes the movement of the camera position. The angle of the camera in the small shell is adjusted by adjusting the bracket to achieve the best recognition angle. ;

[0041] Furthermore, after the brain-computer, ESP32 and OpenMV, lower limb exoskeleton rehabilitation robot and EEG device establish communication, the control logic is designed to control the movement of the lower limb exoskeleton rehabilitation robot, as follows:

[0042] First, set a global variable "computer_status" to track the status of the brain-computer computer, then set it to continuously monitor the messages from OpenMV, that is, the loop of its status is an infinite loop to ensure the sustainable control of the lower limb exoskeleton rehabilitation robot movement, and execute different logics according to the different states of the monitored OpenMV; OpenMV has four states: "recording_data", "stop_walking", "finish", and "get_computer_status". First, set the brain-computer computer state to "stand_by", then the program sends the brain-computer computer state to ESP32. After receiving the information, ESP32 will poll the OpenMV state to see if it has changed; when the OpenMV state changes, the following four situations will occur:

[0043] In the first case, it changes to "recording_data". At this time, the program sets the brain-computer computer state to "counting_data", that is, the data collection state. Then the brain-computer computer prompts the wearer of the lower limb exoskeleton rehabilitation robot to "start imagining", that is, to start collecting the wearer's EEG signal data. After preprocessing the collected EEG signal data, the preprocessed data is classified using the optimal EEGNet convolutional neural network obtained after training, and the EEG classification results are output, waiting to be input into the weighted Bayesian fusion network; next, the program sets the brain-computer computer state to "waiting-OpenMV_data", that is, waiting for the OpenMV visual classification result state; the visual classification results output by OpenMV are forwarded through ESP32 After being given to the brain-computer computer, the program inputs the EEG classification results and visual classification results into the weighted Bayesian fusion network, which will fuse the input classification results; when the weighted Bayesian fusion network outputs the fused classification results, the program sets the brain-computer computer state to "waiting_robot", that is, waiting to send the classification results to the lower limb exoskeleton rehabilitation robot state, and then connects the brain-computer computer and the lower limb exoskeleton rehabilitation robot to communicate by creating a TCP / IP socket; after the connection is successful, the brain-computer computer will send the classification results output by the weighted Bayesian fusion network to the lower limb exoskeleton rehabilitation robot. The classification results are the instructions for controlling the movement of the lower limb exoskeleton rehabilitation robot. The lower limb exoskeleton rehabilitation robot calls the corresponding movement gait data according to the received instructions and executes the corresponding rehabilitation movement gait;

[0044] In the second case, it changes to "stop_walking", and then the brain-computer computer and the lower limb exoskeleton rehabilitation robot are connected and communicated by creating a TCP / IP socket. After the lower limb exoskeleton rehabilitation robot and the brain-computer computer are connected, the program sets the state of the brain-computer computer to "stand_by", and then the brain-computer computer sends the number "0" to the lower limb exoskeleton rehabilitation robot, and the lower limb exoskeleton rehabilitation robot stops moving after receiving the command;

[0045] In the third case, it changes to "finish", and then the brain-computer computer and the lower limb exoskeleton rehabilitation robot are connected and communicated by creating a TCP / IP socket. After the lower limb exoskeleton rehabilitation robot and the brain-computer computer are connected, the program sets the state of the brain-computer computer to "finish", and then the brain-computer computer sends the number "0" to the lower limb exoskeleton rehabilitation robot. After receiving the instruction, the lower limb exoskeleton rehabilitation robot stops moving and stops the DataServerThread thread, terminating the connection with the EEG device and data reception, and the program stops running;

[0046] In the fourth case, it becomes "get_computer_status", and the program returns the current status of the brain-computer computer;

[0047] The control program is provided with infinite loop exit conditions, namely the second and third cases, and the program can also be terminated manually; a switch button is also provided on the lower limb exoskeleton rehabilitation robot, which can actively stop the movement of the lower limb exoskeleton rehabilitation robot.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] 1. The present invention introduces visual classification to assist EEG classification and adopts a weighted Bayesian fusion network to fuse the two classification results. Reasonable weights are added to the EEG classification results and the visual classification results to adjust the contribution to the classification results output by the weighted Bayesian fusion network, so as to make up for the deficiency of using only visual classification results to reduce the subjective rehabilitation intention of patients and the deficiency of using only EEG signal classification results to lead to poor classification results when the EEG signals of some patients are not good, resulting in poor rehabilitation effect for patients.

[0050] 2. The present invention designs a complete interactive process among EEG, vision and lower limb exoskeleton rehabilitation robot modules. The program continuously polls the status of OpenMV, modifies the status of the brain-computer according to the different status of OpenMV, and executes different control logics. Finally, it outputs instructions to the lower limb exoskeleton rehabilitation robot to perform corresponding actions. The process design is reasonable and comprehensive, which can effectively improve the patient's rehabilitation effect.

[0051] 3. Compared with the control systems of other lower limb exoskeleton rehabilitation robots, the overall rehabilitation process design of the present invention is more reasonable. The motion trajectory of the lower limb exoskeleton rehabilitation robot can be modified according to actual conditions and rehabilitation feedback to achieve better rehabilitation effects and stronger subjective rehabilitation awareness. Therefore, it has a wide range of uses in actual scenarios and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the system of the present invention.

[0053] Figure 2 This is an actual scene diagram of the present invention.

[0054] Figure 3 Flowchart of the execution module.

[0055] Figure 4 Design drawing for large shell.

[0056] Figure 5 Design drawing for the large shell casing.

[0057] Figure 6 Design drawing for small shell.

[0058] Figure 7 Design drawing for small shell housing.

[0059] Figure 8 It is a real effect diagram of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0061] The brain-computer configuration of this embodiment adopts: R7 7840H processor, 16GB memory, NVIDIA GeForceGTX4060 graphics card, Windows operating system; the lower limb exoskeleton rehabilitation robot uses the BES-PRO lower limb exoskeleton rehabilitation robot of Aosha Intelligent Company; the EEG device uses the NeuSen W series wireless EEG acquisition system developed by Bo Ruikang Company.

[0062] See also Figure 1 As shown, this embodiment discloses a lower limb exoskeleton rehabilitation robot control system based on multimodal fusion, including the following modules:

[0063] The data acquisition and preprocessing module is used to obtain the original EEG signal dataset, the visual recognition dataset of the actual scene and the gait dataset of the lower limb exoskeleton rehabilitation robot, and preprocess the collected original EEG signal dataset to obtain a clearer, less noisy and stable high-quality EEG signal dataset, and then divide the preprocessed EEG signal dataset into a training set and a validation set according to a set ratio;

[0064] The data processing module is used to process the preprocessed EEG signal dataset, the visual recognition dataset of the actual scene, and the gait dataset of the lower limb exoskeleton rehabilitation robot, and construct a weighted Bayesian fusion network to fuse the EEG classification results and the visual classification results to output the final classification results; in the processing of the EEG signal dataset, the training set is sent to the EEGNet convolutional neural network for training, and the training effect of the network is verified by the verification set to obtain the optimal network, and the optimal network is used for classification in the subsequent EEG classification tasks to obtain the EEG classification results; in the processing of the visual recognition dataset of the actual scene, the collected visual recognition dataset is used The convolutional neural network is input for training to obtain an optimal network, and the optimal network is used for classification in subsequent visual classification tasks to obtain visual classification results; in the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, data points are manually added to the collected gait data to make the discrete data smoother and the gait more standardized, and the processed gait data set is used to standardize the motion gait of the lower limb exoskeleton rehabilitation robot in subsequent motion execution tasks; the weighted Bayesian fusion network introduces a weight factor into the Bayesian fusion network to adjust the contribution of the input signal to the output result of the network, so as to obtain a final classification result that can comprehensively consider the EEG classification result and the visual classification result;

[0065] The execution module controls the motion state of the lower limb exoskeleton rehabilitation robot based on the final classification result output by the data processing module. The control is divided into two steps: the first step is to connect and communicate the devices used in the system, including electroencephalogram devices, visual devices and brain-computers, and set the states for the brain-computers and visual devices respectively in the program, and control the lower limb exoskeleton rehabilitation robot by changing the states of the two; the second step is to set a loop in the program to continuously monitor the messages from the visual devices, where the visual device messages are its states, and execute different control logics according to the different states monitored. When the initial state is changed, the execution process of the control logic of the changed state will be entered, and finally the instructions will be output to the lower limb exoskeleton rehabilitation robot to perform the corresponding actions and complete the rehabilitation exercise.

[0066] Specifically, in the data acquisition and preprocessing module, the acquisition of the EEG electroencephalogram signal data set uses a 64-channel electroencephalogram device to collect data, and the selected channel list is: FC5, FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CP2, CP4 and CP6; the EEG electroencephalogram signal data of each task are collected through the standard paradigm of motor imagery to form a data set, and the data set includes the subject's ground walking motor imagery data, stair climbing motor imagery data and stair descending motor imagery data. Each task is repeated multiple times to obtain sufficient data for subsequent analysis, and the real-time display function is turned on in the EEG acquisition software to view the EEG waveform of each channel, and preliminarily evaluate the signal quality, that is, a normal EEG signal should be smooth and regularly fluctuating, and should not have obvious spikes or violent fluctuations. The preprocessing operation flow includes: resampling, filtering, denoising, artifact removal, baseline calibration, data segmentation, feature extraction, feature selection and dimensionality reduction, classification and regression prediction and result evaluation. The preprocessed EEG signal data set is divided into a training set and a validation set in a ratio of 8:2. The acquisition of the visual recognition data set of the actual scene uses ESP32 and OpenMV, where ESP32 and OpenMV communicate through a serial port connection. OpenMV obtains the visual data of the actual scene, including data on flat ground, stairs, and stairs. It repeatedly shoots different angles of different terrains to obtain data and form a data set. The acquisition of the gait data set of the lower limb exoskeleton rehabilitation robot uses the active rehabilitation mode of the lower limb exoskeleton rehabilitation robot. In this mode, a customizable gait can be achieved by collecting the wearer's gait trajectory data. The wearer performs standardized flat ground walking, stairs, and stairs in the active rehabilitation mode. The lower limb exoskeleton rehabilitation robot will convert the curved motion during movement into discrete data points and record them in the interface program.

[0067] Specifically, in the data processing module, in the processing of the EEG brain wave signal data set, the EEGNet convolutional neural network has three convolution layers, the first layer is Convolution, that is, an ordinary convolution layer operation, called ordinary convolution conv, the second layer is Depthwise Convolution, that is, a channel-by-channel convolution layer operation, called Depthwise conv, and the third layer is separable Convolution, that is, a depth separation convolution layer operation, called separable conv, which is composed of a Depthwise Convolution and a Pointwise Convolution, that is, a point-by-point convolution layer operation; a convolution kernel of Depthwise Convolution is responsible for one channel, that is, a channel is convolved by only one convolution kernel; the shape of the convolution kernel of Pointwise Convolution is 1×1×M×C, M is the number of channels in the previous layer, and C is the number of convolution kernels; the EEGNet convolutional neural network is composed of 1 ordinary convolution conv+1 Depthwise conv+1 separableconv, wherein the separable Convolution is composed of a Depthwise Convolution and a Pointwise Convolution. This construction method can significantly reduce the number of parameters of the convolutional layer. The training set is input into the EEGNet convolutional neural network for training, and the training effect of the network is verified by the validation set to obtain the optimal network for classifying the EEG signal data. The classification results output by the optimal network will be used as one of the data input to the weighted Bayesian fusion network. In the processing of the visual recognition data set of the actual scene, OpenMV is used to collect data of the actual scene, including flat ground, stairs and stairs, and the data is input into the convolutional neural network for training to obtain the optimal network for identifying the actual scene and outputting the classification results. The classification results are forwarded to the weighted Bayesian fusion network using ESP32, that is, the classification results output by the optimal network will be used as one of the data input to the weighted Bayesian fusion network. In the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, the lower limb exoskeleton rehabilitation robot is selected as the active rehabilitation mode. In this mode, a customizable gait can be achieved by collecting the wearer's gait trajectory data. There are three types of rehabilitation modes and they are classified: flat ground walking is category "A", stairs climbing is category "B" and stairs descending is category "C".The data collection function of the lower limb exoskeleton rehabilitation robot collects the trajectory data of the wearer's three different gaits, and writes the collected data into the interface of the active rehabilitation mode, setting the corresponding calling conditions, that is, when receiving the control command "A", call the data of walking on the flat ground; when receiving the control command "B", call the data of going up stairs; when receiving the control command "C", call the data of going down stairs. Then the lower limb exoskeleton rehabilitation robot can call the corresponding data to perform the corresponding action by receiving the corresponding command; at the same time, a stop movement command "0" is also set, that is, when receiving the control command "0", the lower limb exoskeleton rehabilitation robot will stop moving, and the actual scene of the example is as follows. Figure 2 As shown. Since the collected data are discrete data points, the discrete data points are manually filled in to make them smoother overall. The classification results input into the weighted Bayesian fusion network include the EEG classification results output by the EEGNet convolutional neural network and the visual classification results output by OpenMV. Before inputting into the weighted Bayesian fusion network, weights are added to the EEG classification results and the visual classification results. Weight factors are introduced to adjust the contribution of these two classification results to the weighted Bayesian fusion network, so as to obtain the final classification result that comprehensively considers the EEG classification results and the visual classification results. The EEGNet convolutional neural network is the EEG classifier, and OpenMV is the OpenMV visual classifier. The fusion criterion used in the weighted Bayesian fusion network is the maximum a posteriori probability criterion. The meanings of the symbols are as follows:

[0068] The lower limb exoskeleton rehabilitation robot of the present invention has three motion rehabilitation modes: walking on flat ground, climbing stairs and descending stairs, corresponding to three classification results, walking on flat ground is classification A, climbing stairs is classification B, and descending stairs is classification C, so the EEG classifier and the OpenMV visual classifier are both three classifiers, among which the probability of the EEG classifier outputting A is P1, the probability of outputting B is P2, and the probability of outputting C is P3; the probability of the OpenMV visual classifier outputting A is P1′, the probability of outputting B is P2′, and the probability of outputting C is P3′; the EEG classifier accounts for ω1, and the OpenMV visual classifier accounts for ω2, among which ω1+ω2=1, 0<ω1≤ω2<1.

[0069] S1 is the EEG classifier, hereinafter referred to as classifier 1; S2 is the OpenMV visual classifier, hereinafter referred to as classifier 2;

[0070] P(A|S1,S2) represents the probability of occurrence of category A given the observations of classifiers 1 and 2;

[0071] P(B|S1,S2) represents the probability of occurrence of category B given the observations of classifiers 1 and 2;

[0072] P(C|S1,S2) represents the probability of category C occurring given the observations of classifiers 1 and 2;

[0073] P(A) represents the prior probability of category A, that is, the probability of category A occurring in the absence of any observations;

[0074] P(B) represents the prior probability of category B, that is, the probability of category B occurring in the absence of any observations;

[0075] P(C) represents the prior probability of category C, that is, the probability of category C occurring in the absence of any observations;

[0076] P(S1,S2|A) represents the joint observation probability of classifier 1 and classifier 2 in the case of category A;

[0077] P(S1,S2|B) represents the joint observation probability of classifier 1 and classifier 2 in the case of category B;

[0078] P(S1,S2|C) represents the joint observation probability of classifier 1 and classifier 2 in the case of category C;

[0079] P(S1,S2) represents the normalization constant, which is used to ensure that the sum of the posterior probabilities is 1;

[0080] The specific steps of weighted Bayesian fusion network are as follows:

[0081] 1) Define prior probability:

[0082] The setting events of the scene are a section of walking on flat ground, a section of climbing stairs, and a section of descending stairs. The probability of the three events occurring is the same, that is, the prior probabilities are equal, that is, P(A) = P(B) = P(C);

[0083] 2) Calculate the weighted joint probability:

[0084]

[0085]

[0086] in, The probability P1 of the EEG classifier output A is expressed as the power of the weight ω1 of the EEG classifier; The probability P2 of the EEG classifier output B is expressed as the power of the weight ω1 of the EEG classifier; The probability P3 of the EEG classifier output C is expressed as the power of the weight ω1 of the EEG classifier; The probability P1′ of the EEG classifier outputting A is expressed as the power of the weight ω2 of the EEG classifier; The probability P2′ of the EEG classifier outputting A is expressed as the power of the weight ω2 of the EEG classifier; The probability P3′ of the EEG classifier outputting A is expressed as the power of the weight ω2 of the EEG classifier;

[0087] The reason why weighted Bayesian fusion uses power function instead of directly multiplying weights is that it can more reasonably reflect the importance of different classifiers. The use of power function can achieve nonlinear adjustment, so that classifiers with large weights have a greater impact on the final result. In addition, power operation is mathematically reasonable and can better reflect the changes in probability distribution.

[0088] 3) Calculate the normalization constant:

[0089] P(S1,S2)=P(S1,S2|A)·P(A)+P(S1,S2|B)·P(B)+P(S1,S2|C)·P(C)

[0090] 4) Calculate the posterior probability:

[0091]

[0092] Finally, the weighted Bayesian fusion network compares the posterior probabilities of the three and outputs the category with the largest posterior probability.

[0093] Specifically, in the execution module, the process of this module is shown in Figure 3As shown, the control is divided into two steps: the first step is to connect the brain-computer computer, ESP32 and OpenMV, the lower limb exoskeleton rehabilitation robot and the EEG device used in the system, including connecting the brain-computer computer to the WIFI of the lower limb exoskeleton rehabilitation robot, where the brain-computer computer is the current host currently running the program. Then configure the IP address and subnet mask of the current host so that the two are in the same local area network, that is, the current host and the lower limb exoskeleton rehabilitation robot reach communication conditions; then set the configuration file, which includes the IP address of the EEG device, port number port, sampling rate Sampling Rate, EEG electrode location information Chanlocs and ESP32 IP address and port number port and other information, so that the current host, ESP32 and OpenMV and the lower limb exoskeleton rehabilitation robot can communicate smoothly. Initialize a DataServerThread thread in the program, which is responsible for establishing a TCP / IP connection with the EEG device and receiving data. Call the connect() method to try to establish a TCP connection with the EEG device to ensure that EEG signal data can be received; then the current host and OpenMV try to establish a connection. Since there is no direct wireless communication between the two, ESP32 is used as the main control board to act as a router to connect to the current host. OpenMV and ESP32 use serial port connection for serial communication; ESP32 establishes a TCP / IP connection with the current host, and then the host sends a status message "connect_server" to the OpenMV device through the socket, informing that the machine is ready to receive data; then accept the response and wait for the response of the OpenMV device. Only when the returned information is in the "ready" state, it indicates that the connection is successful. When the connection is successful, the program will enable a new thread data_handler, which processes data and logic by calling the function "handle_data". The parameters that this function needs to accept include: OpenMV's TCP socket, DataServerThread thread, EEG device information, and EEGNet convolutional neural network for EEG classification; in the TCP / IP connection established between the lower limb exoskeleton rehabilitation robot and the brain-computer computer, the lower limb exoskeleton rehabilitation robot acts as a server to receive control instructions from the brain-computer computer; the physical gimbal equipped with OpenMV includes a large shell, a large shell cover, a small shell and a small shell cover. The large shell is hollow, and ESP32 and OpenMV are placed in the small shell. The large shell and the large shell cover are fixed with screws, the small shell and the small shell cover are fixed with screws, and the large shell and the small shell are connected by an adjustable angle bracket. The angle of the camera can be adjusted by adjusting the bracket to accurately identify landmarks. The physical design is as follows Figure 4 , Figure 5 , Figure 6 , Figure 7 The gimbal is fixed in front of the waist and abdomen through the waist belt, and the large shell is in contact with the waist and abdomen to alleviate the displacement of the entire gimbal caused by the body during the action, which causes the movement of the camera position. The angle of the camera in the small shell is adjusted by adjusting the bracket to achieve the best recognition angle. The actual effect of the example is shown in Figure 8 shown.

[0094] After the brain-computer, ESP32 and OpenMV, lower limb exoskeleton rehabilitation robot and EEG device establish communication, design the control logic to control the movement of the lower limb exoskeleton rehabilitation robot.

[0095] First, set a global variable "computer_status" to track the status of the brain-computer computer, then set it to continuously monitor the messages from OpenMV, that is, the loop of its status is an infinite loop to ensure the sustainable control of the lower limb exoskeleton rehabilitation robot movement, and execute different logics according to the different states of the monitored OpenMV; OpenMV has four states: "recording_data", "stop_walking", "finish", and "get_computer_status". First, set the brain-computer computer state to "stand_by", then the program sends the brain-computer computer state to ESP32. After receiving the information, ESP32 will poll the OpenMV state to see if it has changed; when the OpenMV state changes, the following four situations will occur:

[0096] In the first case, it changes to "recording_data". At this time, the program sets the brain-computer computer state to "counting_data", that is, the data collection state. Then the brain-computer computer prompts the wearer of the lower limb exoskeleton rehabilitation robot to "start imagining", that is, to start collecting the wearer's EEG signal data. After preprocessing the collected EEG signal data, the preprocessed data is classified using the optimal EEGNet convolutional neural network obtained after training, and the EEG classification results are output, waiting to be input into the weighted Bayesian fusion network; next, the program sets the brain-computer computer state to "waiting-OpenMV_data", that is, waiting for the OpenMV visual classification result state; the visual classification results output by OpenMV are forwarded through ESP32 After being given to the brain-computer computer, the program inputs the EEG classification results and visual classification results into the weighted Bayesian fusion network, which will fuse the input classification results; when the weighted Bayesian fusion network outputs the fused classification results, the program sets the brain-computer computer state to "waiting_robot", that is, waiting to send the classification results to the lower limb exoskeleton rehabilitation robot state, and then connects the brain-computer computer and the lower limb exoskeleton rehabilitation robot to communicate by creating a TCP / IP socket; after the connection is successful, the brain-computer computer will send the classification results output by the weighted Bayesian fusion network to the lower limb exoskeleton rehabilitation robot. The classification results are the instructions for controlling the movement of the lower limb exoskeleton rehabilitation robot. The lower limb exoskeleton rehabilitation robot calls the corresponding movement gait data according to the received instructions and executes the corresponding rehabilitation movement gait;

[0097] In the second case, it changes to "stop_walking", and then the brain-computer computer and the lower limb exoskeleton rehabilitation robot are connected and communicated by creating a TCP / IP socket. After the lower limb exoskeleton rehabilitation robot and the brain-computer computer are connected, the program sets the state of the brain-computer computer to "stand_by", and then the brain-computer computer sends the number "0" to the lower limb exoskeleton rehabilitation robot, and the lower limb exoskeleton rehabilitation robot stops moving after receiving the command;

[0098] In the third case, it changes to "finish", and then the brain-computer computer and the lower limb exoskeleton rehabilitation robot are connected and communicated by creating a TCP / IP socket. After the lower limb exoskeleton rehabilitation robot and the brain-computer computer are connected, the program sets the state of the brain-computer computer to "finish", and then the brain-computer computer sends the number "0" to the lower limb exoskeleton rehabilitation robot. After receiving the instruction, the lower limb exoskeleton rehabilitation robot stops moving and stops the DataServerThread thread, terminating the connection with the EEG device and data reception, and the program stops running;

[0099] In the fourth case, it becomes "get_computer_status", and the program returns the current status of the brain-computer computer;

[0100] The control program is provided with infinite loop exit conditions, namely the second and third cases, and the program can also be terminated manually; a switch button is also provided on the lower limb exoskeleton rehabilitation robot, which can actively stop the movement of the lower limb exoskeleton rehabilitation robot.

[0101] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A lower limb exoskeleton rehabilitation robot control system based on multimodal fusion, characterized in that: include: The data acquisition and preprocessing module is used to obtain the original EEG signal dataset, the visual recognition dataset of the actual scene and the gait dataset of the lower limb exoskeleton rehabilitation robot, and preprocess the collected original EEG signal dataset to obtain a clearer, less noisy and stable high-quality EEG signal dataset, and then divide the preprocessed EEG signal dataset into a training set and a validation set according to a set ratio; The data processing module is used to process the preprocessed EEG signal data set, the visual recognition data set of the actual scene, and the gait data set of the lower limb exoskeleton rehabilitation robot, and construct a weighted Bayesian fusion network to fuse the EEG classification results and the visual classification results to output the final classification results; In the processing of EEG signal data sets, the training set is sent to the EEGNet convolutional neural network for training, and the training effect of the network is verified by the verification set to obtain the optimal network. The optimal network is used for classification in the subsequent EEG classification task to obtain the EEG classification result; in the processing of the visual recognition data set of the actual scene, the collected visual recognition data set is used to input the convolutional neural network for training to obtain the optimal network. The optimal network is used for classification in the subsequent visual classification task to obtain the visual classification result; In the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, data points are manually added to the collected gait data to make the discrete data smoother and the gait more standardized. The processed gait data set is used to standardize the motion gait of the lower limb exoskeleton rehabilitation robot in subsequent motion execution tasks; the weighted Bayesian fusion network introduces weight factors into the Bayesian fusion network to adjust the contribution of the input signal to the output result of the network, so as to obtain the final classification result that can comprehensively consider the EEG classification result and the visual classification result; The execution module controls the motion state of the lower limb exoskeleton rehabilitation robot based on the final classification result output by the data processing module. The control is divided into two steps: the first step is to connect and communicate the devices used in the system, including electroencephalogram devices, visual devices and brain-computers, and set the states for the brain-computers and visual devices respectively in the program, and control the lower limb exoskeleton rehabilitation robot by changing the states of the two; the second step is to set a loop in the program to continuously monitor messages from the visual device, where the visual device message is its state, and different control logics are executed according to the different states monitored. When the initial state is changed, the execution process of the control logic of the changed state will be entered, and finally the instructions will be output to the lower limb exoskeleton rehabilitation robot to perform the corresponding actions and complete the rehabilitation exercise.

2. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 1 is characterized in that: The EEG signal data set is acquired by using a 64-channel EEG device to collect data, and the selected channel list is: FC5, FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CP2, CP4 and CP6; the EEG signal data of each task are collected through the standard paradigm of motor imagery to form a data set, and the data set includes the subject's flat ground walking motor imagery data, stair climbing motor imagery data and stair descending motor imagery data, and each task is important. Repeat several times to obtain enough data for subsequent analysis, and turn on the real-time display function in the EEG acquisition software to view the EEG waveforms of each channel and preliminarily evaluate the signal quality. That is, a normal EEG signal should have smooth and regular fluctuations, and should not have obvious spikes or violent fluctuations. The preprocessing operation process includes: resampling, filtering, denoising, artifact removal, baseline calibration, data segmentation, feature extraction, feature selection and dimensionality reduction, classification and regression prediction, and result evaluation. The preprocessed EEG signal data set is divided into a training set and a validation set in a ratio of 8:

2.

3. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 2 is characterized in that: The acquisition of the visual recognition data set of the actual scene uses ESP32 and OpenMV, wherein ESP32 and OpenMV communicate through a serial port connection, and OpenMV acquires the visual data of the actual scene, including data of flat ground, going up stairs and going down stairs, and repeatedly shoots different angles of different terrains to acquire data and form a data set.

4. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 3 is characterized in that: The gait data set of the lower limb exoskeleton rehabilitation robot is acquired using the active rehabilitation mode of the lower limb exoskeleton rehabilitation robot. In this mode, a customizable gait can be achieved by collecting the wearer's gait trajectory data. The wearer performs standardized flat ground walking, stair climbing and stair descending movements in the active rehabilitation mode. The lower limb exoskeleton rehabilitation robot will convert the curved movements during exercise into discrete data points and record them in the interface program.

5. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 4 is characterized in that: In the processing of EEG brain wave signal data set, the EEGNet convolutional neural network has three convolution layers, the first layer is Convolution, that is, ordinary convolution layer operation, called ordinary convolution conv, the second layer is Depthwise Convolution, that is, channel-by-channel convolution layer operation, called Depthwise conv, and the third layer is separable Convolution, that is, depth separation convolution layer operation, called separable conv, which is composed of a Depthwise Convolution and a Pointwise Convolution, that is, point-by-point convolution layer operation; a convolution kernel of Depthwise Convolution is responsible for one channel, that is, one channel is convolved by only one convolution kernel; the shape of the convolution kernel of Pointwise Convolution is 1×1×M×C, M is the number of channels in the previous layer, and C is the number of convolution kernels; the training set is input into the EEGNet convolutional neural network for training, and the training effect of the network is verified by the verification set to obtain the optimal network for classifying EEG brain wave signal data, and the classification result output by the optimal network will be used as one of the data input into the weighted Bayesian fusion network.

6. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 5 is characterized in that: In the processing of visual recognition data sets in actual scenes, OpenMV is used to collect data on actual scenes, including flat ground, going up stairs and going down stairs, and the data is input into the convolutional neural network for training to obtain the optimal network for identifying actual scenes and outputting classification results. ESP32 is used to forward the classification results to the weighted Bayesian fusion network, that is, the classification results output by the optimal network will be used as one of the data input into the weighted Bayesian fusion network.

7. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 6, characterized in that: In the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, the lower limb exoskeleton rehabilitation robot is selected as the active rehabilitation mode. In this mode, a customizable gait can be achieved by collecting the wearer's gait trajectory data. There are three types of rehabilitation modes set and classified: walking on flat ground is category "A", climbing stairs is category "B" and descending stairs is category "C". The trajectory data of the wearer's three different movement gaits are collected through the data collection function of the lower limb exoskeleton rehabilitation robot, and the collected data are written into the interface of the active rehabilitation mode, and the corresponding calling conditions are set, that is, when receiving the control instruction "A", the data of walking on flat ground is called; when receiving the control instruction "B", the data of climbing stairs is called; when receiving the control instruction "C", the data of descending stairs is called; then, the lower limb exoskeleton rehabilitation robot can call the corresponding data to perform the corresponding action by receiving the corresponding instruction; at the same time, a stop movement instruction "0" is also set, that is, when the control instruction "0" is received, the lower limb exoskeleton rehabilitation robot will stop moving; because the collected data are discrete data points, the discrete data points are made smoother on the whole by manually filling in points.

8. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 7 is characterized in that: The classification results input into the weighted Bayesian fusion network include the EEG classification results output by the EEGNet convolutional neural network and the visual classification results output by OpenMV. Before inputting into the weighted Bayesian fusion network, weights are added to the EEG classification results and the visual classification results. The weight factor is introduced to adjust the contribution of the two classification results to the weighted Bayesian fusion network, so as to obtain the final classification result that can comprehensively consider the EEG classification results and the visual classification results. Among them, the EEGNet convolutional neural network is the EEG classifier, and OpenMV is the OpenMV visual classifier; the fusion criterion used in the weighted Bayesian fusion network is the maximum a posteriori probability criterion, and the meanings of each symbol are as follows: Since the lower limb exoskeleton rehabilitation robot has three movement rehabilitation modes: walking on flat ground, climbing stairs and descending stairs, there are three corresponding classification results, walking on flat ground is classification A, climbing stairs is classification B, and descending stairs is classification C, so the EEG classifier and OpenMV visual classifier are both three classifiers, where the probability of the EEG classifier outputting A is P1, the probability of outputting B is P2, and the probability of outputting C is P3; the probability of the OpenMV visual classifier outputting A is P1′, the probability of outputting B is P2′, and the probability of outputting C is P3′; the EEG classifier accounts for ω1, and the OpenMV visual classifier accounts for ω2, where ω1+ω2=1, 0<ω1≤ω2<1; S1 is the EEG classifier, hereinafter referred to as classifier 1; S2 is the OpenMV visual classifier, hereinafter referred to as classifier 2; P(A|S1,S2) represents the probability of occurrence of category A given the observations of classifiers 1 and 2; P(B|S1,S2) represents the probability of occurrence of category B given the observations of classifiers 1 and 2; P(C|S1,S2) represents the probability of category C occurring given the observations of classifiers 1 and 2; P(A) represents the prior probability of category A, that is, the probability of category A occurring in the absence of any observations; P(B) represents the prior probability of category B, that is, the probability of category B occurring in the absence of any observations; P(C) represents the prior probability of category C, that is, the probability of category C occurring in the absence of any observations; P(S1,S2|A) represents the joint observation probability of classifier 1 and classifier 2 in the case of category A; P(S1,S2|B) represents the joint observation probability of classifier 1 and classifier 2 in the case of category B; P(S1,S2|C) represents the joint observation probability of classifier 1 and classifier 2 in the case of category C; P(S1,S2) represents the normalization constant, which is used to ensure that the sum of the posterior probabilities is 1; The specific steps of weighted Bayesian fusion network are as follows: 1) Define prior probability: The events set in the scene are a section of walking on flat ground, a section of climbing stairs, and a section of descending stairs. The probability of the three events occurring is the same, that is, the prior probabilities are equal, that is, P(A)=P(B)=P(C); 2) Calculate the weighted joint probability: In the formula, The probability P1 of the EEG classifier output A is expressed as the power of the weight ω1 of the EEG classifier; The probability P2 of the EEG classifier output B is expressed as the power of the weight ω1 of the EEG classifier; The probability P3 of the EEG classifier output C is expressed as the power of the weight ω1 of the EEG classifier; The probability P1′ of the EEG classifier outputting A is expressed as the power of the weight ω2 of the EEG classifier; The probability P′2 of the EEG classifier output A is expressed as the power of the weight ω2 of the EEG classifier; The probability P3′ of the EEG classifier outputting A is expressed as the power of the weight ω2 of the EEG classifier; 3) Calculate the normalization constant: P(S1,S2)=P(S1,S2|A)·P(A)+P(S1,S2|B)·P(B)+P(S1,S2|C)·P(C) 4) Calculate the posterior probability: Finally, the weighted Bayesian fusion network compares the posterior probabilities of the three and outputs the category with the largest posterior probability.

9. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 8, characterized in that: Connect the brain-computer, ESP32 and OpenMV, lower limb exoskeleton rehabilitation robot and EEG device used in the system, including connecting the brain-computer to the WIFI of the lower limb exoskeleton rehabilitation robot, where the brain-computer is the current host that is currently running the program; then, configure the IP address and subnet mask of the current host so that the two are in the same local area network, that is, the current host and the lower limb exoskeleton rehabilitation robot can communicate; then set the configuration file, which includes the IP address of the EEG device, port number port, sampling rate Sampling Rate, EEG electrode location information Chanlocs and ESP32's IP address and port number port, etc., to facilitate smooth communication between the current host, ESP32 and OpenMV and the lower limb exoskeleton rehabilitation robot; initialize a DataServerThread thread in the program, which is responsible for establishing a TCP / IP connection with the EEG device and receiving data, and calls the connect() method to try to establish a TCP connection with the EEG device to ensure that EEG signal data can be received; then, the current host and OpenMV try to establish a connection. Since there is no direct wireless communication between the two, ESP32 is used as the main control board to act as a router to connect to the current host, where OpenMV and ESP32 use serial port connection for serial communication; ESP32 establishes a TCP / IP connection with the current host, and then the host sends a status message "connect_server" to the OpenMV device through the socket, informing that the machine is ready to receive data; followed by receiving the response , waiting for the response of the OpenMV device. Only when the returned information is in the "ready" state, it indicates that the connection is successful. When the connection is successful, the program will enable a new thread data_handler, and process the data and logic by calling the function "handle_data". The parameters that this function needs to accept include: OpenMV's TCP socket, DataServerThread thread, EEG device information and EEGNet convolutional neural network for EEG classification. In the TCP / IP connection established between the lower limb exoskeleton rehabilitation robot and the brain-computer computer, the lower limb exoskeleton rehabilitation robot acts as a server to receive control instructions from the brain-computer computer. The physical gimbal equipped with OpenMV includes a large shell, a large shell cover, a small shell and a small shell cover. The large shell is hollow, and ESP32 and OpenMV are placed in the small shell. The large shell and the large shell cover are fixed by screws, and the small shell and the small shell cover are fixed by screws. The large shell and the small shell are connected by an adjustable angle bracket. The angle of the camera can be adjusted by adjusting the bracket to accurately identify the landmark.The gimbal is fixed in front of the waist and abdomen through the waist belt, and the large shell is in contact with the waist and abdomen to alleviate the displacement of the entire gimbal caused by the body during the action, which causes the movement of the camera position. The angle of the camera in the small shell is adjusted by adjusting the bracket to achieve the best recognition angle. ; 10. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 9, characterized in that: After the brain-computer, ESP32 and OpenMV, lower limb exoskeleton rehabilitation robot and EEG device establish communication, design the control logic to control the movement of the lower limb exoskeleton rehabilitation robot, as follows: First, set a global variable "computer_status" to track the status of the brain-computer computer, then set it to continuously monitor the messages from OpenMV, that is, the loop of its status is an infinite loop to ensure the sustainable control of the lower limb exoskeleton rehabilitation robot movement, and execute different logics according to the different states of the monitored OpenMV; OpenMV has four states: "recording_data", that is, recording data state, "stop_walking", that is, stop movement state, "finish", that is, end state, and "get_computer_status", that is, get the state of the brain-computer computer; first, set the brain-computer computer state to "stand_by", that is, wait state, then the program sends the brain-computer computer state to ESP32, and after receiving the information, ESP32 will poll whether the state of OpenMV has changed; when the state of OpenMV changes, the following four situations will occur: In the first case, it becomes "recording_data". At this time, the program sets the brain-computer computer state to "counting_data", that is, the data collection state. Then the brain-computer computer prompts the wearer of the lower limb exoskeleton rehabilitation robot to "start imagining", that is, to start collecting the wearer's EEG signal data. After preprocessing the collected EEG signal data, the optimal EEGNet convolutional neural network obtained after training is used to classify the preprocessed data, output the EEG classification results, and wait for input into the weighted Bayesian fusion network; next, the program sets the brain-computer computer state to "waiting-OpenMV_data", that is, waiting for the OpenMV visual classification result state; the visual classification results output by OpenMV are forwarded through ESP32 After being given to the brain-computer computer, the program inputs the EEG classification results and visual classification results into the weighted Bayesian fusion network, which will fuse the input classification results; when the weighted Bayesian fusion network outputs the fused classification results, the program sets the brain-computer computer state to "waiting_robot", that is, waiting to send the classification results to the lower limb exoskeleton rehabilitation robot state, and then connects the brain-computer computer and the lower limb exoskeleton rehabilitation robot to communicate by creating a TCP / IP socket; after the connection is successful, the brain-computer computer will send the classification results output by the weighted Bayesian fusion network to the lower limb exoskeleton rehabilitation robot. The classification results are the instructions for controlling the movement of the lower limb exoskeleton rehabilitation robot. The lower limb exoskeleton rehabilitation robot calls the corresponding movement gait data according to the received instructions and executes the corresponding rehabilitation movement gait; The second case is "stop_walking". Then, the brain-computer and the lower limb exoskeleton rehabilitation robot are connected and communicated by creating a TCP / IP socket. After the lower limb exoskeleton rehabilitation robot and the brain-computer are connected, the program sets the state of the brain-computer to "stand_by". Then, the brain-computer sends the number "0" to the lower limb exoskeleton rehabilitation robot. After receiving the command, the lower limb exoskeleton rehabilitation robot stops moving. In the third case, it changes to "finish", and then the brain-computer computer and the lower limb exoskeleton rehabilitation robot are connected and communicated by creating a TCP / IP socket. After the lower limb exoskeleton rehabilitation robot and the brain-computer computer are connected, the program sets the state of the brain-computer computer to "finish", and then the brain-computer computer sends the number "0" to the lower limb exoskeleton rehabilitation robot. After receiving the instruction, the lower limb exoskeleton rehabilitation robot stops moving and stops the DataServerThread thread, terminating the connection with the EEG device and data reception, and the program stops running; In the fourth case, it becomes "get_computer_status", and the program returns the current status of the brain-computer computer; The control program is provided with infinite loop exit conditions, namely the second and third cases, and the program can also be terminated manually; a switch button is also provided on the lower limb exoskeleton rehabilitation robot, which can actively stop the movement of the lower limb exoskeleton rehabilitation robot.

Images