Lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion

Abstract

The application discloses a kind of lower limb exoskeleton rehabilitation robot control system based on multimodal fusion, comprising: data acquisition and pre-processing module, for obtaining original EEG electroencephalogram signal dataset, actual scene visual recognition dataset and gait dataset of lower limb exoskeleton rehabilitation robot, and the original EEG electroencephalogram signal dataset is pre-processed, then the EEG electroencephalogram signal dataset after pre-processing is divided into training set and verification set;Data processing module is used to process the EEG electroencephalogram signal dataset after pre-processing, actual scene visual recognition dataset and gait dataset of lower limb exoskeleton rehabilitation robot, and constructs weighted bayesian fusion network to fuse EEG electroencephalogram classification result and visual classification result to output final classification result;Execution module is based on final classification result to control the motion state of lower limb exoskeleton rehabilitation robot.The application can effectively improve the classification instruction accuracy of stroke patients controlling lower limb exoskeleton rehabilitation robot and rehabilitation effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of rehabilitation robot control, in particular to a lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion. BACKGROUND

[0002] "Stroke" is also called "apoplexy", which is an acute cerebrovascular disease. Early rehabilitation training is of great help to the recovery of motor ability of hemiplegic patients, so the lower limb exoskeleton rehabilitation robot has a huge market demand. The traditional passive lower limb exoskeleton rehabilitation robot cannot meet the real-time rehabilitation needs of patients, and the rehabilitation effect is poor. The lower limb exoskeleton rehabilitation robot based on electroencephalogram signal control is a more ideal solution, but due to the instability of the electroencephalogram signal, the instability of the instruction classification is caused, and a new classification method is needed to improve the correctness of the instruction output, therefore, there is an urgent need for a method that can improve the instruction classification to be applied to the control of the lower limb exoskeleton rehabilitation robot.

[0003] Traditionally, brain-computer interface (BCI) realizes direct communication with machines through brain signals, and has been applied to the medical field, such as motor rehabilitation, to help stroke patients achieve physical function recovery. Since EEG signals are easily disturbed by noise, and the control of the lower limb exoskeleton rehabilitation robot applied to stroke patients needs to be accurate, the classification result of the EEGNet convolutional neural network used for EEG signal classification still has a certain distance from the requirements of actual application in terms of accuracy, and needs to be further improved, especially for patients with poor electroencephalogram signals, the accurate classification result is poor, which shows that it is still a challenging problem to construct a classification method and system applied to the lower limb rehabilitation exoskeleton. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion, which can effectively improve the classification instruction accuracy of stroke patients controlling the lower limb exoskeleton rehabilitation robot and the rehabilitation effect. The present application can identify motor imagery EEG brain signals and visual signals and obtain a classification result respectively, and use a weighted Bayesian fusion network to fuse two classification results according to the weight to output a final instruction, so that the movement instruction provided to the lower limb exoskeleton rehabilitation robot is more accurate, and the deficiencies of the subjective rehabilitation intention of the patient caused by using only machine vision classification result and the poor rehabilitation effect of the patient caused by using only electroencephalogram signal for result classification when the electroencephalogram signal of part of the patient is poor can be made up.

[0005] To achieve the above-mentioned purpose, the technical scheme provided by the present application is: a lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion, comprising:

[0006] The data acquisition and preprocessing module is used for acquiring an original EEG brain signal data set, a visual recognition data set of an actual scene and a gait data set of a lower limb exoskeleton rehabilitation robot, and preprocessing the collected original EEG brain signal data set to obtain a high-quality EEG brain signal data set which is clearer, less noisy and more stable, and then dividing the preprocessed EEG brain signal data set into a training set and a verification set according to a set proportion;

[0007] The data processing module is used for processing the preprocessed EEG brain signal data set, the visual recognition data set of the actual scene and the gait data set of the lower limb exoskeleton rehabilitation robot, and constructing a weighted Bayesian fusion network to fuse the EEG brain classification result and the visual classification result to output a final classification result; in the processing of the EEG brain signal data set, the training set is sent into an EEGNet convolutional neural network for training, the training effect of the network is verified through the verification set, so as to obtain an optimal network, and the optimal network is used for classification in a subsequent EEG brain classification task to obtain an EEG brain classification result; in the processing of the visual recognition data set of the actual scene, the collected visual recognition data set is input into a convolutional neural network for training to obtain an optimal network, and the optimal network is used for classification in a subsequent visual classification task to obtain a visual classification result; in the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, data points are manually added to the collected gait data to make the discrete data smoother and the gait more standardized, and the processed gait data set is used to standardize the motion gait of the lower limb exoskeleton rehabilitation robot in a subsequent motion execution task; the weighted Bayesian fusion network is a Bayesian fusion network in which a weight division factor is introduced to adjust the contribution of the input signal to the output result of the network, so as to obtain a final classification result which can comprehensively consider the EEG brain classification result and the visual classification result.

[0008] The execution module controls the motion state of the lower limb exoskeleton rehabilitation robot based on the final classification result output by the data processing module, and controls in two steps: first, the devices used in the system are connected and communicated, including the EEG device, the visual device and the brain-computer computer, and the states of the brain-computer computer and the visual device are set in the program, and the control of the lower limb exoskeleton rehabilitation robot is realized by changing the states; second, a loop for continuously listening to the messages from the visual device is set in the program, wherein the messages from the visual device are the states, different control logics are executed according to the different states, when the initial state is changed, the execution flow of the control logic of the changed state is entered, and finally the instructions are output to the lower limb exoskeleton rehabilitation robot to execute corresponding actions to complete the rehabilitation motion.

[0009] Further, the acquisition of the EEG brain signal data set selects a 64-channel EEG device to collect data, and the selected channel list is: FC5, FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CP2, CP4 and CP6; the EEG brain signal data of each task is collected through the standard paradigm of motor imagery and constitutes a data set, the data set includes motor imagery data of the subject walking on flat ground, motor imagery data of climbing stairs and motor imagery data of descending stairs, each task is repeated multiple times to obtain sufficient data for subsequent analysis, and the real-time display function is opened in the EEG collection software to view the EEG waveform of each channel and preliminarily evaluate the quality of the signal, that is, the normal EEG signal should be smooth and regular fluctuation, and there should be no obvious peak or violent fluctuation, the operation process of pre-processing includes: resampling, filtering, denoising, removing artifacts, baseline calibration, data segmentation, feature extraction, feature selection and dimension reduction, classification and regression prediction and result evaluation, and the pre-processed EEG brain signal data set is divided into a training set and a validation set in a ratio of 8:2.

[0010] Further, the acquisition of the visual recognition data set of the actual scene uses ESP32 and OpenMV, wherein ESP32 and OpenMV communicate through serial connection, OpenMV acquires visual data of the actual scene, including data of flat ground, climbing stairs and descending stairs, and repeatedly shoots different angles of different terrains to acquire data and constitute a data set.

[0011] Further, the acquisition of the gait data set of the lower limb exoskeleton rehabilitation robot uses an active rehabilitation mode of the lower limb exoskeleton rehabilitation robot, in which mode, through the collection of gait trajectory data of the wearer, a customizable gait can be realized, the wearer performs standardized walking on flat ground, climbing stairs and descending stairs in the active rehabilitation mode, and the lower limb exoskeleton rehabilitation robot converts the curve motion during movement into discrete data points and records them in the interface program.

[0012] Further, in the EEG brain electrical signal dataset processing, the EEGNet convolutional neural network has three convolutional layers, the first layer is a Convolution, i.e. a normal convolutional layer operation, referred to as a normal convolution conv, the second layer is a DepthwiseConvolution, i.e. a per-channel convolutional layer operation, referred to as a Depthwise conv, and the third layer is a separableConvolution, i.e. a depth separation convolutional layer operation, referred to as a separable conv, which is composed of a DepthwiseConvolution and a PointwiseConvolution, i.e. a point-by-point convolutional layer operation; a convolution kernel of the DepthwiseConvolution is responsible for a channel, i.e. only one convolution kernel is used for convolution of one channel; the shape of the convolution kernel of the PointwiseConvolution is 1x1xMxC, M is the number of channels of the previous layer, and C is the number of convolution kernels; the training set is input into the EEGNet convolutional neural network for training, and the training effect of the network is verified through the validation set to obtain an optimal network for classifying EEG brain electrical signal data; and the classification result output by the optimal network will be one of the data input into the weighted Bayesian fusion network.

[0013] Further, in the actual scene visual recognition dataset processing, OpenMV is used to collect data of actual scenes, including flat ground, going upstairs and going downstairs, and the data is input into a convolutional neural network for training to obtain an optimal network for recognizing actual scenes and outputting classification results; the classification results are forwarded to the weighted Bayesian fusion network by using ESP32, i.e. the classification results output by the optimal network will be one of the data input into the weighted Bayesian fusion network.

[0014] Further, in the gait data set processing of the lower extremity exoskeleton rehabilitation robot, the lower extremity exoskeleton rehabilitation robot is selected as an active rehabilitation mode, in which mode a customizable gait can be realized by collecting gait trajectory data of the wearer, and three rehabilitation modes are set and classified: walking on flat ground as category A, climbing stairs as category B, and descending stairs as category C; the trajectory data of three different motion gaits of the wearer are collected through the data collection function of the lower extremity exoskeleton rehabilitation robot, and the collected data are written into the interface of the active rehabilitation mode, and the corresponding calling conditions are set, that is, when receiving the control instruction of "A", the data of walking on flat ground are called; when receiving the control instruction of "B", the data of climbing stairs are called; when receiving the control instruction of "C", the data of descending stairs are called; then, the lower extremity exoskeleton rehabilitation robot can execute corresponding actions by receiving corresponding instructions to call corresponding data; at the same time, a stop motion instruction "0" is also set, that is, when receiving the control instruction of "0", the lower extremity exoskeleton rehabilitation robot will stop moving; since the collected data are discrete data points, the discrete data points are made more smooth as a whole by manual point supplementing.

[0015] Further, the classification results input into the weighted Bayesian fusion network include the electroencephalogram classification results output by the EEGNet convolutional neural network and the visual classification results output by the OpenMV, and weights are added to the electroencephalogram classification results and the visual classification results before they are input into the weighted Bayesian fusion network, a weight factor is introduced to adjust the contribution of the two classification results to the weighted Bayesian fusion network, so as to obtain the final classification results that can comprehensively consider the electroencephalogram classification results and the visual classification results, wherein the EEGNet convolutional neural network is an electroencephalogram classifier, and the OpenMV is an OpenMV visual classifier; the fusion criterion used in the weighted Bayesian fusion network is a maximum posterior probability criterion, and the meanings of various symbols are as follows:

[0016] Since the lower extremity exoskeleton rehabilitation robot has three motion rehabilitation modes: walking on flat ground, climbing stairs and descending stairs, there are three classification results, walking on flat ground is classified as A, climbing stairs is classified as B, and descending stairs is classified as C, so the electroencephalogram classifier and the OpenMV visual classifier are both three-classifiers, wherein the probability of the electroencephalogram classifier outputting A is P1, the probability of the electroencephalogram classifier outputting B is P2, and the probability of the electroencephalogram classifier outputting C is P3; the probability of the OpenMV visual classifier outputting A is P1', the probability of the OpenMV visual classifier outputting B is P2', and the probability of the OpenMV visual classifier outputting C is P3'; the proportion of the electroencephalogram classifier is ω1, and the proportion of the OpenMV visual classifier is ω2, wherein ω1+ω2=1, 0<ω1≤ω2<1;

[0017] S1 is an electroencephalogram classifier, hereinafter referred to as classifier 1; S2 is an OpenMV visual classifier, hereinafter referred to as classifier 2;

[0018] P(A|S1, S2) represents the probability of occurrence of class A given the observations of classifier 1 and classifier 2;

[0019] P(B|S1, S2) represents the probability of occurrence of class B given the observations of classifier 1 and classifier 2;

[0020] P(C|S1, S2) represents the probability of occurrence of class C given the observations of classifier 1 and classifier 2;

[0021] P(A) represents the prior probability of class A, i.e. the probability of occurrence of class A without any observation;

[0022] P(B) represents the prior probability of class B, i.e. the probability of occurrence of class B without any observation;

[0023] P(C) represents the prior probability of class C, i.e. the probability of occurrence of class C without any observation;

[0024] P(S1, S2|A) represents the joint observation probability of classifier 1 and classifier 2 given class A;

[0025] P(S1, S2|B) represents the joint observation probability of classifier 1 and classifier 2 given class B;

[0026] P(S1, S2|C) represents the joint observation probability of classifier 1 and classifier 2 given class C;

[0027] P(S1, S2) represents the normalization constant to ensure the sum of posterior probabilities is 1;

[0028] The specific steps of the weighted Bayesian fusion network are as follows:

[0029] 1) Define the prior probability:

[0030] The events set by the scene are a section of flat ground walking, a section of going upstairs and a section of going downstairs, and the probabilities of occurrence of the three events are equal, i.e. the prior probabilities are equal, i.e. P(A) = P(B) = P(C);

[0031] 2) Calculate the weighted joint probability:

[0032]

[0033] In the formula, represents the power of the weight ω1 of the electroencephalogram classifier for the probability P1 of the electroencephalogram classifier output A; represents the power of the weight ω1 of the electroencephalogram classifier for the probability P2 of the electroencephalogram classifier output B; ω1P3 ω2P1' ω2P2' ω2P3'

[0034] The reason why the weighted Bayesian fusion uses the power function instead of directly multiplying the weights is that it more reasonably reflects the importance of different classifiers. The use of the power function can achieve nonlinear adjustment, so that the classifier with a larger weight has a greater impact on the final result. Moreover, the power operation is mathematically reasonable and can better reflect the change of the probability distribution.

[0035] 3) Calculate the normalization constant:

[0036] P(S1,S2)=P(S1,S2|A)·P(A)+P(S1,S2|B)·P(B)+P(S1,S2|C)·P(C)

[0037] 4) Calculate the posterior probability:

[0038]

[0039] Finally, the weighted Bayesian fusion network compares the posterior probabilities of the three, and then outputs the class with the largest posterior probability.

[0040] Further, the brain-computer computer, ESP32, OpenMV, lower extremity exoskeleton rehabilitation robot and electroencephalogram device used by the system are connected, including connecting the brain-computer computer and the WIFI of the lower extremity exoskeleton rehabilitation robot, wherein the brain-computer computer is the current host currently running the program; then, the IP address and subnet mask of the current host are configured so that they are in the same local area network, that is, the current host and the lower extremity exoskeleton rehabilitation robot achieve communication conditions; then the configuration file is set, which includes the IP address of the electroencephalogram device, the port number port, the sampling rate Sampling Rate, the position information Chanlocs of the electroencephalogram electrode and the IP address and port number port of the ESP32 and other information, so as to facilitate the smooth communication between the current host, ESP32, OpenMV and the lower extremity exoskeleton rehabilitation robot; a DataServerThread thread is initialized in the program, which is responsible for establishing TCP / IP connection with the electroencephalogram device and accepting data, calling the connect() method to try to establish TCP connection with the electroencephalogram device to ensure that EEG brain electrical signal data can be accepted; then, the current host and OpenMV try to establish connection, since they cannot directly communicate wirelessly, ESP32 is used as the main control board to act as a router and connect with the current host, wherein OpenMV and ESP32 use serial connection for serial communication; ESP32 establishes TCP / IP connection with the current host, then the host sends a status message "connect_server" to the OpenMV device through the socket socket to inform the local machine that it is ready to accept data; then the response is accepted, waiting for the response of the OpenMV device, only when the returned information is "ready" ready state, the connection is successful; after the connection is successful, a new thread data_handler is enabled, which processes data and logic by calling the function "handle_data", which needs to accept parameters including the TCP socket of OpenMV, the DataServerThread thread, the electroencephalogram device information and the EEGNet convolutional neural network for electroencephalogram classification; in the TCP / IP connection established between the lower extremity exoskeleton rehabilitation robot and the brain-computer computer, the lower extremity exoskeleton rehabilitation robot acts as a server to receive control instructions from the brain-computer computer; the real cloud platform carrying OpenMV includes large shell, large shell cover plate, small shell and small shell cover plate, the large shell is hollow, ESP32 and OpenMV are placed in the small shell, the large shell and the large shell cover plate are fixed by screws, the small shell and the small shell cover plate are fixed by screws, the large shell and the small shell are connected by an adjustable angle support, the angle of the camera is adjusted by adjusting the support to accurately identify the marker;The gimbal is fixed in front of the waist by a belt, and the large shell is in contact with the waist to alleviate the movement of the camera caused by the body during the movement. The angle of the camera in the small shell is adjusted by adjusting the support to achieve the best recognition angle.

[0041] Further, after the brain-computer computer, ESP32, OpenMV, lower limb exoskeleton rehabilitation robot and electroencephalogram device establish communication, the control logic is designed to control the movement of the lower limb exoskeleton rehabilitation robot, as follows:

[0042] First, a global variable "computer_status" is set to track the state of the brain-computer computer, and then a continuous message from OpenMV is set, i.e. the loop of its state is an infinite loop to ensure the sustainability of the control of the lower limb exoskeleton rehabilitation robot movement, and different logic is executed according to the different states of OpenMV; OpenMV has four states: "recording_data" which means recording data state, "stop_walking" which means stop movement state, "finish" which means end state and "get_computer_status" which means get brain-computer computer state state; First, the brain-computer computer state is set to "stand_by" which means waiting state, then the program sends the brain-computer computer state to ESP32, and ESP32 polls the state of OpenMV after receiving the information whether it has changed; When the state of OpenMV changes, the following four situations will occur:

[0043] The first case, i.e. changed to "recording_data", at this time the program sets the brain-computer computer state to "counting_data", i.e. data collection state, then the brain-computer computer prompts the lower extremity exoskeleton rehabilitation robot wearer to "start imagining", i.e. start collecting the wearer's EEG data, after preprocessing the collected EEG data, the optimal EEGNet convolutional neural network obtained after training is used to classify the preprocessed data, and the EEG classification result is output, waiting for input into the weighted Bayesian fusion network; Next, the program sets the brain-computer computer state to "waiting-OpenMV_data", i.e. waiting for OpenMV visual classification result state; After the brain-computer computer receives the visual classification result output by the OpenMV through the ESP32, the program inputs the EEG classification result and the visual classification result into the weighted Bayesian fusion network, which will fuse the input classification results; After the weighted Bayesian fusion network outputs the fused classification result, the program sets the brain-computer computer state to "waiting_robot", i.e. waiting for sending the classification result to the lower extremity exoskeleton rehabilitation robot state, then the brain-computer computer and the lower extremity exoskeleton rehabilitation robot are connected and communicated through the creation of a TCP / IP socket; After the connection is successful, the brain-computer computer sends the classification result output by the weighted Bayesian fusion network to the lower extremity exoskeleton rehabilitation robot, which is the instruction for controlling the motion of the lower extremity exoskeleton rehabilitation robot, and the lower extremity exoskeleton rehabilitation robot calls the corresponding motion gait data according to the received instruction and executes the corresponding rehabilitation motion gait;

[0044] The second case, i.e. changed to "stop_walking", then the brain-computer computer and the lower extremity exoskeleton rehabilitation robot are connected and communicated through the creation of a TCP / IP socket, after the lower extremity exoskeleton rehabilitation robot and the brain-computer computer establish a connection, the program sets the state of the brain-computer computer to "stand_by", then the brain-computer computer sends the number "0" to the lower extremity exoskeleton rehabilitation robot, which will stop moving after receiving the instruction;

[0045] The third case, i.e. changed to "finish", then the brain-computer computer and the lower extremity exoskeleton rehabilitation robot are connected and communicated through the creation of a TCP / IP socket, after the lower extremity exoskeleton rehabilitation robot and the brain-computer computer establish a connection, the program sets the state of the brain-computer computer to "finish", then the brain-computer computer sends the number "0" to the lower extremity exoskeleton rehabilitation robot, which will stop moving and stop the DataServerThread thread, terminate the connection with the EEG device and data reception, and the program stops running;

[0046] The fourth case, i.e. becoming "get_computer_status", the program returns the current state of the brain-computer computer;

[0047] The control program is provided with an exit condition of an infinite loop, i.e. the second case and the third case, and can also be manually terminated; a switch button is also provided on the lower limb exoskeleton rehabilitation robot, and the movement of the lower limb exoskeleton rehabilitation robot can be actively stopped.

[0048] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0049] 1. The present application introduces visual classification to assist electroencephalogram classification and uses a weighted Bayesian fusion network to fuse the two classification results, adds reasonable weights to the electroencephalogram classification results and visual classification results to adjust the contribution of the weighted Bayesian fusion network output classification results, makes up for the deficiency of subjective rehabilitation intention of patients caused by only using visual classification results and the poor classification results caused by using only electroencephalogram classification results when the electroencephalogram of some patients is poor, and the poor rehabilitation effect of patients.

[0050] 2. The present application designs a complete interaction process between electroencephalogram, vision and lower limb exoskeleton rehabilitation robot modules, constantly polls the state of OpenMV through the program, modifies the state of the brain-computer computer according to the obtained different OpenMV states, and executes different control logic, finally outputs instructions to the lower limb exoskeleton rehabilitation robot to execute corresponding actions, the process design is reasonable and considers a wide range of situations, and can effectively improve the rehabilitation effect of patients.

[0051] 3. Compared with other control systems of lower limb exoskeleton rehabilitation robots, the overall rehabilitation process design is more reasonable, the movement trajectory of the lower limb exoskeleton rehabilitation robot can be modified according to the actual situation and rehabilitation feedback to achieve better rehabilitation effect and stronger subjective rehabilitation consciousness, so it has a wide use space in actual scenes and has a broad application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 It is the architecture diagram of the system of the present application.

[0053] Figure 2 It is the actual scene diagram of the present application.

[0054] Figure 3 It is the flowchart of the execution module.

[0055] Figure 4 It is the design drawing of the large shell.

[0056] Figure 5 It is the outer shell design drawing of the large shell.

[0057] Figure 6 Design drawing for small shell.

[0058] Figure 7 Design drawing for small shell housing.

[0059] Figure 8 Actual effect drawing of the present application. DETAILED DESCRIPTION

[0060] The application will be further described in conjunction with the embodiments and the accompanying drawings, but the embodiments of the application are not limited thereto.

[0061] The brain-computer computer configuration of the embodiment adopts: R7 7840H processor, 16GB memory, NVIDIA GeForce GTX4060 graphics card, Windows operating system; the lower limb exoskeleton rehabilitation robot uses BES-PRO lower limb exoskeleton rehabilitation robot of Aoshq Intelligent Company; the electroencephalogram device adopts NeuSen W series wireless electroencephalogram acquisition system developed by Boreikan Company.

[0062] Referring to Figure 1 As shown in the figure, the embodiment discloses a lower limb exoskeleton rehabilitation robot control system based on multi-modal fusion, comprising the following modules:

[0063] The data acquisition and preprocessing module is used for acquiring original EEG electroencephalogram signal data set, visual recognition data set of actual scene and gait data set of lower limb exoskeleton rehabilitation robot, and pre-processing the collected original EEG electroencephalogram signal data set to obtain clearer, less noisy and stable high-quality EEG electroencephalogram signal data set, and then dividing the pre-processed EEG electroencephalogram signal data set into training set and verification set according to the set proportion;

[0064] The data processing module is used for processing the pre-processed EEG brain signal data set, the visual recognition data set of the actual scene and the gait data set of the lower limb exoskeleton rehabilitation robot, and constructing a weighted Bayesian fusion network to fuse the EEG brain classification result and the visual classification result to output the final classification result; in the processing of the EEG brain signal data set, the training set is sent into the EEGNet convolutional neural network for training, the training effect of the network is verified through the verification set, so as to obtain the optimal network, which is used for classification in the subsequent EEG brain classification task to obtain the EEG brain classification result; in the processing of the visual recognition data set of the actual scene, the collected visual recognition data set is input into the convolutional neural network for training to obtain the optimal network, which is used for classification in the subsequent visual classification task to obtain the visual classification result; in the processing of the gait data set of the lower limb exoskeleton rehabilitation robot, the data points are manually added to the collected gait data to make the discrete data more smooth and the gait more standardized, and the processed gait data set is used to standardize the motion gait of the lower limb exoskeleton rehabilitation robot in the subsequent motion execution task; the weighted Bayesian fusion network is a Bayesian fusion network in which a weight division factor is introduced to adjust the contribution of the input signal to the output result of the network, so as to obtain the final classification result which can comprehensively consider the EEG brain classification result and the visual classification result.

[0065] The execution module controls the motion state of the lower limb exoskeleton rehabilitation robot based on the final classification result output by the data processing module, and controls in two steps: the first step is to connect and communicate the devices used in the system, including the EEG device, the visual device and the brain-computer computer, and set the states of the brain-computer computer and the visual device in the program, and control the lower limb exoskeleton rehabilitation robot by changing the states of the two; the second step is to set a loop in the program to continuously listen to the message from the visual device, wherein the message of the visual device is the state thereof, and different control logics are executed according to different states, when the initial state is changed, the execution flow of the control logic of the changed state is entered, finally the instructions are output to the lower limb exoskeleton rehabilitation robot to execute corresponding actions to complete the rehabilitation exercise.

[0066] Specifically, in the data acquisition and preprocessing module, the EEG brain signal data set is acquired by a 64-channel EEG device, and the selected channel list is: FC5, FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CP2, CP4 and CP6; the EEG brain signal data of each task is collected through the standard paradigm of motor imagery and constitutes a data set, the data set includes the motor imagery data of the subject walking on the flat ground, climbing stairs and descending stairs, each task is repeated multiple times to obtain sufficient data for subsequent analysis, and the real-time display function is opened in the EEG acquisition software to view the EEG waveform of each channel and preliminarily evaluate the quality of the signal, that is, the normal EEG signal should be smooth and regular fluctuation, and there should be no obvious peak or violent fluctuation. The operation process of preprocessing includes: resampling, filtering, denoising, removing artifacts, baseline calibration, data segmentation, feature extraction, feature selection and dimension reduction, classification and regression prediction and result evaluation, the EEG brain signal data set after preprocessing is divided into a training set and a validation set in a ratio of 8:2; the acquisition of the visual recognition data set of the actual scene uses ESP32 and OpenMV, wherein ESP32 and OpenMV communicate through serial connection, OpenMV acquires visual data of the actual scene, including data of flat ground, climbing stairs and descending stairs, different angles of different terrains are repeatedly shot to acquire data and constitute a data set; the acquisition of the gait data set of the lower limb exoskeleton rehabilitation robot uses the active rehabilitation mode of the lower limb exoskeleton rehabilitation robot, in this mode, the wearer's gait trajectory data can be collected to realize customizable gait. The wearer performs standardized walking on the flat ground, climbing stairs and descending stairs in the active rehabilitation mode, and the lower limb exoskeleton rehabilitation robot converts the curve motion during movement into discrete data points and records them in the interface program.

[0067] Specifically, in the data processing module, in the EEG brain electrical signal dataset processing, the EEGNet convolutional neural network has three convolutional layers, the first layer is a Convolution, that is, a common convolutional layer operation, called common convolution conv, the second layer is a Depthwise Convolution, that is, a convolutional layer operation of each channel, called Depthwise conv, and the third layer is a separable Convolution, that is, a deep separation convolutional layer operation, called separable conv, which is composed of a Depthwise Convolution and a Pointwise Convolution, that is, a point-by-point convolutional layer operation; a convolution kernel of the Depthwise Convolution is responsible for a channel, that is, only one convolution kernel is used for convolution of one channel; the shape of the convolution kernel of the Pointwise Convolution is 1x1xMxC, M is the number of channels of the previous layer, and C is the number of convolution kernels; the EEGNet convolutional neural network is composed of 1 common convolution conv, 1 Depthwise conv and 1 separable conv, wherein the separable Convolution is composed of a Depthwise Convolution and a Pointwise Convolution. This construction can significantly reduce the number of parameters of the convolutional layer. The training set is input into the EEGNet convolutional neural network for training, and the training effect of the network is verified through the validation set, so as to obtain an optimal network for classifying EEG brain electrical signal data. The classification result output by the optimal network will be one of the data input into the weighted Bayesian fusion network; in the actual scene visual recognition dataset processing, the OpenMV is used to collect the data of the actual scene, including flat ground, upstairs and downstairs, and the data is input into the convolutional neural network for training, so as to obtain an optimal network for identifying the actual scene and outputting a classification result. The classification result is forwarded to the weighted Bayesian fusion network by using the ESP32, that is, the classification result output by the optimal network will be one of the data input into the weighted Bayesian fusion network; in the gait dataset processing of the lower limb exoskeleton rehabilitation robot, the lower limb exoskeleton rehabilitation robot is selected as the active rehabilitation mode, in which mode the wearer's gait trajectory data can be collected to realize the customizable gait, and the rehabilitation mode is set to have three types and is classified: walking on flat ground as category "A", climbing stairs as category "B" and descending stairs as category "C".The data function of the lower extremity exoskeleton rehabilitation robot collects trajectory data of three different motion gaits of the wearer, and writes the collected data into the interface of the active rehabilitation mode, and sets the corresponding calling conditions, that is, when receiving the control instruction of "A", the data of walking on the flat ground is called; when receiving the control instruction of "B", the data of climbing stairs is called; when receiving the control instruction of "C", the data of descending stairs is called. Then the lower extremity exoskeleton rehabilitation robot can execute the corresponding action by receiving the corresponding instruction to call the corresponding data; meanwhile, a stop motion instruction "0" is also set, that is, when receiving the control instruction of "0", the lower extremity exoskeleton rehabilitation robot will stop moving, and the actual scene is shown in the following. Figure 2 The collected data is discrete data points, so the discrete data points are made more smooth as a whole by manual point supplementing; the classification results input into the weighted Bayesian fusion network include the electroencephalogram classification results output by the EEGNet convolutional neural network and the visual classification results output by the OpenMV, and weights are added to the electroencephalogram classification results and the visual classification results before inputting the weighted Bayesian fusion network, a weight factor is introduced to adjust the contribution of the two classification results to the weighted Bayesian fusion network, so as to obtain the final classification results which can comprehensively consider the electroencephalogram classification results and the visual classification results, wherein the EEGNet convolutional neural network is an electroencephalogram classifier, and the OpenMV is an OpenMV visual classifier. The fusion criterion used in the weighted Bayesian fusion network is the maximum posterior probability criterion, and the meanings of various symbols are as follows:

[0068] The lower extremity exoskeleton rehabilitation robot has three motion rehabilitation modes: walking on the flat ground, climbing stairs and descending stairs, and there are three classification results corresponding to the three motion rehabilitation modes, the walking on the flat ground is classification A, the climbing stairs is classification B, and the descending stairs is classification C, so the electroencephalogram classifier and the OpenMV visual classifier are both three-classifiers, wherein the probability of outputting A by the electroencephalogram classifier is P1, the probability of outputting B is P2, and the probability of outputting C is P3; the probability of outputting A by the OpenMV visual classifier is P1', the probability of outputting B is P2', and the probability of outputting C is P3'; the proportion of the electroencephalogram classifier is ω1, and the proportion of the OpenMV visual classifier is ω2, wherein ω1+ω2=1, 0<ω1≤ω2<1.

[0069] S1 is an electroencephalogram classifier, hereinafter referred to as classifier 1; S2 is an OpenMV visual classifier, hereinafter referred to as classifier 2;

[0070] P(A|S1,S2) represents the probability of occurrence of class A given the observation results of the classifier 1 and the classifier 2;

[0071] P(B|S1,S2) represents the probability of occurrence of class B given the observation results of the classifier 1 and the classifier 2;

[0072] P(C|S1, S2) represents the probability of class C given the observations of classifier 1 and classifier 2;

[0073] P(A) represents the prior probability of class A, i.e. the probability of class A without any observation;

[0074] P(B) represents the prior probability of class B, i.e. the probability of class B without any observation;

[0075] P(C) represents the prior probability of class C, i.e. the probability of class C without any observation;

[0076] P(S1, S2|A) represents the joint observation probability of classifier 1 and classifier 2 given class A;

[0077] P(S1, S2|B) represents the joint observation probability of classifier 1 and classifier 2 given class B;

[0078] P(S1, S2|C) represents the joint observation probability of classifier 1 and classifier 2 given class C;

[0079] P(S1, S2) represents the normalization constant to ensure the sum of posterior probabilities is 1;

[0080] The specific steps of the weighted Bayesian fusion network are as follows:

[0081] 1) Define the prior probability:

[0082] The setting events of the scene are a section of flat walking, a section of going upstairs and a section of going downstairs, and the probabilities of the three events are the same, i.e. the prior probabilities are equal, i.e. P(A) = P(B) = P(C);

[0083] 2) Calculate the weighted joint probability:

[0084]

[0085]

[0086] wherein, represents the power of the weight ω1 of the electroencephalogram classifier for the probability P1 of the electroencephalogram classifier output A; represents the power of the weight ω1 of the electroencephalogram classifier for the probability P2 of the electroencephalogram classifier output B; represents the power of the weight ω1 of the electroencephalogram classifier for the probability P3 of the electroencephalogram classifier output C; represents the power of the weight ω2 of the electroencephalogram classifier for the probability P1' of the electroencephalogram classifier output A; ω2 represents the power of the weight of the electroencephalogram classifier in the probability P2' of the electroencephalogram classifier output A; ω2 represents the power of the weight of the electroencephalogram classifier in the probability P3' of the electroencephalogram classifier output A;

[0087] The reason why the weighted Bayesian fusion uses the power function instead of directly multiplying the weights is that it more reasonably reflects the importance of different classifiers, the use of the power function can achieve nonlinear adjustment, so that the classifier with a large weight has a greater impact on the final result, and the power operation is mathematically reasonable and can better reflect the change of the probability distribution;

[0088] 3) Calculate the normalization constant:

[0089] P(S1, S2) = P(S1, S2|A)·P(A) + P(S1, S2|B)·P(B) + P(S1, S2|C)·P(C)

[0090] 4) Calculate the posterior probability:

[0091]

[0092] Finally, the weighted Bayesian fusion network compares the posterior probabilities of the three, and then outputs the class with the largest posterior probability.

[0093] Specifically, in the execution module, the flow of the module is as follows Figure 3As shown, the system is controlled in two steps: the first step is to connect the brain-computer computer, ESP32, OpenMV, lower limb exoskeleton rehabilitation robot and EEG device used by the system, including connecting the brain-computer computer and the WIFI of the lower limb exoskeleton rehabilitation robot, wherein the brain-computer computer is the current host currently running the program. Then configure the IP address and subnet mask of the current host so that they are in the same local area network, that is, the current host and the lower limb exoskeleton rehabilitation robot achieve communication conditions; then set the configuration file, which includes the IP address of the EEG device, the port number port, the sampling rate Sampling Rate, the position information Chanlocs of the EEG electrode and the IP address and port number port of the ESP32 and other information, so that the current host, ESP32 and OpenMV and the lower limb exoskeleton rehabilitation robot can communicate smoothly. Initialize a DataServerThread thread in the program, which is responsible for establishing a TCP / IP connection with the EEG device and accepting data, calling the connect() method to try to establish a TCP connection with the EEG device to ensure that it can accept EEG signal data; then the current host and OpenMV try to establish a connection, since they cannot directly communicate wirelessly, so use ESP32 as the main control board to act as a router and connect with the current host, wherein OpenMV and ESP32 use serial connection for serial communication; ESP32 and the current host establish a TCP / IP connection, then the host sends a status message "connect_server" to the OpenMV device through the socket socket, telling the local machine that it is ready to accept data; then accept the response, wait for the response of the OpenMV device, only when the returned information is "ready" ready state, indicating that the connection is successful. When the connection is successful, the program will enable a new thread data_handler, which processes data and logic by calling the function "handle_data", which needs to accept the following parameters: OpenMV's TCP socket, DataServerThread thread, EEG device information and EEGNet convolutional neural network for EEG classification; in the TCP / IP connection established between the lower limb exoskeleton rehabilitation robot and the brain-computer computer, the lower limb exoskeleton rehabilitation robot acts as a server to receive control instructions from the brain-computer computer; the real cloud platform carrying OpenMV includes large shell, large shell cover plate, small shell and small shell cover plate. The large shell is hollow, and the ESP32 and OpenMV are placed in the small shell. The large shell and the large shell cover plate are fixed by screws, the small shell and the small shell cover plate are fixed by screws, and the large shell and the small shell are connected by an angle-adjustable support. Adjust the angle of the camera to accurately identify the marker. The physical design is as follows Figure 4 、 Figure 5 、 Figure 6 ,Figure 7 The gimbal is fixed on the waist by the waistband, the large shell is in contact with the waist, and the movement of the camera position caused by the offset of the whole gimbal by the body during the movement is relieved. The angle of the camera in the small shell is adjusted by adjusting the support to achieve the best recognition angle. The example physical effect is as shown in Figure 8

[0094] After the brain-computer computer, ESP32, OpenMV, lower limb exoskeleton rehabilitation robot and electroencephalogram device establish communication, the control logic is designed to control the lower limb exoskeleton rehabilitation robot movement.

[0095] First, set a global variable "computer_status" to track the state of the brain-computer computer, then set a continuous message listening from OpenMV, that is, the loop of its state is an infinite loop to ensure the sustainability of the control of the lower limb exoskeleton rehabilitation robot movement, and different logic is executed according to the different states of OpenMV. OpenMV has four states: "recording_data" which means recording data state, "stop_walking" which means stop movement state, "finish" which means end state and "get_computer_status" which means get brain-computer computer state state. First, set the brain-computer computer state to "stand_by" which means standby state, then the program sends the brain-computer computer state to ESP32, and ESP32 receives the information and polls the state of OpenMV whether it changes. When the state of OpenMV changes, the following four situations will occur:

[0096] ​​The first case, i.e. changed to "recording_data", at this time the program sets the brain-computer computer state to "counting_data", i.e. data collection state, then the brain-computer computer prompts the lower extremity exoskeleton rehabilitation robot wearer to "start imagining", i.e. start collecting the wearer's EEG data, after preprocessing the collected EEG data, the optimal EEGNet convolutional neural network obtained after training is used to classify the preprocessed data, and the EEG classification result is output, waiting for input into the weighted Bayesian fusion network; Next, the program sets the brain-computer computer state to "waiting-OpenMV_data", i.e. waiting for OpenMV visual classification result state; After the brain-computer computer receives the visual classification result output by the OpenMV through the ESP32, the program inputs the EEG classification result and the visual classification result into the weighted Bayesian fusion network, which will fuse the input classification results; After the weighted Bayesian fusion network outputs the fused classification result, the program sets the brain-computer computer state to "waiting_robot", i.e. waiting for sending the classification result to the lower extremity exoskeleton rehabilitation robot state, then the brain-computer computer and the lower extremity exoskeleton rehabilitation robot are connected and communicated through the creation of a TCP / IP socket; After the connection is successful, the brain-computer computer sends the classification result output by the weighted Bayesian fusion network to the lower extremity exoskeleton rehabilitation robot, which is the instruction for controlling the motion of the lower extremity exoskeleton rehabilitation robot, and the lower extremity exoskeleton rehabilitation robot executes the corresponding rehabilitation gait according to the received instruction;

[0097] The second case, i.e. changed to "stop_walking", then the brain-computer computer and the lower extremity exoskeleton rehabilitation robot are connected and communicated through the creation of a TCP / IP socket, after the lower extremity exoskeleton rehabilitation robot and the brain-computer computer establish a connection, the program sets the state of the brain-computer computer to "stand_by", then the brain-computer computer sends the number "0" to the lower extremity exoskeleton rehabilitation robot, which will stop moving after receiving the instruction;

[0098] The third case, i.e. changed to "finish", then the brain-computer computer and the lower extremity exoskeleton rehabilitation robot are connected and communicated through the creation of a TCP / IP socket, after the lower extremity exoskeleton rehabilitation robot and the brain-computer computer establish a connection, the program sets the state of the brain-computer computer to "finish", then the brain-computer computer sends the number "0" to the lower extremity exoskeleton rehabilitation robot, which will stop moving and stop the DataServerThread thread, terminate the connection with the EEG device and data reception, and the program stops running;

[0099] The fourth case, i.e. becoming "get_computer_status", the program returns the current state of the brain-computer computer;

[0100] The control program is provided with an exit condition of infinite loop, i.e. the second case and the third case, and can also be manually terminated; a switch button is also provided on the lower limb exoskeleton rehabilitation robot, and the movement of the lower limb exoskeleton rehabilitation robot can be actively stopped.

[0101] The above embodiment is the preferred embodiment of the present application, but the embodiment of the present application is not limited by the above embodiment, and any change, modification, substitution, combination, simplification made without departing from the spirit and principle of the present application should be an equivalent replacement, which is included in the protection scope of the present application.

Claims

1. A control system for a lower limb exoskeleton rehabilitation robot based on multimodal fusion, characterized in that, include: The data acquisition and preprocessing module is used to acquire the original EEG brain signal dataset, the visual recognition dataset of the actual scene, and the gait dataset of the lower limb exoskeleton rehabilitation robot. It preprocesses the collected original EEG brain signal dataset to obtain a clearer, less noisy, and stable high-quality EEG brain signal dataset. Then, the preprocessed EEG brain signal dataset is divided into training set and validation set according to a set ratio. The data processing module is used to process the preprocessed EEG brainwave signal dataset, the visual recognition dataset of the actual scene, and the gait dataset of the lower limb exoskeleton rehabilitation robot, and to construct a weighted Bayesian fusion network to fuse the EEG brainwave classification results and the visual classification results to output the final classification result. In EEG signal dataset processing, the training set is fed into the EEGNet convolutional neural network for training, and the training effect of the network is validated using a validation set to obtain the optimal network. This optimal network is then used in the subsequent EEG classification task to obtain the EEG classification result. In real-world visual recognition dataset processing, the collected visual recognition dataset is input into the convolutional neural network for training to obtain the optimal network. This optimal network is then used in the subsequent visual classification task to obtain the visual classification result. In the gait dataset processing of the lower limb exoskeleton rehabilitation robot, data points are manually added to the collected gait data to make the discrete data smoother and the gait more standardized. The processed gait dataset is then used in subsequent motion execution tasks to standardize the gait of the lower limb exoskeleton rehabilitation robot. The weighted Bayesian fusion network introduces weight factors into the Bayesian fusion network to adjust the contribution of the input signal to the network output, so as to obtain a final classification result that comprehensively considers the EEG classification results and the visual classification results. The execution module controls the motion state of the lower limb exoskeleton rehabilitation robot based on the final classification results output by the data processing module. The control process involves two steps: First, connecting and communicating with the devices used in the system, including EEG devices, vision devices, and a brain-computer interface (BCI). The program sets states for the BCI and vision devices, and controls the lower limb exoskeleton rehabilitation robot by changing these states. Second, a loop is set up in the program to continuously listen for messages from the vision devices, where each message represents its state. Different control logic is executed based on the different states detected. When the initial state changes, the execution flow of the control logic for the changed state is entered, ultimately outputting instructions to the lower limb exoskeleton rehabilitation robot to perform the corresponding actions and complete the rehabilitation exercise.

2. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 1, characterized in that, The EEG signal dataset was acquired using a 64-channel EEG device. The selected channels were: FC5, FC3, FC1, FC2, FC4, FC6, C5, C3, C1, C2, C4, C6, CP5, CP3, CP1, CP2, CP4, and CP6. EEG signal data for each task were collected using a standard paradigm of motor imagery to form a dataset. The dataset included data on subjects' motor imagery of walking on flat ground, climbing stairs, and descending stairs. Each task... The data was repeatedly processed to obtain sufficient data for subsequent analysis. The real-time display function was enabled in the EEG acquisition software to view the EEG waveforms of each channel and to preliminarily assess the signal quality. Normal EEG signals should have smooth and regular fluctuations, without obvious spikes or violent fluctuations. The preprocessing operation process included: resampling, filtering, denoising, artifact removal, baseline calibration, data segmentation, feature extraction, feature selection and dimensionality reduction, classification and regression prediction, and result evaluation. The preprocessed EEG signal dataset was divided into training and validation sets in an 8:2 ratio.

3. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 2, characterized in that, The acquisition of the visual recognition dataset for the actual scene was achieved using ESP32 and OpenMV. ESP32 and OpenMV communicated via a serial port connection. OpenMV acquired visual data of the actual scene, including data from flat ground, going up stairs, and going down stairs. Data was acquired by repeatedly taking pictures of different angles on different terrains and forming the dataset.

4. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 3, characterized in that, The gait dataset of the lower limb exoskeleton rehabilitation robot is acquired using the active rehabilitation mode of the lower limb exoskeleton rehabilitation robot. In this mode, customizable gait can be achieved by collecting the wearer's gait trajectory data. In the active rehabilitation mode, the wearer performs standardized flat walking, climbing stairs, and descending stairs. The lower limb exoskeleton rehabilitation robot will convert the curved movements during the movement into discrete data points and record them in the interface program.

5. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 4, characterized in that, In EEG signal dataset processing, the EEGNet convolutional neural network has three convolutional layers. The first layer is a regular convolutional layer operation, called ordinary convolution (conv). The second layer is a channel-wise convolutional layer operation, called depthwise convolution (conv). The third layer is a separateable convolutional layer operation, called separateable convolution (conv). This layer is composed of one depthwise convolution and one pointwise convolutional layer operation. In depthwise convolution, one convolutional kernel is responsible for one channel, meaning that one channel is convolved by only one convolutional kernel. The shape of the convolutional kernel in pointwise convolution is 1×1×M×C, where M is the number of channels in the previous layer and C is the number of convolutional kernels. The training set is input into the EEGNet convolutional neural network for training, and the training effect of the network is validated through a validation set to obtain the optimal network for classifying EEG signal data. The classification result output by the optimal network will be used as one of the data inputs to the weighted Bayesian fusion network.

6. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 5, characterized in that, In the processing of visual recognition datasets in real-world scenarios, OpenMV is used to collect data on real-world scenarios, including flat ground, going up stairs, and going down stairs. The data is then input into a convolutional neural network for training to obtain the optimal network for recognizing real-world scenarios and outputting classification results. The classification results are then forwarded to a weighted Bayesian fusion network using ESP32. In other words, the classification results output by the optimal network will be used as one of the data inputs to the weighted Bayesian fusion network.

7. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 6, characterized in that, In the gait dataset processing of the lower limb exoskeleton rehabilitation robot, the robot is selected as the active rehabilitation mode. In this mode, customizable gait can be achieved by collecting the wearer's gait trajectory data. Three rehabilitation modes are set and categorized: walking on flat ground (Category A), climbing stairs (Category B), and descending stairs (Category C). The lower limb exoskeleton rehabilitation robot collects trajectory data of the wearer's three different gait types and writes the collected data into the active rehabilitation mode interface. Corresponding calling conditions are set: when the control command "A" is received, the walking on flat ground data is called; when the control command "B" is received, the climbing stairs data is called; and when the control command "C" is received, the descending stairs data is called. Then, the lower limb exoskeleton rehabilitation robot can execute the corresponding action by calling the corresponding data upon receiving the corresponding command. A stop command "0" is also set; when the control command "0" is received, the lower limb exoskeleton rehabilitation robot will stop moving. Since the collected data consists of discrete data points, manual data point addition is used to make the discrete data points appear smoother overall.

8. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 7, characterized in that, The classification results input into the weighted Bayesian fusion network include the EEG classification results output by the EEGNet convolutional neural network and the visual classification results output by OpenMV. Before inputting these results into the weighted Bayesian fusion network, weights are added to the EEG and visual classification results. Weighting factors are introduced to adjust the contributions of these two classification results to the weighted Bayesian fusion network, resulting in a final classification result that comprehensively considers both EEG and visual classification results. Here, the EEGNet convolutional neural network is the EEG classifier, and OpenMV is the OpenMV visual classifier. The fusion criterion used in the weighted Bayesian fusion network is the maximum a posteriori probability criterion. The meanings of the symbols are as follows: Since the lower limb exoskeleton rehabilitation robot has three movement rehabilitation modes: walking on flat ground, climbing stairs, and descending stairs, there are three corresponding classification results: walking on flat ground is category A, climbing stairs is category B, and descending stairs is category C. Therefore, both the EEG classifier and the OpenMV visual classifier are three-classifiers. The probability of the EEG classifier outputting A is... The probability of outputting B is The probability of outputting C is The probability that the OpenMV visual classifier outputs A is: The probability of outputting B is The probability of outputting C is The proportion of EEG classifiers OpenMV visual classifier percentage ,in , ; This is an EEG classifier, hereinafter referred to as Classifier 1; This is the OpenMV visual classifier, hereinafter referred to as Classifier 2; This represents the probability of class A occurring given the observations from classifier 1 and classifier 2. This represents the probability of class B occurring given the observations from classifier 1 and classifier 2. This represents the probability of class C occurring given the observations from classifier 1 and classifier 2. This represents the prior probability of category A, that is, the probability of category A occurring without any observations. This represents the prior probability of category B, that is, the probability of category B occurring without any observations. This represents the prior probability of category C, that is, the probability of category C occurring without any observations. This represents the joint observation probability of classifier 1 and classifier 2 in the case of category A; This represents the joint observation probability of classifier 1 and classifier 2 in the case of category B; This represents the joint observation probability of classifier 1 and classifier 2 in the case of category C; This represents the normalization constant, used to ensure that the sum of the posterior probabilities is 1; The specific steps of the weighted Bayesian fusion network are as follows: 1) Define prior probability: The scenario involves three events: walking on flat ground, climbing stairs, and descending stairs. All three events have the same probability of occurrence, meaning their prior probabilities are equal. ; 2) Calculate the weighted joint probability: ； ； ； In the formula, This represents the probability that the EEG classifier outputs A. The weight of the EEG classifier to the power of; This represents the probability that the EEG classifier outputs B. The weight of the EEG classifier to the power of; This represents the probability that the EEG classifier outputs C. The weight of the EEG classifier to the power of; This represents the probability that the EEG classifier outputs A. The weight of the EEG classifier to the power of; This represents the probability that the EEG classifier outputs A. The weight of the EEG classifier to the power of; This represents the probability that the EEG classifier outputs A. The weight of the EEG classifier to the power of; 3) Calculate the normalization constant: ； 4) Calculate the posterior probability: ； ； ； Finally, the weighted Bayesian fusion network compares the posterior probabilities of these three classes and outputs the class with the highest posterior probability.

9. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 8, characterized in that, The system connects the brain-computer interface (BCI), ESP32 and OpenMV, lower limb exoskeleton rehabilitation robot, and EEG devices. This includes connecting the BCI to the lower limb exoskeleton rehabilitation robot via Wi-Fi, with the BCI serving as the current host for the running program. Next, the host's IP address and subnet mask are configured to ensure they are on the same local area network, enabling communication between the host and the lower limb exoskeleton rehabilitation robot. Finally, a configuration file is set, including the EEG device's IP address, port number, and sampling rate. The program provides information on the rate, EEG electrode location (Chanlocs), and ESP32's IP address and port number to facilitate communication between the host computer, ESP32, OpenMV, and the lower limb exoskeleton rehabilitation robot. A DataServerThread is initialized to establish a TCP / IP connection with the EEG device and receive data. The connect() method is called to attempt a TCP connection, ensuring the reception of EEG signal data. Next, the host computer attempts to establish a connection with OpenMV. Since direct wireless communication between them is not possible, the ESP32 is used as the main control board, acting as a router to connect to the host. OpenMV and ESP32 communicate via serial port. After the ESP32 establishes a TCP / IP connection with the host, the host sends a status message "connect_server" to the OpenMV device via a socket, indicating that it is ready to receive data. Finally, the program receives responses. The system waits for a response from the OpenMV device. A successful connection is indicated only when the returned message is "ready". Once connected, a new thread, `data_handler`, is started. This thread, `handle_data`, processes data and logic by calling the function `handle_data`. This function requires the following parameters: the OpenMV TCP socket, the `DataServerThread`, the EEG device information, and the EEGNet convolutional neural network used for EEG classification. In the TCP / IP connection established between the lower limb exoskeleton rehabilitation robot and the brain-computer interface, the lower limb exoskeleton rehabilitation robot acts as the server, receiving control commands from the brain-computer interface. The physical gimbal equipped with OpenMV includes a large shell, a large shell cover, a small shell, and a small shell cover. The large shell is hollow, and the ESP32 and OpenMV are placed inside the small shell. The large shell and the large shell cover are fixed together with screws, as are the small shells and the small shell cover. The large and small shells are connected by an adjustable bracket, which adjusts the camera angle for accurate landmark recognition.The gimbal is secured to the front of the waist and abdomen via a belt, with the large housing in contact with the waist and abdomen. This helps to mitigate the movement of the camera caused by the body shifting the gimbal during movement. The angle of the camera in the smaller housing is adjusted by adjusting the bracket to achieve the optimal recognition angle.

10. The lower limb exoskeleton rehabilitation robot control system based on multimodal fusion according to claim 9, characterized in that, After establishing communication between the brain-computer interface, ESP32 and OpenMV, the lower limb exoskeleton rehabilitation robot, and the EEG device, control logic is designed to control the movement of the lower limb exoskeleton rehabilitation robot, as follows: First, a global variable "computer_status" is set to track the state of the brain-computer interface (BCI). Then, the system continuously listens for messages from OpenMV, with its state loop being infinite to ensure sustainable control of the lower limb exoskeleton rehabilitation robot's movement. Different logic is executed based on the different OpenMV states monitored. OpenMV has four states: "recording_data", "stop_walking", "finish", and "get_computer_status". First, the BCI state is set to "stand_by". Then, the program sends the BCI state to the ESP32. Upon receiving the information, the ESP32 polls for changes in the OpenMV state. When the OpenMV state changes, one of four scenarios occurs: In the first scenario, the program switches to "recording_data," where it sets the brain-computer interface (BCI) to "counting_data," indicating data collection. The BCI then prompts the wearer of the lower limb exoskeleton rehabilitation robot to "start imagining," initiating the collection of EEG signal data. After preprocessing the collected EEG data, the optimally trained EEGNet convolutional neural network is used to classify the preprocessed data, outputting the EEG classification result, which then awaits input into the weighted Bayesian fusion network. Next, the program sets the BCI to "waiting-OpenMV_data," awaiting the OpenMV visual classification result. The OpenMV visual classification result is then forwarded via ESP32. After the brain-computer interface is connected, the program inputs the EEG classification results and visual classification results into a weighted Bayesian fusion network, which fuses the input classification results. Once the weighted Bayesian fusion network outputs the fused classification result, the program sets the brain-computer interface to "waiting_robot," meaning it is waiting to send the classification result to the lower limb exoskeleton rehabilitation robot. Then, a TCP / IP socket is created to establish a connection between the brain-computer interface and the lower limb exoskeleton rehabilitation robot. After a successful connection, the brain-computer interface sends the classification result output by the weighted Bayesian fusion network to the lower limb exoskeleton rehabilitation robot. This classification result serves as the instruction to control the movement of the lower limb exoskeleton rehabilitation robot. The lower limb exoskeleton rehabilitation robot calls up the corresponding gait data based on the received instruction and executes the corresponding rehabilitation gait. In the second scenario, the brain-computer interface becomes "stop_walking". Then, a TCP / IP socket is created to connect the brain-computer interface with the lower limb exoskeleton rehabilitation robot. After the connection between the lower limb exoskeleton rehabilitation robot and the brain-computer interface is established, the program sets the status of the brain-computer interface to "stand_by". Then, the brain-computer interface sends the number "0" to the lower limb exoskeleton rehabilitation robot. After receiving the instruction, the lower limb exoskeleton rehabilitation robot will stop moving. In the third scenario, the brain-computer interface (BCI) becomes "finish". Then, a TCP / IP socket is created to establish a connection between the BCI and the lower limb exoskeleton rehabilitation robot. After the connection is established, the program sets the BCI to "finish". The BCI then sends the number "0" to the lower limb exoskeleton rehabilitation robot. Upon receiving this instruction, the lower limb exoskeleton rehabilitation robot stops moving and stops the DataServerThread thread, terminating the connection with the EEG device and data reception. The program then stops running. In the fourth case, which is "get_computer_status", the program will return the current state of the brain-computer interface. The control program has an infinite loop exit condition, namely the second and third cases, and the program can also be manually terminated; the lower limb exoskeleton rehabilitation robot is also equipped with a switch button that can actively stop the movement of the lower limb exoskeleton rehabilitation robot.

Images