Control method and system based on multi-mode intelligent brain-controlled wheelchair

Abstract

The invention relates to the technical field of electric wheelchair control, in particular to a control method and system based on a multi-mode intelligent brain-controlled wheelchair, and the method comprises the steps: collecting an electroencephalogram signal of a brain-controlled wheelchair user, carrying out the preprocessing, and generating a verification data set; based on the verification data set, determining that the brain-controlled wheelchair currently shows any one control state of a stop state, a motor imagery state and a moving state according to a judgment condition; blood oxygen data of a brain-controlled wheelchair user is obtained, a judgment threshold value is set, and when the blood oxygen data is lower than the judgment threshold value, emergency braking is carried out, and help seeking is carried out; namely, motor imagery signals are accurately recognized and classified into four motion states of advancing, retreating, left turning and right turning, and the motion of the brain-controlled wheelchair is controlled in real time; the concentration degree in the electroencephalogram signals is monitored, and the moving speed is flexibly controlled; according to the intelligent brain-controlled wheelchair, multi-mode control is carried out, a safety alarm system is arranged in an auxiliary mode in combination with an electroencephalogram signal, concentration control and a blink signal, and the stability, control precision and safety of the intelligent brain-controlled wheelchair are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electric wheelchair control, and particularly relates to a control method and system for a multi-modal intelligent brain-controlled wheelchair. Background Art

[0002] With the accelerating aging of the global population and the continuous improvement of living standards, the travel problems of people with physical disabilities caused by diseases have become increasingly severe. Especially, the travel needs of patients with amyotrophic lateral sclerosis (ALS) and paralysis are more urgent. Usually, the existing brain-controlled wheelchair technology is used to improve the travel experience of this special group. Although the existing brain-controlled wheelchair technology can provide certain help, due to the weak electroencephalogram (EEG) signals and the imperceptible changes, it is difficult to achieve stable and real-time control. Moreover, the existing brain control technologies usually rely on SSVEP (Steady-State Visual Evoked Potential) or P300 to implement the EEG mode. However, the response time of such signals is long, the accuracy rate is not high, it is easy to cause fatigue, and it is highly dependent on external devices and not suitable for long-term control. In addition, for the existing brain control technology, the user needs to stare at the screen for a long time during control, or the brain-controlled wheelchair cannot accurately distinguish the accuracy of the signal emission, resulting in misjudgment, and the feasibility of user operation is relatively low. In addition, the existing brain-controlled wheelchair is not flexible enough in reacting to obstacles or emergencies and cannot make real-time responses according to the user's needs, and the user experience is not strong. Summary of the Invention

[0003] In order to solve the technical problems that the existing brain-controlled wheelchair is difficult to control stably and in real time, is not flexible enough, and has low feasibility for users, the purpose of the present invention is to provide a control method for a multi-modal intelligent brain-controlled wheelchair, and the specific technical solutions adopted are as follows:

[0004] Collect the electroencephalogram signals of the brain-controlled wheelchair user, and perform preprocessing to generate a verification data set;

[0005] Based on the verification data set, determine any one of the control states of the brain-controlled wheelchair currently showing a stop state, a motor imagery state, and a moving state according to the judgment conditions; and obtain the blood oxygen data of the brain-controlled wheelchair user, set a judgment threshold, and when the blood oxygen data is lower than the judgment threshold, perform emergency braking and send out a distress signal;

[0006] If the brain-controlled wheelchair currently shows a stop state, continue to collect the electroencephalogram signals of the user to obtain the blink signal and enter the motor imagery state;

[0007] If the brain-controlled wheelchair currently shows a motor imagery state, extract the feature values of the verification data set, load the weight file, determine the instruction sent by the electroencephalogram signal, and control the brain-controlled wheelchair to enter the moving state based on the instruction;

[0008] If the brain-controlled wheelchair is currently in a moving state, analyze and verify the concentration in the dataset, and control the moving speed of the brain-controlled wheelchair according to the concentration.

[0009] Preferably, the preprocessing includes any one of data cleaning methods such as clutter filtering, integration and amplification, null value completion, and digital-to-analog electrical signal conversion.

[0010] Preferably, based on the verification dataset, determine any one of the control states of the brain-controlled wheelchair as the stop state, motor imagery state, and moving state according to the judgment conditions, including:

[0011] Based on a type of electrode signal in the verification dataset, determine that the brain-controlled wheelchair is currently in a stop state; based on a second type of electrode signal, determine that the brain-controlled wheelchair is currently in a motor imagery state; based on the first type of electrode signal combined with the third electrode signal, determine that the brain-controlled wheelchair is currently in a moving state; the first type of electrode signal includes Fp1 and Fp2, the second type of electrode signal includes C3, C4, and Cz, and the third electrode signal is Fz.

[0012] Preferably, obtain the blood oxygen data of the brain-controlled wheelchair user, set a judgment threshold, and when the blood oxygen data is lower than the judgment threshold, perform emergency braking and send a distress signal, including:

[0013] Obtain a blood oxygen detection device and collect the blood oxygen data of the brain-controlled wheelchair user;

[0014] A safety alarm system is provided on the brain-controlled wheelchair. Set a judgment threshold. When the blood oxygen data is lower than the judgment threshold, perform emergency braking, cut off all power supplies except the safety alarm system, emit an audible and visual prompt based on the safety alarm system, and send location information to the hospital monitoring platform and / or the guardian.

[0015] Preferably, obtain the blink signal, including:

[0016] Mark the time when the brain-controlled wheelchair waits for the blink signal as the first time, and the time when it waits for the next blink signal after the blink signal ends as the second time;

[0017] Set the first screening threshold and the second screening threshold respectively. When the first time is greater than the first screening threshold, determine it as a blink signal and enter the second time; when the first time is less than the second screening threshold, obtain the first time again.

[0018] Preferably, if the brain-controlled wheelchair is currently in a motor imagery state, extract the feature values of the verification dataset, load the weight file, determine the instruction sent by the electroencephalogram signal, and control the brain-controlled wheelchair to enter the moving state based on the instruction, including:

[0019] Build a motor imagery model, input the validation dataset into the motor imagery model to extract eigenvalue, load the weight file stored in the brain-controlled wheelchair, classify the validation dataset to determine the instructions sent by the brain-controlled wheelchair user, and the instructions include forward, backward, left, and right turns;

[0020] Control the brain-controlled wheelchair to enter the corresponding moving state according to different instructions.

[0021] Preferably, build a motor imagery model, and the corresponding calculation formula is:

[0022]

[0023] Wherein, represents the instruction category finally predicted; C represents all categories of the sent instructions; B represents the number of decision trees in the random forest built based on the validation dataset; b represents the b-th tree in the random forest; c b represents the prediction value of the b-th tree; I(c b =c) represents the index function, that is, c b =c, indicating that the value of the index function is 1 or 0.

[0024] Preferably, if the brain-controlled wheelchair is currently in a moving state, analyze the validation dataset to calculate the concentration, and control the moving speed of the brain-controlled wheelchair according to the concentration, including:

[0025] Perform secondary filtering processing based on the validation dataset to determine the concentration;

[0026] Sequentially give the preset low-speed, medium-speed, and high-speed data ranges, compare with the concentration for the data range, and correspondingly control the moving speed of the brain-controlled wheelchair;

[0027] Monitor the blink signal, if there is a blink signal, stop the operation of the brain-controlled wheelchair and return to the stop state.

[0028] To solve the above problems, the present application also provides: A control system for a multi-modal intelligent brain-controlled wheelchair, and the system includes:

[0029] An electroencephalogram acquisition device, used for: acquiring the electroencephalogram signal of the brain-controlled wheelchair user, and performing preprocessing to generate a validation dataset;

[0030] The main control module and the safety alarm system, used for: determining any one of the control states of the stop state, motor imagery state, and moving state of the brain-controlled wheelchair based on the validation dataset according to the judgment condition; and acquiring the blood oxygen data of the brain-controlled wheelchair user, setting a judgment threshold, and when the blood oxygen data is lower than the judgment threshold, performing emergency braking and sending out a distress signal;

[0031] The data analysis and data processing module is used for: if the brain-controlled wheelchair is currently in a stopped state, continuously collect the user's electroencephalogram signal to obtain the blink signal and enter the motor imagery state; if the brain-controlled wheelchair is currently in the motor imagery state, extract the feature values of the verification dataset, load the weight file, determine the instruction sent by the electroencephalogram signal, and control the brain-controlled wheelchair to enter the moving state based on the instruction; if the brain-controlled wheelchair is currently in the moving state, analyze the verification dataset to calculate the concentration, and control the moving speed of the brain-controlled wheelchair according to the concentration.

[0032] The present invention has the following beneficial effects:

[0033] 1. This application collects the user's electroencephalogram signal, performs preprocessing, accurately identifies the motor imagery signal, classifies it into four motion states of forward, backward, left turn, and right turn to control the motion of the brain-controlled wheelchair in real time; at the same time, uses the user's concentration to adjust the speed of the brain-controlled wheelchair, that is, monitors the concentration in the electroencephalogram signal in real time, sets a threshold to divide it into three levels of high, medium, and low, corresponding to the high-speed, medium-speed, and low-speed modes of the brain-controlled wheelchair respectively, providing flexible speed control, enhancing the flexibility of the entire brain-controlled wheelchair operation; and cooperates with obtaining the blink signal, during the movement of the wheelchair, judges whether it is necessary to stop the movement of the wheelchair by monitoring the blink signal, ensures that the user can respond in time when needed, prevents the accuracy defect caused by only using electroencephalogram detection, and improves safety; that is, through the multi-modal control method, combining electroencephalogram signal, concentration control, and blink signal, and also assisting in setting up a safety alarm system, not only reduces the dependence of the brain-controlled wheelchair on external devices, but also improves the user experience and enhances the reliability of the brain-controlled wheelchair in practical applications; thereby improving the stability, control accuracy, and safety of the intelligent brain-controlled wheelchair.

[0034] 2. The present invention also provides a control system for a multi-modal intelligent brain-controlled wheelchair for implementing the control method for a multi-modal intelligent brain-controlled wheelchair provided above. This system has the same beneficial effects as the above-mentioned control method for a multi-modal intelligent brain-controlled wheelchair, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 It is a flowchart of the implementation of a control method for a multi-modal intelligent brain-controlled wheelchair provided by an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of screen display of the safety alarm system of a control method for a multi-modal intelligent brain-controlled wheelchair provided by an embodiment of the present invention. Detailed implementation manners

[0038] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following combines the drawings and preferred embodiments to detail the specific implementation manners, structures, features and effects of a control method and system for a multi-modal intelligent brain-controlled wheelchair proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0040] The following specifically describes the specific solutions of a control method and system for a multi-modal intelligent brain-controlled wheelchair provided by the present invention with reference to the drawings.

[0041] Since the existing intelligent wheelchairs for the travel of people with amyotrophic lateral sclerosis and paralysis are difficult to achieve stable and real-time control, have a long response time, are not flexible enough in reacting to obstacle avoidance or emergencies, cannot make real-time responses according to the needs of users, and the user experience is not strong; the first embodiment of the present invention provides a control method for a multi-modal intelligent brain-controlled wheelchair, which through intelligent settings for the brain-controlled wheelchair, combines electroencephalogram signals, attention and blink signals through a multi-modal control method, makes up for the insufficient effects brought by single-signal recognition of users, reduces the response time of the brain-controlled wheelchair, and cooperates with a safety alarm system. Improve the flexibility and safety of the entire brain-controlled wheelchair; to solve a control method for a multi-modal intelligent brain-controlled wheelchair, the second embodiment of the present invention provides a control system for a multi-modal intelligent brain-controlled wheelchair, which is essentially a software system composed of units that implement corresponding functions, and now details the specific steps in the method.

[0042] Please refer to Figure 1 , which shows an implementation diagram of a control method for a multi-modal intelligent brain-controlled wheelchair provided by an embodiment of the present invention. The method includes:

[0043] Step S1: Collect the electroencephalogram signals of the users of the brain-controlled wheelchair, and perform preprocessing to generate a verification data set;

[0044] Step S2: Based on the validation dataset, determine the current control state of the brain-controlled wheelchair as any one of the stop state, motor imagery state, and movement state according to the judgment conditions; and obtain the blood oxygen data of the user of the brain-controlled wheelchair, set a judgment threshold, and when the blood oxygen data is lower than the judgment threshold, apply emergency braking and send out a distress signal.

[0045] Step S3: If the current state of the brain-controlled wheelchair is the stop state, continue to collect the user's electroencephalogram signal to obtain the blink signal and enter the motor imagery state.

[0046] Step S4: If the current state of the brain-controlled wheelchair is the motor imagery state, extract the feature values of the validation dataset, load the weight file, determine the command sent by the electroencephalogram signal, and control the brain-controlled wheelchair to enter the movement state based on the command.

[0047] Step S5: If the current state of the brain-controlled wheelchair is the movement state, analyze the validation dataset to calculate the concentration and control the moving speed of the brain-controlled wheelchair according to the concentration.

[0048] For better illustration, the research and development of the brain-controlled wheelchair can provide freedom of movement for paralyzed patients and patients with amyotrophic lateral sclerosis, and may also change the patients' lifestyle; in real life, the brain-controlled wheelchair usually controls the wheelchair through brain signals, improving the user's independence and quality of life, reducing the dependence on caregivers, and the brain-controlled wheelchair can be customized according to the specific needs of different patients. For patients with amyotrophic lateral sclerosis, their muscles gradually lose function, but the brain still maintains a certain control ability and can complete the blinking action. Therefore, the control is carried out in combination with the electroencephalogram signal and the blink signal, enhancing the feasibility of the intelligent brain-controlled wheelchair.

[0049] Furthermore, in step S1, the preprocessing includes any one of data cleaning methods such as filtering out clutter, integrating and amplifying, filling in null values, and analog-to-digital signal conversion; specifically, collect the electroencephalogram (EEG, i.e., Electroencephalography) signal of the user of the brain-controlled wheelchair, and preprocess the electroencephalogram signal based on any one of the data cleaning methods such as filtering out clutter, integrating and amplifying, filling in null values, and analog-to-digital signal conversion, so as to eliminate the clutter caused by external devices or physiological noise, ensure that the dynamic range of the electroencephalogram signal meets the requirements of subsequent analysis, fill in null values, ensure the continuity and integrity in the data analysis process, and ensure the reliability of the generated validation dataset.

[0050] Furthermore, in step S2, determining that the current control state of the brain-controlled wheelchair is any one of the stop state, motor imagery state, and movement state based on the validation dataset according to the judgment conditions includes:

[0051] Based on a certain type of electrode signal in the validation dataset, it is determined that the brain-controlled wheelchair is currently in a stopped state. Based on a second type of electrode signal, it is determined that the brain-controlled wheelchair is currently in a motor imagery state. Based on the first type of electrode signal combined with the third electrode signal, it is determined that the brain-controlled wheelchair is currently in a moving state. The first type of electrode signal includes Fp1 and Fp2, the second type of electrode signal includes C3, C4, and Cz, and the third electrode signal is Fz.

[0052] For better illustration, Fp1 and Fp2 are blink signal detection channels, which respectively represent the electrical activities in the frontal pole region and the temporal pole region in the electroencephalogram signal. They are located on both sides of the forehead, forming a specific lead, representing the left and right frontal pole regions, and can participate in processing tasks such as emotion regulation, decision-making, and social cognition. They are mainly used to identify signals related to static states or specific electroencephalogram activities. The Fz electrode signal represents the electroencephalogram activity in the frontal pole region, which is usually related to the movements of the head and face and more complex cognitive activities. C3, C4, and Cz respectively represent the left central region, the right central region, and the central region in the electroencephalogram signal, which are located in the motor area of the cerebral cortex, that is, they record the brain waves related to motor imagery and are used to detect the motor imagery state. Among them, C3 and C4 represent the motor imaging channels, and Cz and Fz represent the auxiliary electrodes. Through the clear division of labor of each channel, the accuracy and efficiency of the current judgment of the movement state of the brain-controlled wheelchair are ensured.

[0053] Specifically, based on the analysis of the validation dataset, first, signal features are extracted to reflect different patterns of the user's electroencephalogram signal, helping the brain-controlled wheelchair understand the user's intention. Then, the machine learning algorithm in the brain-controlled wheelchair is used to determine the brain-controlled wheelchair as the corresponding movement state according to different electrode signals. If the pattern of the first type of electrode signal is recognized as a pattern related to rest, relaxation, or no movement intention, it indicates that the current movement state of the brain-controlled wheelchair is the stopped state. When the pattern of the second type of electrode signal indicates that the user is performing motor imagery, the wheelchair will enter the motor imagery state, that is, the user has the intention to start an action but may not have actually moved yet. When the first type of electrode signal is combined with the third electrode signal, an electroencephalogram signal related to actual action or movement is detected, and it is determined that the brain-controlled wheelchair enters the moving state and starts to control the moving speed according to the user's intention.

[0054] Please refer to Figure 2 , which shows a schematic diagram of the screen display of the safety alarm system of a control method for a multi-modal intelligent brain-controlled wheelchair provided by an embodiment of the present invention. Among them, the contact information between the hospital and the family members is displayed on the screen. Further, in step S2, the blood oxygen data of the brain-controlled wheelchair user is obtained, and a judgment threshold is set. When the blood oxygen data is lower than the judgment threshold, emergency braking is performed and a distress signal is sent, including:

[0055] Obtain a blood oxygen detection device and collect the blood oxygen data of the brain-controlled wheelchair user.

[0056] Preferably, in this embodiment, the blood sample detection device uses the Max30102 module to collect heart rate and blood oxygen data, that is, a high-precision and low-power sensor module integrating a photoelectric sensor and a signal processing circuit is adopted to obtain the blood oxygen data of the brain-controlled wheelchair user more safely and accurately, and improve the recognition efficiency.

[0057] A safety alarm system is provided on the brain-controlled wheelchair, and a judgment threshold is set. When the blood oxygen data is lower than the judgment threshold, an emergency brake is applied, all power supplies except the safety alarm system are cut off, an audible and visual prompt is issued based on the safety alarm system, and positioning information is sent to the hospital monitoring platform and / or the guardian.

[0058] As an alternative implementation, in this embodiment, the judgment threshold is set to 85%; among them, the safety alarm system includes devices such as a display screen, a speaker or a light that can be used to send a distress signal.

[0059] Specifically, assume that the blood oxygen data monitored by the current brain-controlled wheelchair is less than the judgment threshold. At this time, the user's condition may deteriorate, such as the aggravation of respiratory muscle weakness or respiratory tract infection, etc. That is, when it is lower than 85%, it causes a rapid drop in blood oxygen saturation and poses a threat to the user's life. The brain-controlled wheelchair is emergently braked, and all power supplies except the safety alarm system are cut off to prevent the user from making misoperations under stress until the guardian or doctor unlocks it to restore normal functions; and an audible and visual prompt is issued based on the safety alarm system to remind the surrounding people. For example, the alarm light keeps flashing, or a distress message is sent to the surrounding people through the intelligent voice assistant, and the contact information of the guardian or the hospital is displayed on the screen to facilitate the onlookers to take emergency measures to reduce the safety threat to the user; and the safety alarm system includes a GPS (Global Positioning System) locator, which can assist the supporting software to facilitate the hospital monitoring platform and / or the guardian to view the user's positioning information in real time, and package and send the important physiological index data of the patient to the terminal of the supporting software to be able to quickly evaluate the patient's condition and ensure the availability of the positioning information.

[0060] It can be explained that the blink signal refers to the potential change caused by eye blinking monitored by the electrode. In this embodiment, it represents the signal generated by a heavier and deliberate blink, excluding the signal of normal physiological blinking; when blinking, the rapid movement of the eyeball will cause a change in the charge distribution inside and outside the eyeball, and this potential change is usually relatively large, manifested as a larger waveform, which is significantly different from the slower and smaller waveforms generated by the brain; the recognition of the blink signal helps to exclude the artifacts caused by eye movement and ensure more accurate electroencephalogram data analysis.

[0061] Furthermore, in step S3, obtaining the blink signal includes:

[0062] The time for the brain-controlled wheelchair to wait for the blink signal is the first time, and the time to wait for the next blink signal after the end of the blink signal is the second time;

[0063] Set the first screening threshold and the second screening threshold respectively. When the first time is greater than the first screening threshold, it is determined as a blink signal and enters the second time; when the first time is less than the second screening threshold, the first time is acquired again.

[0064] Understandably, to achieve the adaptive adjustment function, the entire brain-controlled wheelchair system provides a way for the user to input their own blink electromyogram signal. The user can, under the guidance of the system, perform multiple blink actions. The system collects the electromyogram signal data during these blinks, and then takes the average value of the peak values of each blink signal and multiplies it by the adjustment coefficient 0.7 to flexibly adjust the first screening threshold according to the user's own conditions, that is, to more accurately determine whether the currently emitted electrical signal is a blink signal through the first screening threshold.

[0065] Specifically, according to different users, the first screening threshold is set correspondingly, and the second screening threshold of the brain-controlled wheelchair is given, that is, the lower limit data value for determining whether it is a blink signal; when the first time is greater than the first screening threshold, it is determined as a blink signal and counted as one blink, and then enters the second time; in practical applications, after entering the second time, the system of the brain-controlled wheelchair waits for 0.1 - 0.2 s and then continues to time to obtain the next blink signal; when the first time is less than the second screening threshold, the first time is acquired again, that is, it means that the currently emitted signal is not a blink signal, preventing the system from misjudging and effectively solving the problem of overcounting the blink signal count and improving the accuracy of identifying the blink signal.

[0066] Further, in step S4, it includes:

[0067] Construct a motor imagery model, input the validation dataset into the motor imagery model to extract feature values, load the weight file stored in the brain-controlled wheelchair, and classify the validation dataset to determine the instructions issued by the user of the brain-controlled wheelchair. The instructions include moving forward, backward, turning left, and turning right;

[0068] Control the brain-controlled wheelchair to enter the corresponding moving state according to different instructions.

[0069] It can be explained that motor imagery means that the user of the brain-controlled wheelchair, without actual movement, only controls the brain-controlled wheelchair to make a response by imagining a certain movement.

[0070] Further, to construct a motor imagery model, the corresponding calculation formula is:

[0071]

[0072] Among them, represents the finally predicted instruction category; C represents all categories of the issued instructions; B represents the number of decision trees in the random forest constructed based on the validation dataset; b represents the b-th tree in the random forest; c b represents the predicted value of the b-th tree; I(c b = c) represents the indicator function, that is, when c b = c, it means the value of the indicator function is 1 or 0.

[0073] It is explained that the acquisition of electroencephalogram signals mainly includes four parts: a signal amplifier, a signal processor, a display, and a host; the frequency of electroencephalogram signals is low and the amplitude is small, so a signal amplifier is used to amplify the collected electroencephalogram signals, and the signal processor is connected to the signal amplifier to filter the electroencephalogram signals. Flash, that is, multimedia content such as navigation and interaction, is played through the display to induce the user to make corresponding imaginations according to the prompts; specifically, the materials for imagination are mainly in the form of Flash animations and pictures, that is, the up, down, left, and right movements of a small ball respectively. The picture shows a small ball and an arrow indicating the direction it is about to move, to induce the user to imagine the movement of the small ball in that direction, and then issue corresponding instructions to convey to the brain-controlled wheelchair for the next operation.

[0074] Specifically, a motor imagery model is constructed based on the implementation of instruction control. First, feature extraction is performed based on the validation dataset. When the experimenter performs unilateral motor imagery, the energy of the mu rhythm (8 - 13 Hz) and beta rhythm (16 - 24 Hz) in the contralateral motor somatosensory area decreases, while the energy in the ipsilateral motor somatosensory area increases; therefore, based on this, the energy features of these two frequency bands are selected as the main features for analysis in the motor imagery state, and the corresponding calculation formula is:

[0075]

[0076] where y j represents the band energy corresponding to the j-th time window; n represents the number of frequency points in the band; P ij (f k ) represents the power spectral density of the i-th channel at the frequency point f k at the j-th time window.

[0077] Then, a random forest is selected as the machine learning method. A random forest is an ensemble learning method mainly composed of multiple decision trees. It is composed of multiple decision trees, and each decision tree can make independent predictions. When new data arrives, all trees can be used for prediction simultaneously, and the judgment process is fast and the accuracy is high, which can meet the real-time judgment requirements of the brain-controlled wheelchair. Even if there are still some noises in the preprocessed validation dataset, the random forest can reduce the influence of noises through majority voting to ensure the reliability of instruction judgment.

[0078] Define the validation dataset as the input dataset D. Randomly sample and replace from the input dataset D to create multiple subsets. Train decision trees on each subset. For each node of the decision tree, select a random subset of the main features, and select the most suitable feature for splitting from the random subset to split according to information gain or Gini impurity to measure the data purity after feature splitting until the feature data in the node is less than the threshold or the decision tree reaches the maximum depth. Notify the split after either of the judgment conditions is met to prevent overfitting in the training of the decision tree and also control the complexity of the decision tree. Combine the prediction results of each subset and perform the final prediction through voting or averaging. After training is completed, all decision trees together form an ensemble model, that is, the motor imagery model.

[0079] It can be explained that the calculation formula based on information gain is:

[0080]

[0081] Among them, D represents the input random subset; A represents the set of features; D v represents the v-th random subset after splitting according to the feature; H(D) represents the entropy of the random subset; p i represents the proportion of the i-th class in the random subset; m represents the number of classes.

[0082] The calculation formula for Gini impurity is:

[0083]

[0084] Make an explanation that the element definitions in this formula are the same as those in information gain; select the most suitable feature for splitting from the random subset to split according to information gain or Gini impurity, that is, split the random subset according to these two criteria of information gain or Gini impurity. Information gain refers to the amount of reduction in the entropy of the entire random subset before and after splitting the random subset. The larger this value, the better the splitting effect, indicating that the feature used for splitting contributes more to classification; while the lower the Gini impurity, the smaller the randomness in the random subset and the clearer the classification result; according to these two relevant evaluation indicators, it is possible to more effectively guide the construction of the motor imagery model.

[0085] To verify the feasibility of the motor imagery model, select some fragment data from the validation dataset in actual application for prediction to obtain the classification result table 1, which is specifically as follows:

[0086] Table 1 Classification accuracy of the motor imagery model

[0087]

[0088]

[0089] Among them, it can be seen that the classification accuracy of the instructions basically remains above 75%, and the average accuracy rate is about 82%, indicating that the reliability of the motor imagery model after training is high.

[0090] Specifically, when it is determined based on the validation data set that the current brain-controlled wheelchair is in the motor imagery state, secondary filtering is performed on the validation data set, that is, time domain and frequency domain methods are used for smoothing and denoising to eliminate eye artifacts, which is applicable to motor imagery with rapid changes; extract eigenvalue, that is, obtain the band energy related to motor imagery; load the weight file, that is, the motor imagery model after being trained through the foregoing steps, match the validation data set with the motor imagery model, receive and judge the electroencephalogram signal of the user in real time as any instruction of up, down, left, or right, and correspondingly control the brain-controlled wheelchair to operate to enter the moving state according to the classification result output by the model to ensure precise motion control.

[0091] Furthermore, in step S5, it includes:

[0092] Perform secondary filtering processing based on the validation data set to determine the concentration;

[0093] Successively give the preset low-speed, medium-speed, and high-speed data ranges, compare the data ranges with the concentration, and correspondingly control the moving speed of the brain-controlled wheelchair;

[0094] Monitor the blink signal. If there is a blink signal, stop the operation of the brain-controlled wheelchair and return to the stop state.

[0095] Specifically, perform secondary filtering on the validation data set to remove eye artifacts and ensure that the extracted features are clearer and more accurate; the concentration refers to evaluating the degree of the user's concentrated attention through the features of the electroencephalogram signal. The higher the concentration, the more concentrated the user's attention. At this time, the moving speed of the brain-controlled wheelchair can be increased accordingly according to the road conditions; that is, when the concentration is low, it means that the user's attention is more scattered, and the brain-controlled wheelchair is controlled to move at a lower speed; when the concentration is moderate, the speed of the brain-controlled wheelchair is at a medium speed; when the concentration is high, it means that the user is concentrating, and the moving speed of the brain-controlled wheelchair can be accelerated to improve the intelligence of the entire brain-controlled wheelchair and ensure that the moving speed conforms to the user's intention; at the same time, monitor the blink signal. If a blink signal is detected, a stop command is triggered, indicating that the current user may issue the next motor instruction different from the current instruction to enter a different motor imagery state; making the entire brain-controlled wheelchair not only be able to flexibly adjust the speed of the brain-controlled wheelchair according to the user's attention level, but also perform safety braking when issuing the next demand instruction, improving the user experience and safety.

[0096] To solve the above technical problems, the second embodiment of the present application provides a control system for a multi-modal intelligent brain-controlled wheelchair, and the system includes:

[0097] An electroencephalogram acquisition device, configured to: acquire the electroencephalogram signals of the brain-controlled wheelchair user, perform preprocessing, and generate a verification data set;

[0098] A main control module and a safety alarm system, configured to: determine any one of the control states of the brain-controlled wheelchair currently presenting as a stopped state, a motor imagery state, and a moving state based on the verification data set according to judgment conditions; and acquire the blood oxygen data of the brain-controlled wheelchair user, set a judgment threshold, and when the blood oxygen data is lower than the judgment threshold, perform an emergency brake and send out a distress signal;

[0099] A data analysis and data processing module, configured to: if the brain-controlled wheelchair currently presents as a stopped state, continue to acquire the electroencephalogram signals of the user to obtain blink signals and enter the motor imagery state; if the brain-controlled wheelchair currently presents as a motor imagery state, extract the feature values of the verification data set, load the weight file, determine the instruction sent by the electroencephalogram signal, and control the brain-controlled wheelchair to enter the moving state based on the instruction; if the brain-controlled wheelchair currently presents as a moving state, analyze the verification data set to calculate the concentration, and control the moving speed of the brain-controlled wheelchair according to the concentration.

[0100] Preferably, in this embodiment, the electroencephalogram acquisition device adopts eight-channel needle-shaped dry electrodes, and the device communicates wirelessly with the brain-controlled wheelchair through Bluetooth, which is convenient and efficient; for the main control module and the safety alarm system, the main control module uses a microcontroller as the core processing unit, which is responsible for coordinating and managing the work of each sub-module to ensure the efficient operation of the entire brain-controlled wheelchair. The safety alarm system is connected to the blood oxygen detection device to respond in a timely manner to the abnormal physiological data of the user; the data analysis and data processing module uses software for machine learning algorithms to improve the efficient management and control of the hardware module to ensure real-time performance and reliability.

[0101] It should be noted that a control system for a multi-modal intelligent brain-controlled wheelchair has the same beneficial effects as the previously provided control method for a multi-modal intelligent brain-controlled wheelchair, and will not be elaborated here.

[0102] It should be noted that: the above sequence of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A control method based on a multimodal intelligent brain-controlled wheelchair, characterized in that: The method comprises: Collect EEG signals from brain-controlled wheelchair users and perform preprocessing to generate a validation data set; Based on the verification data set, determine whether the brain-controlled wheelchair is currently in any of the control states of stop, motor imagination and moving according to the judgment conditions; obtain the blood oxygen data of the brain-controlled wheelchair user, set the judgment threshold, and when the blood oxygen data is lower than the judgment threshold, emergency braking and a call for help are issued; If the brain-controlled wheelchair is currently in a stopped state, continue to collect the user's EEG signal to obtain the blink signal and enter the motor imagery state; If the brain-controlled wheelchair is currently in a motor imagination state, extract the eigenvalues ​​of the verification data set, load the weight file, determine the instructions issued by the EEG signal, and control the brain-controlled wheelchair to enter a moving state based on the instructions; If the brain-controlled wheelchair is currently in a moving state, the verification data set is analyzed to calculate the concentration, and the moving speed of the brain-controlled wheelchair is controlled according to the concentration.

2. A control method based on a multimodal intelligent brain-controlled wheelchair according to claim 1, characterized in that: The preprocessing includes any data cleaning method including filtering out clutter, integrating and amplifying, filling in null values, and converting digital-to-analog electrical signals.

3. The control method based on a multimodal intelligent brain-controlled wheelchair according to claim 1, characterized in that: Based on the validation data set, the brain-controlled wheelchair is currently in any of the following control states: stop state, motor imagination state, and moving state, according to the judgment conditions, including: Based on the first type of electrode signal in the verification data set, it is determined that the brain-controlled wheelchair is currently in a stopped state, based on the second type of electrode signal, it is determined that the brain-controlled wheelchair is currently in a motor imagination state, and based on the first type of electrode signal combined with the third electrode signal, it is determined that the brain-controlled wheelchair is currently in a moving state; the first type of electrode signal includes Fp1 and Fp2, the second type of electrode signal includes C3, C4 and Cz, and the third electrode signal is Fz.

4. The control method based on a multi-modal intelligent brain-controlled wheelchair according to claim 1, characterized in that: Obtain the blood oxygen data of the brain-controlled wheelchair user and set a judgment threshold. When the blood oxygen data is lower than the judgment threshold, emergency braking and a call for help are issued, including: Obtain blood oxygen detection equipment to collect blood oxygen data of brain-controlled wheelchair users; The brain-controlled wheelchair is provided with a safety alarm system and a judgment threshold. When the blood oxygen data is lower than the judgment threshold, emergency braking is performed, all power supplies except the safety alarm system are cut off, sound and light prompts are issued based on the safety alarm system, and positioning information is sent to the hospital monitoring platform and / or the guardian.

5. The control method based on a multimodal intelligent brain-controlled wheelchair according to claim 1, characterized in that: Get blink signals, including: Mark the time the brain-controlled wheelchair waits for a blink signal as the first time, and the time it waits for the next blink signal after the blink signal ends as the second time; The first screening threshold and the second screening threshold are set respectively. When the first time is greater than the first screening threshold, it is determined to be a blink signal and enters the second time; when the first time is less than the second screening threshold, the first time is reacquired.

6. The control method based on a multi-modal intelligent brain-controlled wheelchair according to claim 1, characterized in that: If the brain-controlled wheelchair is currently in a motor imagination state, extract the eigenvalues ​​of the verification data set, load the weight file, determine the instructions issued by the EEG signal, and control the brain-controlled wheelchair to enter a moving state based on the instructions, including: Constructing a motor imagery model, inputting the verification data set into the motor imagery model to extract feature values, loading the weight file stored in the brain-controlled wheelchair, and classifying the verification data set to determine the instructions issued by the user of the brain-controlled wheelchair, wherein the instructions include turning forward, backward, left, and right; According to different instructions, the brain-controlled wheelchair can enter the corresponding moving state.

7. The control method based on a multi-modal intelligent brain-controlled wheelchair according to claim 6, characterized in that: Construct a motor imagery model, and the corresponding calculation formula is: in, represents the instruction category finally predicted; C represents all categories of issued instructions; B represents the number of decision trees in the random forest built based on the validation data set; b represents the bth tree in the random forest; c b represents the predicted value of the bth tree; I(c b =c) represents the indicator function, that is, c b =c, indicating that the value of the indicator function is 1 or 0.

8. The control method based on a multi-modal intelligent brain-controlled wheelchair according to claim 1, characterized in that: If the brain-controlled wheelchair is currently in a moving state, analyze the verification data set to calculate the concentration, and control the movement speed of the brain-controlled wheelchair according to the concentration, including: Perform secondary filtering based on the validation data set to determine the degree of focus; The preset low-speed, medium-speed and high-speed data ranges are given in turn, and the data ranges are compared with the concentration, and the moving speed of the brain-controlled wheelchair is controlled accordingly; Monitor the blink signal. If there is a blink signal, stop the brain-controlled wheelchair and return to the stopped state.

9. A control system based on a multimodal intelligent brain-controlled wheelchair, characterized in that: The system comprises: EEG acquisition equipment, used to: collect EEG signals from brain-controlled wheelchair users, perform preprocessing, and generate a verification data set; The main control module and the safety alarm system are used to: determine whether the brain-controlled wheelchair is currently in a stopped state, a motor imagination state, or a moving state based on the verification data set and the judgment conditions; and obtain the blood oxygen data of the brain-controlled wheelchair user, set the judgment threshold, and when the blood oxygen data is lower than the judgment threshold, emergency braking and a call for help are issued; The data analysis and data processing module is used to: if the brain-controlled wheelchair is currently in a stopped state, continue to collect the user's electroencephalogram signal to obtain a blink signal to enter a motor imagination state; if the brain-controlled wheelchair is currently in a motor imagination state, extract the characteristic values ​​of the verification data set, load the weight file, determine the instructions issued by the electroencephalogram signal, and control the brain-controlled wheelchair to enter a moving state based on the instructions; if the brain-controlled wheelchair is currently in a moving state, analyze the verification data set to calculate the concentration, and control the moving speed of the brain-controlled wheelchair according to the concentration.

Images