Brain-computer fusion grabbing system and method for brain-controlled upper limb rehabilitation robot

Abstract

The invention discloses a brain-computer fusion grabbing system and method for a brain-controlled upper limb rehabilitation robot, relates to the field of brain-computer interfaces, and aims to solve the problems that a traditional robot is single in grabbing control mode and poor in autonomy in a dynamic non-structural environment. The system comprises a brain control module based on fine hand movement, an intention reasoning module based on fine hand movement and a rehabilitation robot body module. Wherein the brain control module based on the fine hand movement is connected with the intention reasoning module based on the fine hand movement through the first wireless communication module, and the intention reasoning module based on the fine hand movement is connected with the rehabilitation robot body module through the second wireless communication module. The brain control module based on the hand fine movement sends a brain control instruction to the intention reasoning module based on the hand fine movement through the first wireless communication module, and the intention reasoning module based on the hand fine movement sends a control instruction to the rehabilitation robot body module through the second wireless communication module.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface, and in particular to a brain-computer fusion grasping system and method for an upper limb rehabilitation robot based on brain control intention and knowledge reasoning for disabled patients. Background Art

[0002] Stroke is a disease of local brain neurological dysfunction caused by cerebral blood circulation disorders, accompanied by symptoms such as language disorders and limb disorders. It has the characteristics of high morbidity and high disability rate. Most stroke patients after recovery have sequelae of hemiplegia. In the rehabilitation treatment of stroke prognosis, traditional rehabilitation robots (such as handle control, voice control and touch screen control, etc.) are often used to rehabilitate the patient's prognosis limb function. However, this method has problems such as poor adaptability, lack of real-time feedback, low patient participation, and complex operation.

[0003] As a cutting-edge human-computer interaction method, brain control technology has the advantage of not relying on the peripheral nervous system and directly establishing a communication channel between the brain and external devices. It can help patients restore basic limb functions by controlling peripheral devices. Helping rehabilitation robots understand the brain control intentions of patients with limb dysfunction can restore the basic self-care ability of patients with motor dysfunction. Therefore, it is very necessary to realize the brain-computer fusion grasping method based on brain control intentions of rehabilitation robots.

[0004] However, the current rehabilitation robot grasping methods based on brain control technology have problems such as single control mode and poor autonomy, and are difficult to adapt to the interaction needs of dynamic environments. Summary of the invention

[0005] The purpose of the present invention is to provide a brain-controlled upper limb rehabilitation robot brain-machine fusion grasping system and method to solve the problems of single grasping control mode of traditional robots and poor autonomy in dynamic unstructured environments.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: In the first aspect, the present invention provides a brain-machine fusion grasping system for a brain-controlled upper limb rehabilitation robot, comprising a brain control module based on fine hand movements, an intention reasoning module based on fine hand movements, and a rehabilitation robot body module; wherein the brain control module based on fine hand movements is connected to the intention reasoning module based on fine hand movements through a first wireless communication module, the intention reasoning module based on fine hand movements is connected to the rehabilitation robot body module through a second wireless communication module, the brain control module based on fine hand movements sends brain control instructions to the intention reasoning module based on fine hand movements through the first wireless communication module, and the intention reasoning module based on fine hand movements sends control instructions to the rehabilitation robot body module through the second wireless communication module.

[0007] Furthermore, the brain control module based on fine hand movements includes an EEG signal acquisition submodule and an EEG signal processing submodule, and the EEG signal acquisition submodule and the EEG signal processing submodule are connected through a third wireless communication module; wherein, the EEG signal acquisition submodule is mainly used to collect EEG signals when the operator performs fine hand movements, and transmit the EEG signals to the EEG signal processing submodule, and the EEG signal processing submodule pre-processes the received EEG signals, extracts multi-domain features, and decodes the operator's brain control intention based on a convolutional neural network.

[0008] Furthermore, the intention reasoning module based on fine hand movements includes a knowledge reasoning submodule and an intention decision submodule, and the knowledge reasoning submodule and the intention decision submodule are signal-connected; wherein, the knowledge reasoning submodule is implemented by relying on an object availability knowledge graph reasoning unit and an expert system reasoning unit based on ward object grasping intention analysis, and the intention decision submodule decides the final object to be grasped based on the graspable target inferred by the knowledge reasoning submodule and the multi-attribute factors of the graspable target, and transmits it to the main body module of the rehabilitation robot through the wireless communication module.

[0009] Furthermore, the rehabilitation robot body module includes a visual perception submodule and a motion control submodule, wherein the visual perception submodule is responsible for photographing and marking items in the item placement area in the ward, and then transmitting them to the item availability knowledge graph reasoning unit through the second wireless communication module for online knowledge graph construction; the motion control submodule receives the control instructions inferred by the intention reasoning module based on fine hand movements through the second wireless communication module to complete the autonomous grasping of items.

[0010] In a second aspect, the present invention provides a brain-machine fusion grasping method for a brain-controlled upper limb rehabilitation robot, which is implemented based on the above-mentioned brain-machine fusion grasping system for a brain-controlled upper limb rehabilitation robot, and comprises the following steps: S1. Identification of the operator's brain control intention based on hand fine motor EEG signal decoding; S2, intention reasoning of rehabilitation robots based on the affordance knowledge graph of graspable objects; S3, expert system knowledge reasoning strategy based on environmental factors; S4. Rehabilitation robot grasping target decision-making based on multi-attribute weight analysis; S5. The rehabilitation robot autonomously controls the grasping motion of the final object to be grasped.

[0011] Furthermore, the specific steps of identifying the operator's brain control intention based on hand fine movement EEG signal decoding include: S11. Design a brain-controlled paradigm based on fine hand movements; Six kinds of hand fine movements, including right hand five-finger closing movement, right index finger single-finger stretching movement gesture, right thumb single-finger stretching movement, left hand five-finger closing movement, left index finger single-finger stretching movement gesture and left thumb single-finger stretching movement, were selected as target movements of the brain control paradigm; S12, EEG signal acquisition of fine hand movements; The operator performs one of the six hand fine movements included in the hand fine movement brain control paradigm according to his own needs, and the EEG signal acquisition submodule acquires the EEG signal of the operator's current hand fine movement; Among them, the EEG signal acquisition submodule selects the 64-lead system under the international standard 10-20 for acquisition; S13, Multi-domain feature extraction of hand fine motor EEG signals; The EEG signals collected in S12 were subjected to 4-40 Hz Butterworth bandpass filtering and trend removal processing, followed by extraction of time domain, frequency domain, spatial domain and time-frequency domain features, and the extracted multi-domain features were optimized using transfer learning and reinforcement learning for subsequent decoding of hand fine motor EEG signals; S14. Construct a convolutional neural network model based on multi-domain feature fusion; The extracted multi-domain features of the EEG signals are fused in the convolutional neural network by using the feature intermediate fusion method, and the network model is used to decode the multi-domain features of the hand fine movement EEG signals obtained by S13 to obtain the brain control command information of the corresponding operator.

[0012] Furthermore, the specific steps of the rehabilitation robot's intention reasoning based on the affordance knowledge graph of graspable items include: S21. Establish an offline knowledge graph of item availability based on grabbable items in the ward environment; According to the definition of item availability, the grasping relationship between common items in the ward and fine hand movements is mapped to item availability to obtain the corresponding relationship between fine hand movements, item availability and items to be grasped, and construct an offline knowledge graph of item availability; S22. Construct an online knowledge graph of the item availability of the graspable items in the dynamic unstructured environment where the rehabilitation robot is located; The visual perception submodule takes pictures of the dynamic unstructured environment in which the rehabilitation robot is located, and uses the YOLOV5 method to detect, mark and output objects that the rehabilitation robot can grasp; S23, matching mechanism of online knowledge graph units; First, based on the offline knowledge graph of item availability constructed in S21, extract the triple information of fine hand movements, item availability and items to be grasped and convert it into an offline graph list represented by dictionary key-value pairs; secondly, compare the item label information extracted in S22 with the offline knowledge graph list to obtain an online knowledge graph list, and finally determine the list 1 of items to be grasped by the operator based on the brain control intention instructions decoded in step S14.

[0013] Furthermore, the expert system knowledge reasoning strategy based on environmental factors is to establish an expert system based on ward object grasping intention analysis according to the environmental factors in the dynamic unstructured environment where the rehabilitation robot is located, and determine the relationship between the environmental factors and the grasped objects. The specific steps include: S31. Establish an expert system knowledge base based on the operator's environmental factors; According to the relationship between the four factors of time, temperature, special events and special time in the operator's environment and the grasped objects, as well as historical grasped object cases, the knowledge base of the expert system is constructed; S32, establishing an expert system reasoning engine based on traversal case search strategy; According to the four factors of time, temperature, special events and special time input into the expert system, the cases contained in the knowledge base are searched and determined to determine the list of objects to be grasped based on environmental factors 2; Furthermore, the specific steps of the rehabilitation robot grasping target decision based on multi-attribute weight analysis include: S41. Determine the multi-attribute factors based on brain control intention, environment, temperature, time, and event importance, and give their weight values ​​based on experience and common sense, and determine the attribute state values ​​of the five attribute decision models based on brain control intention and environmental factors; S42, respectively calculating the state value of each item in the list of items to be grasped 1 inferred from the brain control intention in the online knowledge graph in S23 and the list of items to be grasped 2 inferred from the environmental attribute characteristics of the expert system in S32; S43. Perform weighted sum calculation on the scores of the items to be grasped according to the feature weights set in step S41, and then convert the final score into a probability value. The item corresponding to the maximum probability value is the item to be grasped by the rehabilitation robot.

[0014] Furthermore, the autonomous grasping motion control of the final object to be grasped by the rehabilitation robot body module is achieved by autonomously planning the motion trajectory according to the final object to be grasped result obtained in S43, and converting the motion trajectory into motion control instructions and sending them to its own motion control submodule.

[0015] Compared with the prior art, the beneficial technical effects of the present invention are: (1) The present invention extracts features from EEG signals from four dimensions: time domain, frequency domain, spatial domain, and time-frequency domain. It also uses a feature optimization algorithm that combines transfer learning with reinforcement learning and a feature fusion convolutional neural network model to fully utilize the features of EEG signals. This can effectively improve the decoding accuracy of EEG signals of fine hand movements, and solve the problem of insufficient feature extraction and utilization of EEG signals of fine hand movements in the prior art. (2) The present invention designs a reasoning strategy based on the combination of the knowledge graph of item availability and the expert system, which enables the robot to infer the grasping results of more targets with fewer brain control instructions, effectively improving the rehabilitation robot's reasoning ability about the operator's intentions and solving the problem in the prior art that the brain control strategy has a small number of brain control instructions and cannot meet the requirements of use in unstructured environments; (3) In view of the problem that the current brain-machine fusion control strategy has a single control and low execution efficiency, the present invention combines the brain control strategy based on biological intelligence with the reasoning strategy based on knowledge-driven, so that the brain-controlled grasping system can fully understand the brain control intention of the subject when selecting grasping objects. The intelligence of the brain-controlled rehabilitation robot grasping system is improved, and the execution efficiency of the overall brain-machine interaction system is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a control principle block diagram of a brain-machine fusion grasping system of a brain-controlled upper limb rehabilitation robot of the present invention; Figure 2 It is an algorithm flow chart of a brain-machine fusion grasping method of a brain-controlled upper limb rehabilitation robot of the present invention; Figure 3 Schematic diagram of six kinds of hand fine movements selected by the present invention; Figure 4 Schematic diagram of the electrode arrangement of the EEG signal processing submodule of the present invention; Figure markings: 110 - brain control module based on fine hand movements, 120 - intention reasoning module based on fine hand movements, 130 - first wireless communication module, 131 - second wireless communication module, 133 - third wireless communication module, 140 - rehabilitation robot body module, 210 - EEG signal acquisition submodule, 220 - portable EEG signal processing submodule, 310 - knowledge reasoning submodule, 320 - intention decision submodule, 410 - visual perception submodule, 420 - motion control submodule, 510 - expert system reasoning unit based on ward item grabbing intention analysis, 520 - item availability knowledge graph reasoning unit. DETAILED DESCRIPTION

[0017] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0018] Example 1 Please refer to Figure 1 A brain-controlled upper limb rehabilitation robot brain-machine fusion grasping system of the present embodiment includes a brain control module 110 based on fine hand movements, an intention reasoning module 120 based on fine hand movements, and a rehabilitation robot body module 140; wherein, the brain control module 110 based on fine hand movements is connected to the intention reasoning module 120 based on fine hand movements through a first wireless communication module 130, and the intention reasoning module 120 based on fine hand movements is connected to the rehabilitation robot body module 140 through a second wireless communication module 131, the brain control module 110 based on fine hand movements sends brain control instructions to the intention reasoning module 120 based on fine hand movements through the first wireless communication module 130, and the intention reasoning module 120 based on fine hand movements sends control instructions to the rehabilitation robot body module 140 through the second wireless communication module 131.

[0019] Specifically, the brain control module 110 based on fine hand movements includes an EEG signal acquisition submodule 210 and an EEG signal processing submodule. The EEG signal acquisition submodule 210 and the EEG signal processing submodule are connected through a third wireless communication module 133; wherein, the EEG signal acquisition submodule 210 is mainly used to collect EEG signals when the operator performs fine hand movements, and transmit the EEG signals to the EEG signal processing submodule. The EEG signal processing submodule pre-processes the received EEG signals, extracts multi-domain features, and decodes the operator's brain control intention based on a convolutional neural network to obtain the operator's intention instructions.

[0020] Please refer to Figure 4 In this embodiment, the EEG signal acquisition submodule 210 worn by the operator adopts a portable 64-channel wireless EEG acquisition device, such as a portable wired NeuSen H64 lead EEG cap. The EEG signals of the Fz, FC3, FC1, FCz, FC2, FC4, C5, C3, C1, Cz, C2, C4, C6, CP3, CPz, and CP4 electrode positions under the international standard 10-20 channels are selected.

[0021] Specifically, the intention reasoning module 120 based on fine hand movements includes a knowledge reasoning submodule 310 and an intention decision submodule 320, and the knowledge reasoning submodule 310 and the intention decision submodule 320 are signal-connected; wherein, the knowledge reasoning submodule 310 is implemented by relying on the object availability knowledge graph reasoning unit 520 and the expert system reasoning unit 510 based on the ward object grasping intention analysis, and the intention decision submodule 320 decides the final object to be grasped based on the graspable target inferred by the knowledge reasoning submodule 310 and the multi-attribute factors of the graspable target, and transmits it to the main body module of the rehabilitation robot through the wireless communication module.

[0022] The working principle of the intention reasoning module 120 based on fine hand movements is as follows: first, an offline knowledge graph of item availability based on graspable items in the ward environment is established; secondly, the visual perception submodule 410 in the rehabilitation robot body module 140 identifies the graspable items contained in the dynamic unstructured environment, and transmits the identification results to the intention reasoning module 120 based on fine hand movements through the second wireless communication module 131; the item availability knowledge graph reasoning unit 520 in the knowledge reasoning submodule 310 obtains the item availability online knowledge graph through the trained offline knowledge graph unit and the graspable items identified by the visual perception submodule 410 in the rehabilitation robot body module 140. The subject's brain control intention is matched with the online knowledge graph to obtain the object 1 to be grasped by the operator; the expert system reasoning unit 510 based on the analysis of the intention to grasp items in the ward in the knowledge reasoning submodule 310 establishes a knowledge base and an inference engine according to the environmental factors of the ward to obtain the object 2 to be grasped by the operator; the intention decision submodule 320 performs multi-attribute decision-making on the obtained objects 1 to be grasped and 2 to be grasped, and finally obtains the final object to be grasped by the rehabilitation robot, and transmits it to the rehabilitation robot main body module 140 through the second wireless communication module 131.

[0023] Specifically, the rehabilitation robot body module 140 includes a visual perception submodule 410 and a motion control submodule 420, wherein the visual perception submodule 410 is responsible for photographing and marking items in the item placement area in the ward, and then transmitting them to the item availability knowledge graph reasoning unit 520 through the second wireless communication module 131 for online knowledge graph construction; the motion control submodule 420 receives the control instructions inferred by the intention reasoning module 120 based on fine hand movements through the second wireless communication module 131 to complete the autonomous grasping of items.

[0024] In this embodiment, the rehabilitation robot body module 140 has a seven-degree-of-freedom robotic arm. The motion control submodule 420 in the rehabilitation robot body module 140 autonomously plans the motion trajectory according to the object to be grasped, thereby realizing the grasping motion control of the rehabilitation robot based on brain control intention and knowledge reasoning.

[0025] Example 2 Please refer to Figure 2-4 The specific steps of a brain-machine fusion grasping method of a brain-controlled upper limb rehabilitation robot in this embodiment are: Step 610 : The operator makes fine hand movements, and the EEG signal acquisition submodule 210 acquires EEG signals, and transmits the acquired EEG signals to the EEG signal processing submodule through the third wireless communication module 133 .

[0026] In this embodiment, the EEG signal acquisition submodule 210 adopts a portable wired NeuSen H64 lead EEG cap and selects the EEG signals of the Fz, FC3, FC1, FCz, FC2, FC4, C5, C3, C1, Cz, C2, C4, C6, CP3, CPz, and CP4 electrode positions under the international 10-20 standard for acquisition.

[0027] In this embodiment, the selected fine movements of the hands are: gesture 1: closing movement of the five fingers of the right hand; gesture 2: stretching movement of the right index finger; gesture 3: stretching movement of the right thumb; gesture 4: closing movement of the five fingers of the left hand; gesture 5: stretching movement of the left index finger; gesture 6: stretching movement of the left thumb.

[0028] Step 620: The EEG signal processing submodule pre-processes the collected EEG signal.

[0029] In this embodiment, the collected original EEG signal is subjected to 4-40 Hz fourth-order Butterworth bandpass filtering and trend removal processing.

[0030] Step 630: Extract features and decode the processed EEG signal.

[0031] In this embodiment, the EEG signal processing submodule extracts the time domain, frequency domain, spatial domain and time-frequency domain features of the hand fine movement EEG signals, optimizes the extracted features through the reinforcement learning network NASNet and the transfer learning network CWTNet, and uses the feature intermediate fusion convolutional neural network (FIF-CNN) to effectively fuse the extracted features, thereby improving the EEG decoding accuracy and decoding the operator's intention instructions.

[0032] Step 640: Build an offline knowledge graph of item availability.

[0033] In this embodiment, the item availability knowledge graph reasoning unit 520 selects 60 items commonly used in the ward, and divides the 60 items into three categories of item availability associated with fine hand movements according to the definition of item availability. At the same time, the items are classified according to the inherent attributes of each item, such as category and other attributes. Finally, according to the corresponding relationship between fine hand movements, item availability and items to be grasped, an item availability offline knowledge graph containing 67 nodes is constructed.

[0034] Step 650: Build an online knowledge graph of item availability.

[0035] In this embodiment, the visual perception submodule 410 transmits the captured image of the target environment in which the rehabilitation robot is located into the Yolov5 network model to detect the object targets in the indoor environment, obtain the label of each object in the image, and save and output it; secondly, the object availability knowledge graph reasoning unit 520 combines the hand fine movement nodes and Affordance nodes in the constructed offline object availability knowledge graph and the relationship between the nodes with the identified object label information; finally, the construction of the online knowledge graph of object availability is completed.

[0036] Step 660: Matching mechanism of knowledge graph units.

[0037] In this embodiment, the item availability knowledge graph reasoning unit 520 extracts triple information of fine hand movements, item availability and items to be grasped based on the constructed item availability offline knowledge graph and converts it into dictionary key-value pairs to obtain an offline graph list; secondly, the extracted item label information is compared with the offline knowledge graph list to obtain an online knowledge graph list, and finally, the operator's brain control intention instruction decoded in step 530 is used to finally determine the list 1 of items to be grasped by the operator.

[0038] Step 670: Establish an expert system based on the analysis of the intention to grab items in the ward.

[0039] First, the expert system reasoning unit 510 based on the analysis of the intention to grab items in the ward will establish a knowledge base, that is, a case library constructed according to the relationship between the relevant environmental factors in the operator's environment and the grasped items; then the reasoning engine part is established. The case search strategy set by the expert system used in the present invention is a traversal search strategy, which traverses and searches the cases contained in the knowledge base according to the environmental factors input into the expert system until the traversal is completed, and a list of items to be grabbed 2 is obtained.

[0040] In this embodiment, the knowledge base is represented as , a complete case in the expert system case library for ward object grabbing intention analysis It is composed of four-dimensional environmental feature information and is expressed as ,in, It is the time period information within a day, specifically Indicates the time period from 8 to 9 in the morning. It means the time period from 12:00 to 13:00 at noon. Indicates the time period from 17:00 to 18:00 in the evening; is the temperature of the environment at that time, specifically Indicates moderate temperature. Indicates that the temperature is too high or too low; For special time information, specifically Indicates the time for washing (7:30, 21:30), Indicates the time to take medicine (9:30, 13:30 and 18:30), Indicates no special time; For special event information, specifically It means that you haven't drunk water for a long time. Indicates eating fruit before grabbing the item. It means to grab a tissue before grabbing the item. Said that before taking the remote control board, It means to grab the toiletries before grabbing the item. Indicates that there is no special event. The collected environmental information in the ward at that time is analyzed and the environmental characteristics are input into the expert system. The meaning of the specific environmental characteristics is shown in Table 1. Table 1 Environmental characteristics and their meanings

[0041] Step 580: Determine specific decision factors in multi-attribute decision making.

[0042] In this embodiment, the decision factors are derived from four environmental attribute features (T, W, SE, ST) in the expert system case library and the brain control intention attribute H in the online knowledge graph.

[0043] In this embodiment, according to the knowledge and experience of reasoning objects in the case library of the expert system and the relationship between brain control intention and objects in the online knowledge graph, different weights are set for the five factor attribute characteristics according to experience and common sense. The weights are set according to the importance between each factor and the object to be grasped. Since the brain control intention H is the key to determining whether the patient grasps the object, the largest weight is given to H, that is, ; Special event SE is to reasonably recommend the next item based on the last grabbed item, and the weight value is ; Special time ST is more important than ordinary time T, so the weight value is ; The importance of ordinary time T and temperature W is the same, and the weight is The final weight is set to The states and meanings of specific attribute feature variables are shown in Table 2.

[0044] Table 2 The states and meanings of multi-attribute features

[0045] The item scores are weighted and summed according to the set feature weights, and the final score is converted into a probability value. The item corresponding to the maximum probability value is the final result of the item to be captured.

[0046] Step 690: The rehabilitation robot main body module 140 autonomously plans the motion trajectory according to the final result of the object to be grasped obtained in step 680, and converts the motion trajectory into motion control instructions and sends them to its own motion control submodule 420, thereby realizing the grasping motion control of the rehabilitation robot based on brain control intention and knowledge reasoning.

Claims

1. A brain-controlled upper limb rehabilitation robot brain-machine fusion grasping system, characterized by: It includes a brain control module based on fine hand movements, an intention reasoning module based on fine hand movements, and a rehabilitation robot body module; wherein the brain control module based on fine hand movements is connected to the intention reasoning module based on fine hand movements through a first wireless communication module, the intention reasoning module based on fine hand movements is connected to the rehabilitation robot body module through a second wireless communication module, the brain control module based on fine hand movements sends brain control instructions to the intention reasoning module based on fine hand movements through the first wireless communication module, and the intention reasoning module based on fine hand movements sends control instructions to the rehabilitation robot body module through the second wireless communication module.

2. According to claim 1, a brain-controlled upper limb rehabilitation robot brain-machine fusion grasping system is characterized by: The brain control module based on fine hand movements includes an EEG signal acquisition submodule and an EEG signal processing submodule, and the EEG signal acquisition submodule and the EEG signal processing submodule are connected via a third wireless communication module; wherein the EEG signal acquisition submodule is mainly used to collect EEG signals when the operator performs fine hand movements, and transmit the EEG signals to the EEG signal processing submodule, and the EEG signal processing submodule pre-processes the received EEG signals, extracts multi-domain features, and decodes the operator's brain control intention based on a convolutional neural network.

3. The brain-controlled upper limb rehabilitation robot brain-machine fusion grasping system according to claim 2 is characterized in that: The intention reasoning module based on fine hand movements includes a knowledge reasoning submodule and an intention decision submodule, and the knowledge reasoning submodule and the intention decision submodule are signal-connected; wherein, the knowledge reasoning submodule is implemented by relying on an object availability knowledge graph reasoning unit and an expert system reasoning unit based on ward object grasping intention analysis, and the intention decision submodule decides the final object to be grasped based on the graspable target inferred by the knowledge reasoning submodule and the multi-attribute factors of the graspable target, and transmits it to the main body module of the rehabilitation robot through the wireless communication module.

4. The brain-controlled upper limb rehabilitation robot brain-machine fusion grasping system according to claim 3 is characterized by: The rehabilitation robot body module includes a visual perception submodule and a motion control submodule, wherein the visual perception submodule is responsible for photographing and marking items in the item placement area in the ward, and then transmitting them to the item availability knowledge graph reasoning unit through the second wireless communication module for online knowledge graph construction; the motion control submodule receives the control instructions inferred by the intention reasoning module based on fine hand movements through the second wireless communication module to complete the autonomous grasping of items.

5. A brain-machine fusion grasping method for a brain-controlled upper limb rehabilitation robot is implemented based on the brain-machine fusion grasping system for a brain-controlled upper limb rehabilitation robot according to claim 4, characterized in that: The following steps are included: S1. Identification of the operator's brain control intention based on hand fine motor EEG signal decoding; S2, intention reasoning of rehabilitation robots based on the affordance knowledge graph of graspable objects; S3, expert system knowledge reasoning strategy based on environmental factors; S4. Rehabilitation robot grasping target decision-making based on multi-attribute weight analysis; S5. The rehabilitation robot autonomously controls the grasping motion of the final object to be grasped.

6. The brain-machine fusion grasping method of a brain-controlled upper limb rehabilitation robot according to claim 5, characterized in that: The specific steps of identifying the operator's brain control intention based on hand fine movement EEG signal decoding include: S11. Design a brain-controlled paradigm based on fine hand movements; Six kinds of hand fine movements, including right hand five-finger closing movement, right index finger single-finger stretching movement gesture, right thumb single-finger stretching movement, left hand five-finger closing movement, left index finger single-finger stretching movement gesture and left thumb single-finger stretching movement, were selected as target movements of the brain control paradigm; S12, EEG signal acquisition of fine hand movements; The operator performs one of the six hand fine movements included in the hand fine movement brain control paradigm according to his own needs, and the EEG signal acquisition submodule acquires the EEG signal of the operator's current hand fine movement; Among them, the EEG signal acquisition submodule selects the 64-lead system under the international standard 10-20 for acquisition; S13, Multi-domain feature extraction of hand fine motor EEG signals; The EEG signals collected in S12 were subjected to 4-40 Hz Butterworth bandpass filtering and trend removal, followed by extraction of time domain, frequency domain, spatial domain and time-frequency domain features. Transfer learning and reinforcement learning were used to optimize the extracted multi-domain features for subsequent decoding of hand fine motor EEG signals. S14. Construct a convolutional neural network model based on multi-domain feature fusion; The extracted multi-domain features of the EEG signals are fused in the convolutional neural network by using the feature intermediate fusion method, and the network model is used to decode the multi-domain features of the hand fine movement EEG signals obtained by S13 to obtain the brain control command information of the corresponding operator.

7. The brain-machine fusion grasping method of a brain-controlled upper limb rehabilitation robot according to claim 6, characterized in that: The specific steps of the rehabilitation robot's intention reasoning based on the affordance knowledge graph of graspable items include: S21. Establish an offline knowledge graph of item availability based on graspable items in the ward environment; According to the definition of item availability, the grasping relationship between common items in the ward and fine hand movements is mapped to item availability to obtain the corresponding relationship between fine hand movements, item availability and items to be grasped, and construct an offline knowledge graph of item availability; S22. Construct an online knowledge graph of the item availability of the graspable items in the dynamic unstructured environment where the rehabilitation robot is located; The visual perception submodule takes pictures of the dynamic unstructured environment in which the rehabilitation robot is located, and uses the YOLOV5 method to detect, mark and output objects that the rehabilitation robot can grasp; S23, matching mechanism of online knowledge graph units; First, based on the offline knowledge graph of item availability constructed in S21, extract the triple information of fine hand movements, item availability and items to be grasped and convert it into an offline graph list represented by dictionary key-value pairs; secondly, compare the item label information extracted in S22 with the offline knowledge graph list to obtain an online knowledge graph list, and finally determine the list 1 of items to be grasped by the operator based on the brain control intention instructions decoded in step S14.

8. The brain-machine fusion grasping method of a brain-controlled upper limb rehabilitation robot according to claim 7, characterized in that: The expert system knowledge reasoning strategy based on environmental factors is to establish an expert system based on the analysis of the intention to grasp objects in the ward according to the environmental factors in the dynamic unstructured environment where the rehabilitation robot is located, and determine the relationship between the environmental factors and the grasped objects. The specific steps include: S31. Establish an expert system knowledge base based on the operator's environmental factors; According to the relationship between the four factors of time, temperature, special events and special time in the operator's environment and the grasped objects, as well as historical grasped object cases, the knowledge base of the expert system is constructed; S32, establishing an expert system reasoning engine based on traversal case search strategy; According to the four factors of time, temperature, special events and special time input into the expert system, the cases contained in the knowledge base are searched and a list of items to be grasped based on environmental factors is determined2.

9. The brain-machine fusion grasping method of a brain-controlled upper limb rehabilitation robot according to claim 8, characterized in that: The specific steps of the rehabilitation robot grasping target decision-making based on multi-attribute weight analysis include: S41. Determine the multi-attribute factors based on brain control intention, environment, temperature, time, and event importance, and give their weight values ​​based on experience and common sense, and determine the attribute state values ​​of the five attribute decision models based on brain control intention and environmental factors; S42, respectively calculating the state value of each item in the list of items to be grasped 1 inferred from the brain control intention in the online knowledge graph in S23 and the list of items to be grasped 2 inferred from the environmental attribute characteristics of the expert system in S32; S43. Perform weighted sum calculation on the scores of the items to be grasped according to the feature weights set in step S41, and then convert the final score into a probability value. The item corresponding to the maximum probability value is the item to be grasped by the rehabilitation robot.

10. The brain-machine fusion grasping method of a brain-controlled upper limb rehabilitation robot according to claim 9, characterized in that: The autonomous grasping motion control of the rehabilitation robot for the final object to be grasped is achieved by the rehabilitation robot body module autonomously planning the motion trajectory according to the final object to be grasped result obtained in S43, and converting the motion trajectory into motion control instructions and sending them to its own motion control submodule.

Images