Auxiliary device for disabled people without arms

Abstract

The invention discloses an auxiliary device for armless disabled people, and relates to the technical field of brain-computer interfaces. In order to solve the problems that in the prior art, an invasive type and a semi-invasive type of existing wearable auxiliary devices for providing life assistance for people with upper limb disability need surgical operation, electrodes are implanted into the cerebral cortex, but the surgical risk is large, complications such as infection and bleeding are possibly caused, and long-term maintenance is needed after implantation; in order to solve the technical problems that a non-invasive interbrain interface is relatively small in risk, convenient to use, simple to operate, low in cost and relatively low in control precision, the technical scheme provided by the invention is as follows: the auxiliary system for the armless disabled person comprises a brain signal acquisition module for acquiring an electroencephalogram signal of a user in a non-invasive manner; the visual control module is used for collecting target object positioning information; and the main control module is used for generating grabbing action group signals according to the signals acquired by the brain signal acquisition module and the visual control module. The auxiliary grabbing device is suitable for being applied to auxiliary grabbing work of the arm-free handicapped.

Description

Technical Field

[0001] The invention relates to the field of brain-computer interface technology, and in particular to assisted grasping for armless disabled persons. Background Art

[0002] Due to the loss of upper limbs, armless disabled people face many difficulties in daily life and work, such as limited self-care ability and reduced employment opportunities, which greatly affects their quality of life and social participation. In recent years, with the development of science and technology, a variety of assistive products for armless people have emerged on the market, but they still have shortcomings such as limited user groups, inaccurate control, safety hazards and high prices. Therefore, how to design an assistive device for armless people with complete functions, strong applicability and low cost needs to be solved urgently.

[0003] At present, there are many external devices that can provide life assistance for upper limb disabled people, such as wearable assistive devices, voice control devices, various upper limb prostheses, etc. According to the survey, the most commonly used devices are upper limb prostheses. They can be roughly divided into decorative prostheses and functional prostheses. Decorative prostheses are mainly used to make up for appearance defects and maintain balance, and have no actual functions. In contrast, functional prostheses have actual functions such as grasping, which can better help the lives of patients with limb disabilities. They roughly include three categories: myoelectric prostheses, voice-controlled prostheses, and brain-controlled prostheses. Among them, myoelectric prostheses control the movement of prostheses by receiving bioelectric signals transmitted by the brain through the residual limb muscles. Its control method is close to that of real hands and is easy to use, but the product must rely on the residual limb muscles. If the residual limb muscles of the amputee lose their contraction function, the myoelectric prosthesis cannot be installed. Voice-controlled prostheses are not limited by factors such as the degree of disability of the amputee and the length of time of amputation. However, they are easily restricted by environmental factors and interference from other people's language, with a high possibility of misoperation and relatively low control accuracy. Brain-controlled prostheses mainly use brain-computer interface technology, which controls the movement of the robotic arm by interpreting the user's brain waves or neural signals. Compared with traditional prosthetic control, it is more precise and natural. Brain-computer interface technology can be divided into invasive brain-computer interface, semi-invasive brain-computer interface and non-invasive brain-computer interface according to the source of the signal. Among them, invasive and semi-invasive types require surgical operations to implant electrodes into the cerebral cortex, with high control accuracy, but the surgical risks are relatively high, which may cause complications such as infection and bleeding, and long-term maintenance is required after implantation. Non-invasive brain-computer interfaces have lower risks, are easy to use, simple to operate, and low cost, but their control accuracy is lower. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art that the existing wearable assistive devices for upper limb disabled people to provide life assistance, invasive and semi-invasive ones require surgical operation to implant electrodes into the cerebral cortex, with high control accuracy, but the surgical risk is relatively high, which may cause complications such as infection and bleeding, and long-term maintenance is required after implantation; non-invasive brain-interface has less risk, is easy to use, simple to operate, and low cost, but has low control accuracy, the technical solution provided by the present invention is:

[0005] Assistive system for the armless disabled, including:

[0006] A brain signal acquisition module is used to collect the user's EEG signals in a non-invasive manner;

[0007] A visual control module, used to collect the positioning information of the target object;

[0008] The main control module is used to generate a grasping action group signal according to the signals collected by the brain signal collection module and the visual control module.

[0009] Furthermore, a preferred embodiment is provided, in which the brain signal acquisition module is implemented based on Horizon X3.

[0010] Furthermore, a preferred embodiment is provided, which realizes a non-invasive method through a TGAM brain wave sensor.

[0011] Furthermore, a preferred embodiment is provided, in which the visual control module is implemented through an OPENMV module.

[0012] Based on the same inventive concept, the present invention also provides an assistive device for armless disabled persons, comprising: the system as described above, and

[0013] The robot arm module is used to respond to the grasping action group signal to complete the grasping of the target object.

[0014] Based on the same inventive concept, the present invention also provides an assisted grasping method for armless disabled persons, which is implemented based on the device and includes:

[0015] Steps for collecting user's EEG signals;

[0016] The step of collecting the location information of the target object;

[0017] The step of generating a grasping action group signal according to the user's EEG signal and the target object position information;

[0018] The step of sending the grabbing action group signal.

[0019] Based on the same inventive concept, the present invention also provides an auxiliary grasping device for armless disabled persons, which is implemented based on the above device and includes:

[0020] A module for collecting user's EEG signals;

[0021] A module for collecting location information of target objects;

[0022] A module for generating a grasping action group signal according to the user's EEG signal and the target object position information;

[0023] A module for sending the grab action group signal.

[0024] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, and when the computer reads the computer program, the computer executes the described method.

[0025] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method described.

[0026] Based on the same inventive concept, the present invention also provides a computer program product, which is a computer program. When the computer program is executed, the method described above is implemented.

[0027] Compared with the prior art, the technical solution provided by the present invention is beneficial in that:

[0028] The assistive device for armless disabled people provided by the present invention effectively solves the problem that armless disabled people cannot grasp objects independently by combining EEG control with computer vision technology. The EEG signal acquisition module uses a non-invasive TGAM brain wave sensor, providing a convenient and painless control method. Users only need to wear the sensor to control the robotic arm through brain intention, reducing usage barriers. Compared with traditional myoelectric prostheses, this method does not need to rely on muscle signals, is suitable for patients with impaired muscle function, and avoids the surgical risks brought by invasive brain-computer interfaces.

[0029] The armless disabled assistive device provided by the present invention uses a lightweight target detection algorithm FOMO deployed in the OPENMV visual module, realizing real-time object positioning and grasping functions, and enhancing the autonomy and accuracy of the system. Compared with other high-computing power devices, the FOMO algorithm can run on resource-constrained embedded devices, effectively solving the problem of insufficient computing power of the device. Compared with complex 3D vision systems, the introduction of the FOMO algorithm greatly reduces the cost of the equipment, making it more suitable for large-scale promotion and use.

[0030] The assistive device for armless disabled people provided by the present invention uses a temporal convolutional network (TCN) classification algorithm to process EEG signals, which is superior to traditional RNN and LSTM models in classification accuracy and response speed. TCN captures the temporal characteristics in EEG signals through convolution operations, solving the problem of decreased accuracy in the processing of long sequence signals. Compared with other brain control devices in existing research, this solution has significantly improved classification accuracy (90.2%) and real-time performance, allowing armless disabled people to control the robotic arm more naturally.

[0031] The armless disabled assistive device provided by the present invention solves the problem of visual module failure through a preset action group, ensuring that the device can still maintain the normal operation of basic functions under extreme conditions. Compared with control systems that rely solely on vision, this redundant design improves the reliability of the system and avoids the situation where the device cannot work due to the failure of a single control mode. This semi-manual grasping mode provides users with a simple backup operation method and increases the fault tolerance of the device.

[0032] The assistive device for the armless disabled provided by the present invention is suitable for assisting the armless disabled in grasping work. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a functional framework diagram of the assistive system for armless disabled people;

[0034] Figure 2 This is the overall framework diagram of the assistive system for armless disabled people;

[0035] Figure 3 Workflow diagram of the assistive system for armless disabled people;

[0036] Figure 4 This is the hardware framework diagram of the assistive system for armless disabled people;

[0037] Figure 5 Design framework diagram for assistive system software for armless disabled people;

[0038] Figure 6 This is a schematic diagram of the relationship between EEG classification and robot hand movements;

[0039] Figure 7 Schematic diagram of reading data from brain wave sensor;

[0040] Figure 8 Schematic diagram of the inverse kinematics solution principle. DETAILED DESCRIPTION

[0041] In order to make the advantages and benefits of the technical solution provided by the present invention more clearly reflected, the technical solution provided by the present invention is now further described in detail with reference to the accompanying drawings, specifically:

[0042] Embodiment 1: This embodiment provides an assistive system for armless disabled persons, including:

[0043] A brain signal acquisition module is used to collect the user's EEG signals in a non-invasive manner;

[0044] A visual control module, used to collect the positioning information of the target object;

[0045] The main control module is used to generate a grasping action group signal according to the signals collected by the brain signal collection module and the visual control module.

[0046] Implementation method 2: This implementation method further limits the armless disabled person assistance system provided in implementation method 1, and the brain signal acquisition module is implemented based on Horizon X3 Pie.

[0047] Implementation method three: This implementation method further limits the armless disabled person assistance system provided in implementation method one, and implements a non-invasive method through a TGAM brain wave sensor.

[0048] Implementation method 4: This implementation method further limits the armless disabled person assistance system provided in implementation method 1, and the visual control module is implemented by an OPENMV module.

[0049] Embodiment 5: This embodiment provides an assistive device for armless disabled persons, including: the system provided in embodiment 1, and

[0050] The robot arm module is used to respond to the grasping action group signal to complete the grasping of the target object.

[0051] Specifically, they include:

[0052] Main control module: Horizon X3 Pie is used as the main control chip, which is responsible for processing EEG signals, visual information and controlling the overall movement of the robotic arm. This module is the core of the system and runs deep learning algorithms to analyze EEG signals and control commands.

[0053] Sub-control module: STM32 control board, responsible for receiving instructions sent by the main control, and further controlling the bus servo to perform corresponding actions, such as the robot grabbing and placing objects, etc. The sub-control board is also responsible for storing the action group when the failure occurs, ensuring that the system can operate normally when the vision module fails.

[0054] EEG signal acquisition module: uses non-invasive TGAM EEG sensors to collect the user's EEG signals, transmit the signals to the main control module for classification and processing, and control the movement of the robotic arm. This module is mainly responsible for enabling the armless user to control the robotic arm through the brain.

[0055] Vision Control Module:

[0056] Image acquisition module: Use the OPENMV module and the laser module to locate the object, collect the image of the object and obtain the distance information.

[0057] Image processing module: The image is lightweight processed through the FOMO algorithm to determine the location of the object, and the EEG signal is used to control the robotic arm to complete the grasping task.

[0058] Robotic Arm Module: The robot arm is driven by six bus servos and has an alloy mechanical claw at the end. Through inverse kinematics solution, the robot arm is able to perform precise grab and place actions. The mechanical claw is responsible for performing the grab operation and its size can be adjusted to suit the size of different objects.

[0059] The intelligent algorithms include:

[0060] EEG classification algorithm: Use the temporal convolutional network (TCN) to classify the collected EEG signals and accurately identify the user's movement intention through a deep learning model.

[0061] Object detection algorithm: Deployed in the OPENMV module to detect the position of objects and assist the robotic arm in grasping.

[0062] The coordination relationship between the parts:

[0063] The EEG signal module cooperates with the main control module: the user transmits signals to the main control module through the EEG sensor, the main control module classifies the signals and finally converts them into control instructions for the robotic arm.

[0064] The vision module cooperates with the sub-control module: the vision module provides object positioning information, and the sub-control module drives the robotic arm to perform actions according to the commands of the main control.

[0065] The robotic arm cooperates with the control module: the main control and the auxiliary control communicate with the servo through the bus to ensure that the robotic arm can accurately grasp and place objects.

[0066] Embodiment 6: This embodiment provides an assisted grasping method for armless disabled persons, which is implemented based on the device provided in embodiment 5 and includes:

[0067] Steps for collecting user's EEG signals;

[0068] The step of collecting the location information of the target object;

[0069] The step of generating a grasping action group signal according to the user's EEG signal and the target object position information;

[0070] The step of sending the grabbing action group signal.

[0071] Embodiment 7: This embodiment provides an auxiliary grasping device for armless disabled persons, which is implemented based on the device provided in embodiment 5 and includes:

[0072] A module for collecting user's EEG signals;

[0073] A module for collecting location information of target objects;

[0074] A module for generating a grasping action group signal according to the user's EEG signal and the target object position information;

[0075] A module for sending the grab action group signal.

[0076] Embodiment 8: This embodiment provides a computer storage medium for storing a computer program. When the computer reads the computer program, the computer executes the method provided in embodiment 6.

[0077] Embodiment 9: This embodiment provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method provided in embodiment 6.

[0078] Embodiment 10: This embodiment provides a computer program product, which is a computer program. When the computer program is executed, the method provided in embodiment 1 is implemented.

[0079] Implementation Method XI: Combination Figure 1-8 This embodiment is described in detail by using specific examples to further describe the technical solution provided above. Specifically:

[0080] This embodiment proposes an assistive device for armless people based on the X3 Pie. It combines non-invasive EEG signal control with computer vision control and adopts a portable design. It is more intelligent and convenient to use, while completing basic life tasks at a lower cost.

[0081] The technical difficulties faced by this implementation are:

[0082] How to use non-contact EEG signals to complete the entire grasping process more flexibly and accurately;

[0083] How to match the acquired EEG signals with the expected actions;

[0084] How to solve the problem that the EEG signals of different users are different and the present implementation cannot adapt to every user.

[0085] Based on the above difficulties, the specific functions that need to be implemented in this implementation are as follows:

[0086] 1. Ability to control the device based on EEG signals;

[0087] 2. Ability to detect and locate objects to be grasped;

[0088] 3. Capable of grabbing and placing objects to be grabbed;

[0089] 4. Ability to save device deployment time through software;

[0090] It has good stability and can still work after some functions fail.

[0091] In summary, the overall functional framework of the armless assistive device is as follows: Figure 1 As shown, it is mainly divided into intelligent crawling and placement in the user software interaction part.

[0092] The device mainly combines non-invasive EEG signals with computer vision to control the robot arm to grab and place objects in life, so as to assist the daily life of armless disabled people. When unexpected situations such as visual module failure occur, EEG signals can also be used alone for control, and the robot arm can be debugged according to EEG data through software.

[0093] This implementation is mainly composed of Horizon X3, STM32 control board, EEG sensor, OPENMV vision module, laser module, intelligent algorithm and bus manipulator. The overall framework is as follows Figure 2 As shown in the figure, the X3 Pie is the main control, the STM32 control board is the secondary control, the OPENMV module and laser module are used to obtain image information, and the EEG sensor is used to obtain EEG signals. The EEG signals are transmitted to the main control, the image information is transmitted to OPENMV, and the bus servo manipulator is used to grab and place objects.

[0094] The overall principle of the device is to use the EEG module and the image module to control the manipulator and complete the corresponding functions. The Xuri X3 main control board is equipped with a self-developed EEG signal classification algorithm to process EEG signals. EEG instructions will be used to issue global control instructions, which will be transmitted to the STM32 sub-controller to control the manipulator accordingly. The OPENMV module is equipped with a target detection algorithm, which cooperates with the distance information obtained by the laser module to determine the location of the object, and cooperates with the EEG control instructions to complete the corresponding actions.

[0095] The device will enter the sleep state after 15 seconds of non-use. When the EEG signal is detected, the device enters the initial state.

[0096] In the initial state, EEG signal control has the highest priority, and the visual module control process is locked. The user rotates the robotic arm up, down, left, and right according to the EEG signal. After the OPENMV module can detect the target to be grasped based on the laser guidance, it uses the EEG signal to issue a grasping instruction. At this time, the grasping action has the highest priority, and the EEG signal control process is locked. At this time, the visual module is used to locate and solve the inverse kinematics to grasp the object. After the grasping process is completed, the EEG signal control priority is restored to the highest, and the visual module control process is locked. At this time, the EEG signal can be used to issue a release instruction or perform subsequent actions. The system flow chart is as follows Figure 3 shown.

[0097] Since the visual module may be damaged and the detection network may be unable to detect the object, a set of actions is pre-set. When the detection module fails, the mechanical claw opens and closes twice to give a prompt, and then the system automatically calls the action group for semi-manual grasping to ensure the normal operation of the equipment.

[0098] The overall framework of the system:

[0099] The hardware devices used in this implementation mainly include Horizon X3 Pie, STM32 control board, EEG sensor, OPENMV module, laser module (including laser ranging module and visible laser module), bus servo, alloy manipulator and battery module. The overall architecture of the hardware design system is as follows: Figure 4 As shown;

[0100] In this embodiment, the main control X3 Pie is connected to the EEG sensor and the sub-control STM32 control board at the same time. The X3 Pie has one-way communication with the EEG sensor and the STM32 control board. The OPENMV module is connected to the STM32 control board and the laser module at the same time. The OPENMV module has two-way communication with the STM32 control board and one-way communication with the laser ranging module. There is no direct communication between the OPENMV module and the X3 Pie, but there is a control relationship between the two. The bus servo is connected to the STM32 control board, and the two have two-way communication. The alloy manipulator is connected to the bus servo. The visible laser module, alloy manipulator and battery module do not have communication functions.

[0101] Hardware design of main control and sub-control modules:

[0102] The main control module of this implementation is mainly responsible for processing a large amount of information, transmitting data between various modules, and needs to be equipped with corresponding deep learning algorithms. These conditions require the main control module to have powerful computing power and abundant serial ports. To meet the usage requirements, this implementation uses the Horizon X3 Pie as the main control chip:

[0103] The main parameters corresponding to the Horizon X3 Pie are shown in Table 1:

[0104] Table 1

[0105]

[0106] The sub-control module of this embodiment is mainly responsible for communicating with the main control module, driving devices such as steering gears, and storing a small amount of data. It does not need to carry complex algorithms or powerful computing power. To meet the use requirements, this embodiment uses STM32F103C8T6 as the sub-control chip. Since it needs to be connected to devices such as steering gears, an expansion board is selected to be used with STM32.

[0107] The expansion board has a power supply voltage of 6-16V and has multiple interfaces for easy connection with other devices. This implementation mainly uses the bus servo interface to connect to the bus servo, and at the same time, uses its W25Q64 memory to store the action group when the vision fails. The connection method with other devices will be introduced in this summary and the bus servo summary. The main parameters corresponding to the STM32F103C8T6 chip are shown in Table 2:

[0108] Table 2

[0109]

[0110] The EEG signal acquisition module of this embodiment is mainly used to collect the user's EEG signals and transmit them to the main control chip for processing. It should have the characteristics of convenience, comfort, and stable signal. In order to meet the use requirements, this embodiment adopts a non-invasive TGAM EEG sensor module for EEG acquisition. When in use, the metal electrode contacts the forehead and the ear clip is clamped on the left earlobe.

[0111] The parameters of the TGAM brain wave sensor module are shown in Table 3:

[0112] Table 3

[0113]

[0114] Visual control module hardware design part:

[0115] The visual image acquisition module of this embodiment is mainly used to obtain the image of the object to be grasped, and locate the object with the distance information obtained by the laser ranging module, so as to facilitate the subsequent grasping action. It itself needs to have communication functions and certain data processing and computing capabilities. In order to meet the use requirements, this embodiment uses OPENMV4 H7 Plus to collect and process visual images.

[0116] Table 4

[0117]

[0118] The visual processing module of this embodiment is mainly composed of OPEMMV camera, laser module (including laser ranging module and visible laser module) and sub-control chip. For the convenience of OPENMV visual module and STM32 sub-control connection to convey information, it is used with the expansion version, and the connection mode is serial bus connection. The expansion board bus interface is connected to OPENMV camera UART3. In order to assist users to grab, OPENMV module connects visible laser module, and visible laser module is connected to the 3.3V output pin of OPEMMV module. In addition, laser ranging module is connected to OPENMV for obtaining the distance information of object, and the 3 and 4 pins of ranging module are connected to the 4 and 5 pins of OPENMV.

[0119] The visible laser module has an operating voltage of 3-5V, a size of 6×17.1mm, and is connected to the OPENMV module using DuPont wire.

[0120] The model of the laser ranging module is TOF200F, its operating voltage is 3-5V, the measurement distance is 2m, the measurement blind area is 0-3cm, the infrared emission mechanism is 940nm, the FOV angle of view is 25 degrees, and the size is 206×11mm.

[0121] Robotic arm hardware design part:

[0122] In this embodiment, the robot arm is driven by six servos, the models and quantities of which are ZX15D*3, ZX20D*1, ZX20S*1 and ZX30S*1 respectively. The six bus servos are connected in series, numbered by the host computer, and then connected to the expansion board carried by the STM32 to complete the corresponding actions. Each servo has a built-in 32-bit MCU to support data storage and data readback.

[0123] The alloy mechanical claw module, as the end effector of the robotic arm, will directly affect the accuracy of grasping objects. In order to ensure that common objects can be grasped and released, a claw-type mechanical claw is used.

[0124] The alloy mechanical claw is combined with a servo module, with an opening and closing size of 0 to 230 mm, and can grab most common objects.

[0125] Software design part of the assistive system for armless disabled people:

[0126] The overall framework of software system design is as follows Figure 5 As shown;

[0127] In this embodiment, the main program is the loop and nesting of each subroutine, which is mainly divided into three parts, namely the visual module, the EEG module and the emergency module, each of which receives and sends instructions. The system will be in a dormant state if it is not used for 15s, and the manipulator is in a default state. In the dormant state, the program initial value defaults to aobccc, a, b and c correspond to the EEG module, the visual module and the emergency module respectively, o and c represent turning on or off the program, and only the EEG module works in the dormant state, and the EEG control has the highest priority at this time. When an EEG signal instruction is issued, the manipulator will be in the initial state. When the EEG signal is read to send a grab or release instruction, the EEG module will send an acbocc instruction. At this time, the visual module has the highest priority and completes the relevant actions. After the action is completed, the visual module will send aobccc, and the EEG control priority is restored to the highest at this time, and then the next step of control is continued.

[0128] In addition, if the visual module fails after the EEG signal sends a command and its program cannot obtain the object's location information, the visual module will send acbcco. At this time, the system will enter an emergency state and will call the pre-set grasping action group to cooperate with the user to achieve semi-manual grasping. After the end, it will return to the default state.

[0129] EEG signal classification program design part:

[0130] The data set used in this embodiment is collected autonomously, and the created data set covers 8 types of actions, namely up, down, left, right, grasping, drinking water (placing the grasped object in front of the chest), putting it back (putting the grasped object back to its original place), and releasing the claws.

[0131] At the beginning of the collection, let the subject sit in a comfortable environment, wear the TGAM EEG headband, and start the collection of motor imagery EEG data sets. First, a black screen is displayed on the screen for 10 seconds, and then eight action prompts are released in turn. The display time of each sign is 10 seconds, and a black screen of 5 seconds is played between adjacent prompts. After all actions are collected, the black screen is played for 10 seconds. That is, the collection time of one round is 115 seconds. By analogy, let the subject collect data for 5 rounds. After completion, let the subject rest for 5 minutes, and then collect data again, for a total of 12 times, that is, a total of 8×5×12 actions. The collected data set is not a signal data that can be used directly, but a hexadecimal serial port signal. After the data set is collected, it will be converted through the corresponding program and converted into original sequence data, and then subsequent processing and training will be carried out.

[0132] With the rise of brain-computer interface technology, people's consciousness can be perceived by the outside world through brain waves. By collecting EEG signals, we can obtain a set of time series signals, send the signals into the constructed neural network for processing, training and classification, and thus perform various tasks. In order to effectively capture the long-term dependencies in EEG signals, this embodiment uses a temporal convolutional network (TCN) for classification. TCN was developed on the basis of CNN. It draws on the convolution operation of CNN and adapts it to sequence data by introducing causal convolution and dilated convolution. At the same time, it solves some problems encountered by RNN and LSTM in long sequence processing.

[0133] Considering that the device designed in this embodiment is a portable device, we need a network model with small parameters, low computational consumption and excellent performance.

[0134] The model proposed in this embodiment mainly consists of three modules: convolution module, sliding window module and residual module. The convolution block extracts the main time, depth and spatial information in the EEG signal through double convolution branches, and its output is the primary time series after feature extraction. Due to the small size of the data set, the sliding window module is used to segment the primary time series through the sliding time window for data expansion. Then, more advanced time features are extracted through the residual block designed based on the TCN network, and sent to the fully connected layer, and classified using softmax.

[0135] The data collection process is easily affected by the surrounding environment, blinking, etc., resulting in various noises and artifacts in the collected data set. In order to ensure the accuracy of the classification results, the converted data set is preprocessed by wavelet transform denoising, bandpass filtering, normalization, etc. before training.

[0136] The data set of this embodiment is trained and tested using a ten-fold cross validation method.

[0137] After the training is completed, the model deployment phase begins. The first step is model preparation. The model file obtained after the model training of this implementation mode is saved, and then the model is converted into an ONNX model using the tf2onnx tool. Then comes model verification, which mainly ensures that the proposed algorithm model meets the BPU requirements. This implementation uses the hb_mapperchecker instruction provided by the Horizon development platform to complete the model inspection. For the layers in the program that do not meet the migration requirements, this part is transferred to the CPU for operation through manual adjustment. The last step is model conversion. In this stage, the floating-point model will be converted into a model used by the BPU. Use the function hb_mapper makertbin to complete the conversion. After the conversion is successful, the corresponding bin file is obtained. Copy this file to the X3 Pie, which is the model file required for on-board operation.

[0138] In order to verify the final application effect of the model proposed in this implementation, the converted bin file and reasoning code are copied to the board, and the reasoning code is run using the sudo command. The EEG head ring serial port information is read through the code and a series of conversions and processing are performed, and then the robotic arm is controlled to perform corresponding actions according to different imagination signs.

[0139] In order to solve the problems of low computing power and limited resources of embedded devices, this implementation method chooses to deploy the lightweight target detection algorithm FOMO algorithm in the OPENMV module. The implementation method is: first use the OPENMV module to shoot the data set and upload it to the Edgeimpulse platform, then perform online model training and export, and finally copy the generated main.py and weight files to the OPENMV module.

[0140] Intelligent drive control program design part:

[0141] The robot motion control system is mainly for the control of six bus servos. This section will explain and illustrate the principles of single servo control and six servo coordinated control.

[0142] 1. Control principle of single bus servo

[0143] The servo used in this embodiment is a bus servo, which is connected to the auxiliary control STM32 control board, rotates the corresponding angle according to its instructions, and supports the reading back and saving of the angle. The data will be saved in the memory of the expansion board for subsequent use.

[0144] Each bus servo has two control parameters: PWM value and T. The range of PWM value is 0500-2500, which can be used to control the rotation angle of the servo. The maximum rotation angle of the servo in this embodiment is 270°. The PWM value is proportional to the angle. If 0500 represents -135 degrees, 2500 represents 135 degrees. Parameter T is time, which indicates the movement time required to complete an instruction, and the range is 0-9999ms.

[0145] 2. Cooperative control principle of multiple bus servos

[0146] The six bus servos of this embodiment are controlled by the command of the control panel. The implementation principle is to number each bus servo and control a single servo, and finally achieve the function of coordinated control.

[0147] The process of modifying the bus servo ID number is as follows:

[0148] 1. Connect the bus servo to the control board, and then connect the control board to the computer;

[0149] 2. Use the bus configuration function in the host computer to change the ID number of the connected servo. At the same time, the control board will set the servo angle to the middle position by default to ensure that the manipulator can complete the corresponding action instructions as much as possible.

[0150] After numbering each servo, instructions can be issued according to the required actions.

[0151] In addition, in order to deal with emergencies and set the default posture and initial posture, this embodiment pre-sets three action groups. The action group is stored in the register of the expansion board for offline operation. The action group is a combination of multiple actions. This embodiment takes one action as an example to explain:

[0152] {#000P1500T1250#001P1860T1250#002P2000T1250#003P1000T1250#004P1500T1250#005P1200T1250}

[0153] The above is a command for an action, which contains the control parameters of six servos. #000P1500T1250 means controlling the servo with ID number 000 to rotate from the current position to the position 1500, with a total time of 1250ms.

[0154] EEG control part:

[0155] One of the functions of the auxiliary device is to use different EEG signals to control the robot arm to perform eight different actions. These eight actions are achieved by driving different servo action groups, and their corresponding relationships are as follows: Figure 6 As shown;

[0156] After the brain-computer acquisition device acquires the EEG signal, it transmits the data to the X3 Pai through the Bluetooth serial port. The X3 Pai first converts the hexadecimal serial port data into the original sequence value by parsing it, and then performs data preprocessing operations. Finally, the processed EEG sequence data is sent to the trained and deployed classification network for prediction, and the final prediction result is transmitted to the STM32 microcontroller through the pre-defined serial port. In the STM32 microcontroller, the final prediction result is converted into the corresponding action of the robotic arm, so as to achieve the effect of controlling the robotic arm through the movement imagination of the human brain.

[0157] When using EEG control, the user's brain waves will be extracted by the EEG sensor, which will be processed accordingly and then classified by the EEG classification model. Figure 7 shown.

[0158] Visual control part:

[0159] Visual control is used to compensate for the instability and inaccuracy of EEG control. After identifying an object, it will automatically perform a grabbing action according to the control instructions issued by the EEG signal. This implementation deploys the trained FOMO target detection model to the OPENMV module, which determines the coordinate position after collecting the image, and calculates the approximate position of the object in space with the distance information measured by the laser ranging module. It uses inverse kinematics to solve and calculate the angle that each servo needs to operate, and then transmits this information to the STM32 sub-controller to perform related actions.

[0160] Six-servo inverse kinematics refers to the process of solving the rotation angle of each joint in the robotic arm given the rotation angle of the six servos and the target position and posture of the end of the robotic arm. The principle of inverse kinematics solution is as follows: Figure 8 shown.

[0161] The inverse kinematics instruction in the program of this implementation is: ($KMS:x,y,z,time!), where x, y, and z represent the three-dimensional spatial coordinates of the object, and time represents the time required to move from the current position to the target position.

[0162] This implementation proposes an assistive device for armless disabled people based on Horizon X3, which has brain-electric control, intelligent vision, intelligent grasping, human-computer interaction and intelligent system. It can assist armless disabled people to perform simple daily grasping and placing actions at a relatively low cost, adding color to the lives of armless disabled people, and at the same time contributing a new idea to the design and development of assistive devices for armless disabled people. The problems and technical innovations finally solved by this implementation are summarized as follows:

[0163] 1. An auxiliary device based on the combination of EEG control and computer vision is proposed, which realizes the functions of recognition and positioning, automatic grasping and user EEG control. It excellently completes the daily grasping and releasing tasks with a grasping accuracy of 90.3% and a placement accuracy of 96.3%, reliably and effectively solving the inconvenience caused by armless disabled people in their daily lives.

[0164] 2. A TCN network-based EEG classification algorithm is proposed, which can classify EEG signals extracted by non-invasive EEG sensors with a classification accuracy of up to 90.2%, providing a safer, more accurate and more convenient implementation basis for brain control devices.

[0165] 3. It can be paired with an operating software that integrates EEG data reception, storage, conversion, processing and training, so that users only need to perform simple operations to customize it according to their own data, solving the problem of large differences in EEG signals among different users and unsatisfactory control effects using the same brain control device.

[0166] As the number of disabled people continues to increase, the demand for assistive devices for this group of people will also increase in the future. The armless disabled assistive device designed in this embodiment helps to solve the problems of limited product use conditions, poor control effects and high prices, and provides an effective solution to help more disabled people have a normal life.

[0167] Embodiment 12: This embodiment tests and evaluates the functions of the above-mentioned assistive device for the disabled without arms through specific examples, specifically:

[0168] EEG signal classification test part:

[0169] This embodiment performs sixty sets of motion imagination in sequence, and compares the actual motion of the robot arm with the user's imagined motion. Some of the results are shown in Table 5:

[0170] Table 5

[0171]

[0172] Through the test results of actual use, only 47 misclassifications occurred in a total of 480 motor imaginations. The classification accuracy of the model in practical applications is as high as 90.2%. Although there are misclassifications, since the EEG signal will only control the servo to make a small movement of about 15° each time, the algorithm fully meets the usage requirements of this implementation.

[0173] Pick and place test section:

[0174] This part tests the grasping and placing of different objects to observe whether the auxiliary device can accurately grasp the objects and stably place them in the designated position. This test uses three common objects that need to be grasped in life: a water cup, a medicine box, and a handbag. Each object is tested 100 times.

[0175] The placement process is divided into two types: placing it back to the original position and placing it at another position under EEG control. The placing back to the original position is the reverse movement of the grasping process and is completed by a predetermined program. Placing it at another position is freely controlled by the user. The grasping and placing accuracy rates are shown in Table 6:

[0176] Table 6

[0177]

[0178] As shown in Table 6, the grasping accuracy of the three items is the highest for the medicine box, the middle for the water cup, and the lowest for the handbag. This is because the medicine box and the water cup are large in size and can be deformed, so it is relatively easy to grasp. There are many types of handbags, and their shapes are not fixed, so grasping them is more difficult. However, for armless patients, grasping a handbag is not urgent or necessary, and multiple attempts can be made.

[0179] The placement accuracy was tested when the grasping was successful. The experimental results show that the placement success rates of the handbag and medicine box reached 97.6% and 100% respectively. This is because both can be placed successfully at any angle, but the handbag is made of special materials and may fall. The water cup may fall over due to the movement of the user when it is placed, so the placement success rate is relatively low. However, during the experiment, the placement accuracy of autonomous control is quite impressive.

[0180] In addition, in order to facilitate user use, the device has a preset "drinking water" program, and the instructions are issued by brain electrical control. It will automatically deliver the item to the mouth after grabbing the item. This setting needs to be debugged according to different users and can achieve almost 100% accuracy.

[0181] Visual failure test part:

[0182] Due to various restrictions, the device may have a visual control module failure. In response to this situation, the device has preset automatic grasping actions. When the visual module fails, the mechanical claw will open and close twice as a prompt, and after two seconds, the preset action group will be automatically called for semi-manual grasping.

[0183] This action group is fixed, and the robot arm opens and closes to the maximum by default. Users only need to run in briefly to use this action to grab objects. However, due to the single action, it may not be possible to grab objects in some scenarios.

[0184] In this experiment, the camera is blocked to simulate the failure of the vision module.

[0185] It can be seen that although this solution is used as a solution in emergency situations, it can still maintain a high grasping accuracy and ensure the normal operation of the equipment.

[0186] The technical solution provided by the present invention is further described in detail above through several specific implementation modes in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the several specific implementation modes described above are not intended to be used as limitations on the present invention. Any reasonable modification and improvement of the present invention, combination of implementation modes and equivalent substitution within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An assistive system for armless disabled persons, characterized in that: include: A brain signal acquisition module is used to collect the user's EEG signals in a non-invasive manner; A visual control module, used to collect the positioning information of the target object; The main control module is used to generate a grasping action group signal according to the signals collected by the brain signal collection module and the visual control module.

2. The armless disabled assistance system according to claim 1, characterized in that: The brain signal acquisition module is implemented based on Horizon X3.

3. The armless disabled assistance system according to claim 1, characterized in that: A non-invasive approach is achieved through TGAM brainwave sensors.

4. The armless disabled assistance system according to claim 1, characterized in that: The visual control module is implemented through the OPENMV module.

5. An assistive device for armless disabled persons, characterized in that: include: The system of claim 1, and The robot arm module is used to respond to the grasping action group signal to complete the grasping of the target object.

6. A method for assisting armless disabled persons to grasp, characterized in that: The method is implemented based on the device according to claim 5, and includes: Steps for collecting user's EEG signals; The step of collecting the location information of the target object; The step of generating a grasping action group signal according to the user's EEG signal and the target object position information; The step of sending the grabbing action group signal.

7. An auxiliary grasping device for armless disabled persons, characterized in that: The device is implemented based on the device according to claim 5, and includes: A module for collecting user's EEG signals; A module for collecting location information of target objects; A module for generating a grasping action group signal according to the user's EEG signal and the target object position information; A module for sending the grab action group signal.

8. A computer storage medium for storing a computing program, characterized in that: When the computer reads the computer program, the computer executes the method of claim 6.

9. A computer, comprising a processor and a storage medium, characterized in that: When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 6 .

10. A computer program product, being a computer program, characterized in that When the computer program is executed, the method according to claim 6 is implemented.

Images