Assistive device for armless disabled people

Abstract

The application relates to the technical field of brain-computer interface, and relates to an armless disabled person assisting device. In order to solve the technical problems that the existing wearable assisting device for providing life assistance for upper limb disabled persons is invasive and semi-invasive, needs to be operated, electrodes are implanted into the cerebral cortex, the operation risk is large, complications such as infection and bleeding may be caused, long-term maintenance is needed after implantation, the non-invasive brain interface has small risk, is convenient to use, simple to operate, low in cost, but has low control precision, and the technical scheme provided by the application is as follows: the armless disabled person assisting system comprises a brain signal acquisition module, a visual control module and a main control module. The brain signal acquisition module is used for acquiring the brain electrical signal of a user in a non-invasive manner. The visual control module is used for acquiring target object positioning information. The main control module is used for generating a grasping action group signal according to the signals acquired by the brain signal acquisition module and the visual control module. The application is suitable for application in the work of assisting grasping of armless disabled persons.

Description

Technical Field

[0001] It involves the field of brain-computer interface technology, specifically assistive grasping for people with armless disabilities. Background Technology

[0002] Due to the loss of their upper limbs, people with armless disabilities face numerous difficulties in daily life and work, such as limited self-care abilities and reduced employment opportunities, which significantly impacts their quality of life and social participation. In recent years, with technological advancements, various assistive products for armless individuals have emerged on the market; however, these products still suffer from shortcomings such as limited user groups, inaccurate control, safety hazards, and high prices. Therefore, designing a comprehensive, widely applicable, and cost-effective assistive device for armless individuals is an urgent task.

[0003] Currently, various external devices can provide assistance to people with upper limb disabilities in their daily lives, such as wearable assistive devices, voice-controlled devices, and various types of upper limb prostheses. According to surveys, the most commonly used devices are upper limb prostheses. These can be broadly divided into decorative prostheses and functional prostheses. Decorative prostheses are mainly used to compensate for cosmetic defects and maintain balance, without any practical function. In contrast, functional prostheses have practical functions such as grasping, and can better assist people with limb disabilities in their daily lives. They generally include three categories: myoelectric prostheses, voice-controlled prostheses, and brain-controlled prostheses. Myoelectric prostheses control the movement of the prosthesis by receiving bioelectric signals from the brain through the muscles of the residual limb. Their control method is close to that of a natural hand and is easy to use; however, this product relies on the muscles of the residual limb. If the amputee's residual limb muscles have lost their contractile function, a myoelectric prosthesis cannot be installed. Voice-controlled prostheses are not limited by factors such as the degree of disability or the length of time since amputation. However, they are easily affected by environmental factors and interference from others' speech, have a higher possibility of misoperation, and relatively lower control accuracy. Brain-controlled prostheses primarily utilize brain-computer interface (BCI) technology, which interprets the user's brainwaves or neural signals to control the movement of a robotic arm, offering greater precision and natural control compared to traditional prostheses. BCI technology can be categorized into invasive, semi-invasive, and non-invasive BCIs based on the signal source. Invasive and semi-invasive BCIs require surgical procedures to implant electrodes into the cerebral cortex, offering high control precision but carrying significant surgical risks, potentially leading to complications such as infection and bleeding, and requiring long-term maintenance after implantation. Non-invasive BCIs have lower risks, are convenient to use, simple to operate, and inexpensive, but their control precision is lower. Summary of the Invention

[0004] To address the shortcomings of existing wearable assistive devices for upper limb disabilities, which require surgical intervention to implant electrodes into the cerebral cortex (both invasive and semi-invasive methods offer high precision but carry significant surgical risks, including potential complications such as infection and bleeding, and require long-term maintenance), and the relatively low precision of non-invasive brain-to-brain interfaces (BTIs), this invention provides the following technical solution:

[0005] Assistive systems for people with armless disabilities include:

[0006] The brain signal acquisition module is used to acquire the user's electroencephalogram (EEG) signals in a non-invasive manner.

[0007] The vision control module is used to acquire the positioning information of the target object;

[0008] The main control module is used to generate grasping action group signals based on the signals acquired by the brain signal acquisition module and the visual control module.

[0009] Furthermore, a preferred embodiment is provided in which the brain signal acquisition module is implemented based on the Horizon Sunrise X3 Pie.

[0010] Furthermore, a preferred implementation method is provided, which achieves a non-invasive approach using a TGAM brainwave sensor.

[0011] Furthermore, a preferred implementation is provided in which the vision control module is implemented through an OPENMV module.

[0012] Based on the same inventive concept, the present invention also provides an assistive device for people with armless disabilities, comprising: the aforementioned system, and

[0013] The robotic arm module is used to respond to the grasping action group signal and complete the grasping of the target object.

[0014] Based on the same inventive concept, the present invention also provides an assistive grasping method for people with armless disabilities, the method being implemented based on the aforementioned device, comprising:

[0015] Steps for collecting user's electroencephalogram (EEG) signals;

[0016] The steps for collecting the location information of a target object;

[0017] The step of generating a grasping action group signal based on the user's EEG signal and the target object's position information;

[0018] The step of sending the grabbing action group signal.

[0019] Based on the same inventive concept, the present invention also provides an assistive grasping device for people with armless disabilities, the device being implemented based on the aforementioned device, comprising:

[0020] A module for acquiring users' electroencephalogram (EEG) signals;

[0021] A module for acquiring the location information of the target object;

[0022] A module that generates a set of grasping action signals based on the user's EEG signals and the target object's position information;

[0023] The module that sends the grabbing action group signal.

[0024] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computing program, wherein when the computer reads the computer program, the computer executes the method described thereon.

[0025] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, wherein when the processor reads a computer program stored in the storage medium, the computer executes the method described thereon.

[0026] Based on the same inventive concept, the present invention also provides a computer program product, which, when executed, implements the method described.

[0027] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows:

[0028] This invention provides an assistive device for people with armless disabilities that effectively solves the problem of their inability to independently grasp objects by combining EEG control and computer vision technology. The EEG signal acquisition module uses a non-invasive TGAM brainwave sensor, providing a convenient and painless control method. Users only need to wear the sensor to control the robotic arm through their brain signals, reducing barriers to use. Compared to traditional myoelectric prostheses, this method does not rely on muscle signals, making it suitable for patients with impaired muscle function, and avoids the surgical risks associated with invasive brain-computer interfaces.

[0029] The assistive device for armless disabled persons provided by this invention employs the lightweight object detection algorithm FOMO deployed in the OPENMV vision module, achieving real-time object localization and grasping functions, thus enhancing the system's autonomy and accuracy. Compared to other high-computing-power devices, the FOMO algorithm can run on resource-constrained embedded devices, effectively solving the problem of insufficient device computing power. Compared to complex 3D vision systems, the introduction of the FOMO algorithm significantly reduces device costs, making it more suitable for large-scale deployment and use.

[0030] The assistive device for armless disabled individuals provided by this invention utilizes a Temporal Convolutional Network (TCN) classification algorithm to process electroencephalogram (EEG) signals, achieving superior classification accuracy and response speed compared to traditional RNN and LSTM models. TCN captures temporal features in EEG signals through convolutional operations, addressing the issue of accuracy degradation during long-sequence signal processing. Compared to other brain-controlled devices in existing research, this approach significantly improves classification accuracy (90.2%) and real-time performance, enabling armless disabled individuals to control the robotic arm more naturally.

[0031] The assistive device for armless disabled persons provided by this invention solves the problem of vision module failure through preset action sets, ensuring that the device can maintain normal operation of basic functions even under extreme conditions. Compared with control systems that rely solely on vision, this redundant design improves the system's reliability and avoids situations where the device cannot function due to the failure of a single control mode. This semi-manual grasping mode provides users with a simple backup operation method and increases the device's fault tolerance.

[0032] The assistive device for people with armless disabilities provided by this invention is suitable for use in assistive grasping work for people with armless disabilities. Attached Figure Description

[0033] Figure 1 Functional framework diagram of assistive systems for people with armless disabilities;

[0034] Figure 2 A general framework diagram of assistive systems for people with armless disabilities;

[0035] Figure 3 Workflow diagram for assistive systems for people with armless disabilities;

[0036] Figure 4 Hardware framework diagram of an assistive system for people with armless disabilities;

[0037] Figure 5 Software design framework diagram for assistive systems for people with armless disabilities;

[0038] Figure 6 A schematic diagram illustrating the relationship between EEG classification and robotic arm movements;

[0039] Figure 7 This is a schematic diagram of data reading from an electroencephalogram (EEG) sensor.

[0040] Figure 8 This is a schematic diagram illustrating the principle of solving inverse kinematics. Detailed Implementation

[0041] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically:

[0042] Implementation Method 1: This implementation method provides an assistive system for people with armless disabilities, including:

[0043] The brain signal acquisition module is used to acquire the user's electroencephalogram (EEG) signals in a non-invasive manner.

[0044] The vision control module is used to acquire the positioning information of the target object;

[0045] The main control module is used to generate grasping action group signals based on the signals acquired by the brain signal acquisition module and the visual control module.

[0046] Implementation Method 2: This implementation method is a further limitation of the assistive system for people with armless disabilities provided in Implementation Method 1. The brain signal acquisition module is implemented based on the Horizon Sunrise X3.

[0047] Implementation Method 3: This implementation method further defines the assistive system for people with armless disabilities provided in Implementation Method 1, and achieves a non-invasive approach through the TGAM brainwave sensor.

[0048] Implementation Method 4: This implementation method is a further limitation of the assistive system for people with armless disabilities provided in Implementation Method 1. The visual control module is implemented through the OPENMV module.

[0049] Implementation Method 5: This implementation method provides an assistive device for people with armless disabilities, including: the system provided in Implementation Method 1, and...

[0050] The robotic arm module is used to respond to the grasping action group signal and complete the grasping of the target object.

[0051] Specifically, including:

[0052] Main control module: Using the Horizon Robotics X3 Pie as the main control chip, it is responsible for processing EEG signals, visual information, and controlling the overall movement of the robotic arm. This module is the core of the system, running deep learning algorithms to analyze EEG signals and control commands.

[0053] The secondary control module, an STM32 control board, receives commands from the main controller and further controls the bus servos to perform corresponding actions, such as the robotic arm grasping and placing objects. The secondary control board also stores action sets in case of failure, ensuring the system continues to operate normally even if the vision module malfunctions.

[0054] EEG signal acquisition module: This module uses a non-invasive TGAM brainwave sensor to collect the user's brainwave signals, which are then transmitted to the main control module for classification and processing to control the robotic arm's movements. This module is primarily responsible for enabling armless users to control the robotic arm through their brains.

[0055] Vision control module:

[0056] Image acquisition module: Using the OPENMV module in conjunction with the laser module, the object is located, images of the object are acquired, and distance information is obtained.

[0057] Image processing module: Lightweight processing of images using the FOMO algorithm to determine the position of objects, and combined with EEG signals to control the robotic arm to complete the grasping task.

[0058] Robotic Arm Module: The robotic arm is driven by six bus servos and has an alloy mechanical gripper at its end. Through inverse kinematics solutions, the robotic arm can perform precise grasping and placement actions. The mechanical gripper is responsible for performing the grasping operation, and its size is adjustable to accommodate different object sizes.

[0059] The intelligent algorithms include:

[0060] EEG classification algorithm: The acquired EEG signals are classified using a Temporal Convolutional Network (TCN), and the user's movement intention is accurately identified through a deep learning model.

[0061] Target detection algorithm: Deployed in the OPENMV module, it is used to detect the position of objects and assist the robotic arm in grasping them.

[0062] The coordination relationship between the parts:

[0063] The EEG signal module works in conjunction with the main control module: the user transmits signals to the main control module through the EEG sensor, the main control module classifies the signals and finally converts them into control commands for the robotic arm.

[0064] The vision module works in conjunction with the secondary control module: the vision module provides object positioning information, and the secondary control module drives the robotic arm to perform actions according to the commands of the primary control module.

[0065] The robotic arm works in conjunction with the control module: the main controller and the auxiliary controller communicate with the servo motor via a bus to ensure that the robotic arm can accurately grasp and place objects.

[0066] Implementation Method Six: This implementation method provides an assistive grasping method for people with armless disabilities. The method is based on the device provided in Implementation Method Five and includes:

[0067] Steps for collecting user's electroencephalogram (EEG) signals;

[0068] The steps for collecting the location information of a target object;

[0069] The step of generating a grasping action group signal based on the user's EEG signal and the target object's position information;

[0070] The step of sending the grabbing action group signal.

[0071] Implementation Method Seven: This implementation method provides an assistive grasping device for people with armless disabilities. The device is based on the device provided in Implementation Method Five and includes:

[0072] A module for acquiring users' electroencephalogram (EEG) signals;

[0073] A module for acquiring the location information of the target object;

[0074] A module that generates a set of grasping action signals based on the user's EEG signals and the target object's position information;

[0075] The module that sends the grabbing action group signal.

[0076] Implementation Method 8: This implementation method provides a computer storage medium for storing a computing program. When the computer reads the computer program, the computer executes the method provided in Implementation Method 6.

[0077] Implementation Method Nine: This implementation method provides a computer, including a processor and a storage medium. When the processor reads a computer program stored in the storage medium, the computer executes the method provided in Implementation Method Six.

[0078] Implementation Method 10: This implementation method provides a computer program product. As a computer program, when the computer program is executed, it implements the method provided in Implementation Method 1.

[0079] Implementation Method Eleven: Combination Figure 1-8 This embodiment describes the technical solution provided above in further detail through specific examples. Specifically:

[0080] This embodiment proposes an assistive device for armless individuals based on the Sunrise X3 platform. It utilizes a combination of non-invasive EEG signal control and computer vision control, employing a portable design that is more intelligent and convenient to use while enabling basic daily living tasks at a lower cost.

[0081] The technical challenge faced by this implementation method is:

[0082] How to complete the entire grasping process more flexibly and accurately using non-contact EEG signals;

[0083] How to correlate the acquired EEG signals with the intended actions;

[0084] This implementation method cannot be adapted to every user because of the differences in EEG signals among different users.

[0085] Based on the above difficulties, the specific functions that this implementation method needs to achieve are as follows:

[0086] 1. It can control the device based on brainwave signals;

[0087] 2. It can detect and locate the object to be grasped;

[0088] 3. It can grasp and place objects to be grasped;

[0089] 4. It can save device deployment time through software;

[0090] It has good stability and can still work even after some functions fail.

[0091] In summary, the overall functional framework of the armless assistive device is as follows: Figure 1 As shown, it is mainly divided into intelligent grasping and placement in the user software interaction part.

[0092] This device primarily combines non-invasive EEG signals with computer vision to control a robotic arm to grasp and place objects in daily life, thereby assisting people with armless disabilities in their daily activities. In case of unexpected situations such as visual module failure, EEG signals can be used alone for control, and the robotic arm can be adjusted via software based on EEG data.

[0093] This implementation mainly consists of the Horizon Sunrise X3 platform, STM32 control board, EEG sensor, OPENMV vision module, laser module, intelligent algorithm, and bus-based robotic arm. The overall framework is as follows: Figure 2 As shown in the diagram. The Sunrise X3 chip is the main controller, the STM32 control board is the secondary controller, the OpenMV module and laser module are used for image acquisition, and the EEG sensor is used for EEG signal acquisition. EEG signals are transmitted to the main controller, image information is transmitted to the OpenMV, and the bus-driven servo robotic arm is used for object grasping and placement.

[0094] The overall principle of the device is as follows: It utilizes an EEG module and an image module to control the robotic arm and perform corresponding functions. The Sunrise X3 main control board, equipped with a self-developed EEG signal classification algorithm, processes the EEG signals. EEG commands are used to issue global control instructions, which are then transmitted to the STM32 secondary controller for further control of the robotic arm. The OpenMV module, equipped with a target detection algorithm, works with distance information obtained from the laser module to determine the location of objects. Combined with EEG control commands, this allows the robotic arm to perform the corresponding actions.

[0095] This device will enter a dormant state after 15 seconds when not in use. Upon detecting an EEG signal, the device will return to its initial state.

[0096] Initially, EEG signal control has the highest priority, and the visual module control process is locked. The user rotates the robotic arm in the up, down, left, and right directions based on the EEG signals. After the OPENMV module detects the target to be grasped based on laser guidance, a grasping command is issued using EEG signals. At this point, the grasping action has the highest priority, and the EEG signal control process is locked. The visual module then locates the object and performs inverse kinematics calculations to grasp it. After the grasping process is complete, the EEG signal control priority returns to the highest, and the visual module control process is locked. At this point, a release command or subsequent actions can be issued using EEG signals. The system flowchart is as follows. Figure 3 As shown.

[0097] Because the vision module may malfunction and the detection network may be unable to detect objects, a set of actions is pre-set. When the detection module fails, the mechanical gripper will open and close twice to provide a prompt. After that, the system will automatically call this set of actions to perform semi-manual grasping to ensure the normal operation of the equipment.

[0098] System overall framework:

[0099] The hardware used in this implementation mainly includes the Horizon Sunrise X3 Pie, STM32 control board, EEG sensor, OPENMV module, laser module (including laser ranging module and visual laser module), bus servo motor, alloy manipulator, and battery module. The overall hardware design system architecture is as follows: Figure 4 As shown;

[0100] In this embodiment, the main controller, Sunrise X3, is connected to both the EEG sensor and the secondary controller, the STM32 control board. Communication between the Sunrise X3 and both the EEG sensor and the STM32 control board is unidirectional. The OPENMV module is connected to both the STM32 control board and the laser module. The OPENMV module communicates bidirectionally with the STM32 control board and unidirectionally with the laser ranging module. There is no direct communication between the OPENMV module and the Sunrise X3, but a control relationship exists between them. The bus servo motor is connected to the STM32 control board, and bidirectional communication exists between them. The alloy manipulator is connected to the bus servo motor. The visible laser module, the alloy manipulator, and the battery module do not have communication capabilities.

[0101] Hardware design of the main control and secondary control modules:

[0102] In this implementation, the main control module is primarily responsible for processing large amounts of information and transmitting data between various modules. It also needs to be equipped with corresponding deep learning algorithms. These requirements necessitate that the main control module possess powerful computing capabilities and abundant serial ports. To meet these needs, this implementation selects the Horizon Sunrise X3 chip as the main control chip.

[0103] The main parameters for the Horizon Zero Dawn X3 are shown in Table 1:

[0104] Table 1

[0105]

[0106] In this implementation, the secondary control module is mainly responsible for communicating with the main control module, driving servos and other devices, and storing a small amount of data. It does not require complex algorithms or powerful computing capabilities; therefore, to meet the usage requirements, this implementation uses the STM32F103C8T6 as the secondary control chip. Since it needs to connect with servos and other devices, an expansion board is selected to work in conjunction with the STM32.

[0107] The aforementioned expansion board operates on a 6-16V power supply and features multiple interfaces for easy connection to other devices. This implementation primarily uses the bus servo interface to connect to a bus servo, and utilizes its W25Q64 memory to store action sequences in case of visual failure. Connection methods with other devices will be described in this section and the bus servo section. The main parameters of the STM32F103C8T6 chip are shown in Table 2.

[0108] Table 2

[0109]

[0110] The EEG signal acquisition module in this embodiment is mainly used to acquire the user's EEG signals and transmit them to the main control chip for processing. It should be convenient, comfortable, and provide stable signals. To meet usage requirements, this embodiment uses a non-invasive TGAM EEG sensor module for EEG acquisition. In use, the metal electrodes contact the forehead, and the ear clip is placed on the left earlobe.

[0111] The parameters of the TGAM brainwave sensor module are shown in Table 3:

[0112] Table 3

[0113]

[0114] Hardware design of the vision control module:

[0115] The visual image acquisition module in this embodiment is mainly used to acquire images of the object to be grasped. Combined with distance information obtained from the laser ranging module, it is used to locate the object, facilitating subsequent grasping actions. It needs to have communication capabilities and a certain level of data processing power. To meet the usage requirements, this embodiment uses OpenMV4 H7 Plus for visual image acquisition and processing.

[0116] Table 4

[0117]

[0118] In this embodiment, the vision processing module mainly consists of an OPENMV camera, a laser module (including a laser ranging module and a visible laser module), and a secondary control chip. To facilitate information transmission between the OPENMV vision module and the STM32 secondary control chip, it is used in conjunction with an expansion board via a serial bus connection. The expansion board's bus interface is connected to the OPENMV camera's UART3. To assist the user in grasping objects, the OPENMV module is connected to the visible laser module, which is connected to the 3.3V output pin of the OPENMV module. Furthermore, the laser ranging module is connected to the OPENMV to acquire distance information of objects; pins 3 and 4 of the ranging module are connected to pins 4 and 5 of the OPENMV module.

[0119] The visual laser module operates at 3-5V and measures 6×17.1mm. It is connected to the OPENMV module using DuPont wires.

[0120] The laser ranging module is model TOF200F. Its operating voltage is 3-5V, the measurement distance is 2m, the measurement blind zone is 0-3cm, the infrared emission mechanism is 940nm, the FOV field of view is 25 degrees, and the size is 206×11mm.

[0121] Robotic arm hardware design:

[0122] In this embodiment, the robotic arm is driven by six servo motors, specifically ZX15D*3, ZX20D*1, ZX20S*1, and ZX30S*1. These six servo motors are connected serially via a bus, numbered by the host computer, and then connected to an expansion board on the STM32 microcontroller to perform corresponding actions. Each servo motor has a built-in 32-bit MCU, supporting data saving and data readback.

[0123] As the end effector of the robotic arm, the alloy mechanical gripper module directly affects the accuracy of grasping objects. To ensure that commonly used items can be grasped and placed, a claw-type mechanical gripper is adopted.

[0124] This alloy mechanical gripper, in conjunction with a servo module, has an opening and closing size of 0–230 mm and can grasp most commonly used objects.

[0125] Software design for assistive systems for people with armless disabilities:

[0126] The overall framework diagram of the software system design is as follows: Figure 5 As shown;

[0127] In this implementation, the main program consists of a loop and nested structure of subroutines, primarily divided into three parts: a vision module, an EEG module, and an emergency module. Each program receives and sends commands. The system enters a sleep state after 15 seconds of inactivity, with the robotic arm in its default state. In sleep mode, the initial program value is aobccc, where 'a', 'b', and 'c' correspond to the EEG module, vision module, and emergency module, respectively. 'o' and 'c' represent enabling or disabling the program. Only the EEG module operates in sleep mode, with EEG control having the highest priority. When an EEG signal command is issued, the robotic arm returns to its initial state. Upon receiving a grasp or release command from the EEG signal, the EEG module sends the acbocc command. At this point, the vision module has the highest priority and completes the relevant action. After the action is finished, the vision module sends aobccc, restoring the highest priority to EEG control, allowing the next control step to proceed.

[0128] In addition, if the visual module malfunctions after the EEG signal issues a command and its program cannot obtain the object's position information, the visual module will send acbcco. At this time, the system enters an emergency state and will call a pre-set grasping action group to cooperate with the user to achieve semi-manual grasping. After the operation is completed, it will return to the default state.

[0129] EEG signal classification program design:

[0130] The dataset used in this implementation method is self-collected, and the dataset covers 8 types of actions, namely up, down, left, right, grab, drink (placing the grabbed object in front of the chest), put back (placing the grabbed object back in its original position), and release.

[0131] At the start of the data collection, subjects were seated in a comfortable environment and fitted with the TGAM EEG headband. The collection of motor imagery EEG datasets began. First, a 10-second black screen was displayed, followed by eight sequential action cues, each displayed for 10 seconds, with a 5-second black screen between each cue. After all actions were collected, another 10-second black screen was displayed. One round of data collection lasted 115 seconds. This process was repeated for five rounds, followed by a 5-minute rest period before the data collection resumed. A total of 12 rounds were conducted, resulting in 8 × 5 × 12 actions. The collected dataset was not directly usable signal data, but rather hexadecimal serial signals. After collection, the dataset was converted into raw sequence data by a suitable program before further processing and training.

[0132] With the rise of brain-computer interface technology, people's consciousness can be perceived by the outside world through brainwaves. By collecting brainwave signals, we can obtain a set of time-series signals, which can then be fed into a constructed neural network for processing, training, and classification to perform various tasks. To effectively capture the long-term dependencies present in brainwave signals, this implementation uses a Temporal Convolutional Network (TCN) for classification. TCN is developed based on CNN, borrowing the convolutional operations of CNN and adapting it to sequential data by introducing causal convolution and dilated convolution, while solving some problems encountered by RNN and LSTM in long sequence processing.

[0133] Considering that the device designed in this embodiment is a portable device, we need a network model with a small number of parameters, low computational cost, and excellent performance.

[0134] The model proposed in this implementation consists of three main modules: a convolutional module, a sliding window module, and a residual module. The convolutional block extracts the main temporal, depth, and spatial information from the EEG signal through dual convolutional branches, and its output is the primary time series after feature extraction. Due to the small dataset size, a sliding window module is used to segment the primary time series using a sliding time window to augment the data. Then, a residual block based on a TCN network extracts higher-level temporal features and sends them to a fully connected layer for classification using softmax.

[0135] Data acquisition is easily affected by the surrounding environment, blinking, etc., resulting in various noises and artifacts in the acquired dataset. In order to ensure the accuracy of classification results, the transformed dataset is preprocessed with wavelet transform denoising, bandpass filtering, normalization and other methods before training.

[0136] The dataset used in this implementation method is trained and tested using ten-fold cross-validation.

[0137] After training, the model deployment phase begins. First, model preparation involves saving the model file obtained after training in this implementation method, and then converting it to an ONNX model using the tf2onnx tool. Next is model validation, primarily ensuring the proposed algorithm model meets BPU requirements. This implementation method uses the hb_mapperchecker instruction provided by the Horizon Robotics development platform to perform model checks. For layers in the program that do not meet transfer requirements, manual adjustments are made to redirect these parts to CPU execution. Finally, model conversion occurs, converting the floating-point model to a model used by the BPU. The conversion is completed using the hb_mapper makertbin function. After successful conversion, the corresponding bin file is obtained, which is copied to the Sunrise X3 platform; this is the model file required for onboard operation.

[0138] To verify the final application effect of the model proposed in this embodiment, the converted bin file and inference code were copied to the board. The inference code was run using the sudo command. The code reads the EEG headband serial port information and performs a series of conversions and processing, thereby controlling the robotic arm to perform corresponding actions based on different imagined symbols.

[0139] To address the issues of low computing power and limited resources in embedded devices, this implementation method deploys the lightweight object detection algorithm FOMO within the OPENMV module. The method is as follows: first, the dataset is captured using the OPENMV module and uploaded to the Edgeimpulse platform; then, online model training and export are performed; finally, the generated main.py file and weight files are copied to the OPENMV module.

[0140] Intelligent drive control program design section:

[0141] The motion control system of the robotic arm mainly controls six bus servo motors. This section will explain and illustrate the principles of single servo motor control and six servo motor coordinated control.

[0142] I. Control Principle of a Single Bus Servo

[0143] The servo used in this embodiment is a bus servo, which is connected to the secondary STM32 control board. It rotates the corresponding angle according to the command of the servo, and supports the reading and saving of the angle. This data will be stored in the memory of the expansion board for later use.

[0144] Each bus servo has two main control parameters: PWM value and T. The PWM value ranges from 0500 to 2500, and this value can be used to control the rotation angle of the servo. In this embodiment, the maximum rotation angle of the servo is 270°. The PWM value is proportional to the angle; 0500 represents -135 degrees, and 2500 represents 135 degrees. The parameter T is time, representing the movement time required to complete one command, ranging from 0 to 9999 ms.

[0145] II. Cooperative Control Principle of Multiple Bus Servos

[0146] In this embodiment, the six bus servos are controlled via commands from the control board. The principle behind this is to number each bus servo, control each individual servo, and ultimately achieve coordinated control.

[0147] The procedure for modifying the bus servo ID number is as follows:

[0148] 1. Connect the bus servo to the control board, and then connect the control board to the computer;

[0149] 2. Using the bus configuration function in the host computer, change the ID number of the connected servo motor. At the same time, the control board will set the servo motor angle to the middle position by default to ensure that the robot can complete the corresponding action commands as much as possible.

[0150] After numbering each servo, commands can be issued according to the required actions.

[0151] In addition, to handle emergencies and set default and initial postures, this embodiment pre-configures three action groups. These action groups are stored in the expansion board's registers for offline execution. An action group is a combination of multiple actions; this embodiment will explain using a single action as an example:

[0152] {#000P1500T1250#001P1860T1250#002P2000T1250#003P1000T1250#004P1500T1250#005P1200T1250}

[0153] The above is a command for one action, which includes control parameters for six servos. #000P1500T1250 indicates that the servo with ID number 000 is controlled to rotate from the current position to the 1500 position, with a total time of 1250ms.

[0154] EEG control section:

[0155] One of the functions of the assistive device is to control the robotic arm to perform eight different movements using different EEG signals. These eight movements are achieved by driving different servo motor sets, and their correspondence is as follows: Figure 6 As shown;

[0156] After acquiring EEG signals, the brain-computer interface (BCI) device transmits the data to the X3Pie via Bluetooth serial port. The X3Pie first parses the hexadecimal serial data, converting it into raw sequence values, then performs data preprocessing. Finally, the processed EEG sequence data is fed into a pre-trained and deployed classification network for prediction, and the final prediction result is transmitted to the STM32 microcontroller via a pre-defined serial port. In the STM32 microcontroller, the final prediction result is translated into corresponding movements of the robotic arm, thus achieving the effect of controlling the robotic arm through the user's motor imagery.

[0157] When using brainwave control, the user's brainwaves are extracted by a brainwave sensor, processed accordingly, and then classified by a brainwave classification model. Brainwave sensor data includes... Figure 7 As shown.

[0158] Visual control section:

[0159] Visual control compensates for the instability and inaccuracy of brainwave control. Upon object detection, it automatically performs a grasping action based on control commands issued by the brainwave signal. In this implementation, the trained FOMO target detection model is deployed to the OPENMV module. After acquiring images, the model determines the coordinates and, in conjunction with distance information measured by the laser ranging module, calculates the approximate position of the object in space. Using inverse kinematics, it calculates the required angle of rotation for each servo motor and transmits this information to the STM32 slave controller to perform the relevant actions.

[0160] Six-servo inverse kinematics refers to the process of solving for the rotation angle of each joint in a robotic arm, given the rotation angles of the six servos and the target position and attitude at the end of the robotic arm. A schematic diagram of the inverse kinematics solution principle is shown below. Figure 8 As shown.

[0161] The inverse kinematics instruction in this implementation program is: ($KMS:x,y,z,time!), where x, y, and z represent the three-dimensional spatial coordinates of the object, and time represents the time required to move from the current position to the target position.

[0162] This embodiment proposes an assistive device for armless disabled individuals based on the Horizon Sunrise X3 platform. This device features EEG control, intelligent vision, intelligent grasping, human-computer interaction, and an intelligent system. It can assist armless disabled individuals in performing simple daily tasks such as grasping and placing objects at a relatively low cost, adding color to their lives and contributing a new approach to the design and development of assistive devices for armless disabled individuals. The problems ultimately solved and the technological innovations of this embodiment are summarized as follows:

[0163] 1. An assistive device based on the combination of EEG control and computer vision is proposed, which realizes functions such as recognition and positioning, automatic grasping and user EEG control. It has successfully completed daily grasping and placing tasks with a grasping accuracy of 90.3% and a placement accuracy of 96.3%, and reliably and effectively solves the inconvenience caused by the lack of arms in the daily life of people with armless disabilities.

[0164] 2. A TCN-based EEG classification algorithm is proposed, which can classify EEG signals extracted by non-invasive EEG sensors with a classification accuracy of up to 90.2%, providing a safer, more accurate and more convenient foundation for the implementation of brain control devices.

[0165] 3. It can be paired with an operating software that integrates EEG data reception, storage, conversion, processing and training, allowing users to customize their own data with simple operations, solving the problem of large differences in EEG signals among different users and the unsatisfactory control effect of using the same brain control device.

[0166] With the increasing number of people with disabilities, the demand for assistive devices for this group will also increase in the future. The assistive device for armless people with disabilities designed in this embodiment helps to solve the problems of limited use, poor control effect and high price of current products, and provides an effective solution to help more people with disabilities live a normal life.

[0167] Implementation Method Twelve: This implementation method tests and evaluates the various functions of the above-mentioned assistive device for people with armless disabilities through specific embodiments. Specifically:

[0168] Electroencephalogram (EEG) signal classification test section:

[0169] This implementation method sequentially performs sixty sets of motion visualizations and compares the actual movements of the robotic arm with the imagined movements of the user. Some results are shown in Table 5:

[0170] Table 5

[0171]

[0172] Based on real-world testing, out of a total of 480 motor imagery attempts, only 47 misclassifications occurred. The model achieved a classification accuracy of 90.2% in practical applications. Although there were misclassifications, since the EEG signals only control the servo motor to make small movements of about 15° each time, the algorithm fully meets the usage requirements of this implementation method.

[0173] Grab and place test section:

[0174] This section tests the grasping and placement of different objects to observe whether the auxiliary device can accurately grasp items and stably place them in designated positions. The tests use three common everyday items that need to be grasped: a water cup, a medicine box, and a handbag. Each item is tested 100 times.

[0175] The placement process is divided into two types: returning to the original position and placing in other positions under EEG control. Returning to the original position is the reverse of the grasping process and is completed by a pre-programmed sequence. Placing in other positions is freely controlled by the user. The accuracy rates of grasping and placement are shown in Table 6.

[0176] Table 6

[0177]

[0178] Table 6 shows that the accuracy of grasping the three items is highest for medicine boxes, middle for water cups, and lowest for handbags. This is because medicine boxes and water cups are relatively large and can deform, making them easier to grasp. Handbags come in many varieties and have inconsistent shapes, making them more difficult to grasp. However, for armless patients, grasping handbags is not urgent or necessary, and multiple attempts can be made.

[0179] Placement accuracy was tested after successful grabbing. The results showed that the success rates for placing the handbag and the medicine box reached 97.6% and 100%, respectively. This is because both can be placed successfully from any angle, but the handbag's material makes it susceptible to falling. The water cup had a relatively lower success rate due to potential tipping over after placement caused by user movement. However, the accuracy rate was quite good when placed manually.

[0180] In addition, for user convenience, the device has a preset "drinking water" program. The command is issued by brainwave control and will automatically bring the object to the mouth after it is grasped. This setting needs to be adjusted according to different users and can achieve almost 100% accuracy.

[0181] Visual impairment test section:

[0182] Due to various limitations, the vision control module of this device may malfunction. In response to this situation, the device is pre-programmed with automatic grasping actions. When the vision module malfunctions, the mechanical gripper will open and close twice to provide a warning. Two seconds later, the device will automatically call the preset action group to perform semi-manual grasping.

[0183] This motion sequence is fixed, and the robotic arm defaults to its maximum opening and closing. Users only need a short period of practice to learn how to use this motion to grasp objects. However, due to the limited range of motion, it may be unable to grasp objects in some scenarios.

[0184] This experiment involves obstructing the camera to simulate a visual module malfunction.

[0185] It is evident that although this solution is intended for emergency situations, it can still maintain a high grasping accuracy and ensure the normal operation of the equipment.

[0186] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An assisted system for armless disabled persons, characterized by, The system comprises: a brain signal acquisition module, configured to acquire brain electrical signals of a user in a non-invasive manner, and during the acquisition, the subject is allowed to sit in a comfortable environment, wear a TGAM brain wave head ring, and start collecting motor imagery brain electrical data sets; a visual control module, configured to collect target object positioning information; a main control module, configured to generate a grasping action group signal according to signals collected by the brain signal acquisition module and the visual control module; Specifically, the brain signal acquisition module is configured to collect brain electrical signals of a user, transmit the signals to the main control module for classification and processing, and control the action of the mechanical arm; a brain electrical classification algorithm: using a time convolution network to classify the collected brain electrical signals, and accurately identifying the user's motor intention through a deep learning model; The model mainly consists of three modules: a convolution module, a sliding window module and a residual module; the convolution module extracts the main time, depth and spatial information in the brain electrical signals through double convolution branches, and the output is the primary time sequence after feature extraction; due to the small size of the data set, the sliding window module is used to divide the primary time sequence through a sliding time window for data augmentation; then a residual module based on the TCN network design is used to extract higher-level time features, and sent to the fully connected layer for classification using softmax; The visual control module comprises: an image acquisition module: using an OPENMV module to cooperate with a laser module to position the object, collect the image of the object and obtain distance information; an image processing module: performing lightweight processing on the image through the FOMO algorithm to determine the position of the object, and combining the brain electrical signals to control the mechanical arm to complete the grasping task; The interaction logic between the main control module and the visual control module is as follows: In the initial state, the brain electrical signal control has the highest priority, and the visual module control process is locked; the user rotates the mechanical arm in the up, down, left and right directions according to the brain electrical signals, and after the OPENMV module detects the target to be grasped according to the laser guidance, the brain electrical signal is used to issue a grasping instruction; at this time, the grasping action has the highest priority, and the brain electrical signal control process is locked; at this time, the object is grasped by positioning and inverse kinematics solving through the visual module; after the grasping process is completed, the brain electrical signal control priority is restored to the highest, and the visual module control process is locked; at this time, the brain electrical signal is used to issue a release instruction or perform subsequent actions.

2. The armless handicapped person assisting system according to claim 1, characterized by, The brain signal acquisition module is based on Horizon Sun X3.

3. The armless handicapped person assisting system according to claim 1, characterized by, The non-invasive manner is realized by a TGAM brain wave sensor.

4. The armless handicapped person assisting system according to claim 1, characterized by, The visual control module is realized by an OPENMV module.

5. An assistive device for armless disabled persons, characterized by, The system of claim 1, and a mechanical arm module, configured to complete the grasping of the target object in response to the grasping action group signal. The method is realized based on the device of claim 5, comprising:

6. A method of assisting a person with no arms in grasping, characterized by, a step of collecting brain electrical signals of a user; a step of collecting target object positioning information; a step of generating a grasping action group signal according to the brain electrical signals of the user and the target object positioning information; a step of sending the grasping action group signal. The auxiliary grasping device is realized based on the device of claim 5, comprising:

7. An assisted gripping device for armless disabled persons, characterized in that, a module for collecting brain electrical signals of a user; ​ A module for collecting position information of a target object; A module for generating a grasping action group signal according to the user's electroencephalogram signal and the position information of the target object; A module for sending the grasping action group signal.

8. Computer storage medium for storing a computer program, characterized in that When a computer reads the computer program, the computer executes the method of claim 6.

9. A computer comprising a processor and a storage medium, characterized in that When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 6.

10. Computer program product as computer program, characterized in that When the computer program is executed, the method of claim 6 is implemented.

Images