Electroencephalogram control mechanical arm system

Abstract

The invention discloses an electroencephalogram control mechanical arm system, which belongs to the technical field of brain-computer interfaces, and comprises a head band and electrode electroencephalogram cap combination device and a mechanical arm device, the head band and electrode electroencephalogram cap combined device is used for obtaining electroencephalogram signal features and head attitude angle data of a wearer and recognizing facial emotion features and gesture features of a communication object. The electroencephalogram signal features, the head attitude angle data and the facial emotion features and gesture features of the communication object are transmitted to the mechanical arm device so as to execute corresponding mechanical arm actions; the mechanical arm device is further used for conducting voice interaction with the wearer through the voice module. The problems that an existing mechanical arm is single in function and not convenient and fast to carry and use are solved.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface technology, and in particular relates to a brain-computer control robotic arm system. Background Technology

[0002] The key to EEG-controlled robotic arms lies in the precise acquisition and processing of EEG signals. Non-invasive brain-computer interface systems have shown broad application prospects in medical rehabilitation, human-computer interaction, and neuroscience research, but significant technological challenges remain. Traditional wet electrode EEG acquisition systems rely on conductive gel, resulting in poor comfort during long-term monitoring and low signal stability. Therefore, improving the accuracy and real-time performance of EEG signal acquisition, as well as ensuring system stability, are key areas requiring further research.

[0003] Existing brain-controlled robotic arm systems have many limitations. For example, most robotic arms use a fixed base design, generally relying on heavy mechanical supports, metal floor-mounted rails, or dedicated workbenches for support, making them unable to move with the user and providing assistance, resulting in poor portability. Secondly, existing brain-controlled robotic arms have relatively limited functions, only capable of basic motor control. Users can only complete simple operations by repeatedly triggering single brainwave commands, unable to express complex task intentions through natural language, and unable to adapt to the environment and the behavior and emotions of the communicator, lacking the ability for deep human-computer interaction. Summary of the Invention

[0004] In view of the above-mentioned shortcomings in the prior art, the present invention provides a brain-computer interface-controlled robotic arm system, which solves the problems of existing robotic arms having limited functions and being inconvenient to carry and use.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: The present invention provides an EEG-controlled robotic arm system, comprising a headband and electrode EEG cap combination device and a robotic arm device; The headband and EEG cap combination device is used to acquire the wearer's EEG signal characteristics and head posture angle data, and to identify the facial emotion characteristics and gesture characteristics of the communication object. The EEG signal characteristics, head posture angle data, and facial emotion characteristics and gesture characteristics of the communication object are transmitted to the robotic arm device to execute the corresponding robotic arm action. The robotic arm device is also used to conduct voice interaction with the wearer through the voice module.

[0006] Furthermore, the headband and EEG cap assembly includes a headband, an EEG cap body, a gyroscope sensor module, eight dry electrodes for data acquisition, an EEG signal acquisition board, an EEG signal processing board, a signal control board, a camera module, two ear clips, an EEG cap fixing strap, and two tension adjusters. The eight dry electrodes and the EEG signal acquisition board are fixedly mounted on the main body of the EEG cap. Each dry electrode consists of a base and several electrode pins. The electrode pins can be attached to the surface of the scalp to acquire the wearer's brainwave signals and transmit them to the EEG signal acquisition board. The EEG signal acquisition board amplifies the brainwave signals through an operational amplifier to obtain amplified brainwave signals, which are then transmitted to the EEG signal processing board via an FPC flexible flat cable. The camera module is positioned in front of the headband to capture facial and hand images of the person being communicated with, and to identify the facial emotional and gesture characteristics of the person being communicated with. The images are then transmitted to the main control board, which controls the robotic arm to perform corresponding robotic arm actions based on the facial emotional and gesture characteristics of the person being communicated with. The gyroscope sensor module is located on the top of the headband and is used to monitor the wearer's head posture angle and transmit the head posture angle data to the signal control board. The EEG signal processing board and the main signal control board are fixedly mounted on the headband. The EEG signal processing board is used to differentially amplify, filter, perform voltage follower analysis, and digitally convert the amplified EEG signal to obtain a digital EEG signal. The main signal control board is wirelessly connected to the robotic arm device. The main signal control board uses the wearer's digital EEG signal as the EEG signal feature, and combines it with head posture angle data, facial emotion features, and gesture features of the person being communicated with to transmit it to the robotic arm device to perform corresponding robotic arm movements. The two ear clips are connected to the left and right sides of the headband respectively, and are used to provide a reference voltage for the dry electrode by clipping on the earlobe; The EEG cap fixing strap is located below the headband, and the tightness of the headband and EEG cap combination device can be adjusted from the left and right sides by two tension adjusters.

[0007] Furthermore, the camera module uses the MaixPy library and the pyAI-K210 camera to capture images of the face and hands of the person being communicated with, and deploys an emotion recognition model and a gesture recognition model to obtain the facial emotion features and gesture features of the person being communicated with through facial and hand images.

[0008] Furthermore, the method for facial emotion feature recognition using emotion recognition models includes the following steps: Feature extraction and emotion category probability prediction are performed on the facial images of the communication partners to obtain multiple emotion category probability values; Perform Argmax maximum value retrieval on the probability values ​​of each emotion category, and iterate through all emotion category probability values ​​to filter out the confidence level and the corresponding index; Emotional labels are obtained through index mapping, which serve as facial emotion features of the communication subject.

[0009] Furthermore, the gesture recognition model is based on the YOLOv2 model for gesture recognition; The method for recognizing gesture features using a gesture recognition model includes the following steps: The gesture recognition model inputs a hand image into Darknet-19 for forward propagation and extracts a depth feature map; Multi-scale detection is performed at each grid location on the depth feature map, and predicted bounding boxes are output, where each grid corresponds to multiple anchor points; The predicted bounding boxes are decoded, and the relative offsets are converted into actual image coordinates to obtain the detection boxes; Filter out detection frames with a confidence level below 0.5; Non-maximum suppression is performed on the remaining detection boxes, and highly overlapping detection boxes are filtered out based on an IoU threshold of 0.3 to obtain the hand feature detection results; Probability prediction and label mapping are performed on the hand feature detection results to obtain gesture labels, which are used as the gesture features of the communication object.

[0010] Furthermore, the robotic arm device includes a robotic arm module consisting of 6 servo motors, a robotic arm base, a robotic upper arm, a robotic lower arm, and a robotic gripper, a motion control board, a voice module, and a hard-shell backpack; The six servo motors include a first servo motor, a second servo motor, a third servo motor, a fourth servo motor, a fifth servo motor, and a sixth servo motor. The first servo motor is fixedly mounted on the base of the robotic arm and is used to control the rotation of the robotic arm around the base. The second servo motor is mounted on the first servo motor and is used to control the forward and backward tilting motion of the robotic arm. One end of the robotic arm is connected to the second servo motor, and the other end is connected to the third servo motor to extend the arm's reach. The third servo motor is used to control the inward and outward movement of the robotic forearm relative to the robotic arm. One end of the robotic forearm is connected to the third servo motor, and the other end is connected to the fourth servo motor to extend the arm's reach. The fourth servo motor is used to control the inward and outward movement of the gripper relative to the robotic forearm. One end of the fifth servo motor is fixedly mounted on the fourth servo motor. The upper end is connected to the root of the mechanical gripper, and the other end is used to control the rotation of the mechanical gripper; the sixth servo motor is set on the gripper side and is used to control the opening and closing of the mechanical gripper; the base of the robotic arm is fixedly connected to the outside of the hard-shell backpack by bolts and nuts; the motion control board and the voice module are set inside the hard-shell backpack; the motion control board is connected to the signal control board via a wireless serial port; the motion control board generates corresponding motion control commands based on the wearer's EEG signal characteristics, head posture angle data, facial emotion characteristics and gesture characteristics of the person being communicated with, and controls and drives the six servos to execute the corresponding robotic arm movements through the onboard CAN bus; the voice module is used to wake up the robotic arm and perform voice human-computer interaction.

[0011] Furthermore, the voice model includes an MCU module of model ESP32S3, and a microphone and speaker connected to the MCU module; The method for voice models to interact with wearers via voice includes the following steps: The wearer's original audio is recorded via microphone; The original audio is compressed using OPUS encoding via the MCU module to obtain compressed audio. The compressed audio is uploaded to the server using the WebSocket protocol via the MCU module. The server backend decodes the OPUS format to recover the original audio, converts the original audio into text information, and uses the large language model API to analyze the text information before outputting the reply text. The server converts the response text into response audio and sends it to the MCU module according to the WebSocket protocol so that the response audio can be played through the speaker, completing the voice interaction with the wearer.

[0012] The beneficial effects of this invention are as follows: The EEG-controlled robotic arm system provided by this invention, through eight-channel non-invasive acquisition dry electrodes, combined with filtering, noise reduction, voltage tracking analysis, and other processing, significantly improves the signal-to-noise ratio of EEG signal acquisition, enabling more intuitive and visual acquisition of effective EEG signals, preserving various characteristic frequency band information of EEG, and reducing interference from contact impedance and environmental noise. This significantly reduces the difficulty of communicating and controlling the robotic arm. The EEG-controlled robotic arm system provided by this invention deploys an emotion recognition model, a gesture recognition model, and a voice module, effectively expanding the functionality of the brain-controlled robotic arm. The camera recognizes the facial emotion and gesture features of the person being interacted with, forming multi-source data fusion with the acquired EEG signal features and head posture angle data of the wearer, enabling the robotic arm device to dynamically adjust its movements. This invention allows the wearer to interact with a large voice model while wearing the robotic arm, acquiring knowledge and emotional value, thus compensating for the shortcomings of traditional EEG-controlled robotic arms that only focus on physiological assistance while neglecting psychological needs.

[0013] Other advantages of the present invention will be analyzed in more detail in the following embodiments. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a block diagram of a brain-controlled robotic arm system according to an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the structure of the headband and electrode EEG cap combination device in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the robotic arm device in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides an EEG-controlled robotic arm system, including a headband and electrode EEG cap combination device and a robotic arm device; The headband and EEG cap combination device is used to acquire the wearer's EEG signal characteristics and head posture angle data, and to identify the facial emotion characteristics and gesture characteristics of the communication object. The EEG signal characteristics, head posture angle data, and facial emotion characteristics and gesture characteristics of the communication object are transmitted to the robotic arm device to execute the corresponding robotic arm action. The robotic arm device is also used to conduct voice interaction with the wearer through the voice module.

[0020] like Figure 2 As shown, the headband and EEG cap assembly includes a headband, an EEG cap body, a gyroscope sensor module, eight dry electrodes, an EEG signal acquisition board, an EEG signal processing board, a signal control board, a camera module, two ear clips, an EEG cap fixing strap, and two tension adjusters. When brain neurons are active synchronously, they generate electrophysiological signals. These electrophysiological signals are transmitted to the scalp surface in the form of weak currents. This solution uses non-invasive dry electrodes as metal conductive elements, which directly contact the scalp surface at multiple specific locations. By utilizing the conductivity, a multi-channel acquisition array is formed to collect the weak potential changes on the scalp surface caused by the synchronous activity function of neurons, thus simultaneously acquiring brainwave signals from different regions of the wearer's brain.

[0021] The eight dry electrodes and the EEG signal acquisition board are fixedly mounted on the main body of the EEG cap. Each dry electrode consists of a base and several electrode pins. The electrode pins can be attached to the scalp surface to acquire the wearer's EEG signals and transmit them to the EEG signal acquisition board. The EEG signal acquisition board amplifies the EEG signals through an operational amplifier to obtain amplified EEG signals, which are then transmitted to the EEG signal processing board via an FPC flexible flat cable. In this scheme, Ag or AgCl dry electrodes are used as the acquisition dry electrodes, acquiring the wearer's EEG signals through direct contact with the scalp. The electrode pins can be attached to the scalp surface, thereby reducing interference caused by contact impedance. The camera module is positioned in front of the headband to capture facial and hand images of the person being communicated with, and to identify the facial emotional and gesture characteristics of the person being communicated with. The images are then transmitted to the main control board, which controls the robotic arm to perform corresponding robotic arm actions based on the facial emotional and gesture characteristics of the person being communicated with. The gyroscope sensor module is located on the top of the headband and is used to monitor the wearer's head posture angle and transmit the head posture angle data to the signal control board. In this embodiment, the gyroscope sensor module adopts an attitude angle sensor of model MPU-6050; The EEG signal processing board and the main signal control board are fixedly mounted on the headband. The EEG signal processing board is used to differentially amplify, filter, perform voltage follower analysis, and digitally convert the amplified EEG signal to obtain a digital EEG signal. The main signal control board is wirelessly connected to the robotic arm device. The main signal control board uses the wearer's digital EEG signal as the EEG signal feature, and combines it with head posture angle data, facial emotion features, and gesture features of the person being communicated with to transmit it to the robotic arm device to perform corresponding robotic arm movements. In this embodiment, the EEG signal processing board amplifies the signal and suppresses high-frequency interference through a differential amplifier and a second-order Sallen-Key low-pass filter, and converts it into a digital EEG signal through an ADC chip; the EEG signal processing board is connected to the signal main control board through the SPI bus to transmit the digital EEG signal to the signal main control board. The acquired EEG signals are weak, typically at the μV level, and contain interference signals from environmental electromagnetic interference and physiological artifacts. Therefore, this solution uses an operational amplifier, differential amplifier, low-pass filter, voltage follower, and ADC chip to amplify, adaptively filter, analyze voltage, and digitally convert the EEG signals in sequence. This identifies and filters out noise interference from the environment and physiological artifacts, improves the signal-to-noise ratio, selects characteristic frequency bands related to brain activity, and converts them into digital EEG signals as EEG signal features. In this embodiment, the EEG signal processing board uses an ESP32-R8 dual-core microcontroller; the signal control board uses an STM32F103C8T6 microprocessor; the GPIO17 and GPIO18 pins of the ESP32-R8 dual-core microcontroller are connected to the PA10 and PA9 pins of the STM32F103C8T6 microprocessor respectively; the SCL and SDA ports of the MPU-6050 attitude angle sensor are connected to the PB6 and PB7 pins of the STM32F103C8T6 microprocessor, and the wearer's head attitude angle data is transmitted to the signal control board using the I2C protocol; The two ear clips are connected to the left and right sides of the headband respectively, and are used to provide a reference voltage for the dry electrode by clipping on the earlobe; The EEG cap fixing strap is located below the headband, and the tightness of the headband and EEG cap combination device can be adjusted from the left and right sides using two tension adjusters. In this design, the two tension adjusters can effectively ensure the stable acquisition of the wearer's EEG signals by the dry electrodes.

[0022] The camera module uses the MaixPy library and a pyAI-K210 camera to capture images of the face and hands of the person being communicated with. It also deploys an emotion recognition model and a gesture recognition model to obtain the facial emotion features and gesture features of the person being communicated with through facial and hand images.

[0023] In this solution, the Flash storage capability of the K210 chip is utilized to pre-store pre-trained lightweight emotion recognition and gesture recognition models at corresponding fixed addresses. Upon startup, the model data is directly read from the corresponding addresses via a specified interface of MaixPy, thereby achieving accurate matching of facial emotion feature recognition and gesture recognition. In this embodiment, the camera module communicates with the signal control board via the UART protocol, transmitting the facial emotion features and gesture features of the person being interacted with to the signal control board.

[0024] The method for facial emotion recognition models to identify facial emotion features includes the following steps: Feature extraction and emotion category probability prediction are performed on the facial images of the communication partners to obtain multiple emotion category probability values; Perform Argmax maximum value retrieval on the probability values ​​of each emotion category, and iterate through all emotion category probability values ​​to filter out the confidence level and the corresponding index; Emotional labels are obtained through index mapping, which serve as facial emotion features of the communication subject.

[0025] The gesture recognition model is based on the YOLOv2 model for gesture recognition. The method for recognizing gesture features using a gesture recognition model includes the following steps: The gesture recognition model inputs a hand image into Darknet-19 for forward propagation and extracts a depth feature map; Multi-scale detection is performed at each grid location on the depth feature map, and predicted bounding boxes are output, where each grid corresponds to multiple anchor points; The predicted bounding boxes are decoded, and the relative offsets are converted into actual image coordinates to obtain the detection boxes; Filter out detection frames with a confidence level below 0.5; Non-maximum suppression is performed on the remaining detection boxes, and highly overlapping detection boxes are filtered out based on an IoU threshold of 0.3 to obtain the hand feature detection results; Probability prediction and label mapping are performed on the hand feature detection results to obtain gesture labels, which are used as the gesture features of the communication object.

[0026] like Figure 3 As shown, the robotic arm device includes a robotic arm module consisting of 6 servo motors, a robotic arm base, a robotic upper arm, a robotic lower arm, and a robotic gripper, a motion control board, a voice module, and a hard-shell backpack. The six servo motors include a first servo motor, a second servo motor, a third servo motor, a fourth servo motor, a fifth servo motor, and a sixth servo motor. The first servo motor is fixedly mounted on the base of the robotic arm and is used to control the rotation of the robotic arm around the base. The second servo motor is mounted on the first servo motor and is used to control the forward and backward tilting motion of the robotic arm. One end of the robotic arm is connected to the second servo motor, and the other end is connected to the third servo motor to extend the arm's reach. The third servo motor is used to control the inward and outward movement of the robotic forearm relative to the robotic arm. One end of the robotic forearm is connected to the third servo motor, and the other end is connected to the fourth servo motor to extend the arm's reach. The fourth servo motor is used to control the inward and outward movement of the gripper relative to the robotic forearm. One end of the fifth servo motor is fixedly mounted on the fourth servo motor. The upper end is connected to the root of the mechanical gripper, and the other end is used to control the rotation of the mechanical gripper; the sixth servo motor is set on the gripper side and is used to control the opening and closing of the mechanical gripper; the base of the robotic arm is fixedly connected to the outside of the hard-shell backpack by bolts and nuts; the motion control board and the voice module are set inside the hard-shell backpack; the motion control board is connected to the signal control board via a wireless serial port; the motion control board generates corresponding motion control commands based on the wearer's EEG signal characteristics, head posture angle data, facial emotion characteristics and gesture characteristics of the person being communicated with, and controls and drives the six servos to execute the corresponding robotic arm movements through the onboard CAN bus; the voice module is used to wake up the robotic arm and perform voice human-computer interaction.

[0027] In this design, the hard-shell backpack is made of rigid plastic, which can both securely connect to the robotic arm and provide insulation when worn on the back of the user.

[0028] In this solution, the robotic arm's movements include handshake or hugging, enabling users to interact with others. It can also drive the servo motor to control the robotic arm's movements by controlling the head's posture angle. For example, when the user moves their head left or right or nods, the robotic arm can move left or right accordingly, amplifying the user's behavioral characteristics.

[0029] The voice model includes an ESP32S3 MCU module, and a microphone and speaker connected to the MCU module; The method for voice models to interact with wearers via voice includes the following steps: The wearer's original audio is recorded via microphone; The original audio is compressed using OPUS encoding via the MCU module to obtain compressed audio. The compressed audio is uploaded to the server using the WebSocket protocol via the MCU module. The server backend decodes the OPUS format to recover the original audio, converts the original audio into text information, and uses the large language model API to analyze the text information before outputting the reply text. The server converts the response text into response audio and sends it to the MCU module according to the WebSocket protocol so that the response audio can be played through the speaker, completing the voice interaction with the wearer.

[0030] In use, the EEG cap of the headband and electrode EEG cap combination device is worn on the head, and the dry electrodes for data acquisition are positioned so that the motor palpation device can adhere to the scalp surface. The tightness adjuster is adjusted so that the EEG cap conforms to the jawline. Then, the headband is put on, so that the camera module is positioned in the center of the front of the wearer's head, and the gyroscope sensor module is kept horizontally positioned in the center of the top of the head. The ear clips on both sides are clipped to the earlobes to ensure the stability and accuracy of EEG data acquisition. Subsequently, the wearer puts on a hard-shell backpack and adjusts the position of the robotic arm module, thus completing the wearing of the EEG-controlled robotic arm system. This allows the system to acquire the wearer's EEG signal characteristics and head posture angle data, recognize the facial emotional characteristics and gesture characteristics of the person being communicated with, and then execute corresponding robotic arm actions, as well as conduct voice interaction with the wearer through the voice module.

[0031] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A brain-computer interface controlled robotic arm system, characterized in that, Includes a combination of a headband and an EEG cap, and a robotic arm device; The headband and EEG cap combination device is used to acquire the wearer's EEG signal characteristics and head posture angle data, and to identify the facial emotion characteristics and gesture characteristics of the communication object. The EEG signal characteristics, head posture angle data, and facial emotion characteristics and gesture characteristics of the communication object are transmitted to the robotic arm device to execute the corresponding robotic arm action. The robotic arm device is also used to conduct voice interaction with the wearer through the voice module.

2. The EEG-controlled robotic arm system according to claim 1, characterized in that, The headband and EEG cap assembly includes a headband, an EEG cap body, a gyroscope sensor module, eight dry electrodes, an EEG signal acquisition board, an EEG signal processing board, a signal control board, a camera module, two ear clips, an EEG cap fixing strap, and two tension adjusters. The eight dry electrodes and the EEG signal acquisition board are fixedly mounted on the main body of the EEG cap. Each dry electrode consists of a base and several electrode pins. The electrode pins can be attached to the surface of the scalp to acquire the wearer's brainwave signals and transmit them to the EEG signal acquisition board. The EEG signal acquisition board amplifies the brainwave signals through an operational amplifier to obtain amplified brainwave signals, which are then transmitted to the EEG signal processing board via an FPC flexible flat cable. The camera module is positioned in front of the headband to capture facial and hand images of the person being communicated with, and to identify the facial emotional and gesture characteristics of the person being communicated with. The images are then transmitted to the main control board, which controls the robotic arm to perform corresponding robotic arm actions based on the facial emotional and gesture characteristics of the person being communicated with. The gyroscope sensor module is located on the top of the headband and is used to monitor the wearer's head posture angle and transmit the head posture angle data to the signal control board. The EEG signal processing board and the main signal control board are fixedly mounted on the headband. The EEG signal processing board is used to differentially amplify, filter, perform voltage follower analysis, and digitally convert the amplified EEG signal to obtain a digital EEG signal. The main signal control board is wirelessly connected to the robotic arm device. The main signal control board uses the wearer's digital EEG signal as the EEG signal feature, and combines it with head posture angle data, facial emotion features, and gesture features of the person being communicated with to transmit it to the robotic arm device to perform corresponding robotic arm movements. The two ear clips are connected to the left and right sides of the headband respectively, and are used to provide a reference voltage for the dry electrode by clipping on the earlobe; The EEG cap fixing strap is located below the headband, and the tightness of the headband and EEG cap combination device can be adjusted from the left and right sides by two tension adjusters.

3. The EEG-controlled robotic arm system according to claim 2, characterized in that, The camera module uses the MaixPy library and a pyAI-K210 camera to capture images of the face and hands of the person being communicated with. It also deploys an emotion recognition model and a gesture recognition model to obtain the facial emotion features and gesture features of the person being communicated with through facial and hand images.

4. The EEG-controlled robotic arm system according to claim 3, characterized in that, The method for facial emotion recognition models to identify facial emotion features includes the following steps: Feature extraction and emotion category probability prediction are performed on the facial images of the communication partners to obtain multiple emotion category probability values; Perform Argmax maximum value retrieval on the probability values ​​of each emotion category, and iterate through all emotion category probability values ​​to filter out the confidence level and the corresponding index; Emotional labels are obtained through index mapping, which serve as facial emotion features of the communication subject.

5. The EEG-controlled robotic arm system according to claim 4, characterized in that, The gesture recognition model is based on the YOLOv2 model for gesture recognition. The method for recognizing gesture features using a gesture recognition model includes the following steps: The gesture recognition model inputs a hand image into Darknet-19 for forward propagation and extracts a depth feature map; Multi-scale detection is performed at each grid location on the depth feature map, and predicted bounding boxes are output, where each grid corresponds to multiple anchor points; The predicted bounding boxes are decoded, and the relative offsets are converted into actual image coordinates to obtain the detection boxes; Filter out detection frames with a confidence level below 0.5; Non-maximum suppression is performed on the remaining detection boxes, and highly overlapping detection boxes are filtered out based on an IoU threshold of 0.3 to obtain the hand feature detection results; Probability prediction and label mapping are performed on the hand feature detection results to obtain gesture labels, which are used as the gesture features of the communication object.

6. The EEG-controlled robotic arm system according to claim 5, characterized in that, The robotic arm device includes a robotic arm module consisting of 6 servo motors, a robotic arm base, a robotic upper arm, a robotic lower arm, and a robotic gripper, a motion control board, a voice module, and a hard-shell backpack. The six servo motors include a first servo motor, a second servo motor, a third servo motor, a fourth servo motor, a fifth servo motor, and a sixth servo motor. The first servo motor is fixedly mounted on the base of the robotic arm and is used to control the rotation of the robotic arm around the base. The second servo motor is mounted on the first servo motor and is used to control the forward and backward tilting of the robotic arm. One end of the robotic arm is connected to the second servo motor, and the other end is connected to the third servo motor to extend the arm's reach. The third servo motor controls the inward and outward movements of the robotic forearm relative to the robotic upper arm; one end of the robotic forearm is connected to the third servo motor, and the other end is connected to the fourth servo motor to extend the arm span; the fourth servo motor controls the inward and outward movements of the robotic gripper relative to the robotic forearm; one end of the fifth servo motor is fixedly mounted on the fourth servo motor, and the other end is connected to the root of the robotic gripper to control the rotation of the robotic gripper; the sixth servo motor is located on the gripper side to control the opening and closing of the robotic gripper; the robotic arm base is fixedly connected to the outside of the hard-shell backpack by bolts and nuts; the motion control board and the voice module are located inside the hard-shell backpack; the motion control board communicates with the signal control board via a wireless serial port; the motion control board generates corresponding motion control commands based on the wearer's EEG signal characteristics, head posture angle data, facial emotion characteristics, and gesture characteristics of the person being interacted with, and controls and drives the six servos to execute the corresponding robotic arm movements via the onboard CAN bus; the voice module is used to wake up the robotic arm and perform voice human-computer interaction.

7. The EEG-controlled robotic arm system according to claim 6, characterized in that, The voice model includes an ESP32S3 MCU module, and a microphone and speaker connected to the MCU module; The method for voice models to interact with wearers via voice includes the following steps: The wearer's original audio is recorded via microphone; The original audio is compressed using OPUS encoding via the MCU module to obtain compressed audio. The compressed audio is uploaded to the server using the WebSocket protocol via the MCU module. The server backend decodes the OPUS format to recover the original audio, converts the original audio into text information, and uses the large language model API to analyze the text information before outputting the reply text. The server converts the response text into response audio and sends it to the MCU module according to the WebSocket protocol so that the response audio can be played through the speaker, completing the voice interaction with the wearer.

Images