Mechanical arm control method and system based on motor imagery and eye movement tracking

Abstract

The invention relates to the technical field of mechanical arm control, in particular to a mechanical arm control method and system based on motor imagery and eye movement tracking. (2) constructing an electroencephalogram signal classification model and performing training; (3) calibrating an eye movement tracking module and a mechanical arm control module; (4) inputting data collected in real time into an electroencephalogram signal classification model and an eye movement tracking module, and generating a user intention and a to-be-grabbed target label; and (5) an instruction decision is made according to the intention of the user, the to-be-grabbed target label is input into a mechanical arm control module, a mechanical arm control instruction is generated, the mechanical arm control instruction is driven to execute the grabbing action, and the control process of the mechanical arm is achieved. According to the scheme, more efficient man-machine interaction is achieved, through combination of motor imagery electroencephalogram decoding and eye movement tracking, mechanical arm selection target grabbing operation can be conducted through brain control and eye movement without manual operation, and the real-time performance and stability of a control system are improved through optimization of the decoding process.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arm control, and particularly to a robotic arm control method and system based on motor imagery and eye movement tracking. Background Art

[0002] In traditional robotic arm control systems, users use input devices such as mice, keyboards, joysticks, or teaching pendants to control the devices. However, with the expansion of interaction scenarios and the development of intelligent machinery applications, traditional control methods have gradually revealed drawbacks. First, when the interaction task changes, it often requires users to re-set and program input tools such as teaching pendants, resulting in low efficiency. Second, complex input tools increase the user's learning and training costs. Finally, for elderly or users with motor disabilities, it is difficult to perform interactive control quickly and accurately.

[0003] In this context, brain-computer interface technology can achieve direct mapping of neural intentions to external devices by collecting and decoding electroencephalogram (EEG) signals. This technology does not rely on human muscle tissue and nervous system, and directly establishes communication between the brain and external devices, thus providing help for people with motor disabilities caused by congenital diseases or accidents. Brain-computer interface technology can be divided into invasive, semi-invasive, and non-invasive according to the signal acquisition method. Among them, non-invasive signal electroencephalogram collected by scalp surface electrodes has advantages such as low cost, portability, non-invasiveness, and high safety. The experimental paradigm based on EEG sensorimotor rhythm is called motor imagery. In a control system based on motor imagery, users imagine specific limb movements without actually performing them, and the system decodes the EEG signals to obtain the corresponding intentions of the users. This control system constructed based on motor imagery can often provide an intuitive and simple interaction logic. As an endogenous EEG signal, the motor imagery paradigm does not rely on the visual channel, thus avoiding visual fatigue during long-term operation. However, the free movement of the user's eyes also causes the motor imagery signals to be easily interfered by eye movement artifacts. At the same time, the classification accuracy of motor imagery is significantly affected by individual differences, and it is difficult to increase the number of decoded categories. Currently, the commonly used motor imagery paradigms are mostly binary classification tasks of left hand and right hand or four-classification tasks of left hand, right hand, tongue, and feet. The limited number of decoded categories brings limitations to the construction of a high-degree-of-freedom robotic arm control system.

[0004] In summary, there are still some limitations in existing robotic arm control methods. The traditional robotic arm control methods based on teaching pendants, keyboard and mouse input have low interaction efficiency, high learning costs, and limited applicability. And the robotic arm control method based on a single motor imagery paradigm has the problem of insufficient interaction freedom caused by limited number of decoded categories, and it is difficult to meet the operation requirements of complex tasks.

[0005] Therefore, there is an urgent need for a new technical solution to solve the above technical problems. Summary of the Invention

[0006] The object of the present invention is to overcome the above problems of the prior art, and provides a robotic arm control method and system based on motor imagery and eye tracking, so as to solve the technical problems that the traditional robotic arm control method based on a teach pendant, keyboard and mouse input has low interaction efficiency, high learning cost and limited applicability, while the robotic arm control method based on a single motor imagery paradigm has insufficient interaction freedom due to limited decoding category numbers and is difficult to meet the operation requirements of complex tasks.

[0007] The above object is achieved by the following technical solutions:

[0008] A robotic arm control method and system based on motor imagery and eye tracking, comprising:

[0009] Step 1: Collect data required for a task, including electroencephalogram (EEG) signal data, eye movement camera video stream data, field of view camera video stream data, and scene camera video stream data;

[0010] Step 2: Construct an EEG signal classification model, including a channel selection layer, a compressive sensing layer, and a feature extraction network, and use the EEG signal data to train the EEG signal classification model;

[0011] Step 3: Calibrate an eye tracking module and a robotic arm control module through the eye movement camera video stream data, the field of view camera video stream data, and the scene camera video stream data;

[0012] Step 4: Input the real-time collected data into the EEG signal classification model and the eye tracking module to generate a user intention and a to-be-grabbed target label respectively;

[0013] Step 5: Make an instruction decision according to the user intention, and input the to-be-grabbed target label into the robotic arm control module to generate a robotic arm control instruction and drive it to perform a grabbing action.

[0014] Further, the specific steps of Step 1 include:

[0015] Step 101: The user wears an EEG cap, imagines the movement of the left or right hand according to the instructions of the researcher, and saves the EEG signal data and the instruction label to construct a motor imagery EEG signal data set;

[0016] Step 102: Turn on the eye movement camera, the field of view camera, and the scene camera, and collect eye images, user field of view images, and environmental scene images respectively.

[0017] Further, the specific steps of Step 2 include:

[0018] Step 201: Through a sampling weight of w c,kThe K selective neurons form a channel selection layer, which realizes the differentiable approximation of discrete EEG channel parameters through the reparameterization method, and samples the weight w c,k The calculation formula is as follows:

[0019]

[0020] Among them, the trainable importance parameter α c,k corresponds to the channel weight, and G c,k is a random variable that is independent of each combination of c and k and follows a Gumbel distribution;

[0021] The output calculation formula of the k-th selective neuron is as follows:

[0022] z k = w k T X

[0023] Among them, X is the neuron input, and w k T is the transposed sampled weight matrix, and w k is obtained by summing c from 1 to C for w c,k ;

[0024] Step 202: Construct a compressive sensing layer through a fully connected layer. The output dimension of the compressive sensing layer is N×Q, where N is the number of selected EEG channels and Q is the compressed time dimension. The calculation formula is as follows:

[0025] Q = G × CR

[0026] Among them, G is the time dimension before compression, and the compression ratio CR represents the degree of data compression in the time dimension;

[0027] Step 203: Construct a feature extraction network through a convolutional neural network, and parallelly use convolutional layers with different kernel sizes to extract multi-band features of EEG signals;

[0028] Step 204: Use the motor imagery EEG signal dataset constructed in Step 101 to perform end-to-end training on the EEG signal classification model; during the training process, use the Adam optimizer, combined with the learning rate decay strategy and the dropout strategy of the selection layer, to avoid model overfitting and obtain the optimal weight prediction model.

[0029] Furthermore, the specific content of the said Step 3 includes:

[0030] Step 301: Convert the input eye movement camera image into a grayscale image, apply Gaussian blur to remove noise, and then perform binarization to extract the pixels in the entire image with grayscale values less than the threshold; calculate the average position of these black pixels to determine the center of the region of interest; after determining the region of interest, perform Canny edge detection and then use the HoughCircles algorithm to detect the user's pupil edge, and finally obtain the position of the user's pupil in the eye movement camera image;

[0031] Step 302: Display a red dot on the screen to guide the user to fixate on the red dot, and use the vision camera image to extract the position of the red dot through threshold detection; calculate the homogeneous transformation matrix between the pupil position in the eye movement camera and the red dot position in the vision camera by the least squares method, and map the pupil position to the gaze coordinates in the environmental coordinate system to complete the calibration of the eye movement tracking module;

[0032] Step 303: Send multiple movement instructions to the robotic arm, record the coordinates of the robotic arm's movement position in the scene camera and the movement instructions; calculate the homogeneous transformation matrix between the robotic arm movement instructions and the movement position in the scene camera by the least squares method to determine the conversion relationship between the robotic arm instructions and the scene camera coordinates, and complete the calibration of the robotic arm control module.

[0033] Furthermore, step 4 specifically includes:

[0034] Step 401: Input the real-time collected electroencephalogram (EEG) signal data into the trained EEG signal classification model to obtain the EEG signal classification result output by the model; judge the user's intention according to the EEG signal classification result. If the classification result shows that the user is imagining the movement of the left hand, it is judged that the user hopes to execute the robotic arm grasping operation; if the classification result shows that the user is imagining the movement of the right hand, it is judged that the user hopes to cancel the robotic arm grasping operation;

[0035] Step 402: When it is judged that the user hopes to execute the robotic arm grasping operation, input the vision camera video stream image into the YOLOv8 object detection model to generate object bounding boxes and labels; the eye movement tracking module calculates the pupil position at this time and maps it to the gaze coordinates, and confirms the object within the bounding box that contains the gaze coordinates as the target to be grasped, and saves the object label of the target to be grasped.

[0036] Furthermore, step 5 specifically includes:

[0037] Step 501: According to the user's intention, when it is judged through the EEG signal classification result that the user hopes to execute the robotic arm grasping operation, input the object label of the target to be grasped into the robotic arm control module;

[0038] Step 502: Input the video stream image of the scene camera into the YOLOv8 object detection model, match the target label to be grasped with the detection result, and output the center coordinates of the matched object.

[0039] Step 503: The robotic arm control module generates servo control instructions according to the object center coordinates and drives the robotic arm to perform grasping.

[0040] A robotic arm control system based on motor imagery and eye tracking for implementing the method of claims 1-6, comprising:

[0041] Perception and interaction layer: The perception and interaction layer is used to collect and preprocess the EEG and eye movement data during user interaction, including an EEG decoding module and an eye tracking module; the EEG decoding module real-time collects the EEG signals of the user performing motor imagery through an EEG cap, and performs the training and inference of the EEG signal classification model, and outputs the motor imagery intention of the left or right hand; the eye tracking module includes an eye movement camera and a field of view camera, detects the pupil position through the video stream image and maps it to the gaze direction in the field of view camera, and can detect the object label that the user is looking at at the same time.

[0042] Intelligent decision-making layer: The intelligent decision-making layer includes a control decision-making module for coordinating multi-modal signals, receiving the user intention output by the EEG signal decoding module and judging whether to trigger the target matching process, and determining whether the object label output by the eye tracking module is the target label to be grasped.

[0043] Execution terminal layer: The execution terminal layer includes an instruction encoding module and a robotic arm control module; the instruction encoding module maps the robotic arm grasping instruction to the rotation angles of the servos of each joint of the robotic arm through the robotic arm kinematic model, encodes it and outputs it to the robotic arm main board through the serial port; the robotic arm control module matches the target label to be grasped through the scene camera to obtain the target coordinates, and converts it into a robotic arm grasping instruction to realize the grasping operation of the robotic arm on the target.

[0044] A robotic arm control method and system based on motor imagery and eye tracking provided by the present invention can achieve a more efficient human-machine interaction method. Through the combination of a brain signal decoding module and an eye tracking module, users can interact with the robotic arm to select and grasp targets through brain control and eye movement without manual operation. Different from traditional eye tracking systems that require strict constraints on the user's head posture or preset interaction targets, the present invention realizes the coordinated perception between the user and the environment, expands the interaction targets while the user's head can move freely. In addition, it can also achieve a more lightweight brain signal decoding process. Through the use of channel selection based on the reparameterization method, the dynamic optimization of channel selection is realized, enabling the network to automatically identify the most informative electroencephalogram channels for the classification task, thereby reducing the data dimension and the computational burden, and achieving the optimization goals of lightweight and low latency for the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of a robotic arm control method based on motor imagery and eye tracking according to the present invention;

[0046] Figure 2 is a schematic diagram of the network structure of an electroencephalogram signal classification model in a robotic arm control method based on motor imagery and eye tracking according to the present invention;

[0047] Figure 3 is a block diagram of the architecture of a robotic arm control system based on motor imagery and eye tracking according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The present invention will be further described in detail below with reference to the drawings and embodiments. The described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0049] As Figure 1 shown, the present solution provides a robotic arm control method based on motor imagery and eye tracking, including the following steps:

[0050] Step 1: Collect data required for the task, including electroencephalogram signal data, eye tracking camera video stream data, field of view camera video stream data, and scene camera video stream data;

[0051] Step 2: Construct an electroencephalogram signal classification model, including a channel selection layer, a compressive sensing layer, and a feature extraction network, and use the electroencephalogram signal data to train the electroencephalogram signal classification model;

[0052] Step 3: Calibrate the eye movement tracking module and the robotic arm control module based on the eye movement camera video stream data, the field of view camera video stream data, and the scene camera video stream data;

[0053] Step 4: Input the real-time collected data into the electroencephalogram signal classification model and the eye movement tracking module to generate user intentions and target labels to be grasped respectively;

[0054] Step 5: Make instruction decisions according to the user's intention, and input the target label to be grasped into the robotic arm control module to generate a robotic arm control instruction and drive it to perform the grasping action.

[0055] In this embodiment, step 1 specifically includes:

[0056] Step 101: The user wears an EEG cap. First, according to the instructions of the researcher, imagine the movement of the left or right hand, and save the EEG signal data and the instruction label to construct a motor imagery EEG signal dataset; during the acquisition process, the user needs to face a display screen, and a green cross icon is displayed on the screen to prompt the user's gaze focus position; at the 2nd second of the start of the acquisition, a sound signal is provided to prompt the user that a red arrow will appear after 1 second; the left and right directions of the arrow respectively indicate the motor imagery tasks of the left or right hand; once the red arrow appears, the user immediately starts the corresponding motor imagery task and maintains this state until the screen goes black again;

[0057] Step 102: The user wears glasses integrated with an eye movement camera and a field of view camera, and turns on the eye movement camera, the field of view camera, and the scene camera to collect eye images, user field of view images, and environmental scene images respectively.

[0058] In this embodiment, step 2 specifically includes:

[0059] Step 201: Construct an electroencephalogram signal classification model, including a channel selection layer, a compressive sensing layer, and a feature extraction network, as Figure 2 shown. First, form a channel selection layer by sampling K selection neurons with a weight of w c,k to realize the differentiable approximation of discrete EEG channel parameters through the reparameterization method. The calculation formula of the sampling weight w c,k is as follows:

[0060]

[0061] where the trainable importance parameter α c,k corresponds to the channel weight, and G c,kis a random variable that is independent of each combination of c and k and obeys the Gumbel distribution; when τ increases, the output gradually tends to a uniform distribution; conversely, when τ decreases, the output gradually tends to a one-hot distribution;

[0062] The output calculation formula of the k-th selection neuron is as follows:

[0063] z k =w k T X

[0064] Among them, X is the neuron input, w k T is the transposed sampling weight matrix, w k By c,k The sum of c from 1 to C is obtained; the input dimension of the channel selection layer is C×G, and the output dimension is N×G, where G represents the number of time samples, C represents the total number of channels for collecting EEG signals, and the spatial dimension N represents the number of selected channels, which is equal to the number of neurons K selected by the channel selection layer.

[0065] Step 202: construct a compressed sensing layer through a fully connected layer. The output dimension of the compressed sensing layer is N×Q, where N is the number of EEG channels after selection and Q is the compressed time dimension. The calculation formula is as follows:

[0066] Q=G×CR

[0067] Among them, G is the time dimension before compression, and the compression ratio CR represents the degree of data compression in the time dimension;

[0068] Step 203: construct a feature extraction network through a convolutional neural network, which includes multiple convolutional layers and pooling layers. The specific structure is as follows:

[0069] (1) Convolutional layer: The feature extraction network includes a parallel convolutional layer and a subsequent spatial convolutional layer. In the parallel convolutional layer, four parallel convolutional layers using convolution kernels of different sizes are used to perform convolution operations in the time domain. This multi-scale convolution design enables the network to effectively extract feature information of EEG activity at different frequencies. In the subsequent stage of feature extraction, a spatial convolutional layer is used to perform convolution processing on the data in the spatial domain to enhance the network's feature extraction capability for spatial information.

[0070] (2) Batch Normalization Layer: Each convolutional layer is followed by a batch normalization layer to improve the stability of training and accelerate convergence. Batch normalization reduces internal covariate shift by standardizing the output of each mini-batch;

[0071] (3) Average pooling layer: Reduces the feature dimension through temporal average pooling, enhancing computational efficiency while retaining important information. This operation also helps mitigate overfitting;

[0072] (4) Dropout layer: During training, randomly deactivates some neurons, enabling the network to be independent of the outputs of some specific neurons, thereby enhancing the model's robustness and generalization ability and preventing overfitting during model training;

[0073] (5) Fully connected layer: The last part of the feature extraction network is the fully connected layer, which is responsible for mapping the extracted features to classification results; After flattening the features and entering the fully connected layer, the feature extraction network can learn the overall representation of the motor imagery EEG signals and finally output the probability distribution of each class.

[0074] Step 204: Use the motor imagery EEG signal dataset constructed in Step 101 to perform end-to-end training on the EEG signal classification model; During training, use the Adam optimizer, combined with the learning rate decay strategy and the selective layer dropout strategy, to avoid model overfitting and obtain the optimal weight prediction model.

[0075] In this embodiment, Step 3 specifically includes:

[0076] Step 301: Perform pupil position localization; Convert the input eye movement camera image into a grayscale image, apply Gaussian blur to remove noise, and then perform binarization to extract the pixels with grayscale values less than the threshold in the entire image; Calculate the average position of these black pixels to determine the center of the region of interest; After determining the region of interest, perform Canny edge detection and then use the HoughCircles algorithm to detect the user's pupil edge; For each edge point, the HoughCircles algorithm calculates all possible center coordinates and corresponding radii of the edge point. Each possible combination of center and radius will obtain a vote value in the parameter space, indicating the likelihood of the combination appearing in the image. Each calculation will increase the vote value for each pair of center coordinates and radius in the parameter space. After processing all edge points, return the combination with the highest cumulative vote value as the position of the user's pupil in the eye movement camera image;

[0077] Step 302: Calibrate the eye tracking module; First, draw a red dot on the screen of the black background image and keep it for a set time so that the user can fixate steadily; At this time, the program will record the moment when the red dot is displayed and start capturing the video streams from the spatial camera and the eye movement camera; The input data of each camera will be saved as an image in real time; Extract the position of the red dot through threshold detection in the vision camera image; Calculate the homogeneous transformation matrix between the pupil position in the eye movement camera and the red dot position in the vision camera by the least squares method, and map the pupil position to the gaze coordinates in the environment coordinate system to complete the calibration of the eye tracking module;

[0078] Step 303: Calibrate the robotic arm control module, and the specific implementation steps are as follows:

[0079] (1) Place white paper: Fix the white paper for recording positions within the working range of the robotic arm to ensure that the scene camera can capture the entire working range of the robotic arm;

[0080] (2) Record the moving positions of the robotic arm: Send a moving instruction to the robotic arm and make a mark at the position where the end of the robotic arm touches the white paper. To ensure the calibration accuracy, repeat this step twelve times and ensure that the marking order is accurately recorded to correspond to the moving instructions;

[0081] (3) Collect scene camera images: After the robotic arm movement is completed, save the images captured by the scene camera. And obtain the relative positions of each mark in this image;

[0082] (4) Determine the correspondence: Establish the connection between the coordinate system of the robotic arm and the coordinate system of the camera by calculating the correspondence between the mark positions in the scene camera and the moving instructions. Solve the transformation matrix by the least squares method to determine the conversion relationship between the robotic arm instruction coordinates and the scene camera coordinates.

[0083] Step 4 in this embodiment specifically includes:

[0084] Step 401: Input the real-time collected electroencephalogram signal data into the trained electroencephalogram signal classification model to obtain the electroencephalogram signal classification result output by the model; Judge the user's intention according to the electroencephalogram signal classification result. If the classification result shows that the user is imagining the movement of the left hand, judge that the user hopes to execute the robotic arm grasping operation; If the classification result shows that the user is imagining the movement of the right hand, judge that the user hopes to cancel the robotic arm grasping operation;

[0085] Step 402: When it is determined that the user hopes to perform a robotic arm grasping operation, input the video stream image of the vision camera into the YOLOv8 object detection model to generate object bounding boxes and labels; the eye tracking module calculates the pupil position at this time and maps it to the gaze coordinates, and determines the object within the bounding box that contains the gaze coordinates as the target to be grasped, and saves the object label of the target to be grasped.

[0086] In this embodiment, step 5 specifically includes:

[0087] Step 501: According to the user's intention, when it is determined through the EEG signal classification result that the user hopes to perform a robotic arm grasping operation, input the object label of the target to be grasped into the robotic arm control module;

[0088] Step 502: Input the video stream image of the scene camera into the YOLOv8 object detection model, match the target to be grasped label with the detection result, and output the center coordinates of the matched object;

[0089] Step 503: The robotic arm control module generates servo control instructions according to the object center coordinates and drives the robotic arm to perform grasping.

[0090] As Figure 3 shown, this solution also provides a robotic arm control system based on motor imagery and eye tracking for implementing the above-mentioned robotic arm control method based on motor imagery and eye tracking. The system includes a perception and interaction layer, an intelligent decision-making layer, and an execution terminal layer, where:

[0091] The perception and interaction layer is used to collect and preprocess EEG and eye movement data during user interaction, including an EEG decoding module and an eye tracking module; in the EEG decoding module, the user needs to wear an EEG cap to collect EEG signals in real time, save the EEG signals and labels when the user performs motor imagery to construct an EEG signal dataset for training the EEG signal classification model. The network model weights obtained after training are used to calculate the motor imagery intention of the user's left or right hand during real-time inference; in the eye tracking module, the device worn by the user includes an eye movement camera and a vision camera; among them, the video stream image of the eye movement camera is used for pupil position detection; after the vision camera is calibrated with the eye movement camera in the coordinate system, the pupil position can be mapped to the gaze direction in the field of view to detect the object label that the user is looking at;

[0092] The intelligent decision-making layer includes a control decision module for coordinating multi-modal signals; after receiving the user intention output by the EEG signal decoding module, the control decision module determines whether to trigger the target matching process; if it is confirmed to perform the target matching operation, the object label output by the eye tracking module is set as the target to be grasped label and transmitted to the execution terminal layer;

[0093] The execution terminal layer includes an instruction encoding module and a robotic arm control module; in the instruction encoding module, the mapping between the robotic arm grasping instruction and the rotation angles of the servos of each joint of the robotic arm is realized through the robotic arm kinematic model, and then the grasping instruction can be encoded and output to the robotic arm main board through the serial port to drive the robotic arm; the robotic arm control module performs target detection on the scene camera image, matches the target label to be grasped and obtains the target coordinates, and finally converts them into robotic arm grasping instructions to realize the grasping operation of the robotic arm on the target.

[0094] The above is only to illustrate the embodiments of the present invention and is not intended to limit the present invention. For those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A robotic arm control method based on motor imagery and eye tracking, characterized in that: include: Step 1: Collect the data required for the task, including EEG signal data, eye movement camera video stream data, field of view camera video stream data, and scene camera video stream data; Step 2: constructing an EEG signal classification model, including a channel selection layer, a compressed sensing layer, and a feature extraction network, and using the EEG signal data to train the EEG signal classification model; Step 3: Calibrate the eye tracking module and the robotic arm control module through the eye camera video stream data, the field of view camera video stream data and the scene camera video stream data; Step 4: By inputting the real-time collected data into the EEG signal classification model and the eye tracking module, the user intention and the target label to be captured are generated respectively; Step 5: Make command decisions based on user intentions, and input the target label to be grasped into the robot control module to generate a robot control command and drive it to perform a grasping action.

2. A method for controlling a robotic arm based on motor imagery and eye tracking according to claim 1, characterized in that: The step 1 specifically includes: Step 101: The user wears an EEG cap and imagines the movement of the left or right hand according to the researcher's instructions, and saves the EEG signal data and instruction labels to construct a movement imagery EEG signal dataset; Step 102: Turn on the eye movement camera, the field of view camera and the scene camera to collect eye images, user field of view images and environmental scene images respectively.

3. A method for controlling a robotic arm based on motor imagery and eye tracking according to claim 2, characterized in that: The step 2 specifically includes: Step 201: By sampling weight w c,k The K selection neurons of the channel selection layer form a differentiable approximation of discrete EEG channel parameters through the reparameterization method. The sampling weight w c,k The calculation formula is as follows: Among them, the trainable importance parameter α c,k Corresponding channel weight, G c,k is a random variable that is independent of each combination of c and k and follows a Gumbel distribution; The output calculation formula of the k-th selection neuron is as follows: z k =w k T X Among them, X is the neuron input, w k T is the transposed sampling weight matrix, w k By c,k The sum of c from 1 to C; Step 202: construct a compressed sensing layer through a fully connected layer, the output dimension of the compressed sensing layer is N×Q, where N is the number of EEG channels after selection, and Q is the compressed time dimension. The calculation formula is as follows: Q=G×CR Among them, G is the time dimension before compression, and the compression ratio CR represents the degree of data compression in the time dimension; Step 203: constructing a feature extraction network through a convolutional neural network, and using convolutional layers with different kernel sizes in parallel to extract multi-band features of the EEG signal; Step 204: Use the motor imagery EEG signal dataset constructed in step 101 to perform end-to-end training on the EEG signal classification model; during the training process, use the Adam optimizer, combined with the learning rate decay strategy and the selection layer random inactivation strategy to avoid model overfitting and obtain the optimal weight prediction model.

4. The method for controlling a robotic arm based on motor imagery and eye tracking according to claim 3, characterized in that: The step 3 specifically includes: Step 301: convert the input eye-tracking camera image into a grayscale image, apply Gaussian blur to remove noise, and then perform binarization processing to extract pixels whose grayscale values ​​of the entire image are less than a threshold; calculate the average position of these black pixels to determine the center of the region of interest; after determining the region of interest, perform Canny edge detection and then use the HoughCircles algorithm to detect the edge of the user's pupil, and finally obtain the position of the user's pupil in the eye-tracking camera image; Step 302: Display a red dot on the screen to guide the user to gaze at the red dot, and use the field of view camera image to extract the position of the red dot through threshold detection; calculate the homogeneous transformation matrix of the pupil position in the eye movement camera and the red dot position in the field of view camera by the least square method, map the pupil position to the gaze coordinates in the environment coordinate system, and complete the calibration of the eye tracking module; Step 303: Send multiple movement instructions to the robot arm, record the coordinates of the robot arm's moving position in the scene camera and the movement instructions; calculate the homogeneous transformation matrix of the moving position of the robot arm's movement instruction in the scene camera by the least squares method, determine the conversion relationship between the robot arm instruction and the scene camera coordinates, and complete the calibration of the robot arm control module.

5. The method for controlling a robotic arm based on motor imagery and eye tracking according to claim 1, characterized in that: The step 4 specifically includes: Step 401: inputting the real-time collected EEG signal data into the trained EEG signal classification model to obtain the EEG signal classification result output by the model; judging the user intention according to the EEG signal classification result, if the classification result shows that the user is imagining the movement of the left hand, then judging that the user wants to perform the robot arm grasping operation; if the classification result shows that the user is imagining the movement of the right hand, then judging that the user wants to cancel the robot arm grasping operation; Step 402: When it is determined that the user wants to perform a robotic arm grasping operation, the field of view camera video stream image is input into the YOLOv8 target detection model to generate an object bounding box and label; the eye tracking module calculates the pupil position at this time and maps it to gaze coordinates, and identifies the object in the bounding box containing the gaze coordinates as the target to be grasped, and saves the object label of the target to be grasped.

6. A method and system for controlling a robotic arm based on motor imagery and eye tracking according to claim 1, characterized in that: The step 5 specifically includes: Step 501: According to the user's intention, when it is determined through the EEG signal classification result that the user wants to perform a robotic arm grasping operation, the object label of the target to be grasped is input into the robotic arm control module; Step 502: Input the scene camera video stream image into the YOLOv8 target detection model, match the target label to be captured with the detection result, and output the center coordinates of the matched object; Step 503: The robotic arm control module generates a steering gear control instruction according to the center coordinates of the object and drives the robotic arm to perform grasping.

7. A robotic arm control system based on motor imagery and eye tracking, used to implement the method of claims 1-6, characterized in that: include: Perception interaction layer: The perception interaction layer is used to collect EEG and eye movement data during user interaction and perform preprocessing, including an EEG decoding module and an eye movement tracking module; The EEG decoding module collects EEG signals of the user when performing motor imagery in real time through an EEG cap, performs training and reasoning of an EEG signal classification model, and outputs the motor imagery intention of the left or right hand; the eye tracking module includes an eye camera and a field of view camera, detects the pupil position through a video stream image and maps it to the gaze direction in the field of view camera, and can also detect the label of the object the user is looking at; Intelligent decision layer: The intelligent decision layer includes a control decision module, which is used to coordinate multimodal signals, receive the user intention output by the EEG signal decoding module and determine whether to trigger the target matching process, and determine whether the object label output by the eye tracking module is the target label to be captured; Execution terminal layer: The execution terminal layer includes an instruction encoding module and a robotic arm control module; the instruction encoding module maps the robotic arm grasping instruction to the rotation angle of the servo of each joint of the robotic arm through the robotic arm kinematic model, encodes it and outputs it to the robotic arm mainboard through the serial port; the robotic arm control module obtains the target coordinates by matching the target label to be grasped through the scene camera, and converts it into the robotic arm grasping instruction to realize the robotic arm's grasping operation on the target.

Images