Robotic arm grasping method based on adaptive update of stimulation control

Abstract

The application discloses a mechanical arm grabbing method based on stimulus control adaptive updating, which can be used for matching a steady-state visual evoked potential (SSVEP) based rehabilitation auxiliary brain-computer interface system for different application requirements. The method process is as follows: a mask region model for the task is obtained by training a self-built data set, target recognition and mask segmentation of the experimental scene image are carried out by using the mask region model, the classification, the bounding box and the mask of the object in the image are obtained, adaptive frequency and phase distribution are carried out based on the same, the stimulus control is updated in real time, the user induces the SSVEP signal by staring at the corresponding stimulus control of the target object, the recognition result is intuitively fed back to the user through display mask and the like after classification analysis, and the mechanical arm is controlled to assist the user to complete the grabbing and the like.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology for robotic arms, specifically relating to a robotic arm grasping method based on adaptive updates of stimulus controls. Background Technology

[0002] Brain-computer interface (BCI) systems are cutting-edge technologies that enable direct human-computer interaction by decoding central nervous system activity. These systems primarily utilize non-invasive electroencephalography (EEG), invasive electrocorticography (ECoG), or functional near-infrared spectroscopy (fNIRS) to record and analyze the user's neurophysiological activity in real time. Through advanced signal processing and machine learning algorithms, BCI systems can decode the user's brain signals into control commands, driving external devices or computer systems to function. This helps users with special needs overcome motor impairments, interact effectively with their environment, enhance motor rehabilitation, and improve quality of life, demonstrating significant advantages over traditional rehabilitation methods. Furthermore, BCI technology shows great potential in the field of mental health. For example, through emotion-inducing experimental tasks and EEG feature analysis, it can assist in symptom quantification and neurofeedback therapy, providing new technological means for mental health rehabilitation.

[0003] Considering safety, cost, and performance, Steady-State Visual Evoked Potential (SSVEP) stands out among various EEG technologies. By analyzing SSVEP signals in the occipital region of the user's brain, the user's motor intentions can be accurately captured, thereby enabling effective control of assistive rehabilitation robot systems. However, the understanding and application of SSVEP-based BCI systems still need to be deepened. Traditional technologies typically employ visual stimulation paradigms with fixed frequencies and phase encodings, relying heavily on simple square flashing or LED arrays, making it difficult to dynamically adjust to meet real-time task requirements and limiting the degree of freedom of interaction. Therefore, accurately identifying motor intentions and user needs, and developing more flexible, personalized, and interactive rehabilitation assistive technologies, remains one of the main challenges. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a robotic arm grasping method based on adaptive updating of stimulus controls. This method trains a mask region model tailored to the specific task using a self-built dataset. This model is then used to perform target recognition and mask segmentation on experimental scene images, obtaining object classifications, bounding boxes, and masks. Based on this, adaptive frequency and phase allocation is performed to form real-time matched and updated stimulus controls. The user induces an SSVEP signal by gazing at the stimulus control corresponding to the target object. After classification and analysis, the recognition results are intuitively fed back to the user through methods such as displaying a mask. Simultaneously, the robotic arm is controlled to assist the user in completing grasping tasks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A robotic arm grasping method based on adaptive updating of stimulus controls, the robotic arm grasping method comprising the following steps:

[0007] S1. Select N objects to participate in this robotic arm grasping task, take M images to form an initial dataset, mark the object edges of each image, and convert them into corresponding mask images, label visualization images, and category list files to form a self-built dataset.

[0008] S2. Initialize the Mask R-CNN model based on convolutional neural network. The Mask R-CNN model based on convolutional neural network is referred to as the mask region model. By training the mask region model on different components using different learning rates on a self-built dataset, a mask region model for this robotic arm grasping task is obtained.

[0009] S3. Capture experimental scene images using a depth camera, and use a mask region model for this robotic arm grasping task to perform target recognition and mask segmentation on the experimental scene images to obtain the classification, bounding boxes, and masks of L objects in the experimental scene images, and calculate the center position of each object.

[0010] S4. Based on the number and center position of the objects, perform adaptive frequency and phase allocation online, and place a stimulation circle of a specified size at the center position of each object as a stimulation control. Display the bounding boxes of all objects for the first threshold time to prompt the user to select targets for this round of experiment. Then, according to the allocated stimulation paradigm, perform flashing stimulation for the second threshold time on the stimulation interface.

[0011] S5. The user gazes at the stimulation circle placed on the target object position in the stimulation interface, generating a corresponding steady-state visual evoked potential signal, hereinafter referred to as SSVEP. The user's SSVEP signal is collected, and baseline correction and filtering preprocessing are performed. The data is then classified and analyzed to obtain the EEG recognition result. The result is fed back to the user by displaying the mask object and bounding box of the target object for a third threshold time. At the same time, the robotic arm is controlled to perform the corresponding operation. Then, S3 to S5 are repeated until the robotic arm grasping task is completed.

[0012] Furthermore, step S1 is performed as follows:

[0013] S11. Determine the requirements for this robotic arm grasping task. For each object, the front view, side view and top view of each object are repeatedly photographed from multiple angles to cover the possible pose and lighting changes of each object in the experimental scene, so as to alleviate the difference between the general dataset and the actual task requirements, and finally form an initial dataset containing M images.

[0014] S12. Use a point-by-point polyline annotation method along the visible contour of the object to annotate the object edge of each image, generate M JSON files, and improve the learning accuracy of the subsequent head network.

[0015] S13. Use automated scripts to convert each JSON file into a corresponding mask image, label visualization image, and category list file, and check the labeling accuracy through the label visualization image to shorten the manual review time and ensure labeling quality.

[0016] S14. By organizing and placing M original images, M JSON files, M mask images, and M category list files separately, a self-built dataset is formed, which is convenient for model reading and training.

[0017] Furthermore, the mask region model comprises four components: a backbone network, a Region Proposal Network (RPN), a Region of Interest Alignment (ROIAlign), and a head network. The backbone network employs a feature pyramid structure to extract features from the input image, obtaining a multi-scale feature map to balance small target details with high-level semantics and alleviate information loss. The RPN generates candidate target regions on the multi-scale feature map based on a multi-scale anchor box mechanism. That is, multiple candidate boxes of specific sizes and aspect ratios are pre-placed in each cell of the multi-scale feature map, and then candidate boxes containing the target are selected and their positions and sizes are fine-tuned to avoid missed detections. After mapping the candidate target regions to the multi-scale feature map, ROIAlign extracts alignment features from the multi-scale feature map through bilinear interpolation to avoid edge misalignment from increasing errors. The head network performs target recognition and end-to-end pixel-level prediction mask segmentation based on the alignment features, ultimately obtaining accurate classification, bounding boxes, and masks of objects in the input image.

[0018] Furthermore, step S2 is as follows:

[0019] S21. Load the self-built dataset, modify the category definition based on the objects in the dataset, and use the random hierarchical partitioning method to perform random partitioning independently for each category, so that the ratio of training set to validation set is 8:2, ensuring that each category maintains the same proportion in the training and validation stages, and improving the generalization performance of the mask region model.

[0020] S22. Initialize the pre-trained mask region model in the deep learning framework, retain the weight parameters of the backbone network, RPN and ROIAlign, adjust the output dimension of the head network to N, freeze the pre-trained weights of the first three parts, and only fine-tune the head network. This can significantly reduce training time while avoiding catastrophic forgetting and quickly adapt to new objects.

[0021] S23. Set the head network learning rate to... To accelerate the convergence of the weight parameters of the head network's output layer, the learning rates of the backbone network, RPN, and ROIAlign are set to... To stabilize the gradient magnitude of these three weights during backpropagation, a mask region model is trained on a self-built dataset. Finally, a mask region model for this robotic arm grasping task is output, enabling the model to achieve high-precision classification, bounding boxes, and mask output with limited data.

[0022] Furthermore, step S3 is as follows:

[0023] S31. To address the issue that traditional stimulus paradigms cannot dynamically perceive changes in objects within the experimental scene, a depth camera is placed directly above the experimental scene. This camera can simultaneously provide the two-dimensional pixel coordinates and corresponding depth values ​​of objects, providing complete three-dimensional spatial information for subsequent robotic arm grasping. After the experiment begins, the current frame of the experimental scene image is captured. A mask region model tailored to this robotic arm grasping task is then used to perform target recognition and mask segmentation on this experimental scene image, obtaining the target area within the experimental scene image. The object classification, bounding box, and mask information are as follows: ;

[0024] S32. To address the complex problem of calculating the geometric center of irregular objects, the center point of the object's bounding box is used to represent the object's center point for rapid coordinate calculation. This is based on the bounding box coordinate information of the i-th object. , Calculate the coordinates of the center point of the bounding box to represent the center point coordinates of the i-th object:

[0025]

[0026]

[0027] in, This represents a Cartesian coordinate system with its origin at the top left corner. It is the x-coordinate of the top-left corner of the bounding box of the i-th object in the Cartesian coordinate system. It is the ordinate of the top-left corner of the bounding box of the i-th object in the Cartesian coordinate system. It is the x-coordinate of the lower right corner of the bounding box of the i-th object in the Cartesian coordinate system. It is the ordinate of the bottom right corner of the bounding box of the i-th object in the Cartesian coordinate system. This represents the x-coordinate of the i-th object's center point in the Cartesian coordinate system. This represents the ordinate of the i-th object's center point in the Cartesian coordinate system;

[0028] S33. Because the bounding box information obtained after target recognition and mask segmentation of the experimental scene image using the mask region model for this robotic arm grasping task is in a Cartesian coordinate system with the top left corner as the origin, while the experimental window is based on a right-handed coordinate system with the center as the origin, in order to correctly display the flashing stimulus in the experimental window, it is necessary to change the center point coordinates. Transform from a Cartesian coordinate system with its origin at the top left corner to a right-handed coordinate system with its origin at the center:

[0029]

[0030]

[0031] in, This represents a right-handed coordinate system with the center as the origin. This represents the x-coordinate of the i-th object's center point in a right-handed coordinate system. This represents the ordinate of the i-th object's center point in a right-handed coordinate system. It is the width of the experimental scene image. It is the height of the experimental scene image;

[0032] S34. Convert the mask information of L objects into mask objects of a specified format. The mask information of the objects is a two-dimensional array. It is necessary to create L arrays to store RGBA format image data. Fill the positions in the i-th array where the mask information of the i-th object is greater than 0.5 with the specified color information and fill the positions where the mask information is less than or equal to 0.5 with the transparency information. Then convert the array into a mask object acceptable to the experimental window. In S52, the mask object of the target object corresponding to the recognition result can be correctly displayed in the experimental window, which is presented intuitively and significantly reduces the user's learning cost and operation complexity.

[0033] Furthermore, step S4 is as follows:

[0034] S41. Based on the number of objects Generate a frequency sequence and set a minimum frequency value. and the highest value The intervals of the frequency sequences in this round of experiments can be calculated online using the following formula. :

[0035]

[0036] in, It is a rounding function, representing the rounding operation. The result is rounded to one decimal place, depending on the number of objects. Minimum frequency and interval The frequency sequence of this round of experiments :

[0037]

[0038] in, This represents the frequency of the k-th object; unlike traditional stimulus paradigms that use a fixed number of objects and frequency sequences, this method sets the minimum frequency value based on the optimal stimulus band of SSVEP. and the highest value The number of objects was calculated to be Time-frequency sequence interval Then, a frequency sequence suitable for this round of experiments is obtained. This improves the robustness of the experimental stimulus paradigm and the accuracy of the recognition results;

[0039] S42. Assign a corresponding frequency and phase to each object according to its position, and calculate the center point of the i-th object in the right-hand coordinate system. With the origin of the right-hand coordinate system distance :

[0040]

[0041] in, Represents the midpoint of the right-hand coordinate system With the origin of the right-hand coordinate system The geometric distances between them are reordered in descending order. Let represent the index of the i-th object in the new sorting. Then, the distance of the i-th object is re-recorded as . Then, according to this new sorting, the frequency of the j-th object is assigned sequentially. and phase :

[0042]

[0043] in, This represents the phase of the j-th object in the new sorting. This allows for rapid frequency and phase allocation while reducing interference to SSVEP signal generation when adjacent frequencies are close in position. To avoid the impact of different mask sizes and colors on the accuracy and fairness of the stimulation results, a stimulation circle of a specified size is uniformly used as the stimulation control and placed at the geometric center of the object. The blinking stimulation pattern of each object's stimulation control is set to a stimulation circle with a radius ranging from [40 pixels to 70 pixels]. The stimulus circle for the j-th object The coordinates of the center point of the circle, As a stimulus circle The frequency of the flashing stimulus, As The phase of the flashing stimulus;

[0044] S43. Display the bounding boxes of all objects for the first threshold time to prompt the user that there are selectable targets in this round of experiment. Then, according to the assigned stimulus paradigm, perform flashing stimulation on the stimulus interface for the second threshold time.

[0045] Furthermore, step S5 is as follows:

[0046] S51. The user generates a corresponding SSVEP signal by gazing at the stimulus circle placed at the target object position on the stimulus interface. The two-dimensional data of the acquired SSVEP signal are processed. Baseline correction is performed by averaging the two-dimensional data using a sliding window;

[0047] Two-dimensional data of input SSVEP signal ,in This refers to the number of SSVEP signal acquisition channels. It is the number of sampling points of the SSVEP signal within the second threshold time, for two-dimensional data. Perform baseline correction:

[0048]

[0049] in, To adjust the sliding window size, This indicates that two-dimensional data Divided into The size is The time window is defined and the average value of each time window is calculated. Subtracting the average value of each sampling point within its time window from the average value of that sampling point yields two-dimensional data that has eliminated baseline drift. This effectively solved the baseline fluctuation problem in SSVEP signals caused by physiological noise or equipment drift; then, the two-dimensional data... A 50Hz notch filter and a 1Hz to 100Hz bandpass filter are applied to eliminate power frequency interference while retaining the main frequency range of the SSVEP signal, further purifying the SSVEP signal to improve the accuracy of subsequent processing.

[0050] S52. Processed two-dimensional data The EEG recognition results are obtained through classification and analysis. Based on the EEG recognition results, the mask object and bounding box of the corresponding target object are displayed on the experimental window for a third threshold time to provide feedback to the user. This real-time feedback mechanism ensures that the user is aware of the system's recognition results, improving the user-friendliness and reliability of the interaction. At the same time, the robotic arm is controlled to perform corresponding operations in combination with depth information to ensure the accuracy of grasping. At this point, one round of experiment is completed. Then, steps S3 to S5 are repeated until the robotic arm grasping task is completed.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] 1. Dynamic and adaptive stimulus paradigm with flexible scenarios: In experimental scenarios with the same or similar tasks, it breaks through the limitations of traditional pre-arranged stimulus paradigms with fixed object types, quantities, and positions. Through a mask region model, it performs real-time target recognition and mask segmentation, adaptively generating stimulus controls in the stimulus interface. Furthermore, it automatically forms frequency and phase allocation with minimal interference based on the spatial distribution of objects, thereby driving the dynamic update of the stimulus interface. It supports the addition, removal, or movement of objects during the experiment without the need for repeated manual modifications to the stimulus paradigm and control program, thus improving the flexibility of experimental scenarios and stimulus paradigms.

[0053] 2. Task-driven model customization with strong portability: The mask region model can be dynamically adjusted according to different needs. Transfer training can be performed based on task scenarios and self-built datasets to quickly obtain task-oriented mask region models, reducing the adaptation cost of new tasks, improving the adaptability and portability of the experimental platform, and meeting the needs of different tasks, different users, and different fields.

[0054] 3. Simple and clear interface with user-friendly interaction: Users no longer need to learn complicated stimulus paradigms. They only need to set three threshold times (cues / stimuli / feedback) and look at the target object in the stimulus interface. The robotic arm will automatically complete the remaining tasks and can be easily controlled in real time. After each round of recognition, a mask feedback mechanism is set up to intuitively present the recognition results and the basis for system decision-making, reducing the learning cost for users and making it easy for users with different physical and cognitive conditions to use. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the design structure of the method of the present invention;

[0057] Figure 2 This is a schematic flowchart of the method of the present invention;

[0058] Figure 3 This is a schematic diagram of the mask region model in the method of the present invention;

[0059] Figure 4 This is a schematic diagram of the stimulation interface for a grasping task based on six objects in Embodiment 1 of the present invention.

[0060] Figure 5 This is a schematic diagram of the feedback interface for a grasping task based on six objects in Embodiment 1 of the present invention;

[0061] Figure 6 This is a schematic diagram of the stimulation interface for a grasping task based on five objects in Embodiment 2 of the present invention.

[0062] Figure 7 This is a schematic diagram of the feedback interface for a five-object grasping task in Embodiment 2 of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0064] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0065] Example 1

[0066] In this embodiment, the experimental scenario includes 6 different objects. The following is a detailed description of the robotic arm grasping process using the present invention for adaptive updating of stimulus controls.

[0067] As attached Figure 1 As shown, the robotic arm grasping method based on adaptive updating of stimulus controls disclosed in this embodiment includes the following steps:

[0068] Step S1: Select 6 objects to participate in this robotic arm grasping task, take 300 images to form an initial dataset, mark the object edges in each image, and convert them into corresponding mask images, label visualization images, and category list files to form a self-built dataset. The process is as follows:

[0069] Step S11: The task of this robotic arm grasping is to control the robotic arm to assist in grasping a water cup. Six common objects of different sizes and shapes in daily life are selected, namely a water cup, a pen, a mouse, a notebook, a tissue, and a knife, to construct an experimental scenario. The front view, side view, and top view of each object are repeatedly photographed from multiple angles, and finally an initial dataset of 300 images is formed, with 50 images of each object.

[0070] Step S12: Mark the object edges in each image and generate 300 JSON files;

[0071] Step S13: Convert each JSON file to generate a corresponding mask image, label visualization image, and category list file, and check the labeling accuracy through the label visualization image;

[0072] Step S14: By organizing and placing 300 original images, 300 JSON files, 300 mask images, and 300 category list files respectively, a self-built dataset is formed.

[0073] Step S2: Initialize the mask region model. By training the mask region model on a self-built dataset with different learning rates for different components, a mask region model suitable for this robotic arm grasping task is obtained. The process is as follows:

[0074] Step S21: Load the self-built dataset, modify the category definition based on the objects in the dataset, and use the random hierarchical partitioning method to perform random partitioning independently for each category, so that the ratio of training set to validation set is 8:2.

[0075] Step S22: Initialize the pre-trained mask region model in the deep learning framework, retain the weight parameters of the backbone network, RPN and ROIAlign, and adjust the output dimension of the head network to 6.

[0076] Step S23: Set the learning rate of the head network to 0.005, and the learning rates of the backbone network, RPN and ROIAlign to 0.0005. Then train the mask region model on the self-built dataset, and finally output the mask region model for this robotic arm grasping task.

[0077] Step S3: Capture images of the experimental scene using a depth camera. Use a mask region model tailored to this robotic arm grasping task to perform target recognition and mask segmentation on the experimental scene images, obtaining the classification, bounding boxes, and masks of the six objects in the experimental scene images. Calculate the center position of each object. The process is as follows:

[0078] Step S31: Place a depth camera directly above the experimental scene. After the experiment starts, capture the current frame of the experimental scene image. The width of the experimental scene image is... Experimental scene image height The instance segmentation model designed for this task was used to perform target recognition and mask segmentation on the experimental scene image, resulting in the classification, bounding boxes, and mask information of six objects in the image.

[0079] Step S32: Based on the bounding box information of the six objects, calculate the center point coordinates of the six objects in the Cartesian coordinate system with the upper left corner as the origin, as shown in Table 1.

[0080] Step S33: Based on the coordinates of the center points of the six objects in the Cartesian coordinate system with the upper left corner as the origin and the size information of the experimental scene image, calculate the coordinates of the center points of the six objects in the right-hand coordinate system with the center as the origin, as shown in Table 1.

[0081] Table 1. Coordinate information in different coordinate systems in Example 1

[0082]

[0083] Note: All coordinate values ​​are rounded to integers.

[0084] Step S34: Convert the mask information of the objects into mask objects that can be displayed and called in the experimental window. Create 6 arrays for storing RGBA format image data based on the mask information of 6 objects. Fill the positions in the array where the mask information is greater than 0.5 with the specified color information and fill the positions where the mask information is less than or equal to 0.5 with the transparency information. Finally, convert the array into a mask object in visual.ImageStim format.

[0085] Step S4: Based on the number and center position of the objects, perform adaptive frequency and phase allocation online, and place a stimulus circle of a specified size at the center position of each object. Display the bounding boxes of all objects for 2 seconds to indicate to the user the selectable targets for this round of experiment. Then, according to the allocated stimulus paradigm, perform a 3-second flashing stimulus on the stimulus interface. The process is as follows:

[0086] Step S41, according to the formula Calculations yielded , , Intervals of frequency sequences in the experiment Based on the number of objects Minimum frequency and interval The frequency sequence of this round of experiments ;

[0087] Step S42: Calculate the distance between the center point of the object and the origin in the right-hand coordinate system and sort them from largest to smallest. Assign the frequency and phase in this order, and use the center point coordinates, frequency and phase of each object as the center point coordinates, flashing stimulus frequency and flashing stimulus phase of each object's stimulation circle, as shown in Table 2.

[0088] Table 2. Object stimulus parameter allocation table for Example 1

[0089]

[0090] Note: All coordinate values ​​are integers.

[0091] Step S43: Display the bounding boxes of all objects for 2 seconds to prompt the user that there are selectable targets for this round of experimentation. During this time, the user moves their gaze to the water glass and keeps it still. Then, as shown in the attached figure... Figure 4 As shown, a 3-second flashing stimulus is applied to the stimulus interface according to the assigned stimulus paradigm.

[0092] Step S5: The user generates a corresponding SSVEP signal by gazing at the stimulation circle placed on the target object position in the stimulation interface; the user's SSVEP signal is collected, baseline correction and filtering preprocessing are performed, classification and analysis are conducted to obtain the EEG recognition result, and the result is fed back to the user by displaying the mask object and bounding box of the target object for 2 seconds. At the same time, the robotic arm is controlled to perform corresponding operations. The process is as follows:

[0093] Step S51: The user generates a corresponding SSVEP signal by gazing at the stimulation circle placed on the water cup position in the stimulation interface. The collected SSVEP signal is then subjected to baseline correction and 50Hz notch filtering and 1Hz to 100Hz bandpass filtering.

[0094] Step S52: Classify and analyze the processed SSVEP signal to obtain an EEG recognition result of 12.4Hz, as shown in the attached figure. Figure 5 As shown, the system feeds back the results to the user by displaying the corresponding recognition result—the mask object of the water cup and the bounding box—for 2 seconds, while simultaneously controlling the robotic arm to perform corresponding operations based on depth information.

[0095] Example 2

[0096] In this embodiment, the same experimental steps as in Embodiment 1 are used, but the experimental scenario is changed to 5 different objects. The robotic arm grasping process with adaptive updating of stimulus controls in this embodiment is described in detail below.

[0097] Step S1: Select 5 objects to participate in this robotic arm grasping task, take 250 images to form an initial dataset, mark the object edges in each image, and convert them into corresponding mask images, label visualization images, and category list files to form a self-built dataset. The process is as follows:

[0098] Step S11: The task of this robotic arm grasping is to control the robotic arm to assist in grasping cookies. Five common objects of different sizes and shapes in daily life are selected: mobile phone, cookie, supplement container, sticky note, and stationery jar to construct the experimental scene. The front view, side view and top view of each object are repeatedly photographed from multiple angles, and finally the initial dataset of 250 images is formed with 50 images of each object.

[0099] Step S12: Mark the object edges in each image and generate 250 JSON files;

[0100] Step S13: Convert each JSON file to generate a corresponding mask image, label visualization image, and category list file, and check the labeling accuracy through the label visualization image;

[0101] Step S14: By organizing and placing 250 original images, 250 JSON files, 250 mask images, and 250 category list files respectively, a self-built dataset is formed.

[0102] Step S2: Initialize the mask region model. By training the mask region model on a self-built dataset with different learning rates for different components, a mask region model suitable for this robotic arm grasping task is obtained. The process is as follows:

[0103] Step S21: Load the self-built dataset, modify the category definition based on the objects in the dataset, and use the random hierarchical partitioning method to perform random partitioning independently for each category, so that the ratio of training set to validation set is 8:2.

[0104] Step S22: Initialize the pre-trained mask region model in the deep learning framework, retain the weight parameters of the backbone network, RPN and ROIAlign, and adjust the output dimension of the head network to 5.

[0105] Step S23: Set the learning rate of the head network to 0.005, and the learning rates of the backbone network, RPN and ROIAlign to 0.0005. Then train the mask region model on the self-built dataset, and finally output the mask region model for this robotic arm grasping task.

[0106] Step S3: Capture the experimental scene image using a depth camera. Use the mask region model designed for this robotic arm grasping task to perform target recognition and mask segmentation on the experimental scene image, obtaining the classification, bounding boxes, and masks of the five objects in the experimental scene image, and calculating the center position of each object. The process is as follows:

[0107] Step S31: Place a depth camera directly above the experimental scene. After the experiment starts, capture the current frame of the experimental scene image. The width of the experimental scene image is... Experimental scene image height The instance segmentation model designed for this task was used to perform target recognition and mask segmentation on the experimental scene image, resulting in the classification, bounding boxes, and mask information of five objects within the image.

[0108] Step S32: Based on the bounding box information of the five objects, calculate the center point coordinates of the five objects in the Cartesian coordinate system with the upper left corner as the origin, as shown in Table 3.

[0109] Step S33: Based on the coordinates of the center points of the five objects in the Cartesian coordinate system with the upper left corner as the origin and the size information of the experimental scene image, calculate the coordinates of the center points of the five objects in the right-hand coordinate system with the center as the origin, as shown in Table 3.

[0110] Table 3. Coordinate information in different coordinate systems in Example 2

[0111]

[0112] Note: All coordinate values ​​are integers.

[0113] Step S34: Convert the mask information of the objects into mask objects that can be displayed and called in the experimental window. Create 5 arrays for storing RGBA format image data based on the mask information of 5 objects. Fill the positions in the array where the mask information is greater than 0.5 with the specified color information and fill the positions where the mask information is less than or equal to 0.5 with the transparency information. Finally, convert the array into a mask object in visual.ImageStim format.

[0114] Step S4: Based on the number and center position of the objects, perform adaptive frequency and phase allocation online, and place a stimulus circle of a specified size at the center position of each object. Display the bounding boxes of all objects for 2 seconds to indicate to the user the selectable targets for this round of experiment. Then, according to the allocated stimulus paradigm, perform a 3-second flashing stimulus on the stimulus interface. The process is as follows:

[0115] Step S41, according to the formula Calculations yielded , , Intervals of frequency sequences in the experiment Based on the number of objects Minimum frequency and interval The frequency sequence of this round of experiments ;

[0116] Step S42: Calculate the distance between the center point of the object and the origin in the right-hand coordinate system and sort them from largest to smallest. Assign the frequency and phase in this order, and use the center point coordinates, frequency and phase of each object as the center point coordinates, flashing stimulus frequency and flashing stimulus phase of each object's stimulation circle, as shown in Table 4.

[0117] Table 4. Object stimulus parameter allocation table for Example 2

[0118]

[0119] Note: All coordinate values ​​are integers.

[0120] Step S43: Display the bounding boxes of all objects for 2 seconds to indicate to the user the selectable targets for this round of experiment. During this time, the user should move their gaze to the cookie and keep it still, then proceed as shown in the attached diagram. Figure 6 As shown, a 3-second flashing stimulus is applied to the stimulus interface according to the assigned stimulus paradigm.

[0121] Step S5: The user generates a corresponding SSVEP signal by gazing at the stimulation circle placed on the target object position in the stimulation interface; the user's SSVEP signal is collected, baseline correction and filtering preprocessing are performed, classification and analysis are conducted to obtain the EEG recognition result, and the result is fed back to the user by displaying the mask object and bounding box of the target object for 2 seconds. At the same time, the robotic arm is controlled to perform corresponding operations. The process is as follows:

[0122] Step S51: The user generates the corresponding SSVEP signal by gazing at the stimulation circle placed on the cookie position in the stimulation interface. The collected SSVEP signal is then subjected to baseline correction and 50Hz notch filtering and 1Hz to 100Hz bandpass filtering.

[0123] Step S52: Classify and analyze the processed SSVEP signal to obtain an EEG recognition result of 11.7Hz, as shown in the attached figure. Figure 7 As shown, the corresponding recognition result—the mask object of the cookie and the bounding box—is displayed to the user for 2 seconds, and the robotic arm is controlled to perform corresponding operations in combination with the depth information.

[0124] In summary, the above embodiments utilize a mask region model for real-time target recognition and mask segmentation. The mask region model can be dynamically adjusted according to different needs. Transfer training based on task scenarios and self-built datasets rapidly yields task-oriented mask region models, significantly improving model adaptability and construction efficiency while reducing training time and resource consumption. Users no longer need to learn complex stimulus paradigms; stimulus controls are adaptively generated within the stimulus interface. Users only need to set three threshold times (cues / stimuli / feedback) and gaze at the target object in the stimulus interface, greatly reducing the difficulty of user operation and improving the user experience. Furthermore, the mask region model automatically forms the minimum interference frequency and phase allocation based on the spatial distribution of objects, driving the dynamic update of the stimulus interface. This supports the addition, removal, or movement of objects during experiments, effectively improving the system's stability and reliability in complex environments and enhancing its universality for various experimental tasks and scenarios.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A robotic arm grasping method based on adaptive updating of stimulus controls, characterized in that, The robotic arm grasping method includes the following steps: S1. Select N objects to participate in this robotic arm grasping task, take M images to form an initial dataset, mark the object edges of each image, and convert them into corresponding mask images, label visualization images, and category list files to form a self-built dataset. S2. Initialize the Mask R-CNN model based on convolutional neural network. The Mask R-CNN model based on convolutional neural network is referred to as the mask region model. By training the mask region model on different components using different learning rates on a self-built dataset, a mask region model for this robotic arm grasping task is obtained. S3. Capture images of the experimental scene using a depth camera. Use a mask region model tailored to this robotic arm grasping task to perform target recognition and mask segmentation on these images. Obtain the classification, bounding boxes, and masks of L objects in the experimental scene image, and calculate the center position of each object. The process is as follows: S31. Place a depth camera directly above the experimental scene. After the experiment begins, capture the current frame of the experimental scene image. Use the mask region model designed for this robotic arm grasping task to perform target recognition and mask segmentation on the experimental scene image to obtain the target area within the experimental scene image. The object classification, bounding box, and mask information are as follows: N represents the number of objects; S32. Based on the bounding box coordinates of the i-th object , Calculate the coordinates of the center point of the bounding box to represent the center point coordinates of the i-th object: ， ， in, This represents a Cartesian coordinate system with its origin at the top left corner. It is the x-coordinate of the top-left corner of the bounding box of the i-th object in the Cartesian coordinate system. It is the ordinate of the top-left corner of the bounding box of the i-th object in the Cartesian coordinate system. It is the x-coordinate of the lower right corner of the bounding box of the i-th object in the Cartesian coordinate system. It is the ordinate of the bottom right corner of the bounding box of the i-th object in the Cartesian coordinate system. This represents the x-coordinate of the i-th object's center point in the Cartesian coordinate system. This represents the ordinate of the i-th object's center point in the Cartesian coordinate system; S33. Correctly display the flashing stimulus in the experimental window, and set the center point coordinates. Transform from a Cartesian coordinate system with its origin at the top left corner to a right-handed coordinate system with its origin at the center: ， ， in, This represents a right-handed coordinate system with the center as the origin. This represents the x-coordinate of the i-th object's center point in a right-handed coordinate system. This represents the ordinate of the i-th object's center point in a right-handed coordinate system. It is the width of the experimental scene image. It is the height of the experimental scene image; S34. Convert the mask information of L objects into a mask object of a specified format; S4. Based on the number and center position of the objects, adaptive frequency and phase allocation are performed online. A stimulus circle of a specified size is placed at the center position of each object as a stimulus control. The bounding boxes of all objects are displayed for a first threshold time to indicate to the user the selectable targets for this round of experiment. Then, according to the allocated stimulus paradigm, flashing stimulation for a second threshold time is performed on the stimulus interface. The process is as follows: S41. Based on the number of objects Generate a frequency sequence and set a minimum frequency value. and the highest value The intervals of the frequency sequences in this round of experiments can be calculated online using the following formula. : ， in, It is a rounding function, representing the rounding operation. The result is rounded to one decimal place, depending on the number of objects. Minimum frequency and interval The frequency sequence of this round of experiments : ， in, This represents the frequency of the k-th object; S42. Assign a corresponding frequency and phase to each object according to its position, and calculate the center point of the i-th object in the right-hand coordinate system. With the origin of the right-hand coordinate system distance : ， in, Represents the midpoint of the right-handed coordinate system With the origin of the right-hand coordinate system The geometric distances between them are reordered in descending order. Let represent the index of the i-th object in the new sorting. Then, the distance of the i-th object is re-recorded as . Then, according to this new sorting, the frequency of the j-th object is assigned sequentially. and phase : ， in, This represents the phase of the j-th object in the new sort. The blinking stimulus pattern of each object's stimulus control is uniformly set to a stimulus circle with a radius ranging from [40 pixels to 70 pixels]. The stimulus circle for the j-th object The coordinates of the center point of the circle, As a stimulus circle The frequency of the flashing stimulus, As The phase of the flashing stimulus; S43. Display the bounding boxes of all objects for the first threshold time to prompt the user that there are selectable targets in this round of experiment. Then, according to the assigned stimulus paradigm, perform flashing stimulation on the stimulus interface for the second threshold time. S5. The user gazes at the stimulation circle placed on the target object position in the stimulation interface, generating a corresponding steady-state visual evoked potential signal, hereinafter referred to as SSVEP. The user's SSVEP signal is collected, and baseline correction and filtering preprocessing are performed. The data is then classified and analyzed to obtain the EEG recognition result. The result is fed back to the user by displaying the mask object and bounding box of the target object for a third threshold time. At the same time, the robotic arm is controlled to perform the corresponding operation. Then, S3 to S5 are repeated until the robotic arm grasping task is completed.

2. The robotic arm grasping method based on adaptive updating of stimulus controls according to claim 1, characterized in that, The process of step S1 is as follows: S11. Determine the requirements for this robotic arm grasping task. For each object, the front view, side view, and top view of each object are repeatedly photographed from multiple angles, and finally an initial dataset containing M images is formed. S12. Mark the object edges in each image and generate M JSON files; S13. Convert each JSON file to generate a corresponding mask image, label visualization image, and category list file, and check the labeling accuracy through the label visualization image; S14. By organizing and placing M original images, M JSON files, M mask images, and M category list files respectively, a self-built dataset is formed.

3. The robotic arm grasping method based on adaptive updating of stimulus controls according to claim 1, characterized in that, The mask region model comprises four components: a backbone network, a region generation network, a region of interest alignment (ROIAlign), and a head network. The region generation network is referred to as RPN, and the region of interest alignment is referred to as ROIAlign. The backbone network extracts features from the input image to obtain a multi-scale feature map. The RPN generates candidate target regions on the multi-scale feature map. ROIAlign maps the candidate target regions to the multi-scale feature map and extracts alignment features from the multi-scale feature map. The head network performs target recognition and mask segmentation based on the alignment features, ultimately obtaining the classification, bounding boxes, and masks of objects in the input image.

4. The robotic arm grasping method based on adaptive updating of stimulus controls according to claim 3, characterized in that, The process of step S2 is as follows: S21. Load the self-built dataset, modify the category definition based on the objects in the dataset, and use the random hierarchical partitioning method to perform random partitioning independently for each category, so that the ratio of training set to validation set is 8:

2. S22. Initialize the pre-trained mask region model, retain the weight parameters of the backbone network, RPN and ROIAlign, and adjust the output dimension of the head network to N; S23. Set the head network learning rate to... The learning rates for the backbone network, RPN, and ROIAlign are set to... Then, a mask region model is trained on a self-built dataset, and finally, a mask region model for this robotic arm grasping task is output.

5. The robotic arm grasping method based on adaptive updating of stimulus controls according to claim 1, characterized in that, The process of step S5 is as follows: S51. The user generates a corresponding SSVEP signal by gazing at the stimulus circle placed at the target object position on the stimulus interface. The two-dimensional data of the acquired SSVEP signal are processed. Baseline correction is performed by averaging through a sliding window to eliminate baseline drift in the two-dimensional data. Two-dimensional data of input SSVEP signal ,in This refers to the number of SSVEP signal acquisition channels. It is the number of sampling points of the SSVEP signal within the second threshold time, for two-dimensional data. Perform baseline correction: in, To adjust the sliding window size, This indicates that two-dimensional data Divided into The size is The time window is defined and the average value of each time window is calculated. Subtracting the average value of each sampling point within its time window from the average value of that sampling point yields two-dimensional data that has eliminated baseline drift. Next, the two-dimensional data Perform 50Hz notch filtering and 1Hz to 100Hz bandpass filtering; S52. Processed two-dimensional data The EEG recognition results are obtained through classification and analysis. Based on the EEG recognition results, the mask object and bounding box of the corresponding target object are displayed on the experimental window for a third threshold time to provide feedback to the user. At the same time, the robotic arm is controlled to perform corresponding operations in combination with depth information. Thus, one round of experiment ends. Then, steps S3 to S5 are repeated until the robotic arm grasping task is completed.

Images