Robot control method, robot, storage medium, and gripping system

Abstract

The application discloses a robot control method, a robot, a storage medium and a grabbing system. The method comprises the following steps: acquiring an image of a target area; identifying the image by using a YOLO algorithm to determine a target position of a target object in the image; generating a target instruction based on the target position, and calculating a running track in a pulse neural network based on the target instruction; and controlling the robot to grab the target object according to the running track. In the above manner, the robot based on the pulse neural network has the image recognition function, and the robot is more intelligent.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to robot control methods, robots, storage media, and grasping systems. Background Technology

[0002] There are many severely paralyzed patients in the world who can only perform some essential daily activities, such as drinking water, with the help of others. With the continuous development of artificial intelligence and robotics, more and more research results are being applied to assist these people in order to improve their quality of life. Among them, the field of brain-computer interface (BCI), as a branch of neural engineering, is developing rapidly and has broad prospects, which has aroused a surge of research enthusiasm in the field of brain-computer interface.

[0003] Current robot control technology can only perform simple or even preset robotic arm movements, failing to fully leverage its advantages. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a robot control method, a robot, a storage medium, and a grasping system that enable robots based on spiking neural networks to have image recognition capabilities, thereby making the robots more intelligent.

[0005] To address the aforementioned issues, this application provides a robot control method comprising: acquiring an image of a target area; using the YOLO algorithm to identify the image and determine the target location of a target object in the image; generating a target instruction based on the target location and calculating a running trajectory in a spiking neural network based on the target instruction; and controlling the robot to grasp the target object according to the running trajectory.

[0006] The process of acquiring images of the target region includes: acquiring a first image and a second image of the target region; and using the YOLO algorithm to identify the images in order to determine the target location of the target object in the images, including: using the YOLO algorithm to identify the first image in order to determine the first target location of the target object in the first image, and identifying the second image in order to determine the second target location of the target object in the second image.

[0007] Among them, generating target instructions based on target location includes: generating a third target location of the target item in the world coordinate system based on the first target location and the second target location; and generating target instructions based on the third target location.

[0008] The process of generating a third target position for a target item in the world coordinate system based on the first and second target positions includes: determining an initial third target position; obtaining multiple corresponding fourth target positions using the initial third target position; and determining the initial third target position as the final third target position when any fourth target position satisfies a preset condition with the first target position and when any fourth target position satisfies a preset condition with the second target position.

[0009] The process of obtaining multiple fourth target positions using the initial third target position includes: acquiring the parameter matrix of the image acquisition device corresponding to the first or second image; and obtaining multiple fourth target positions using the initial third target position and the parameter matrix.

[0010] The process of using the YOLO algorithm to identify images and determine the target location of target objects in the images includes: using a trained image recognition model to identify images and determine the target location of target objects in the images, wherein the trained image recognition model is obtained by training sample images of target objects using the YOLO algorithm.

[0011] The process of controlling a robot to grasp a target object according to its operating trajectory includes: generating a first instruction based on the operating trajectory, the first instruction being used to control the robot to move to a first position on the operating trajectory; acquiring feedback data of the robot moving to the first position; calculating trajectory correction data in a spiking neural network based on the first instruction and the feedback data; generating a second instruction based on the operating trajectory, the second instruction and the trajectory correction data being used to control the robot to move from the first position to a second position on the operating trajectory; and then controlling the robot's robotic arm to grasp the target object.

[0012] To address the aforementioned issues, another technical solution adopted in this application is to provide a robot comprising a processor and a memory coupled to the processor; wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method provided by the above technical solution.

[0013] To address the aforementioned problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the method provided by the above technical solution.

[0014] To address the aforementioned issues, another technical solution adopted in this application is to provide a grasping system, which includes: an image acquisition device for acquiring an image of a target area; a controller connected to the image acquisition device for using the YOLO algorithm to identify the image and determine the target position of a target object in the image; generating a target instruction based on the target position and calculating a running trajectory in a spiking neural network based on the target instruction; and a robot connected to the controller for grasping the target object according to the running trajectory.

[0015] The beneficial effects of this application are as follows: Unlike existing technologies, this application provides a robot control method that includes: acquiring an image of a target area; using the YOLO algorithm to recognize the image and determine the target position of a target object in the image; generating a target command based on the target position and calculating a running trajectory in a spiking neural network based on the target command; and controlling the robot to grasp the target object according to the running trajectory. Through this method, a spiking neural network-based robot can possess image recognition capabilities, making the robot more intelligent. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the first embodiment of the robot control method provided in this application;

[0017] Figure 2 This is a flowchart illustrating the second embodiment of the robot control method provided in this application;

[0018] Figure 3 This is a flowchart illustrating an embodiment of step 24 provided in this application;

[0019] Figure 4 This is a flowchart illustrating an embodiment of step 242 provided in this application;

[0020] Figure 5 This is a flowchart illustrating an embodiment of step 26 provided in this application;

[0021] Figure 6 This is a flowchart illustrating an embodiment of step 263 provided in this application;

[0022] Figure 7 This is a flowchart illustrating an embodiment of step 2631 provided in this application;

[0023] Figure 8 This is a flowchart illustrating an embodiment of step 26312 provided in this application;

[0024] Figure 9 This is a structural schematic diagram of an embodiment of the robot provided in this application;

[0025] Figure 10This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium provided in this application;

[0026] Figure 11 This is a schematic diagram of an embodiment of the crawling system provided in this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] See Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the robot control method provided in this application. The method includes:

[0031] Step 11: Obtain an image of the target area.

[0032] In some embodiments, acquiring an image of the target area can be done using an image acquisition component. For example, the image acquisition component is located at the head of the robot and can acquire an image of the entire target area. Alternatively, the image acquisition component is located at the end of the robot's robotic arm, and by controlling the robotic arm, the image acquisition component at the end of the robotic arm can acquire an image of the entire target area. The image acquisition component can be a camera capable of acquiring depth images.

[0033] In other embodiments, the image acquisition component and the robot are set separately, and the image acquisition component only needs to be set above the target area.

[0034] In other embodiments, the image acquisition component may be a binocular camera.

[0035] In some embodiments, the target area may be a desktop, a cleaning surface, or a sorting surface, etc.

[0036] Step 12: Use the YOLO algorithm to identify the image in order to determine the target location of the target object in the image.

[0037] This can be achieved by building an image recognition model based on the YOLO algorithm and training the model. The trained model is then used to identify objects within the image to determine their locations.

[0038] The trained image recognition model is obtained by training sample images of the target item using the YOLO algorithm.

[0039] For example, if the target item is grapes, a dataset of multiple grape images is created, and the images are labeled using the image annotation tool LabelImg. YOLOv5 is then trained on a server with two 2080ti graphics cards to identify the target item in the collected images in real time, and the image coordinates of the acquired target object are transformed to obtain the world coordinates of the target item.

[0040] The YOLO algorithm can be different versions of the YOLO algorithm, such as YOLOv2, YOLOv3, YOLOv4, or YOLOv5.

[0041] Step 13: Generate target instructions based on the target location, and calculate the running trajectory in the spiking neural network based on the target instructions.

[0042] The robot is controlled by a spiking neural network. After determining the target position, the spiking neural network calculates the trajectory from the robot's current position to the target position based on the joint structure between the robots.

[0043] In step 13, the most reasonable motion trajectory can be established based on the target position and the starting position. If there is an obstacle between the target position and the starting position, the preset motion trajectory can be made to bypass the obstacle.

[0044] Step 14: Control the robot to grab the target item according to the running trajectory.

[0045] In one application scenario, a brain-computer interface (BCI) is used to acquire the electroencephalogram (EEG) signals of a test animal and analyze these signals to obtain corresponding parsed signals. For example, if the test animal wants to grasp an item on a table, such as grapes, the BCI will acquire and analyze the animal's EEG signals. At this point, a robot acquires an image of the table and uses the YOLO algorithm to identify the grapes' target location. Based on the target location, a target command is generated, and a trajectory is calculated in a spiking neural network. The robot then follows this trajectory to grasp the target item. After grasping the grapes, the robot moves them to the target object.

[0046] In this embodiment, by acquiring an image of the target area; using the YOLO algorithm to identify the image to determine the target location of the target object in the image; generating a target instruction based on the target location; and calculating the running trajectory in the spiking neural network based on the target instruction; and controlling the robot to grasp the target object according to the running trajectory, the robot based on the spiking neural network can have the function of image recognition, making the robot more intelligent.

[0047] In one application scenario, a brain-computer interface (BCI) is used to acquire the electroencephalogram (EEG) signals of special users, such as paralyzed or intellectually disabled individuals, and then analyze these signals to obtain corresponding analytical signals. For example, if a special user wants to grasp an object on a table, such as an apple, the BCI will acquire and analyze the user's EEG signal. At this point, an image acquisition device captures an image of the table, and the YOLO algorithm is used to identify the target location of the grape in the image. Based on the target location, a target command is generated, and a trajectory is calculated in a spiking neural network based on the target command. The robot is then controlled to grasp the target object according to the trajectory. After grasping the grape, it is moved to the special user's location. This allows for better utilization of the robot's advantages, enabling the use of BCI and image recognition to improve the quality of life for special users and enhance their ability to live independently.

[0048] See Figure 2 , Figure 2 This is a flowchart illustrating a second embodiment of the robot control method provided in this application. The method includes:

[0049] Step 21: Obtain the first and second images of the target area.

[0050] In this embodiment, a binocular camera is used, so each camera can acquire one image. That is, a first image and a second image of the target area are acquired.

[0051] Step 22: Use the YOLO algorithm to identify the first image to determine the first target location of the target object in the first image.

[0052] Specifically, an image recognition model can be built based on the YOLO algorithm and trained. Then, the trained image recognition model is used to recognize the image. After recognizing the target object, the first target position of the target object in the first image can be determined.

[0053] Step 23: Use the YOLO algorithm to identify the second image to determine the second target location of the target object in the second image.

[0054] Specifically, an image recognition model can be built based on the YOLO algorithm and trained. Then, the trained image recognition model is used to recognize the image. After recognizing the target object, the second target position of the target object in the second image can be determined.

[0055] Step 24: Generate the third target position of the target item in the world coordinate system based on the first target position and the second target position.

[0056] In this process, the binocular camera can be calibrated in advance to obtain the corresponding camera parameters such as focal length and intrinsic parameter matrix. Then, coordinate transformation can be performed using the camera parameters such as focal length and intrinsic parameter matrix to obtain the third target position of the target object in the world coordinate system.

[0057] In some embodiments, see Figure 3 Step 24 can be the following process:

[0058] Step 241: Determine the initial location of the third target.

[0059] The third target location is in three-dimensional coordinates in the world coordinate system.

[0060] Step 242: Use the initial third target position to obtain the corresponding multiple fourth target positions.

[0061] The fourth target location is the image pixel coordinates.

[0062] In some embodiments, the initial third target position can be transformed to obtain the corresponding coordinates in the camera, the coordinates in the camera can be transformed to obtain the physical coordinates of the image, and then the physical coordinates of the image can be transformed to obtain the image pixel coordinates.

[0063] In some embodiments, see Figure 4 Step 242 can be the following process:

[0064] Step 2421: Obtain the parameter matrix of the image acquisition device corresponding to the first image or the second image.

[0065] The parameter matrix includes an intrinsic parameter matrix and an extrinsic parameter matrix.

[0066] Step 2422: Use the initial third target position and parameter matrix to obtain multiple fourth target positions.

[0067] Step 243: When any fourth target position satisfies the preset condition with the first target position, and when any fourth target position satisfies the preset condition with the second target position, the initial third target position is determined as the final third target position.

[0068] If none of the preset conditions are met between the fourth target position and the first target position, and none of the preset conditions are met between the fourth target position and the second target position, the value of the initial third target position is modified, and then steps 242-243 are executed again. This process continues until any fourth target position meets the preset conditions with either the first or second target position, at which point the initial third target position is determined as the final third target position.

[0069] In some embodiments, the fourth target location can be determined using the following formula:

[0070]

[0071] Where s represents the scaling factor, and s is not zero; dX represents the physical size of the pixel in the X-axis direction, dY represents the physical size of the pixel in the Y-axis direction; (u0, v0) represents the principal pixel coordinates, f represents the effective focal length, i.e., the distance from the optical center to the image plane, R represents a 3*3 rotation matrix, and t represents a 3*1 translation vector. Represented as homogeneous coordinates in the world coordinate system; a x =f / dX, a y =f / dY represent the scale factors of the u and v axes in the pixel coordinates, respectively. M1 is the camera's intrinsic parameter matrix, M2 is the camera's extrinsic parameter matrix, and M is the projection matrix.

[0072] In this process, by iteratively changing the value of the initial third target position, the fourth target position corresponding to the first and second target positions is obtained.

[0073] In this process, Newton's method can be used to determine the location of the fourth target. The formula for Newton's method is as follows:

[0074]

[0075] Where Ax represents the aforementioned MXw Let 'b' represent the first or second target position. When f(x) converges to the preset requirement, the corresponding third target position in the world coordinate system can be determined. Newton's method can control the error between the final third target position and the first or second target position to within one centimeter.

[0076] Step 25: Generate target instructions based on the third target position, and calculate the running trajectory in the spiking neural network based on the target instructions.

[0077] Step 26: Control the robot to grab the target item according to the running trajectory.

[0078] In some embodiments, see Figure 5 Step 26 can be the following process:

[0079] Step 261: Generate a first instruction based on the running trajectory. The first instruction is used to control the robot to move to the first position on the running trajectory.

[0080] In some embodiments, the trajectory consists of a series of coordinate points.

[0081] Since a robot has many joints, the first instruction can be a control signal for each joint. By controlling each joint, the robot can move to the first position on its trajectory.

[0082] In this embodiment, the robot moves according to a given target position.

[0083] If the robot is a robotic arm, it can be moved to a designated target location.

[0084] Step 262: Obtain feedback data on the robot's movement to the first position.

[0085] In some embodiments, when the robot moves to a first position based on a first command, position information of each joint is collected by the robot's sensors. The sensors can be encoders at the robot joints or motor ends to acquire joint position information. Further processing of the position information allows the robot to obtain its current speed, direction, etc. Based on this data collected by the sensors, the robot's actual trajectory data can be obtained. These sensors can also be used to detect the robot's current speed.

[0086] It is understandable that different robots acquire different data from their sensors. Therefore, it is important to obtain appropriate data based on the characteristics of the robot.

[0087] In some embodiments, when the robot moves to the first position, due to errors in the robot's own structure, the actual position of the robot when it moves to the first position may not be the first position. In this case, the feedback data may be the robot's current actual position, as well as its actual speed, actual direction, actual joint torque, etc.

[0088] Step 263: Calculate trajectory correction data in the spiking neural network based on the first instruction and feedback data.

[0089] In some embodiments, the spiking neural network can be constructed based on the Hodgkin-Huxley model, or it can be constructed based on the Leaky Integrate and Fire model or the Izhikevich model.

[0090] Spiking neural networks can be trained using unsupervised learning algorithms and / or supervised learning algorithms.

[0091] Step 264: Generate a second instruction based on the running trajectory. The second instruction and trajectory correction data are used to control the robot to move from the first position to the second position on the running trajectory, thereby controlling the robot's robotic arm to grasp the target item.

[0092] After obtaining trajectory correction, the robot combines the trajectory correction data with the second instruction to generate the optimal second instruction to control the robot to move from the first position to the second position on the running trajectory.

[0093] It is understandable that by combining trajectory correction data and second commands, the robot can move from the first position to the second position on the running trajectory, making the actual position closer to the second position.

[0094] In this way, other positions on the trajectory can be processed in the same manner to obtain trajectory correction data based on the previous position, so as to compensate for the actual position of the current position and make the actual position tend to the current position in the trajectory.

[0095] In some embodiments, see Figure 6 Step 263 above can be the following process:

[0096] Step 2631: Update the weights of the spiking neural network using the first instruction and feedback data.

[0097] In some embodiments, see Figure 7 Step 2631 can be the following process:

[0098] Step 26311: Encode the first instruction and feedback data to obtain the activity of neurons in the spiking neural network.

[0099] Specifically, the activity of neurons can be represented by the following formula:

[0100] a = G[αe·x];

[0101] Where G[·] is a nonlinear neural activation function, α is a scaling factor (gain) associated with the neuron, e is the neuron's encoder, and x is the vector to be encoded, i.e., the first instruction and feedback data.

[0102] Step 26312: Calculate the decoder using the activity of neurons.

[0103] In some embodiments, see Figure 8 Step 26312 can be performed using the following process to calculate the decoder:

[0104] Step 263121: Calculate the first parameter using the first instruction, feedback data, and neuron activity.

[0105] Specifically, step 263121 can use the following formula to calculate the first parameter:

[0106] r=∫a j xdx.

[0107] Among them, a j is the activity of neuron j, x is the first input instruction and feedback data, and r is the first parameter.

[0108] Step 263122: Calculate the second parameter using the activity of multiple neurons.

[0109] Specifically, step 263122 can use the following formula to calculate the second parameter:

[0110] T ij =∫a i a j dx.

[0111] Among them, a j It is the activity of neuron j, a i It is the activity of neuron i, T ij This is the second parameter between neuron j and neuron i.

[0112] Step 263123: Calculate the decoder using the first and second parameters.

[0113] Specifically, step 263123 can use the following formula to calculate the decoder:

[0114] d = r -1 T.

[0115] Step 26313: Calculate the decoding estimate using the activity of the decoder and neurons.

[0116] Specifically, the decoder estimate is obtained by performing a dot product of the activities of the decoder and neurons. This can be expressed using the following formula:

[0117]

[0118] Step 26314: Obtain the first difference using the decoded estimation and feedback data.

[0119] It is understandable that the result of decoding and estimation is the optimal motion data of the robot predicted by the spiking neural network. This data can then be compared with the actual motion data in the feedback data to obtain the first difference between the optimal motion data and the actual motion data in the feedback data.

[0120] Step 26315: Obtain the weight correction value of the spiking neural network using the first difference and the activity of the neurons.

[0121] In some embodiments, online supervised learning rules can be used to determine the weight adjustment values.

[0122] Specifically, it can be expressed using the following formula:

[0123] Δd i =κEa i ;

[0124] Δω ij =κα j e j ·Ea i ;

[0125] Where, Δω ij The weight adjustment value represents the connection weight between neuron j and neuron i, κ is the scalar learning rate, and E represents the first difference, i.e., the decoding estimate. The difference between x and x.

[0126] It is understandable that different neurons have different decoders, so the decoder correction value Δd corresponding to that neuron can be obtained based on the first difference. i .

[0127] In some embodiments, unsupervised learning rules can be used to determine the weight adjustment values.

[0128] Specifically, it can be expressed using the following formula:

[0129] Δω ij =a i a j (a j -θ);

[0130] Where, Δω ijθ represents the weight adjustment value for the connection weight between neuron j and neuron i, and θ represents the modification threshold used to limit the modification range of neuron j.

[0131] In some embodiments, a combination of unsupervised learning rules and online supervised learning rules can be used to determine the weight adjustment value.

[0132] Specifically, the weight adjustment value is calculated using the following formula:

[0133] Δω ij =κα j a i (Se j ·E+(1-S)a j (a j -θ)).

[0134] Where κ represents the scalar learning rate, α j a represents the scaling factor of neuron j. i Let S represent the activity of neuron i, S represent the control parameter used to represent the relative weighting of supervised learning terms with respect to unsupervised learning terms, E represent the first difference, and θ represent the modification threshold.

[0135] Step 26316: Update the weights of the spiking neural network using the weight correction values.

[0136] In a spiking neural network, weights are set between neurons, and correction values ​​can be used to update these weights. A negative correction value indicates that the original weights need to be reduced, while a positive correction value indicates that the original weights need to be increased.

[0137] Step 2632: Calculate the trajectory correction data using the updated spiking neural network.

[0138] Trajectory correction data is obtained using the updated weights, decoder, and neuron activity.

[0139] Specifically, trajectory correction data can be calculated by multiplying the activity of neurons by their weights and then multiplying them by the decoder.

[0140] Specifically, using the following formula:

[0141]

[0142] Where a represents the activity of the neuron, ω represents the updated weights, d represents the decoder, and Γ adapt This represents trajectory correction data.

[0143] Step 264 above can calculate the torque required for each joint of the robot based on the second instruction and trajectory correction data.

[0144] Specifically, the torque for robot movement control in the second instruction can be calculated using the following formula.

[0145]

[0146] Where q represents the coordinates of each joint of the robot. Let M(q) represent the angular velocity of each joint of the robot, and M(q) represent the inertial force on each joint caused by the acceleration of the motion of each joint. Γ represents the inertial force exerted on other joints by the velocity of each joint of the robot, also known as the Coriolis force or centrifugal force. G(q) represents the weight of the robot arm itself that each joint needs to overcome. adapt Γ represents the trajectory correction data, and Γ represents the torque that each joint actuator needs to apply to make the joints move along a predetermined trajectory (position, velocity, acceleration) according to the robot dynamics model.

[0147] By using the above method, the robot's trajectory is corrected in real time using a spiking neural network, enabling the robot to move stably and accurately, improving the stability and robustness of robot control. On the other hand, using a spiking neural network can improve the computational efficiency of trajectory correction data, thereby improving the robot's motion efficiency.

[0148] See Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the robot provided in this application. The robot 90 includes a processor 91 and a memory 92 coupled to the processor 91.

[0149] The memory 92 is used to store computer programs, and the processor 91 is used to execute the computer programs to implement the following methods:

[0150] The robot acquires an image of the target area; uses the YOLO algorithm to identify the target object in the image to determine its location; generates a target command based on the target location, and calculates the running trajectory in a spiking neural network based on the target command; and controls the robot to grasp the target object according to the running trajectory.

[0151] It is understood that the processor 91 in this embodiment is also used to execute computer programs to implement the methods in any of the above embodiments. The specific implementation steps can be referred to the above embodiments, and will not be repeated here.

[0152] In some embodiments, robot 90 is a robotic arm.

[0153] See Figure 10 , Figure 10This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 100 is used to store a computer program 101, which, when executed by a processor, implements the following methods:

[0154] The robot acquires an image of the target area; uses the YOLO algorithm to identify the target object in the image to determine its location; generates a target command based on the target location, and calculates the running trajectory in a spiking neural network based on the target command; and controls the robot to grasp the target object according to the running trajectory.

[0155] It is understood that when the computer program 101 in this embodiment is executed by the processor, it is also used to implement the method in any of the above embodiments. The specific implementation steps can be referred to the above embodiments, and will not be repeated here.

[0156] See Figure 11 , Figure 11 This is a schematic diagram of an embodiment of the grasping system provided in this application. The grasping system 110 includes: an image acquisition device 111, a controller 112, and a robot 90.

[0157] The image acquisition device 111 is used to acquire images of the target area.

[0158] The controller 112 is connected to the image acquisition device 111 and is used to identify the image using the YOLO algorithm to determine the target position of the target object in the image; and to generate a target command based on the target position, and calculate the running trajectory in the spiking neural network based on the target command.

[0159] Robot 90 is connected to controller 112 and is used to grasp target items according to the running trajectory.

[0160] The image acquisition device 111 can be a binocular camera, and the controller 112 is also used to acquire a first image and a second image of the target area; to use the YOLO algorithm to identify the first image to determine the first target position of the target item in the first image, and to identify the second image to determine the second target position of the target item in the second image.

[0161] The controller 112 is also used to generate a third target position of the target item in the world coordinate system based on the first target position and the second target position; and to generate target instructions based on the third target position.

[0162] The controller 112 is also used to determine the initial third target position; obtain a plurality of corresponding fourth target positions using the initial third target position; and determine the initial third target position as the final third target position when any fourth target position satisfies a preset condition with the first target position and when any fourth target position satisfies a preset condition with the second target position.

[0163] The controller 112 is also used to acquire the parameter matrix of the image acquisition device corresponding to the first image or the second image; and to obtain multiple fourth target positions using the initial third target position and the parameter matrix.

[0164] The controller 112 is also used to identify the image using a trained image recognition model to determine the target location of the target object in the image, wherein the trained image recognition model is obtained by training sample images of the target object based on the YOLO algorithm.

[0165] The controller 112 is also used to generate a first instruction based on the running trajectory, the first instruction being used to control the robot to move to a first position on the running trajectory; to acquire feedback data of the robot moving to the first position; to calculate trajectory correction data in a spiking neural network based on the first instruction and the feedback data; to generate a second instruction based on the running trajectory, the second instruction and the trajectory correction data being used to control the robot to move from the first position to a second position on the running trajectory; and to control the robot's robotic arm to grasp the target object.

[0166] In some embodiments, the controller 112 is integrated with the robot 90.

[0167] The grasping system 110 in this embodiment is based on a target recognition module using the YOLO algorithm, which can achieve real-time recognition and transmit the image coordinates of the target objects identified in the left and right cameras to the coordinate transformation module. The coordinate transformation module, knowing the image coordinates of the target objects in the left and right cameras, randomly initializes the world coordinates and uses Newton's method based on the coordinate transformation formula to continuously approximate the real world coordinates, and transmits the world coordinates to the robotic arm control module. After obtaining the world coordinates, the robotic arm control module based on the spiking neural network can move to the position of the target object for grasping, and has a stronger and faster adaptive capability to environmental interference.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated units in the other embodiments described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for controlling a robot, characterized in that, The method includes: The acquired user's electroencephalogram (EEG) signals are analyzed to obtain corresponding analytical signals; wherein, the analytical signals are used to characterize the grasping of the target object; The first and second images of the target region are obtained based on the parsed signal; The first image is identified using the YOLO algorithm or an image recognition model trained based on the YOLO algorithm to determine the first target location of the target object in the first image, and the second image is identified to determine the second target location of the target object in the second image. Determine the initial location of the third target; Obtain the parameter matrix of the image acquisition device corresponding to the first image or the second image; Multiple fourth target positions are obtained using the initial third target position and the parameter matrix; wherein, Newton's method is used to determine the fourth target positions. When any of the fourth target positions satisfies a preset condition with the first target position, and when any of the fourth target positions satisfies a preset condition with the second target position, the initial third target position is determined as the final third target position; A target instruction is generated based on the third target location, and the running trajectory is calculated in a spiking neural network based on the target instruction; The robot is controlled according to the stated trajectory to grab the target item and move it to the user.

2. The method according to claim 1, characterized in that, The step of controlling the robot to grasp the target item according to the running trajectory includes: A first instruction is generated based on the running trajectory, and the first instruction is used to control the robot to move to a first position on the running trajectory; Obtain feedback data showing that the robot has moved to the first position; Based on the first instruction and the feedback data, trajectory correction data is calculated in the spiking neural network; A second instruction is generated based on the running trajectory. The second instruction and the trajectory correction data are used to control the robot to move from the first position to a second position on the running trajectory; thereby controlling the robot's robotic arm to grasp the target item.

3. A robot, characterized in that, The robot includes a processor and a memory coupled to the processor; The memory is used to store a computer program, and the processor is used to execute the computer program to implement the method as described in any one of claims 1-2.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a processor, is used to implement the method as described in any one of claims 1-2.

5. A grasping system, characterized in that, The crawling system includes: An image acquisition device is used to acquire a first image and a second image of a target area based on an analytical signal; wherein the analytical signal is obtained by analyzing the acquired electroencephalogram (EEG) signal of the user; the analytical signal is used to characterize the grasping of a target object; A controller, connected to the image acquisition device, is used to identify the first image using the YOLO algorithm or an image recognition model trained based on the YOLO algorithm to determine the first target position of the target object in the first image, and to identify the second image to determine the second target position of the target object in the second image; determine an initial third target position; acquire a parameter matrix of the image acquisition device corresponding to the first image or the second image; obtain multiple fourth target positions using the initial third target position and the parameter matrix; wherein, Newton's method is used to determine the fourth target positions; when any of the fourth target positions satisfies a preset condition with the first target position, and when any of the fourth target positions satisfies a preset condition with the second target position, the initial third target position is determined as the final third target position; a target instruction is generated based on the third target position, and a running trajectory is calculated in a spiking neural network based on the target instruction; The robot, connected to the controller, is used to grab the target item according to the running trajectory and move it to the user.

Images