Manipulator control method based on pulse neural network, manipulator and storage medium

Abstract

The application relates to the technical field of mechanical control, and discloses a mechanical arm control method based on a pulse neural network, a mechanical arm and a storage medium. The mechanical arm control method comprises the following steps: obtaining a grabbing instruction for a target object; controlling at least two fingers of the mechanical arm to grab the target object based on the grabbing instruction; determining a target finger in the at least two fingers in a grabbing process; wherein the target finger is a finger that contacts the target object; determining joint angle correction data of the target finger in a pulse neural network based on the grabbing instruction and feedback data of the target finger; and continuing to control the at least two fingers to grab the target object according to the joint angle correction data and the grabbing instruction. Through the above method, the acting force of the mechanical arm when grabbing the target object is improved, the adaptability of the mechanical arm when grabbing the object is improved, and the abrasion of the mechanical arm is reduced.

Description

Technical Field

[0001] The present application relates to the field of mechanical control technology, and in particular to a manipulator control method, a manipulator, and a storage medium based on a pulse neural network. Background Art

[0002] With the advancement of technology, robots are being used in a variety of scenarios. For example, in industrial applications such as dispensing, welding, and monitoring the movement of conveyor belts, robots must quickly converge to a given trajectory. In the field of service robotics, mobile robots must follow humans at the same speed in real time or move at a given speed along a given trajectory.

[0003] For example, when the robot is a manipulator, most applications of the manipulator use suction cups, two fingers or customized grippers. These manipulators lack adaptability when grasping objects and are prone to causing damage to the objects and the manipulator itself. Summary of the Invention

[0004] The present application provides a manipulator control method, a manipulator, and a storage medium based on a pulse neural network, which can improve the force of the manipulator in grasping a target object, improve the adaptability of the manipulator in grasping objects, and reduce the wear of the manipulator.

[0005] A technical solution adopted in the present application is to provide a manipulator control method based on a pulse neural network, which includes: obtaining a grasping instruction for a target object; based on the grasping instruction, controlling at least two fingers of the manipulator to grasp the target object; during the grasping process, determining a target finger among at least two fingers; wherein the target finger is the finger that touches the target object; based on the grasping instruction and feedback data of the target finger, determining the joint angle correction data of the target finger in the pulse neural network; and continuing to control at least two fingers to grasp the target object according to the joint angle correction data and the grasping instruction.

[0006] Among them, the joint angle correction data of the target finger is determined in the pulse neural network based on the grasping instruction and the feedback data of the target finger, including: using the grasping instruction and the feedback data to update the weight of the pulse neural network; and using the updated pulse neural network to determine the joint angle correction data of the target finger.

[0007] Among them, the weights of the pulse neural network are updated using the capture instructions and feedback data, including: encoding the capture instructions and feedback data to obtain the activities of neurons in the pulse neural network; using the activities of neurons to determine a decoder; using the decoder and the activities of neurons to calculate a decoding estimate; using the decoding estimate and feedback data to obtain a first difference; using the first difference and the activities of neurons to obtain a weight correction value of the pulse neural network; and using the weight correction value to update the weight of the pulse neural network.

[0008] The weight correction value of the pulse neural network is obtained by using the first difference and the activity of the neuron, including: calculating the weight correction value using the following formula: Δω ij =κα j a i (Se j ·E+(1-S)a j (a j -θ)); where κ represents the scalar learning rate, α j represents the scaling factor of neuron j, a i represents the activity of neuron i, S represents the control parameter, E represents the first difference, and θ represents the modification threshold.

[0009] The joint angle correction data of the target finger is determined using the updated pulse neural network, including: obtaining the joint angle correction data of the target finger using the updated weights, decoders and neuron activities.

[0010] The joint angle correction data of the target finger is obtained by using the updated weights, decoders, and neuron activities, including: calculating the joint angle correction data using the following formula: Where a represents the activity of the neuron, ω represents the updated weight, d represents the decoder, and θ adapt Indicates joint angle correction data.

[0011] Among them, during the grasping process, determining the target finger among at least two fingers includes: during the grasping process, obtaining the first joint position and first force corresponding to each finger at the current moment, and the second joint position and second force at the previous moment; determining the target finger among at least two fingers based on the first joint position and the second joint position, and the first force and the second force.

[0012] The step of obtaining a grabbing instruction for a target object includes: identifying the type of the target object; and determining a grabbing instruction based on the type.

[0013] Another technical solution adopted in the present application is to provide a manipulator based on a pulse neural network, which includes: at least two fingers; a processor connected to at least two fingers; and a memory connected to the processor; wherein the memory is used to store program data and the processor is used to execute program data to implement the method provided by the above technical solution.

[0014] Another technical solution adopted in the present application is to provide a computer-readable storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the method provided by the above technical solution.

[0015] The beneficial effects of the present application are: different from the existing technology, the present application's manipulator control method, manipulator and storage medium based on pulse neural network, the manipulator control method includes: obtaining a grasping instruction for the target object; based on the grasping instruction, controlling at least two fingers of the manipulator to grasp the target object; during the grasping process, determining the target finger among the at least two fingers; wherein the target finger is the finger that touches the target object; determining the joint angle correction data of the target finger in the pulse neural network based on the grasping instruction and the feedback data of the target finger; and continuing to control at least two fingers to grasp the target object according to the joint angle correction data and the grasping instruction. Through the above method, the target finger that contacts the target object is determined during the grasping process, and then the joint angle correction data of the target finger is determined using the pulse neural network, so that the joint correction data is used to correct the grasping instruction, improve the force of the manipulator to grasp the target object, improve the adaptability of the manipulator to grasp objects, and reduce the wear of the manipulator. Furthermore, the pulse neural network is used to correct the joint angle of the manipulator in real time, improve the stability and robustness of the manipulator control, and the pulse neural network can be used to improve the calculation efficiency of the joint angle correction data, thereby improving the grasping efficiency of the manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0017] Figure 1 This is a flow chart of an embodiment of a manipulator control method based on a spiking neural network provided by the present application;

[0018] Figure 2 This is a flowchart of an embodiment of step 11 provided in this application;

[0019] Figure 3 This is a flow chart of another embodiment of the manipulator control method based on a spiking neural network provided by the present application;

[0020] Figure 4 This is a flowchart of an embodiment of step 34 provided by this application;

[0021] Figure 5 This is a flowchart of an embodiment of step 342 provided by this application;

[0022] Figure 6 This is a flow chart of another embodiment of the manipulator control method based on a spiking neural network provided by the present application;

[0023] Figure 7 This is a schematic structural diagram of an embodiment of a manipulator based on a spiking neural network provided by the present application;

[0024] Figure 8 This is a schematic structural diagram of another embodiment of a manipulator based on a spiking neural network provided by the present application;

[0025] Figure 9 This is a schematic diagram of the working principle of the first controller provided by this application;

[0026] Figure 10 This is a schematic diagram of the working principle of the manipulator based on the pulse neural network provided by this application;

[0027] Figure 11 It is a structural diagram of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0029] The terms "first," "second," and the like in this application are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0030] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] Most applications of manipulators use suction cups, two fingers or customized grippers. These manipulators lack adaptability when grasping objects and usually grasp objects directly, which can easily cause damage to the objects and the manipulator itself. Based on this, this application proposes to segment the process of manipulator grasping objects. When any target finger is detected to be in contact with the target object, the joint angle correction data of the target finger is determined in the pulse neural network based on the grasping instruction and the feedback data of the target finger. The joint correction data is used to correct the grasping instruction, thereby improving the force of the manipulator to grasp the target object. For details, please refer to the following technical solution.

[0032] See Figure 1 , Figure 1 This is a flow chart of an embodiment of a manipulator control method based on a spiking neural network provided by this application. The manipulator control method includes:

[0033] Step 11: Get the crawling instruction for the target object.

[0034] In some embodiments, the target object may be an object, such as an apple, an orange, a cube, a cylinder, etc.

[0035] Since the manipulator has many joints, the grasping instruction can be a control signal for each joint. By controlling each joint, the grasping instruction can grasp the target object.

[0036] In some embodiments, see Figure 2 , step 11 can be the following process:

[0037] Step 111: Identify the type of the target object.

[0038] In some embodiments, the manipulator is connected to an image acquisition device, which captures an image of a target object and then uses image recognition technology to identify the type of the target object. For example, the target object may be a strip, a cylinder, or a sphere.

[0039] Step 112: Determine a fetch instruction based on the type.

[0040] The robot pre-sets the corresponding grabbing instructions according to different types. After identifying the type of the target object, the grabbing instructions are directly determined according to the type.

[0041] Step 12: Based on the grasping instruction, control at least two fingers of the manipulator to grasp the target object.

[0042] In some embodiments, the robotic hand may include at least two fingers. Each finger may be modeled after a human finger. Thus, each finger has different joints. For example, the robotic hand may include five fingers, each corresponding to the human thumb, index finger, middle finger, ring finger, and pinky finger. The movement of each finger may be controlled by a corresponding transmission mechanism, such as a motor, gears, or belts.

[0043] The grasping command controls the coordinated movement of different joints of each finger to grasp the target object.

[0044] Step 13: During the grasping process, determine a target finger among the at least two fingers; wherein the target finger is the finger that touches the target object.

[0045] During the grasping process, the fingers of the manipulator may contact the target object, and thus the target fingers that contact the target object are determined.

[0046] Step 14: Determine the joint angle correction data of the target finger in the pulse neural network based on the grasping instruction and the feedback data of the target finger.

[0047] Because the grasping instruction is set in advance, the grasping control signal at each moment is fixed. However, after the finger contacts the target object, an interaction force exists between the finger and the target object. Therefore, the fixed grasping control signal is not suitable and may cause damage between the finger and the target object. Therefore, this embodiment proposes that when any target finger is detected to have contacted the target object, the target finger's joint angle correction data is determined in the spiking neural network based on the grasping instruction and the target finger's feedback data. In this way, when any target finger contacts the target object, the target finger is suppressed.

[0048] The feedback data may be the current joint angle and joint position of the target finger of the manipulator, or the current of the motor that controls the manipulator.

[0049] In some embodiments, the spiking neural network may be constructed based on a Hodgkin-Huxley model, or may be constructed based on a Leaky Integrate and Fire model or an Izhikevich model.

[0050] The spiking neural network can work based on an unsupervised learning algorithm and / or a supervised learning algorithm.

[0051] Step 15: Continue to control at least two fingers to grasp the target object based on the joint angle correction data and the grasping instruction.

[0052] After obtaining the joint angle correction data, the target finger of the manipulator will generate the optimal sub-grasping instruction based on the joint angle correction data and the grasping instruction to control the manipulator.

[0053] During the grasping process, at least two fingers typically contact the target object sequentially. The finger that contacts the target object first generates the optimal sub-grasp instruction based on joint angle correction data and the grasp instruction to control the robot hand. If the finger that contacts the target object later has not yet contacted the target object when another finger contacts it, it does not need to use the joint angle correction data and only needs to continue to move according to the corresponding grasp instruction.

[0054] Only when the finger touches the target object will the joint angle correction data of the target finger be determined using the pulse neural network.

[0055] In this way, the other fingers of the robot can operate in the same way as described above. When any target finger is detected to be in contact with the target object, the joint angle correction data of the target finger is determined in the pulse neural network based on the grasping instruction and the feedback data of the target finger to compensate for the joint angle in the sub-grasping instruction, so that the joint angle when grasping the target object is more adapted to the structure of the target object.

[0056] In this embodiment, the target finger that contacts the target object is determined during the grasping process, and then the joint angle correction data of the target finger is determined using the pulse neural network, so that the grasping instruction is corrected using the joint correction data, the force of the manipulator grasping the target object is improved, the adaptability of the manipulator in grasping objects is improved, and the wear of the manipulator is reduced. Furthermore, the pulse neural network is used to perform real-time joint angle correction on the manipulator, thereby improving the stability and robustness of the manipulator control, and the pulse neural network can be used to improve the calculation efficiency of the joint angle correction data, thereby improving the grasping efficiency of the manipulator.

[0057] See Figure 3 , Figure 3 This is a flow chart of an embodiment of a manipulator control method based on a spiking neural network provided by this application. The manipulator control method includes:

[0058] Step 31: Get the crawling instruction for the target object.

[0059] Step 32: Based on the grasping instruction, control at least two fingers of the manipulator to grasp the target object.

[0060] Step 31 and step 32 have the same or similar technical solutions as those in the above embodiment and are not described in detail here.

[0061] Step 33: During the grasping process, determine a target finger among the at least two fingers; wherein the target finger is the finger that touches the target object.

[0062] Step 34: Use the captured instructions and feedback data to update the weights of the pulse neural network.

[0063] In some embodiments, see Figure 4 , step 34 may be the following process:

[0064] Step 341: Encode the capture instructions and feedback data to obtain the activities of neurons in the pulse neural network.

[0065] Specifically, the activity of a neuron can be expressed as follows:

[0066] a=G[αe·x];

[0067] where G[·] is the nonlinear neural activation function, α is the scaling factor (gain) associated with the neuron, e is the neuron’s encoder, and x is the vector to be encoded, i.e., the capture instructions and feedback data.

[0068] Step 342: Determine a decoder using the activity of the neurons.

[0069] In some embodiments, see Figure 5 , step 342 can use the following process to calculate the decoder:

[0070] Step 3421: Calculate the first parameter using the capture instruction, feedback data, and neuron activity.

[0071] Specifically, step 3421 can use the following formula to calculate the first parameter:

[0072] r=∫a j xdx.

[0073] Among them, a j is the activity of neuron j, x is the input capture instruction and feedback data, and r is the first parameter.

[0074] Step 3422: Calculate a second parameter using the activities of multiple neurons.

[0075] Specifically, step 3422 can use the following formula to calculate the second parameter:

[0076] T ij =∫a i a j dx.

[0077] Among them, a j is the activity of neuron j, a iis the activity of neuron i, T ij is the second parameter between neuron j and neuron i.

[0078] Step 3423: Calculate a decoder using the first parameter and the second parameter.

[0079] Specifically, step 3423 can use the following formula to find the decoder:

[0080] d=r -1 T.

[0081] Step 343: Calculate the decoding estimate using the decoder and neuron activities.

[0082] It can be understood that the decoding estimation result is the optimal motion data of the robot's fingers predicted by the pulse neural network, which can be compared with the actual motion data in the feedback data to obtain a first difference between the optimal motion data and the actual motion data in the feedback data.

[0083] Specifically, we perform a dot product between the decoder and the neuron activity to obtain the decoding estimate. This can be expressed as follows:

[0084]

[0085] Step 344: Obtain a first difference using the decoded estimate and the feedback data.

[0086] Step 345: Obtain a weight correction value of the spiking neural network using the first difference and the activity of the neuron.

[0087] In some embodiments, an online supervised learning rule may be used to determine the weight revision values.

[0088] Specifically, it can be expressed using the following formula:

[0089] Δd i =κEa i ;

[0090] Δω ij =κα j e j ·Ea i ;

[0091] Among them, Δω ij represents the weight correction value of the connection weight between neuron j and neuron i, κ is the scalar learning rate, and E represents the first difference, i.e., the decoding estimate The difference between and x.

[0092] It is understandable that different neurons have different decoders, and the decoder correction value Δd corresponding to the neuron can be obtained according to the first difference.i .

[0093] In some embodiments, an unsupervised learning rule may be used to determine the weight correction values.

[0094] Specifically, it can be expressed using the following formula:

[0095] Δω ij =a i a j (a j -θ);

[0096] Among them, Δω ij represents the weight correction value of the connection weight between neuron j and neuron i, and θ represents the modification threshold, which is used to limit the modification range of neuron j.

[0097] In some embodiments, the weight correction value may be determined by combining unsupervised learning rules and online supervised learning rules.

[0098] Specifically, the weight correction value is calculated using the following formula:

[0099] Δω ij =κα j a i (Se j ·E+(1-S)a j (a j -θ)).

[0100] Where κ represents the scalar learning rate, α j represents the scaling factor of neuron j, a i represents the activity of neuron i, S represents the control parameter, E represents the first difference, and θ represents the modification threshold.

[0101] Step 346: Update the weights of the spiking neural network using the weight correction values.

[0102] In a spiking neural network, weights are set between neurons, so correction values ​​can be used to update the weights between neurons. If the weight correction value is negative, it means the original weight needs to be reduced, while a positive weight correction value means the original weight needs to be increased.

[0103] Step 35: Use the updated spiking neural network to determine the joint angle correction data of the target finger.

[0104] The updated weights, decoder, and neuron activities are used to obtain the joint angle correction data of the target finger.

[0105] Specifically, the trajectory correction data can be calculated by multiplying the activity of the neuron by the weight and then dot-multiplying the decoder.

[0106] Specifically, the following formula is used:

[0107]

[0108] Where a represents the activity of the neuron, ω represents the updated weight, d represents the decoder, and θ adapt Indicates joint angle correction data.

[0109] Step 36: Continue to control at least two fingers to grasp the target object according to the joint angle correction data and the grasping instruction.

[0110] Furthermore, the torque required for each joint of the manipulator's fingers can be calculated based on the grasping instruction and joint angle correction data.

[0111] The corresponding motor of each finger is controlled to generate a corresponding torque force on the finger so that the finger of the robot can grasp the target object.

[0112] In this embodiment, when the target finger contacts the target object during the grasping process, the pulse neural network is used to determine the joint angle correction data of the target finger, so as to use the joint correction data to correct the grasping instruction, improve the force of the manipulator to grasp the target object, improve the adaptability of the manipulator to grasp objects, and reduce the wear of the manipulator. Furthermore, the pulse neural network is used to correct the joint angle of the manipulator in real time, thereby improving the stability and robustness of the manipulator control, and the pulse neural network can improve the calculation efficiency of the joint angle correction data, thereby improving the grasping efficiency of the manipulator.

[0113] See Figure 6 , Figure 6 This is a flow chart of another embodiment of the manipulator control method based on a spiking neural network provided by the present application. The manipulator control method includes:

[0114] Step 61: Get a crawling instruction for the target object.

[0115] Step 62: Based on the grasping instruction, control at least two fingers of the robot arm to grasp the target object.

[0116] Step 61 - Step 62 have the same or similar technical solutions as any of the above embodiments, and are not described in detail here.

[0117] Step 63: During the grasping process, obtain the first joint position and first force corresponding to each finger at the current moment, and the second joint position and second force at the previous moment.

[0118] The first joint position and the first force at the current moment, and the second joint position and the second force at the previous moment can be determined from the feedback data. That is, the feedback data includes the joint position of the target finger and the force of the target finger.

[0119] Step 64: Determine a target finger among the at least two fingers based on the first joint position and the second joint position, and the first force and the second force.

[0120] For example, if the difference between the first joint position and the second joint position is less than a threshold value, it is determined that the target finger has not moved. However, when the first force is greater than the second force, the target finger should theoretically move. Therefore, when the first force is greater than the second force and the difference between the first joint position and the second joint position is less than a threshold value, it is determined that the target finger is in contact with the target object.

[0121] Based on this, the above steps 63 and 64 are performed for each finger to determine the target finger among the at least two fingers.

[0122] Step 65: Determine the joint angle correction data of the target finger in the spiking neural network based on the grasping instruction and the feedback data of the target finger.

[0123] Step 66: Continue to control at least two fingers to grasp the target object according to the joint angle correction data and the grasping instruction.

[0124] Step 65-Step 66 have the same or similar technical solutions as any of the above embodiments and are not described in detail here.

[0125] In this embodiment, based on the first joint position and the second joint position, as well as the first force and the second force, it is determined that the target finger is in contact with the target object, and then the target finger is suppressed. The joint angle correction data of the target finger is determined by the pulse neural network, and the grasping instruction is corrected using the joint correction data to improve the force of the manipulator grasping the target object, improve the adaptability of the manipulator in grasping objects, and reduce the wear of the manipulator. Furthermore, the pulse neural network is used to perform real-time joint angle correction of the manipulator, improve the stability and robustness of the manipulator control, and the pulse neural network can improve the calculation efficiency of the joint angle correction data, thereby improving the grasping efficiency of the manipulator.

[0126] See Figure 7 , Figure 7 Schematic diagram of the structure of an embodiment of a spiking neural network-based manipulator provided by this application. Manipulator 70 includes at least two fingers 71; a processor 72 connected to at least two fingers 71; and a memory 73 connected to processor 72. Memory 73 is used to store program data, and processor 72 is used to execute the program data to implement the following method:

[0127] Obtain a grasping instruction for a target object; based on the grasping instruction, control at least two fingers of the manipulator to grasp the target object; during the grasping process, determine a target finger among the at least two fingers; wherein the target finger is the finger that contacts the target object; based on the grasping instruction and feedback data of the target finger, determine joint angle correction data of the target finger in a pulse neural network; according to the joint angle correction data and the grasping instruction, continue to control at least two fingers to grasp the target object.

[0128] It can be understood that the processor 72 in this embodiment is also used to execute program data to implement the method in any of the above embodiments. The specific implementation steps can refer to the above embodiments and will not be repeated here.

[0129] See Figure 8 , Figure 8 FIG. 8 is a schematic structural diagram of another embodiment of the manipulator provided by the present application. The manipulator 80 includes a recognition module 81 , a hand controller 82 , a finger controller 83 and a finger 84 .

[0130] The recognition module 81 is used to recognize the type of the target object. The recognition module 81 may include an image acquisition device.

[0131] The hand controller 82 is connected to the recognition module 81 and is used to determine a grabbing instruction according to the type of the target object.

[0132] The finger controller 83 is connected to the hand controller 82 and the finger 84, and is used to control at least two fingers 84 of the manipulator 80 to grasp the target object based on the grasping instruction; when it is detected that any target finger 84 contacts the target object, the joint angle correction data of the target finger 84 is determined in the pulse neural network based on the grasping instruction and the feedback data of the target finger 84; according to the joint angle correction data and the grasping instruction, at least two fingers 84 are controlled to grasp the target object.

[0133] The actual implementation of the robot 80 is described below:

[0134] The robot 80 includes a recognition module 81 , a hand controller 82 , a finger controller 83 , and fingers 84 .

[0135] There is a hand primitive in the hand controller 82, and the hand primitive is used to represent different functions of coordinating fingers.

[0136] Among them, the motion primitive-based approach assumes that human motion can be decomposed into a set of primitive actions, called motion primitives. Imitation, which consists of learning models of primitives and their sequencing, can be used to reproduce human-like motion. After human motion data is captured and preprocessed, the motion data is segmented, and each segment is implemented to build a motion primitive while ensuring the stability of the motion posture. The trajectory composed of motion primitives is used as the reference trajectory of the robot.

[0137] The finger controller 83 includes a finger primitive, which is used to represent the different functions of coordinating fingers. The finger primitive is used to represent the joint coordination when the fingers are closed.

[0138] For each finger 84, there is a corresponding finger primitive in the finger controller 83, which represents the joint coordination between the finger joints in the closing movement. Figure 9 As shown in the figure, there are five finger primitives: thumb, index finger, middle finger, ring finger, and pinky finger. A finger primitive defines the minimum initial value and maximum final value of each active joint in a trajectory. It is modeled as an activation parameter u∈[0,1] that maps the activations of a series of joints during the execution of the movement.

[0139] The activation function is: Among them, u represents the activation parameter.

[0140] It is important to have smooth initial and final phases for f(u) to prevent wear in the motors and gearboxes in the manipulator. This type of function is often used for interpolation in robots. The activation function must be mapped to the robot kinematics.

[0141] Use the activation function to generate appropriate crawling instructions. It is expressed by the following formula:

[0142] g(f(u))=f(u)*(θ max -θ min )+θ min θ max Indicates the maximum final value, θ min Indicates the minimum initial value.

[0143] Finger controller 83 contains two controllers: a first controller and a second controller. When the first controller detects that a target finger has contacted a target object, it uses the second controller to control that finger. The second controller is used to determine joint angle correction data for target finger 84 within a spiking neural network based on the grasping instruction and feedback data from target finger 84.

[0144] The second controller controls the target finger 84 according to the joint angle correction data and the grasping instruction.

[0145] The first controller continues to control the other fingers 84 according to the grasping instruction to grasp the target object.

[0146] The second controller uses the force of the motor to control the force that can be applied by the finger.

[0147] The recognition module 81 creates activation patterns of grabbing instructions of different functions for the hand controller 82 .

[0148] The hand controller 82 contains a hand primitive, which represents the coordinated functions of different fingers. Hand movement is also simulated using motion primitives, but rather than controlling joints, they control the activation parameters of the finger primitives in the finger control 83. The hand primitive supports different grasping functions, such as spheres, cylinders, and rectangles, based on the recognition results of the recognition module 81.

[0149] By using motion primitives for grasping actions, the complexity of hand control is reduced to a single activation parameter for each functional support.

[0150] During action execution, the hand primitive is modeled as a mapping from u to a series of finger primitive activations. Based on expert experience, typically biologists studying human hand movements to grasp objects, different grasping functions are supported, with different initial grasping poses (pre-shapes) and final poses when the hand is closed. Primitives are defined using the minimum open and maximum closed activation parameters of the finger primitives. Each hand primitive is connected to all finger primitives. A second controller can adjust the finger motions online to adapt to the shape of the object.

[0151] Recognition module 81 generates the functional activation pattern for the hand primitive. An external activation signal is used to activate the hand primitive. This module uses a YOLOv5-based object recognition algorithm, capable of identifying long, cylindrical, and spherical objects. The hand primitive is then activated to select the grasping function for objects of different shapes.

[0152] The first controller and the second controller are the parts that provide adaptability and flexibility to the grasping motion, which is required for compliant grasping.

[0153] The finger 84 joint angle θ is not measured, only the motor current is used for contact detection. The first controller is modeled as an alternative selection mechanism of the interneuron network in the spinal cord. It combines inhibitory and excitatory connections, such as Figure 9 The manipulator feedback is used to calculate Δθ as the time variation of the finger joints, using the current joint position θ t and the previous joint position θ provided by the delay loop connection t-1The first controller is excited by the force feedback from the motor and inhibited by Δθ. This ensures that the first controller detects finger contact with the target object only when the force increases and the corresponding joint does not move, thus ignoring the force changes caused by the nonlinearity of the robot dynamics.

[0154] When the first controller detects contact between a target finger and a target object, two types of reflexes can be triggered. The first provides inhibition and motion halting for the target finger. When contact is detected on a finger, the reflex inhibits the respective finger primitive and the contact joint position is mapped to the new target position.

[0155] The second reflex mechanism is used to activate the second controller. By measuring the motor current, the actual force of each joint of the target finger can be estimated, and the target force in the grasping instruction can be obtained. The second controller combines the actual force and the target force, and uses the pulse neural network to output the joint angle correction data, that is, force feedback is used as a control parameter to control the force that can be applied by the finger. When it is detected that the target finger contacts the target object, the second controller will be activated, and when there is no contact or the hand is open, the second controller will be suppressed. The target force can be set individually to each joint to determine the force and sensitivity of the grasp. Thus. The second controller can change the strength of the finger in real time through the pulse neural network, with higher flexibility.

[0156] The second controller is a controller based on SNN (Spiking Neuron Networks). Due to the interaction between the finger and the object, the initial contact point must change when the object moves or deforms. Ideally, there is a part in the second controller that can learn online to compensate for these changes without calculating the exact contact point or inverse kinematics. To this end, this application proposes an adaptive control scheme, such as Figure 10 For the second controller, the target force is used as training data, the actual force fed back by the manipulator 80 is used as learning data, and the combined learning rule the homeostatic Prescribed Error Sensitivity (hPES) is used as the weight update rule in the spiking neural network.

[0157] Among them, the activity of neurons can be expressed as:

[0158] a=G[αe·x].

[0159] where G[·] is the nonlinear neural activation function, α is the scaling factor (gain) associated with the neuron, e is the neuron’s encoder, and x is the vector to be encoded, i.e., our input data.

[0160] Decoding estimate, is the sum of the activities of each neuron, weighted by the n-dimensional decoder.

[0161]

[0162] Where d is the decoder and a is the activity of the neuron. The decoder is found by least squares minimizing the difference between the decoded estimate and the actual encoded vector.

[0163] The decoder d can be calculated according to the following formula:

[0164] d=r -1 T;

[0165] T ij =∫a i a j dx;

[0166] r=∫a j xdx;

[0167] Among them, d is the decoder, a i is the activity of neuron i, a j is the activity of neuron j, x is the input data, r is the first parameter, T ij is the second parameter between neuron j and neuron i.

[0168] The weight correction value is obtained using the following formula:

[0169] Δω ij =κα j a i (Se j ·E+(1-S)a j (a j -θ)).

[0170] Wherein, 0≤S≤1, S is the relative weight of the online supervised learning item relative to the unsupervised learning item, that is, the control parameter in the above embodiment.

[0171] Among them, the weight connection calculation formula between neuron i and neuron j is:

[0172] ω ij =α j e j d i .

[0173] Among them, α j Scaling factor (gain) of neuron j, e j is the encoder of neuron j, d i is the decoder of neuron i.

[0174] The weights between neurons are obtained based on the weight correction value, and then the joint angle correction data is obtained by using the weights, decoder and neuron activities. The joint angle correction data can be calculated according to the following formula:

[0175]

[0176] Where a is the activity of the neuron, which is encoded by the input data, ω is the connection weight between neurons, d is the neuron decoder, and θ adapt Indicates joint angle correction data.

[0177] Finally, the second controller outputs joint angle correction data, which is then integrated with the target angle output by the first controller to obtain the actual target angle. Using PID control, this actual target angle is calculated to determine the required force and corresponding current. This current is then applied to the motor, causing the finger to generate the appropriate force to grasp the target object.

[0178] In this embodiment, the above-mentioned method can improve the ability of multiple joints of the finger 84 to move simultaneously and collaboratively, making the movement of the finger 84 more flexible and improving the grasping adaptability.

[0179] See Figure 11 , Figure 11 1 is a schematic diagram of the structure of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 110 is used to store program data 113. When the program data 113 is executed by a processor, it is used to implement the following method steps:

[0180] Obtain a grasping instruction for a target object; based on the grasping instruction, control at least two fingers of the manipulator to grasp the target object; during the grasping process, determine a target finger among the at least two fingers; wherein the target finger is the finger that contacts the target object; based on the grasping instruction and feedback data of the target finger, determine joint angle correction data of the target finger in a pulse neural network; according to the joint angle correction data and the grasping instruction, continue to control at least two fingers to grasp the target object.

[0181] It can be understood that the computer-readable storage medium 110 in this embodiment is applied to the manipulator 70 or the manipulator 80 in the above embodiments, and its specific implementation steps can refer to the above embodiments and will not be repeated here.

[0182] In summary, the present invention discloses a manipulator control method, manipulator, and storage medium based on a pulse neural network. The manipulator control method includes: obtaining a grasping instruction for a target object; controlling at least two fingers of the manipulator to grasp the target object based on the grasping instruction; determining a target finger among the at least two fingers during the grasping process; wherein the target finger is the finger that contacts the target object; determining joint angle correction data of the target finger in the pulse neural network based on the grasping instruction and feedback data of the target finger; and continuing to control the at least two fingers to grasp the target object according to the joint angle correction data and the grasping instruction. Through the above-mentioned method, the target finger that contacts the target object is determined during the grasping process, and then the joint angle correction data of the target finger is determined in the pulse neural network, so as to correct the grasping instruction using the joint correction data, improve the force of the manipulator grasping the target object, improve the adaptability of the manipulator grasping the object, and reduce the wear of the manipulator. Furthermore, the pulse neural network is used to perform real-time joint angle correction on the manipulator, thereby improving the stability and robustness of the manipulator control, and the pulse neural network can improve the calculation efficiency of the joint angle correction data, thereby improving the grasping efficiency of the manipulator.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0184] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0185] In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0186] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0187] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A manipulator control method based on pulse neural network, characterized in that: The manipulator control method comprises: Get the crawling instruction for the target object; Based on the grasping instruction, controlling at least two fingers of the manipulator to grasp the target object; During the grasping process, the first joint position and first force corresponding to each finger at the current moment, as well as the second joint position and second force at the previous moment, are obtained; Determining a target finger among the at least two fingers according to the first joint position and the second joint position, and the first force and the second force; wherein the target finger is the finger that contacts the target object; Determining joint angle correction data of the target finger in a spiking neural network based on the grasping instruction and feedback data of the target finger; The at least two fingers are continuously controlled to grasp the target object according to the joint angle correction data and the grasping instruction.

2. The method according to claim 1, characterized in that Determining the joint angle correction data of the target finger in a spiking neural network based on the grasping instruction and the feedback data of the target finger includes: Updating the weights of the spiking neural network using the capture instruction and the feedback data; The updated spiking neural network is used to determine joint angle correction data of the target finger.

3. The method according to claim 2, characterized in that The updating of the weight of the spiking neural network using the capture instruction and the feedback data includes: Encoding the capture instruction and the feedback data to obtain the activity of neurons in the spiking neural network; Determine a decoder using the activity of the neurons; calculating a decoding estimate using the decoder and the activity of the neuron; Obtaining a first difference using the decoded estimate and the feedback data; Obtaining a weight correction value of the spiking neural network using the first difference and the activity of the neuron; The weight of the spiking neural network is updated using the weight correction value.

4. The method according to claim 3, characterized in that The obtaining of a weight correction value of the spiking neural network by using the first difference and the activity of the neuron includes: The weight correction value is calculated using the following formula: Give ij =ka j a i (Se j ·E+(1-S)a j (a j -i)); Where κ represents the scalar learning rate, α j represents the scaling factor of neuron j, a i represents the activity of neuron i, S represents the control parameter, E represents the first difference, and θ represents the modification threshold.

5. The method according to claim 3, characterized in that The method of determining the joint angle correction data of the target finger by using the updated spiking neural network includes: The joint angle correction data of the target finger is obtained using the updated weights, the decoder and the activity of the neurons.

6. The method according to claim 5, characterized in that The obtaining of the joint angle correction data of the target finger by using the updated weights, the decoder, and the activity of the neurons includes: The joint angle correction data is calculated using the following formula: Where a represents the activity of the neuron, ω represents the updated weight, d represents the decoder, θ adapt represents the joint angle correction data.

7. The method according to claim 1, characterized in that The step of obtaining a crawl instruction for a target object includes: Identify the type of target object; The fetch instruction is determined based on the type.

8. A manipulator based on a pulse neural network, characterized in that: The manipulator comprises: At least two fingers; a processor connected to the at least two fingers; a memory connected to the processor; The memory is used to store program data, and the processor is used to execute the program data to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program data, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 7.

Images