Mechanical arm self-adaptive grabbing method and system based on reinforcement learning and imitation learning

Abstract

The invention relates to the technical field of robot grabbing, and discloses a mechanical arm self-adaptive grabbing method and system based on reinforcement learning and imitation learning, and the method comprises the steps: constructing a first feature vector of a target object based on the perception of multi-dimensional attributes of the target object; based on a pre-trained imitation learning neural network and the first feature vector, a first grabbing parameter set of the mechanical arm is generated, and the first grabbing parameter set comprises at least one of a grabbing position parameter, a grabbing posture parameter and a grabbing force parameter; in the process that the mechanical arm makes contact with the target object based on the first grabbing parameter set, real-time force feedback data between the target object and the mechanical arm is collected; and on the basis of the reinforcement learning strategy network and the real-time force feedback data, the first grabbing parameter set is optimized online, a second grabbing parameter set is obtained, and the second grabbing parameter set is used for controlling the mechanical arm to execute grabbing operation on the target object. Therefore, the self-adaptability and the flexibility of grabbing of the mechanical arm can be improved.

Description

Technical Field

[0001] This invention relates to the field of robot grasping technology, and in particular to an adaptive grasping method and system for a robotic arm based on reinforcement learning and imitation learning. Background Technology

[0002] Robotic arm grasping technology is one of the core issues in the field of robotics research. Its goal is to endow robotic arms with the ability to intelligently and adaptively grasp diverse objects.

[0003] However, when faced with complex, unstructured real-world environments, traditional methods typically rely on accurate 3D object models and pre-programmed grasping strategies. These methods require searching for the optimal grasping point through mechanical analysis (such as force closure and shape closure) given the known object model. Their main drawback is: Poor environmental adaptability: It relies on prior knowledge and has difficulty coping with unknown objects or dynamic environments where the position or posture of objects changes.

[0004] Lack of flexibility in strategy: Fixed grasping strategy can easily lead to grasping failure or damage to objects.

[0005] Therefore, it is particularly important to propose a technical solution to improve the adaptability and flexibility of robotic arm grasping. Summary of the Invention

[0006] This invention provides an adaptive grasping method and system for robotic arms based on reinforcement learning and imitation learning, which can improve the adaptability and flexibility of robotic arm grasping.

[0007] To address the aforementioned technical problems, the first aspect of this invention discloses an adaptive grasping method for a robotic arm based on reinforcement learning and imitation learning, the method comprising: Based on the perception of the multi-dimensional attributes of the target object, a first feature vector of the target object is constructed; Based on the pre-trained imitation learning neural network and the first feature vector, a first set of grasping parameters for the robotic arm is generated. The first set of grasping parameters includes at least one of grasping position parameters, grasping posture parameters, and grasping force parameters. During the process of the robotic arm contacting the target object based on the first set of grasping parameters, real-time force feedback data between the target object and the robotic arm is collected; Based on the reinforcement learning policy network and the real-time force feedback data, the first grasping parameter set is optimized online to obtain a second grasping parameter set, which is used to control the robotic arm to perform a grasping operation on the target object.

[0008] As an optional implementation, in the first aspect of the present invention, constructing the first feature vector of the target object based on the perception of its multi-dimensional attributes includes: The point cloud data of the target object is acquired by controlling the vision sensor corresponding to the robotic arm, and geometric attribute parameters for characterizing the macroscopic structure of the target object are extracted based on the point cloud data. The tactile sensor corresponding to the robotic arm is controlled to contact the target object in a preset manner, and pressure distribution signals and vibration signals are collected during the contact process. Based on the pressure distribution signals, a hardness attribute parameter is obtained to characterize the softness and hardness of the target object, and based on the vibration signals, a roughness attribute parameter is obtained to characterize the surface texture of the target object. Based on the geometric attribute parameters, the hardness attribute parameters, and the roughness attribute parameters, a first feature vector of the target object is constructed.

[0009] As an optional implementation, in the first aspect of the present invention, generating the first set of grasping parameters for the robotic arm based on the pre-trained imitation learning neural network and the first feature vector includes: A multimodal teaching dataset consisting of human demonstration operations is obtained. The multimodal teaching dataset includes a second feature vector corresponding to a successful grasping event, a gripper motion trajectory sequence, and a grasping force sequence. The second feature vector corresponds to the first feature vector. Extract a sequence of exemplary data segments from the gripper motion trajectory sequence and the gripping force sequence to characterize key stages of the gripping operation. The key stages include at least approaching the object, the moment of contact, stable gripping, and lifting movement. Using the second feature vector as input data, and the gripper pose and grasping force contained in the corresponding demonstration data segment sequence as supervision labels, the preset imitation learning neural network is trained until convergence. The first feature vector is input into the imitation learning neural network to generate the first set of grasping parameters for the robotic arm.

[0010] As an optional implementation, in a first aspect of the invention, the step of collecting real-time force feedback data between the target object and the robotic arm during the process of the robotic arm contacting the target object based on the first grasping parameter set includes: The original force and torque signals during the contact process are continuously collected by controlling the multi-axis force sensor installed on the end effector of the robotic arm. The original force and torque signals include the contact force and torque acting on the gripper in various directions. The original force and torque signals are filtered in real time to eliminate high-frequency noise, and force feedback characteristic parameters for characterizing the stability of the contact state are calculated. The force feedback characteristic parameters include at least the magnitude of the resultant force, the direction of the force vector, and the gradient of the torque change. The filtered original force and torque signals, along with the force feedback characteristic parameters, are fused to form the real-time force feedback data.

[0011] As an optional implementation, in the first aspect of the present invention, the online optimization of the first grasping parameter set based on the reinforcement learning policy network and the real-time force feedback data to obtain the second grasping parameter set includes: The first feature vector, the first grasping parameter set, the real-time force feedback data, and the current state information of the robotic arm are fused to construct a state vector that characterizes the current grasping environment. The state vector is processed by a reinforcement learning policy network to generate an action vector for fine-tuning the first grasping parameter set, the action vector corresponding to the first grasping parameter set; The first set of grabbing parameters is updated according to the action vector to obtain a set of candidate grabbing parameters and grabbing attempts are performed. At the same time, a reward signal is calculated based on the grabbing results. The reward signal is used to comprehensively evaluate the grabbing success rate, stability and efficiency. Based on the reward signal, the candidate crawling parameter set is optimized online to obtain a second crawling parameter set.

[0012] As an optional implementation, in the first aspect of the present invention, the step of optimizing the candidate crawling parameter set online based on the reward signal to obtain a second crawling parameter set includes: The state vector, the action vector, the reward signal, and the new state vector obtained after the grasping attempt are combined to form a quadruple of experience data, and the experience data is stored as historical experience data in the experience replay buffer. A preset number of historical experience data are randomly sampled from the experience replay buffer, and the temporal difference error of the current reinforcement learning policy network is calculated based on the historical experience data. A policy loss function is constructed based on the temporal difference error, and the parameters of the reinforcement learning policy network are updated through the backpropagation algorithm to minimize the policy loss function. The process iteratively executes the operation of constructing a quadruple of experience data from the state vector, the action vector, the reward signal, and the new state vector obtained after the grasping trial, and storing the experience data as historical experience data in the experience replay buffer, until the operation of updating the parameters of the reinforcement learning policy network through the backpropagation algorithm to minimize the policy loss function, until the reward signal corresponding to the new action vector output by the reinforcement learning policy network converges to a stable interval. At this point, the optimal grasping parameter set generated by the reinforcement learning policy network is determined as the second grasping parameter set.

[0013] As an optional implementation, in the first aspect of the present invention, the method further includes: After the robotic arm completes the grasping operation based on the second grasping parameter set, the target object displacement information and the time information are determined. The target object displacement information and the time information correspond to the range of steps from when the robotic arm contacts the target object based on the first grasping parameter set to when the grasping operation is completed. The target object displacement information, the time information, and the force feedback feature parameters are compared with their respective preset thresholds to obtain the comparison results. Based on the comparison results, a policy optimization instruction is generated. The policy optimization instruction is used to add the first feature vector, the second set of grasping parameters, and the corresponding grasping result data corresponding to this grasping task to the training dataset of the imitation learning neural network and / or adjust the network parameters of the reinforcement learning policy network.

[0014] A second aspect of this invention discloses an adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning, the system comprising: The attribute-aware module is used to construct the first feature vector of the target object based on the perception of its multi-dimensional attributes. The imitation learning control module is used to generate a first set of grasping parameters for the robotic arm based on a pre-trained imitation learning neural network and the first feature vector. The first set of grasping parameters includes at least one of grasping position parameters, grasping posture parameters, and grasping force parameters. The force feedback module is used to collect real-time force feedback data between the target object and the robotic arm during the process of the robotic arm contacting the target object based on the first grasping parameter set; An adaptive optimization control module is used to optimize the first grasping parameter set online based on the reinforcement learning policy network and the real-time force feedback data to obtain a second grasping parameter set. The second grasping parameter set is used to control the robotic arm to perform a grasping operation on the target object.

[0015] A third aspect of this invention discloses another adaptive grasping system for robotic arms based on reinforcement learning and imitation learning, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the adaptive grasping method for robotic arms based on reinforcement learning and imitation learning disclosed in the first aspect of the present invention.

[0016] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the adaptive grasping method for robotic arms based on reinforcement learning and imitation learning disclosed in the first aspect of the present invention.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, a first feature vector of the target object is constructed based on the perception of its multi-dimensional attributes. A first set of grasping parameters for the robotic arm is generated based on a pre-trained imitation learning neural network and the first feature vector. This first set of grasping parameters includes at least one of grasping position parameters, grasping posture parameters, and grasping force parameters. During the process of the robotic arm contacting the target object based on the first set of grasping parameters, real-time force feedback data between the target object and the robotic arm is collected. Based on a reinforcement learning policy network and the real-time force feedback data, the first set of grasping parameters is optimized online to obtain a second set of grasping parameters. This second set of grasping parameters is used to control the robotic arm to perform the grasping operation on the target object. Therefore, implementing this invention can improve sample training efficiency by integrating imitation learning and reinforcement learning paradigms, thereby shortening the actual interaction time required for the robotic arm to obtain an effective grasping strategy. This, in turn, helps reduce wear and tear on hardware and energy consumption, achieving highly efficient and energy-saving system training. This system can generate high-quality initial grasping strategies (the first set of grasping parameters) by imitating neural networks, improving the starting point and safety of strategy exploration. This helps avoid numerous random, invalid, and even dangerous grasping attempts when reinforcement learning starts from scratch, thus protecting the robotic arm and the target object and achieving safe and reliable learning initialization. It can also optimize the initial strategy online based on real-time force feedback data, improving the adaptability and precision of the grasping strategy. This allows the robotic arm to dynamically adjust its grasping force, position, and posture according to real physical interactions, thus better handling uncertainties such as object slippage and deformation, achieving robust grasping of dynamic environments and different objects, and improving the adaptability and flexibility of the robotic arm's grasping. Furthermore, it can decompose the grasping process into sequentially linked stages of perception, initial strategy generation, and online optimization, improving the modularity and logical clarity of the system design. This facilitates independent development and debugging of each module, thereby enhancing the maintainability and scalability of the entire system, resulting in a clearly structured and easily implemented engineering solution. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an adaptive grasping method for a robotic arm based on reinforcement learning and imitation learning, as disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning, as disclosed in an embodiment of the present invention. Figure 3 This is a schematic diagram of another adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning disclosed in an embodiment of the present invention. Figure 4 This is a schematic diagram of another adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning disclosed in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention discloses an adaptive grasping method and system for robotic arms based on reinforcement learning and imitation learning. By integrating these two paradigms, it improves sample training efficiency, thereby shortening the actual interaction time required for the robotic arm to acquire an effective grasping strategy. This, in turn, reduces wear and tear on hardware and energy consumption, achieving efficient and energy-saving system training. It can generate high-quality initial grasping strategies (first grasping parameter set) through imitation learning neural networks, improving the starting point and safety of strategy exploration. This helps avoid numerous random, invalid, and even dangerous grasping attempts when reinforcement learning starts from scratch, thus protecting the robotic arm and the target object and achieving safe and reliable learning initialization. It can optimize the initial strategy online based on real-time force feedback data, improving the adaptability and precision of the grasping strategy. This allows the robotic arm to dynamically adjust grasping force, position, and posture according to real physical interactions, thus better handling uncertainties such as object slippage and deformation, achieving robust grasping in dynamic environments and with different objects, and improving the adaptability and flexibility of the robotic arm's grasping capabilities. This approach breaks down the data capture process into sequentially linked stages: perception, initial strategy generation, and online optimization. This improves the modularity and logical clarity of the system design, facilitating independent development and debugging of each module. Ultimately, it enhances the maintainability and scalability of the entire system, resulting in a clearly structured and easily implemented solution. These are detailed below.

[0024] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating an adaptive grasping method for a robotic arm based on reinforcement learning and imitation learning, as disclosed in an embodiment of the present invention. Figure 1 The described adaptive grasping method for robotic arms based on reinforcement learning and imitation learning can be applied to robots and also to intelligent devices associated with robots. These intelligent devices include, but are not limited to, one or more of the following: switching devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. It is suitable for application scenarios such as industrial assembly, logistics sorting, home services, and agricultural harvesting. This invention does not limit the scope of the application. Figure 1 As shown, the adaptive grasping method for robotic arms based on reinforcement learning and imitation learning can include the following operations: 101. Based on the perception of the multi-dimensional attributes of the target object, construct the first feature vector of the target object; In this embodiment of the invention, optionally, a comprehensive set of physical properties of an object can be obtained through the fusion of multiple sensors (such as vision and touch), rather than relying solely on vision. The purpose is to transform a specific, unknown object into a digital feature representation (i.e., a "first feature vector") that can be understood and processed by a subsequent neural network. This vector contains key information that determines how to grasp the object.

[0025] The system first controls the robotic arm's vision sensors (such as an RGB-D camera) to scan the working area, obtaining point cloud data containing the target object. By segmenting, denoising, and reconstructing the point cloud, the geometric attributes of the object (such as approximate length, width, height, main orientation, and overall shape category) are extracted. Then, the robotic arm is controlled to move, causing the tactile sensor mounted at the end effector to gently contact the object's surface with a preset force or trajectory. By analyzing the pressure distribution and vibration signals generated during contact, the object's hardness and surface roughness are estimated. Finally, these attribute parameters (geometry, hardness, roughness) from different sources and with different dimensions are normalized and concatenated to form a unified digital vector, the "first feature vector."

[0026] In this embodiment of the invention, as an optional implementation, the above-mentioned construction of the first feature vector of the target object based on the perception of multi-dimensional attributes of the target object includes: The point cloud data of the target object is acquired by controlling the vision sensor corresponding to the robotic arm, and geometric attribute parameters for characterizing the macroscopic structure of the target object are extracted based on the point cloud data. The tactile sensors corresponding to the control robot arm contact the target object in a preset manner and collect pressure distribution signals and vibration signals during the contact process. Based on the pressure distribution signals, the hardness attribute parameters used to characterize the softness and hardness of the target object are obtained, and based on the vibration signals, the roughness attribute parameters used to characterize the surface texture of the target object are obtained. The first feature vector of the target object is constructed based on the geometric property parameters, hardness property parameters, and roughness property parameters.

[0027] In this embodiment of the invention, optionally, a visual sensor (such as a depth camera) can be fixed to a robotic arm or above the workspace. The robotic arm can move to multiple viewpoints or directly acquire RGB and depth images of the scene from the fixed camera, the latter being converted into a 3D point cloud.

[0028] The point cloud processing workflow includes: removing outliers using statistical filters; downsampling using voxel mesh filters to reduce data volume; removing background elements such as desktops using planar segmentation algorithms (e.g., RANSAC); and separating different object point clouds using Euclidean clustering. For the point cloud of the target object, its 3D bounding box is calculated to obtain dimensional parameters such as length, width, and height. Furthermore, principal component analysis (PCA) can be used to calculate the principal orientation of the point cloud, and the surface can be fitted to estimate shape parameters such as overall curvature.

[0029] Alternatively, a "preset method" refers to a standardized contact detection procedure. For example, a robotic arm controls a gripper to approach an object's surface vertically at a constant low speed until a tactile sensor detects pressure exceeding a very small threshold, indicating that contact has occurred.

[0030] Hardness estimation: After contact, the grippers continue to apply a small and slowly increasing displacement along the normal direction, while recording the pressure value. The higher the hardness of the object, the greater the pressure required to produce the same displacement. By analyzing the slope of this "pressure-displacement" curve or the pressure value at which a specific displacement is achieved, the relative hardness properties can be estimated. For example, the hardness index can be calculated based on the indentation depth and pressure: H = F / d, where F is the pressure and d is the indentation depth. Furthermore, the hardness index can be normalized to the range of 0-1, where 0 represents very soft (such as a sponge) and 1 represents very hard (such as a metal).

[0031] Roughness estimation: After contact, the gripper can slide a short distance on the object's surface at a constant speed and pressure. A vibration sensor (such as an accelerometer) mounted on the tactile sensor records the high-frequency vibration signal during the sliding process. Rough surfaces produce stronger, higher-frequency vibrations. The surface roughness is estimated by calculating the root mean square (RMS) value of the vibration signal or analyzing its power spectral density (PSD) energy in the high-frequency range.

[0032] Alternatively, the geometric parameters (such as length, width, height, curvature, etc.), hardness estimates, and roughness estimates obtained in the above steps can be normalized (e.g., scaled to the [0,1] interval) to eliminate the influence of different dimensions. Then, these normalized values ​​are concatenated into a one-dimensional array in a predetermined order, which is the "first eigenvector".

[0033] As can be seen, implementing this optional embodiment can improve the comprehensiveness and redundancy of target object attribute perception by fusing visual sensor point cloud data and tactile sensor physical signals. This facilitates obtaining a comprehensive object description that transcends single vision, including geometry and physical material properties. Consequently, it provides a richer and more reliable information foundation for subsequent strategy generation, enabling more comprehensive environmental perception of objects. By actively contacting and measuring hardness and roughness through tactile sensors, the perception accuracy of key physical properties of objects (such as fragility and surface friction coefficient) can be improved. This helps distinguish objects with similar visual shapes but different materials (such as real apples and plastic apples), thus facilitating the development of differentiated grasping force strategies and enabling precise grasping based on the object's physical characteristics. Furthermore, by extracting multi-dimensional parameters such as geometry, hardness, and roughness and constructing a unified feature vector, the structure and processability of perceived information can be improved. This facilitates the transformation of heterogeneous sensor data into standardized input suitable for neural network processing, thereby improving the input quality and learning efficiency of subsequent imitation learning and reinforcement learning, and achieving efficient integration of perceived information and learning strategies.

[0034] 102. Based on a pre-trained imitation learning neural network and a first feature vector, a first set of grasping parameters for the robotic arm is generated. The first set of grasping parameters includes at least one of grasping position parameters, grasping posture parameters, and grasping force parameters. In this embodiment of the invention, optionally, experience learned from human expert instruction can be utilized to quickly generate a high-quality, executable initial grasping plan for the currently perceived object. This avoids the large amount of random exploration that reinforcement learning involves starting from scratch, significantly improving the initial success rate and training efficiency.

[0035] Alternatively, a neural network (such as a multilayer perceptron) can be pre-trained using a large amount of data on successful human grasping of different objects (recording object attributes, grasping pose, grasping force, etc.). During application, the "first feature vector" obtained in the previous step is input into this pre-trained network. Based on the "knowledge" it has learned from human demonstrations (i.e., the mapping relationship between attributes and grasping strategies), the network directly infers and outputs a set of recommended grasping parameters, such as the gripper's posture, its position on the object, and the initial force used for grasping. This set of parameters constitutes the "first grasping parameter set."

[0036] In this embodiment of the invention, as another optional implementation, the above-mentioned generation of the first grasping parameter set of the robotic arm based on the pre-trained imitation learning neural network and the first feature vector includes: Obtain a multimodal teaching dataset consisting of human demonstration operations. The multimodal teaching dataset includes a second feature vector, a gripper motion trajectory sequence, and a gripping force sequence corresponding to a successful grasping event. The second feature vector corresponds to the first feature vector. Extract a sequence of exemplary data segments from the gripper motion trajectory sequence and gripping force sequence to characterize the key stages of the gripping operation. The key stages include at least approaching the object, the moment of contact, stable gripping, and lifting movement. Using the second feature vector as input data and the gripper pose and grasping force contained in the corresponding demonstration data segment sequence as supervision labels, the preset imitation learning neural network is trained until convergence. The first feature vector is input into the imitation learning neural network to generate the first set of grasping parameters for the robotic arm.

[0037] In this embodiment of the invention, optionally, teaching data can be collected in several ways: a) The operator controls the slave hand (robotic arm) to complete the grasping in real time through a master hand device. b) The operator grasps the object with their bare hand, and their hand movements are recorded by a motion capture system. c) The movements of the human hand and the object are analyzed from a third-person perspective video.

[0038] Optionally, each teaching session records not only the motion trajectory and grasping force of the robotic arm or human hand, but also the attribute information of the object being grasped at that time. This attribute information may be obtained during collection through additional measuring devices (such as weighing) or post-collection annotation, and processed into the same format as the "first feature vector" structure defined above; this is the "second feature vector." Its correspondence with the "first feature vector" lies in the identical data structure; the only difference is the source (one is real-time sensing, the other is historical data).

[0039] Alternatively, a complete capture trajectory can be very large and contain redundancy. An algorithm can automatically identify and extract data points from several key frames or key segments. For example: Approaching an object: The instant just before the gripper is about to make contact with the object.

[0040] Moment of contact: The moment when the force sensor first detects a non-zero contact force.

[0041] Stable grip: The stage where the gripping force is stable and the object is firmly held.

[0042] Lifting and moving: The moment when an object is successfully picked up and begins to move.

[0043] Extract the gripper pose (position and attitude) and gripping force data corresponding to these key stages as the essential demonstration data for this gripping operation.

[0044] Alternatively, the above-mentioned "using the second feature vector as input data and the corresponding ... as supervision labels to train a preset imitation learning neural network until convergence" is a standard supervised learning (behavior cloning) process. The input to the neural network is the "second feature vector" (object attributes), and the expected output (i.e., supervision labels) is the grasping parameters extracted in the key stages (such as grasping position, posture, force). The neural network can be a multilayer perceptron structure containing multiple hidden layers.

[0045] The training goal is to teach the network to map inputs to outputs, that is, to learn to "see" the attributes of an object and predict how a human expert would grasp it. Using a loss function such as mean squared error (MSE), the network weights are iteratively optimized using gradient descent algorithms (such as Adam) until the loss function value stabilizes at a low level. At this point, the network is considered to have "converged" and learned the essence of human grasping strategies.

[0046] Optionally, regarding the aforementioned "inputting the first feature vector into the imitation learning neural network to generate the first grasping parameter set for the robotic arm," during the deployment phase, once the system perceives a new object in real time and generates its "first feature vector," it directly inputs it into the trained imitation learning network. The network's forward propagation calculation immediately outputs a set of grasping parameters, which is the "first grasping parameter set" generated based on human experience.

[0047] Further, optionally, the training process for imitation learning neural networks may include: Step 1: Data Collection Human teaching data is collected through remote operation. The operator controls the robotic arm to grasp objects using a 6-DOF control handle.

[0048] Data collection settings: Object types: 50 kinds of everyday objects (cups, bottles, boxes, balls, books, tools, etc.); Each object: 20-30 successful grasps; Total number of grasps: 1200; Recorded data: Object attributes (20-dimensional), grasping position (3-dimensional), grasping posture (4-dimensional, quaternion), grasping force (1-dimensional), gripper opening degree (1-dimensional), approach speed (1-dimensional); Step 2: Data Preprocessing Preprocessing of trajectory data: Normalization: Normalize all features to the range of 0-1; Filtering: Smooth the trajectory using a moving average filter; Resampling: Resample trajectories of different lengths to a uniform length (20 frames); Keyframe extraction: Extract four keyframes: approach, contact, capture, and lift. Data augmentation: Position perturbation: Add a random perturbation of ±5mm to the gripping position; Attitude perturbation: Add a random perturbation of ±5 degrees to the gripping attitude; Force perturbation: Add a random perturbation of ±10% to the gripping force. Dataset split: Training set: 1000 crawls (83.3%); Validation set: 100 crawls (8.3%); Test set: 100 crawls (8.3%). Step 3: Network Construction Behavioral Cloning Network Structure: Input Layer: 20 neurons (object attributes); Hidden Layer 1: 256 neurons, ReLU activation function, Dropout 0.2; Hidden Layer 2: 128 neurons, ReLU activation function, Dropout 0.2; Hidden Layer 3: 64 neurons, ReLU activation function; Output Layer: 10 neurons (grasping parameters). Output layer activation function: Grasp location (3D): Linear activation; Grasping posture (4D): Normalization (quaternion constraint); Grasping force (1D): Sigmoid activation (output 0-1, mapped to 0-50N); Gripper opening degree (1D): Sigmoid activation (output 0-1, mapped to 0-85mm); Approach velocity (1D): Sigmoid activation (output 0-1, mapped to 10-100mm / s). Step 4: Network Training Training settings: Loss function: Mean Squared Error (MSE); Optimizer: Adam, learning rate 0.001, β1=0.9, β2=0.999; Batch size: 64; Number of Epochs: 100; Early stopping: Stop when the validation set loss does not decrease for 10 consecutive epochs; Training process: Epochs 1-20: Training set loss decreases rapidly from 0.5 to 0.1; Epochs 21-50: Training set loss decreases slowly from 0.1 to 0.05; Epochs 51-80: Training set loss stabilizes, fluctuating between 0.04 and 0.05; Epoch 81: Validation set loss begins to rise, triggering early stopping. Final model: Training set loss: 0.042; validation set loss: 0.048; test set loss: 0.051; Step 5: Network Verification: The initial grasping strategy was evaluated on the test set: grasping success rate: 73% (73 / 100); average grasping time: 4.2 seconds; position error: average 8 mm; posture error: average 6 degrees; force error: average 2.5 N; meeting the requirement of initial success rate ≥ 70%.

[0049] As can be seen, implementing this optional embodiment can improve the reliability and anthropomorphism of the initial grasping strategy by using a multimodal teaching dataset composed of human demonstration operations for training. This facilitates the direct transfer of efficient human grasping experience and skills to the robotic arm, making the initial grasping action of the robotic arm more natural and efficient, and enabling rapid startup based on human expert knowledge. It can extract demonstration data fragments of key stages from continuous trajectories, improving the condensation and representativeness of training data. This allows for focusing on the most decisive operational moments in the grasping task, reducing the interference of redundant data on model training, and thus improving the training efficiency and generalization ability of the imitation learning neural network, achieving more efficient learning of the essential grasping actions. By establishing an end-to-end mapping from object attributes (second feature vector) to grasping parameters (demonstration data), it improves the directness and automation of strategy generation, thus eliminating complex mechanical analysis and manual parameter tuning processes, and reducing the system's dependence on domain expert knowledge, achieving a data-driven, automated grasping strategy generation method.

[0050] 103. During the process of the robotic arm contacting the target object based on the first set of grasping parameters, real-time force feedback data between the target object and the robotic arm is collected; In this embodiment of the invention, optionally, during the execution of the initial grasping strategy, a high-precision force sensor can be used to monitor the interaction force between the robotic arm and the object in real time. This provides real, dynamic feedback signals for subsequent optimization, allowing for the assessment of the grasping strategy's effectiveness (e.g., whether excessive force is applied or whether slippage occurs).

[0051] Alternatively, as the robotic arm moves and contacts the object according to the "first grasping parameter set," a six-dimensional force / torque sensor mounted on the robotic arm's wrist or gripper continuously collects force and torque data in three directions at a high frequency. These raw force signals, after being filtered to remove noise, are encapsulated into "real-time force feedback data," which reflects the real-time state of the grasping action.

[0052] In this embodiment of the invention, optionally, the aforementioned visual sensor may be an RGB-D camera such as the Intel RealSense D435i; the aforementioned tactile sensor may be a sensor with pressure and vibration sensing capabilities such as OptoForce; the aforementioned force sensor may be a six-dimensional force / torque sensor such as the ATI Nano17; and the aforementioned robotic arm may be a 7-DOF collaborative robotic arm.

[0053] In another optional implementation of this invention, the process of collecting real-time force feedback data between the target object and the robotic arm during the contact process between the robotic arm and the target object based on the first grasping parameter set includes: By controlling the multi-axis force sensor installed on the end effector of the robotic arm, the original force and torque signals during the contact process are continuously collected. The original force and torque signals include the contact force and torque acting on the gripper in various directions. The original force and torque signals are filtered in real time to eliminate high-frequency noise, and force feedback characteristic parameters for characterizing the stability of the contact state are calculated. The force feedback characteristic parameters include at least the magnitude of the resultant force, the direction of the force vector, and the gradient of the torque change. The filtered original force, torque signal, and force feedback characteristic parameters are fused to form real-time force feedback data.

[0054] In this embodiment of the invention, optionally, the aforementioned force sensor (typically a six-axis force / torque sensor) can be mounted between the end flange and the gripper of the robotic arm. It is capable of simultaneously measuring forces (Fx, Fy, Fz) applied to it in three orthogonal directions and torques (Tx, Ty, Tz) in three directions. These raw signals are continuously sampled at frequencies up to hundreds or thousands of hertz throughout the entire process of the robotic arm contacting the object.

[0055] Alternatively, the original signal may contain high-frequency electronic noise and mechanical vibrations. In digital systems, digital filters (such as low-pass filters or moving average filters) are typically used for real-time filtering to retain only the low-frequency useful signals that reflect the actual contact force.

[0056] Further, optionally, the calculation of the above force feedback characteristic parameters includes: Magnitude of resultant force: Calculate the magnitude of the vector sum of forces in three directions, i.e., F_total = sqrt(Fx^2 + Fy^2 + Fz^2), which represents the total magnitude of the contact force.

[0057] Force vector direction: The direction of force can be determined by calculating the ratio of each component force to the resultant force or by directly calculating its spatial angle. This is crucial for determining whether the contact is normal.

[0058] Torque gradient: Calculates the rate of change of torque over time. Dramatic fluctuations in torque may indicate that an object is rotating or slipping.

[0059] Alternatively, the filtered six-dimensional force / torque raw data can be combined with the calculated derived feature parameters (such as resultant force, direction angle, and gradient) to form a richer and more comprehensive force feedback data package. This data package contains both raw information and high-level feature information, which together serve as "real-time force feedback data" provided to subsequent optimization modules.

[0060] As can be seen, implementing this optional embodiment can continuously acquire raw force and torque signals through multi-axis force sensors, improving the real-time and multi-dimensional perception of the interaction force between the robotic arm and the object. This facilitates the comprehensive capture of the dynamic changes of force / torque in three-dimensional space during the grasping process, thereby providing rich tactile feedback information for online optimization and achieving fine perception of the contact state. It can filter the raw signal in real time and calculate force feedback feature parameters, improving the quality and interpretability of the force feedback data. This helps to filter out sensor noise and mechanical vibration interference, extracting essential features reflecting grasping stability (such as resultant force, direction, and gradient), thus facilitating a more accurate assessment of the grasping state and realizing the transformation from raw signals to high-level semantic features. It can fuse the filtered raw signal with feature parameters, improving the information richness of the constructed real-time force feedback data. This allows for the simultaneous provision of low-level detailed information and high-level abstract indicators for the optimization algorithm, thus meeting the different input information requirements of different optimization modules (such as model-based control or model-free reinforcement learning) and achieving more compatible feedback information construction.

[0061] 104. Based on the reinforcement learning policy network and real-time force feedback data, the first grasping parameter set is optimized online to obtain the second grasping parameter set. The second grasping parameter set is used to control the robotic arm to perform grasping operations on the target object.

[0062] In this embodiment of the invention, optionally, the grasping task can be modeled as a reinforcement learning problem, whereby the agent (robotic arm) learns to adjust and optimize its grasping strategy based on environmental feedback (force signals, grasping results). The purpose is to enable the system to adapt to uncertainty and continuously improve grasping performance through repeated trials, ultimately obtaining a customized grasping strategy for the current object and environment that is superior to the initial strategy.

[0063] Optionally, a reinforcement learning policy network can be maintained. It takes the current state (including the "first feature vector," the "first grasping parameter set," real-time force feedback data, and information about the robotic arm's joints) as input. The policy network outputs an action representing a fine-tuning of the "first grasping parameter set" (e.g., adjusting the grasping position 1 mm to the left, or increasing the grasping force by 0.1 N). After executing this fine-tuned action, a reward signal is calculated based on the grasping result (success / failure, stability, and time). Through extensive experience with this "state-action-reward-new state" cycle, the policy network continuously updates its parameters, aiming to learn a policy that maximizes cumulative rewards. When the policy converges or meets performance requirements, the final grasping parameters generated by this policy become the optimized "second grasping parameter set." The reinforcement learning policy network can use a deep Q-network or a proximal policy optimization algorithm; its training process employs an experience replay mechanism, and training can be stabilized by setting a target network.

[0064] Further optionally, after obtaining the second set of grasping parameters, the robotic arm is controlled to perform a grasping operation, wherein: a fifth-order polynomial interpolation can be used to plan the motion trajectory from the current position to the grasping point; a numerical iterative method (such as the damped least squares method) can be used to solve the inverse kinematics; and an impedance control algorithm or a force PID controller can be used to implement compliant force control during the grasping process.

[0065] In this embodiment of the invention, optionally, the robotic arm adaptive gripping system of the present invention has broad industrial application value and can be applied to the following fields: 1. Industrial assembly: assembly of electronic products, assembly of automotive parts, assembly of precision instruments, etc. 2. Logistics sorting: express sorting, warehouse picking, cargo handling, etc. 3. Household services: chores assistance, organization, elderly care, etc. 4. Agricultural picking: fruit picking, vegetable picking, flower picking, etc. 5. Medical assistance: Surgical instrument delivery, drug sorting, sample processing, etc.

[0066] As can be seen, implementing the embodiments of the present invention can improve sample training efficiency by integrating imitation learning and reinforcement learning paradigms, thereby shortening the actual interaction time required for the robotic arm to acquire an effective grasping strategy. This, in turn, helps reduce wear and tear on hardware and energy consumption, achieving efficient and energy-saving system training. It can generate high-quality initial grasping strategies (first grasping parameter set) through imitation learning neural networks, improving the starting point and safety of strategy exploration. This helps avoid numerous random, invalid, or even dangerous grasping attempts when reinforcement learning starts from scratch, thus protecting the robotic arm and the target object and achieving safe and reliable learning initialization. It can optimize the initial strategy online based on real-time force feedback data, improving the adaptability and precision of the grasping strategy. This allows the robotic arm to dynamically adjust the grasping force, position, and posture according to real physical interactions, thus helping to cope with uncertainties such as object slippage and deformation, achieving robust grasping of dynamic environments and different objects, and improving the adaptability and flexibility of the robotic arm's grasping. It can break down the crawling process into sequential stages of perception, initial strategy generation, and online optimization, improving the modularity and logical clarity of the system design. This facilitates the independent development and debugging of each module, thereby enhancing the maintainability and scalability of the entire system and achieving a solution with a clear structure that is easy to implement in engineering.

[0067] As an optional implementation, the above-mentioned reinforcement learning policy network and real-time force feedback data are used to optimize the first grasping parameter set online to obtain a second grasping parameter set, including: The first feature vector, the first set of grasping parameters, real-time force feedback data, and the current state information of the robotic arm are fused to construct a state vector that characterizes the current grasping environment. The state vector is processed by a reinforcement learning policy network to generate an action vector for fine-tuning the first set of grasping parameters. The action vector corresponds to the first set of grasping parameters. The first set of grabbing parameters is updated based on the action vector to obtain a set of candidate grabbing parameters and a grabbing attempt is performed. At the same time, a reward signal is calculated based on the grabbing result. The reward signal is used to comprehensively evaluate the grabbing success rate, stability and efficiency. Based on the reward signal, the candidate crawling parameter set is optimized online to obtain the second crawling parameter set.

[0068] In this embodiment of the invention, optionally, the "state" in reinforcement learning needs to comprehensively describe the current state of the environment. Therefore, multiple information sources are integrated: Static property of an object: "First eigenvector".

[0069] Current execution strategy: "First set of fetch parameters" (or its current optimized version).

[0070] Dynamic interactive information: Filtered raw force / torque data and its characteristic parameters (from the above embodiments).

[0071] Robot body information: current angles, speeds, etc. of each joint of the robotic arm.

[0072] After normalizing these heterogeneous data separately, they are concatenated into a longer one-dimensional vector, which is the "state vector".

[0073] Alternatively, the policy network (e.g., a deep neural network) can take a pre-constructed "state vector" as input. Its output layer is designed to correspond to the dimensions of the grasping parameters that need to be optimized. For example, the output could be a seven-dimensional "action vector" containing three translation increments, three rotation increments, and one grasping force increment. This action vector is not an absolute grasping parameter, but rather an adjustment to the current set of grasping parameters.

[0074] Alternatively, the motion vector (fine-tuning amount) can be added to the current set of grasping parameters to obtain a new set of "candidate grasping parameters". The robotic arm then immediately attempts to perform a grasping motion using this new set of parameters.

[0075] Optionally, after the grasping attempt, a scalar reward value is calculated based on the result. The design of the reward function is crucial; for example: Successful grasp: A large positive reward is given if the object is successfully grasped and held for a period of time. Grasping stability: Additional rewards are given based on the smoothness (small fluctuations) of the force signal during the grasping process. Grasping efficiency: The shorter the time taken to complete the grasp, the higher the reward. Penalty: A large negative reward is given for collisions, object slippage, or grasping failure. This reward signal guides the reinforcement learning agent to learn towards "fast, stable, and successful" grasping.

[0076] As can be seen, implementing this optional embodiment can improve the completeness of environmental representation by constructing a state vector by fusing the first feature vector, grasping parameters, force feedback, and robotic arm state. This allows for the inclusion of object attributes, current strategy, real-time interaction, and ontology state, providing a comprehensive decision-making basis for strategy optimization and achieving panoramic modeling of the grasping environment. It can generate action vectors for fine-tuning through a policy network, improving the precision and controllability of strategy optimization. This facilitates small, gradual improvements to the initial grasping strategy rather than a complete overhaul, ensuring the stability and convergence of the optimization process and achieving robust online strategy iteration. Furthermore, it can calculate a comprehensive reward signal based on the grasping results, improving the guidance of the optimization direction. This allows for the simultaneous balancing of multiple performance indicators such as grasping success rate, stability, and efficiency, guiding the strategy towards optimal overall performance and achieving multi-objective balanced grasping strategy optimization.

[0077] In this optional embodiment, as an optional implementation, the above-mentioned online optimization of the candidate crawling parameter set based on the reward signal to obtain a second crawling parameter set includes: The state vector, action vector, reward signal, and new state vector obtained after the grasping trial are combined into a quadruple of experience data, and the experience data is stored as historical experience data in the experience replay buffer. A predetermined number of historical experience data are randomly sampled from the experience replay buffer, and the temporal difference error of the current reinforcement learning policy network is calculated based on the historical experience data. The policy loss function is constructed based on the temporal difference error, and the parameters of the reinforcement learning policy network are updated through the backpropagation algorithm to minimize the policy loss function. The iterative execution of the operation of constructing a quadruple of empirical data from the state vector, action vector, reward signal, and new state vector obtained after the grasping trial, and storing the empirical data as historical empirical data in the experience replay buffer, and the operation of updating the parameters of the reinforcement learning policy network through the backpropagation algorithm to minimize the policy loss function, until the reward signal corresponding to the new action vector output by the reinforcement learning policy network converges to the stable interval, the optimal grasping parameter set generated by the reinforcement learning policy network at this time is determined as the second grasping parameter set.

[0078] In this embodiment of the invention, optionally, each grabbing attempt generates one piece of experience data, in the format (s, a, r, s'), where s is the current state, a is the action performed, r is the reward obtained, and s' is the new state entered after performing the action. These experiences are stored in a fixed-size first-in-first-out (FIFO) buffer.

[0079] Further, alternatively, in order to break the correlation between data and improve training stability, instead of using experience sequentially, a small batch of historical experience is randomly drawn from the buffer.

[0080] Temporal difference error (TD) is a core concept in reinforcement learning. It evaluates the accuracy of the current policy network's estimate of the action's value. The specific calculation typically involves a target network used for stabilization training. The error is calculated as: TD Error = Target Q-value - Current Q-value, where the target Q-value is calculated based on the target network, and the current Q-value is calculated based on the current policy network.

[0081] Alternatively, the square or absolute value of the temporal difference error is commonly used as the loss function. The gradient of the loss function with respect to the policy network parameters is calculated using the backpropagation algorithm, and then the network parameters are fine-tuned using gradient descent. The goal is to reduce the TD error and make the network's prediction of action value more accurate.

[0082] Optionally, the above process (collecting experience, storing, sampling, calculating errors, and updating the network) is continuously repeated. With the accumulation of experience and the network updates, the quality of the actions output by the policy network increases, and the rewards become larger. When, after multiple consecutive attempts, the reward signal no longer increases significantly but stabilizes at a relatively high level, the policy is considered to have "converged." At this point, the set of grasping parameters generated by the policy network is the optimal "second grasping parameter set" after online optimization.

[0083] In this embodiment of the invention, optionally, the reinforcement learning policy network uses a deep Q-network algorithm: State space definition: The state vector s (dimension 40) includes: object attributes: 20 dimensions, robotic arm joint angles: 7 dimensions, end position: 3 dimensions, end pose: 4 dimensions (quaternion), force feedback: 6 dimensions (3-axis force + 3-axis torque).

[0084] Action space definition: Action vector a (dimension 8) includes: gripping position increment: 3-dimensional, range ±10mm; gripping posture increment: 3-dimensional, range ±10 degrees; gripping force increment: 1-dimensional, range ±5N; gripper opening and closing degree increment: 1-dimensional, range ±10mm.

[0085] The action space is a continuous space. Action discretization is used, with each dimension discretized into 5 values. The total number of actions is 5^8 = 390625 (too large).

[0086] Improvement solution: Adopt the Actor-Critic architecture, where the Actor network outputs continuous actions.

[0087] Reward function design: The reward function r(s, a) includes: Successful grasp reward: +10 for successfully grasping and holding for 3 seconds; failure: 0; Grasping stability reward: calculated based on the standard deviation of force feedback; the smaller the standard deviation of force feedback, the higher the reward; Grasping efficiency reward: calculated based on grasping time; the shorter the grasping time, the higher the reward; Collision penalty, such as the robotic arm colliding with the environment; Slip penalty, such as an object slipping from the gripper; Total reward = successful grasp reward + grasping stability reward + grasping efficiency reward + collision penalty + slip penalty.

[0088] Q-network structure: The input of the Q-network is the state vector s (40-dimensional) and the action vector a (8-dimensional), and the output is the Q value (1-dimensional).

[0089] Network structure: Input layer: 48 neurons (40+8); Hidden layer 1: 512 neurons, ReLU activation function; Hidden layer 2: 256 neurons, ReLU activation function; Hidden layer 3: 128 neurons, ReLU activation function; Hidden layer 4: 64 neurons, ReLU activation function; Output layer: 1 neuron (Q-value), linear activation.

[0090] Training process: Training settings: Experience replay buffer capacity: 10000; Batch size: 64; Learning rate: 0.0001; Discount factor γ: 0.99; Exploration rate ε: decays from 1.0 to 0.1, decaying in 5000 steps; Target network update frequency: every 100 steps; Total training steps: 10000; Training Process: Step 1: Initialize the Q-network and target Q-network with identical parameters; Step 2: Initialize the experience replay buffer; Step 3: For each episode: Reset the environment and randomly place objects; Obtain the initial state s; For each time step t: Select action a according to the ε-greedy policy; Execute action a, observe the reward r and the next state s'; Store (s, a, r, s') in the experience replay buffer; Randomly sample 64 samples from the buffer; Calculate the target Q-value; Calculate the loss; Update the Q-network parameters; If the capture is successful or fails, end the episode; Update the target network every 100 steps; The training results, based on the experiment, are as follows: Steps 0-2000: Success rate increased from 73% to 80%; Steps 2000-5000: Success rate increased from 80% to 88%; Steps 5000-10000: Success rate increased from 88% to 92%; Final success rate: 92%; Compared to pure reinforcement learning (which requires 100,000 steps to reach 90% from scratch), this invention only requires 10,000 steps to reach 92%, improving training efficiency by 10 times.

[0091] As can be seen, implementing this optional embodiment can improve the utilization efficiency of training data and break the temporal correlation between data by storing experience replay buffers and randomly sampling historical experience data. This is beneficial for improving the stability and convergence speed of training, and thus helps to avoid the policy getting stuck in local optima due to continuous similar failure experiences, achieving stable and efficient offline policy learning. The network can be updated by calculating temporal difference errors and minimizing the policy loss function, improving the accuracy and directionality of policy network parameter updates. This helps the policy network estimate the value of actions more closely to the true long-term reward, thereby guiding the policy to improve towards maximizing long-term cumulative rewards, achieving incremental policy optimization based on value estimation. By iteratively executing the process of experience collection, storage, and learning until convergence, the performance optimality of the final second grasping parameter set can be improved. This helps to ensure that the online optimization process can fully utilize historical interaction experience, continuously approaching the optimal grasping strategy for a specific object in the current environment, achieving autonomous performance improvement of the grasping strategy.

[0092] In another alternative embodiment, the method described above further includes: After the robotic arm completes the grasping operation based on the second grasping parameter set, the target object displacement information and time information are determined. The target object displacement information and time information correspond to the range of steps from when the robotic arm contacts the target object based on the first grasping parameter set to when the grasping operation is completed. The displacement information, time information, and force feedback characteristic parameters of the target object are compared with their respective preset thresholds to obtain the comparison results. Based on the comparison results, a policy optimization instruction is generated. The policy optimization instruction is used to add the first feature vector, the second set of grasping parameters, and the corresponding grasping result data corresponding to this grasping task to the training dataset of the imitation learning neural network and / or adjust the network parameters of the reinforcement learning policy network.

[0093] In this embodiment of the invention, optionally, the target object displacement information is determined by comparing the positional change of the object before grasping (positioned by the vision system) and after successful grasping (e.g., after the object is placed at the target position). This determines whether the grasping was successful and the degree of success (e.g., whether any undue slippage occurred).

[0094] Time information: Records the total time spent from the start of the grasping action (e.g., from the "approaching object" stage) to the completion of the grasp (e.g., the object is lifted to the predetermined height).

[0095] Alternatively, the system can preset some performance thresholds. For example: displacement threshold (to determine whether the capture was successful), time threshold (to determine whether it is efficient), and force fluctuation threshold (to determine whether it is stable).

[0096] Compare the measured values ​​with thresholds. For example: Is the object displacement greater than the success threshold (if so, it's successful)? Is the total time less than the efficiency threshold? Is the force fluctuation less than the stability threshold? The comparison result is a set of Boolean values ​​or scores used to quantify the overall performance of this capture.

[0097] Further, optionally, a decision can be made based on the comparison results, generating a "strategy optimization instruction": If the grasping is highly successful and efficient, the (first feature vector, second grasping parameter set) can be added as a new successful sample to the training dataset of the imitation learning network. This continuously enriches the knowledge base of the pre-trained model, enabling it to handle more objects.

[0098] If the capture is successful but certain metrics (such as stability) are poor, the reward function of the reinforcement learning policy network can be adjusted. For example, increasing the reward weight for stability can guide the network to pay more attention to stability in subsequent learning.

[0099] Alternatively, the successful experience data (as a high-quality experience) can be directly used to update the parameters of the reinforcement learning policy network itself.

[0100] This closed loop ensures that the system can not only optimize online in a single task, but also continuously learn from experience throughout its lifecycle, becoming increasingly "intelligent" and adaptable.

[0101] As can be seen, implementing this optional embodiment can improve the objectivity and quantification of the performance evaluation of a single grasping task by determining whether the grasping is successful, recording the time taken, and comparing the fluctuation indicators of computing power with thresholds. This facilitates a multi-dimensional and quantifiable accurate evaluation of the grasping effect, providing clear directions for strategy optimization and achieving a data-driven scientific evaluation of grasping performance. It can generate strategy optimization instructions based on the evaluation results, add successful data to the imitation learning dataset or adjust the parameters of the reinforcement learning network, improving the system's continuous learning ability and knowledge accumulation. This helps to transform successful grasping experience into permanent knowledge, enrich the initial strategy library, or adjust online optimization criteria, enabling the system to learn from each grasping attempt and achieve lifelong growth and continuous enhancement of system performance and adaptability. It can form a complete closed loop of "perception-decision-execution-evaluation-optimization," improving the intelligence and autonomy of the entire system. This helps to reduce dependence on external intervention, allowing the system to self-improve in long-term operation and better adapt to unknown challenges such as new objects and new environments.

[0102] In this embodiment of the invention, optionally, the specific process of grasping various everyday objects using this embodiment of the invention is as follows: Test object: Choose 50 everyday objects and divide them into 5 categories: Rigid objects (10 types): metal cups, glass bottles, wooden blocks, plastic boxes, etc.; Flexible objects (10 types): sponges, towels, plastic bags, rubber balls, etc.; Fragile objects (10 types): eggs, fruits, ceramic cups, light bulbs, etc.; Irregular objects (10 types): pliers, scissors, keys, pens, etc.; Large objects (10 types): books, shoe boxes, basketballs, pillows, etc.

[0103] Test process: Perform 10 grasping tests on each object and record the success rate, grasping time, and object damage rate.

[0104] Step 1: Object Placement: Randomly place the object on the workbench, with random position and orientation.

[0105] Step 2: Object Perception: A depth camera acquires point cloud data to identify the position and shape of objects. A tactile sensor measures hardness and roughness.

[0106] Step 3: Strategy Generation: Generate grasping parameters based on the object's attributes and the optimized grasping strategy.

[0107] Step 4: Grasping Execution: The robotic arm performs the grasping action, and the force feedback control module adjusts the grasping force in real time.

[0108] Step 5: Result Evaluation: Determine whether the grasp was successful, and record the grasp time and object status.

[0109] The experimental results are as follows: Overall results (50 objects, 10 times each, 500 times in total): Success rate: 91.2% (456 / 500); Average grasping time: 4.5 seconds; Object damage rate: 3.8% (19 / 500). Classification results: Rigid objects: 96% success rate, average time 4.2 seconds, 0% damage rate; Flexible objects: 88% success rate, average time 5.1 seconds, 2% damage rate; Fragile objects: 85% success rate, average time 4.8 seconds, 8% damage rate; Irregular objects: 92% success rate, average time 4.3 seconds, 1% damage rate; Large objects: 95% success rate, average time 4.8 seconds, 0% damage rate. Comparative experiment: Using the same test with a pure imitation learning method (without reinforcement learning optimization): Grasping success rate: 68% (340 / 500); Average grasping time: 5.2 seconds; Object damage rate: 12% (60 / 500). Experiments show that, compared with pure imitation learning methods, the present invention improves the grasping success rate by 34% (from 68% to 91.2%) and reduces the object damage rate by 68% (from 12% to 3.8%).

[0110] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of a robotic arm adaptive grasping system based on reinforcement learning and imitation learning, disclosed in an embodiment of the present invention. This robotic arm adaptive grasping system based on reinforcement learning and imitation learning can be applied to robots, and also to intelligent devices associated with robots. These intelligent devices include, but are not limited to, one or more of the following: switching devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. It is suitable for application scenarios such as industrial assembly, logistics sorting, home services, and agricultural harvesting. The embodiments of the present invention do not limit its application. Figure 2 As shown, the adaptive grasping system for robotic arms based on reinforcement learning and imitation learning may include: The attribute-aware module 201 is used to construct the first feature vector of the target object based on the perception of the multi-dimensional attributes of the target object; The imitation learning control module 202 is used to generate a first set of grasping parameters for the robotic arm based on a pre-trained imitation learning neural network and a first feature vector. The first set of grasping parameters includes at least one of grasping position parameters, grasping posture parameters, and grasping force parameters. Force feedback module 203 is used to collect real-time force feedback data between the target object and the robotic arm during the process of the robotic arm contacting the target object based on the first grasping parameter set; The adaptive optimization control module 204 is used to optimize the first grasping parameter set online based on the reinforcement learning policy network and real-time force feedback data to obtain the second grasping parameter set. The second grasping parameter set is used to control the robotic arm to perform grasping operations on the target object.

[0111] As can be seen, implementing the embodiments of the present invention can improve sample training efficiency by integrating imitation learning and reinforcement learning paradigms, thereby shortening the actual interaction time required for the robotic arm to acquire an effective grasping strategy. This, in turn, helps reduce wear and tear on hardware and energy consumption, achieving efficient and energy-saving system training. It can generate high-quality initial grasping strategies (first grasping parameter set) through imitation learning neural networks, improving the starting point and safety of strategy exploration. This helps avoid numerous random, invalid, or even dangerous grasping attempts when reinforcement learning starts from scratch, thus protecting the robotic arm and the target object and achieving safe and reliable learning initialization. It can optimize the initial strategy online based on real-time force feedback data, improving the adaptability and precision of the grasping strategy. This allows the robotic arm to dynamically adjust the grasping force, position, and posture according to real physical interactions, thus helping to cope with uncertainties such as object slippage and deformation, achieving robust grasping of dynamic environments and different objects, and improving the adaptability and flexibility of the robotic arm's grasping. It can break down the crawling process into sequential stages of perception, initial strategy generation, and online optimization, improving the modularity and logical clarity of the system design. This facilitates the independent development and debugging of each module, thereby enhancing the maintainability and scalability of the entire system and achieving a solution with a clear structure that is easy to implement in engineering.

[0112] In this embodiment of the invention, as an optional implementation, the attribute perception module 201 constructs the first feature vector of the target object based on the perception of its multi-dimensional attributes in the following specific ways: The point cloud data of the target object is acquired by controlling the vision sensor corresponding to the robotic arm, and geometric attribute parameters for characterizing the macroscopic structure of the target object are extracted based on the point cloud data. The tactile sensors corresponding to the control robot arm contact the target object in a preset manner and collect pressure distribution signals and vibration signals during the contact process. Based on the pressure distribution signals, the hardness attribute parameters used to characterize the softness and hardness of the target object are obtained, and based on the vibration signals, the roughness attribute parameters used to characterize the surface texture of the target object are obtained. The first feature vector of the target object is constructed based on the geometric property parameters, hardness property parameters, and roughness property parameters.

[0113] As can be seen, implementing this optional embodiment can improve the comprehensiveness and redundancy of target object attribute perception by fusing visual sensor point cloud data and tactile sensor physical signals. This facilitates obtaining a comprehensive object description that transcends single vision, including geometry and physical material properties. Consequently, it provides a richer and more reliable information foundation for subsequent strategy generation, enabling more comprehensive environmental perception of objects. By actively contacting and measuring hardness and roughness through tactile sensors, the perception accuracy of key physical properties of objects (such as fragility and surface friction coefficient) can be improved. This helps distinguish objects with similar visual shapes but different materials (such as real apples and plastic apples), thus facilitating the development of differentiated grasping force strategies and enabling precise grasping based on the object's physical characteristics. Furthermore, by extracting multi-dimensional parameters such as geometry, hardness, and roughness and constructing a unified feature vector, the structure and processability of perceived information can be improved. This facilitates the transformation of heterogeneous sensor data into standardized input suitable for neural network processing, thereby improving the input quality and learning efficiency of subsequent imitation learning and reinforcement learning, and achieving efficient integration of perceived information and learning strategies.

[0114] In this embodiment of the invention, as another optional implementation, the specific method by which the above-mentioned imitation learning control module 202 generates the first grasping parameter set of the robotic arm based on the pre-trained imitation learning neural network and the first feature vector includes: Obtain a multimodal teaching dataset consisting of human demonstration operations. The multimodal teaching dataset includes a second feature vector, a gripper motion trajectory sequence, and a gripping force sequence corresponding to a successful grasping event. The second feature vector corresponds to the first feature vector. Extract a sequence of exemplary data segments from the gripper motion trajectory sequence and gripping force sequence to characterize the key stages of the gripping operation. The key stages include at least approaching the object, the moment of contact, stable gripping, and lifting movement. Using the second feature vector as input data and the gripper pose and grasping force contained in the corresponding demonstration data segment sequence as supervision labels, the preset imitation learning neural network is trained until convergence. The first feature vector is input into the imitation learning neural network to generate the first set of grasping parameters for the robotic arm.

[0115] As can be seen, implementing this optional embodiment can improve the reliability and anthropomorphism of the initial grasping strategy by using a multimodal teaching dataset composed of human demonstration operations for training. This facilitates the direct transfer of efficient human grasping experience and skills to the robotic arm, making the initial grasping action of the robotic arm more natural and efficient, and enabling rapid startup based on human expert knowledge. It can extract demonstration data fragments of key stages from continuous trajectories, improving the condensation and representativeness of training data. This allows for focusing on the most decisive operational moments in the grasping task, reducing the interference of redundant data on model training, and thus improving the training efficiency and generalization ability of the imitation learning neural network, achieving more efficient learning of the essential grasping actions. By establishing an end-to-end mapping from object attributes (second feature vector) to grasping parameters (demonstration data), it improves the directness and automation of strategy generation, thus eliminating complex mechanical analysis and manual parameter tuning processes, and reducing the system's dependence on domain expert knowledge, achieving a data-driven, automated grasping strategy generation method.

[0116] In this embodiment of the invention, as another optional implementation, the force feedback module 203 collects real-time force feedback data between the target object and the robotic arm during the process of the robotic arm contacting the target object based on the first grasping parameter set, specifically in the following ways: By controlling the multi-axis force sensor installed on the end effector of the robotic arm, the original force and torque signals during the contact process are continuously collected. The original force and torque signals include the contact force and torque acting on the gripper in various directions. The original force and torque signals are filtered in real time to eliminate high-frequency noise, and force feedback characteristic parameters for characterizing the stability of the contact state are calculated. The force feedback characteristic parameters include at least the magnitude of the resultant force, the direction of the force vector, and the gradient of the torque change. The filtered original force, torque signal, and force feedback characteristic parameters are fused to form real-time force feedback data.

[0117] As can be seen, implementing this optional embodiment can continuously acquire raw force and torque signals through multi-axis force sensors, improving the real-time and multi-dimensional perception of the interaction force between the robotic arm and the object. This facilitates the comprehensive capture of the dynamic changes of force / torque in three-dimensional space during the grasping process, thereby providing rich tactile feedback information for online optimization and achieving fine perception of the contact state. It can filter the raw signal in real time and calculate force feedback feature parameters, improving the quality and interpretability of the force feedback data. This helps to filter out sensor noise and mechanical vibration interference, extracting essential features reflecting grasping stability (such as resultant force, direction, and gradient), thus facilitating a more accurate assessment of the grasping state and realizing the transformation from raw signals to high-level semantic features. It can fuse the filtered raw signal with feature parameters, improving the information richness of the constructed real-time force feedback data. This allows for the simultaneous provision of low-level detailed information and high-level abstract indicators for the optimization algorithm, thus meeting the different input information requirements of different optimization modules (such as model-based control or model-free reinforcement learning) and achieving more compatible feedback information construction.

[0118] As an optional implementation, the adaptive optimization control module 204 described above optimizes the first grasping parameter set online based on the reinforcement learning policy network and real-time force feedback data to obtain the second grasping parameter set in the following specific ways: The first feature vector, the first set of grasping parameters, real-time force feedback data, and the current state information of the robotic arm are fused to construct a state vector that characterizes the current grasping environment. The state vector is processed by a reinforcement learning policy network to generate an action vector for fine-tuning the first set of grasping parameters. The action vector corresponds to the first set of grasping parameters. The first set of grabbing parameters is updated based on the action vector to obtain a set of candidate grabbing parameters and a grabbing attempt is performed. At the same time, a reward signal is calculated based on the grabbing result. The reward signal is used to comprehensively evaluate the grabbing success rate, stability and efficiency. Based on the reward signal, the candidate crawling parameter set is optimized online to obtain the second crawling parameter set.

[0119] As can be seen, implementing this optional embodiment can improve the completeness of environmental representation by constructing a state vector by fusing the first feature vector, grasping parameters, force feedback, and robotic arm state. This allows for the inclusion of object attributes, current strategy, real-time interaction, and ontology state, providing a comprehensive decision-making basis for strategy optimization and achieving panoramic modeling of the grasping environment. It can generate action vectors for fine-tuning through a policy network, improving the precision and controllability of strategy optimization. This facilitates small, gradual improvements to the initial grasping strategy rather than a complete overhaul, ensuring the stability and convergence of the optimization process and achieving robust online strategy iteration. Furthermore, it can calculate a comprehensive reward signal based on the grasping results, improving the guidance of the optimization direction. This allows for the simultaneous balancing of multiple performance indicators such as grasping success rate, stability, and efficiency, guiding the strategy towards optimal overall performance and achieving multi-objective balanced grasping strategy optimization.

[0120] In this optional embodiment, as an optional implementation method, the adaptive optimization control module 204 optimizes the candidate crawling parameter set online based on the reward signal to obtain the second crawling parameter set in the following specific ways: The state vector, action vector, reward signal, and new state vector obtained after the grasping trial are combined into a quadruple of experience data, and the experience data is stored as historical experience data in the experience replay buffer. A predetermined number of historical experience data are randomly sampled from the experience replay buffer, and the temporal difference error of the current reinforcement learning policy network is calculated based on the historical experience data. The policy loss function is constructed based on the temporal difference error, and the parameters of the reinforcement learning policy network are updated through the backpropagation algorithm to minimize the policy loss function. The iterative execution of the operation of constructing a quadruple of empirical data from the state vector, action vector, reward signal, and new state vector obtained after the grasping trial, and storing the empirical data as historical empirical data in the experience replay buffer, and the operation of updating the parameters of the reinforcement learning policy network through the backpropagation algorithm to minimize the policy loss function, until the reward signal corresponding to the new action vector output by the reinforcement learning policy network converges to the stable interval, the optimal grasping parameter set generated by the reinforcement learning policy network at this time is determined as the second grasping parameter set.

[0121] As can be seen, implementing this optional embodiment can improve the utilization efficiency of training data and break the temporal correlation between data by storing experience replay buffers and randomly sampling historical experience data. This is beneficial for improving the stability and convergence speed of training, and thus helps to avoid the policy getting stuck in local optima due to continuous similar failure experiences, achieving stable and efficient offline policy learning. The network can be updated by calculating temporal difference errors and minimizing the policy loss function, improving the accuracy and directionality of policy network parameter updates. This helps the policy network estimate the value of actions more closely to the true long-term reward, thereby guiding the policy to improve towards maximizing long-term cumulative rewards, achieving incremental policy optimization based on value estimation. By iteratively executing the process of experience collection, storage, and learning until convergence, the performance optimality of the final second grasping parameter set can be improved. This helps to ensure that the online optimization process can fully utilize historical interaction experience, continuously approaching the optimal grasping strategy for a specific object in the current environment, achieving autonomous performance improvement of the grasping strategy.

[0122] In another alternative embodiment, such as Figure 3 As shown, the system also includes: The strategy optimization module 205 is used to determine the target object displacement information and time information after the robotic arm completes the grasping operation based on the second grasping parameter set. The target object displacement information and time information correspond to the range of steps from when the robotic arm contacts the target object based on the first grasping parameter set to when the grasping operation is completed. The strategy optimization module 205 is also used to compare the target object displacement information, time information and force feedback characteristic parameters with their respective preset thresholds to obtain comparison results. The strategy optimization module 205 is also used to generate strategy optimization instructions based on the comparison results. The strategy optimization instructions are used to add the first feature vector, the second set of grasping parameters, and the corresponding grasping result data corresponding to this grasping task to the training dataset of the imitation learning neural network and / or adjust the network parameters of the reinforcement learning strategy network.

[0123] As can be seen, implementing this optional embodiment can improve the objectivity and quantification of the performance evaluation of a single grasping task by determining whether the grasping is successful, recording the time taken, and comparing the fluctuation indicators of computing power with thresholds. This facilitates a multi-dimensional and quantifiable accurate evaluation of the grasping effect, providing clear directions for strategy optimization and achieving a data-driven scientific evaluation of grasping performance. It can generate strategy optimization instructions based on the evaluation results, add successful data to the imitation learning dataset or adjust the parameters of the reinforcement learning network, improving the system's continuous learning ability and knowledge accumulation. This helps to transform successful grasping experience into permanent knowledge, enrich the initial strategy library, or adjust online optimization criteria, enabling the system to learn from each grasping attempt and achieve lifelong growth and continuous enhancement of system performance and adaptability. It can form a complete closed loop of "perception-decision-execution-evaluation-optimization," improving the intelligence and autonomy of the entire system. This helps to reduce dependence on external intervention, allowing the system to self-improve in long-term operation and better adapt to unknown challenges such as new objects and new environments.

[0124] Example 3 Please see Figure 4 , Figure 4 This is a schematic diagram of another adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning, disclosed in an embodiment of the present invention. This adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning can be applied to robots, and also to intelligent devices associated with robots. These intelligent devices include, but are not limited to, one or more of the following: switching devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. It is suitable for application scenarios such as industrial assembly, logistics sorting, home services, and agricultural harvesting. The embodiments of the present invention do not limit its application. Figure 4 As shown, the adaptive grasping system for robotic arms based on reinforcement learning and imitation learning may include: Memory 301 that stores executable program code.

[0125] Processor 302 coupled to memory 301.

[0126] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the adaptive grasping method of the robotic arm based on reinforcement learning and imitation learning described in Embodiment 1 of the present invention.

[0127] Example 4 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the adaptive grasping method for a robotic arm based on reinforcement learning and imitation learning described in Embodiment 1 of this invention.

[0128] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0129] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0130] Finally, it should be noted that the adaptive grasping method and system for robotic arms based on reinforcement learning and imitation learning disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive grasping method for robotic arms based on reinforcement learning and imitation learning, characterized in that, The method includes: Based on the perception of the multi-dimensional attributes of the target object, a first feature vector of the target object is constructed; Based on the pre-trained imitation learning neural network and the first feature vector, a first set of grasping parameters for the robotic arm is generated. The first set of grasping parameters includes at least one of grasping position parameters, grasping posture parameters, and grasping force parameters. During the process of the robotic arm contacting the target object based on the first set of grasping parameters, real-time force feedback data between the target object and the robotic arm is collected; Based on the reinforcement learning policy network and the real-time force feedback data, the first grasping parameter set is optimized online to obtain a second grasping parameter set, which is used to control the robotic arm to perform a grasping operation on the target object.

2. The adaptive grasping method for robotic arms based on reinforcement learning and imitation learning according to claim 1, characterized in that, The construction of the first feature vector of the target object based on the perception of its multi-dimensional attributes includes: The point cloud data of the target object is acquired by controlling the vision sensor corresponding to the robotic arm, and geometric attribute parameters for characterizing the macroscopic structure of the target object are extracted based on the point cloud data. The tactile sensor corresponding to the robotic arm is controlled to contact the target object in a preset manner, and pressure distribution signals and vibration signals are collected during the contact process. Based on the pressure distribution signals, a hardness attribute parameter is obtained to characterize the softness and hardness of the target object, and based on the vibration signals, a roughness attribute parameter is obtained to characterize the surface texture of the target object. Based on the geometric attribute parameters, the hardness attribute parameters, and the roughness attribute parameters, a first feature vector of the target object is constructed.

3. The adaptive grasping method for robotic arms based on reinforcement learning and imitation learning according to claim 1 or 2, characterized in that, The first set of grasping parameters for the robotic arm, generated based on a pre-trained imitation learning neural network and the first feature vector, includes: A multimodal teaching dataset consisting of human demonstration operations is obtained. The multimodal teaching dataset includes a second feature vector corresponding to a successful grasping event, a gripper motion trajectory sequence, and a grasping force sequence. The second feature vector corresponds to the first feature vector. Extract a sequence of exemplary data segments from the gripper motion trajectory sequence and the gripping force sequence to characterize key stages of the gripping operation. The key stages include at least approaching the object, the moment of contact, stable gripping, and lifting movement. Using the second feature vector as input data, and the gripper pose and grasping force contained in the corresponding demonstration data segment sequence as supervision labels, the preset imitation learning neural network is trained until convergence. The first feature vector is input into the imitation learning neural network to generate the first set of grasping parameters for the robotic arm.

4. The adaptive grasping method for robotic arms based on reinforcement learning and imitation learning according to claim 1, characterized in that, During the process of the robotic arm contacting the target object based on the first grasping parameter set, real-time force feedback data between the target object and the robotic arm is collected, including: The original force and torque signals during the contact process are continuously collected by controlling the multi-axis force sensor installed on the end effector of the robotic arm. The original force and torque signals include the contact force and torque acting on the gripper in various directions. The original force and torque signals are filtered in real time to eliminate high-frequency noise, and force feedback characteristic parameters for characterizing the stability of the contact state are calculated. The force feedback characteristic parameters include at least the magnitude of the resultant force, the direction of the force vector, and the gradient of the torque change. The filtered original force and torque signals, along with the force feedback characteristic parameters, are fused to form the real-time force feedback data.

5. The adaptive grasping method for robotic arms based on reinforcement learning and imitation learning according to claim 4, characterized in that, The first grasping parameter set is optimized online based on the reinforcement learning policy network and the real-time force feedback data to obtain the second grasping parameter set, including: The first feature vector, the first grasping parameter set, the real-time force feedback data, and the current state information of the robotic arm are fused to construct a state vector that characterizes the current grasping environment. The state vector is processed by a reinforcement learning policy network to generate an action vector for fine-tuning the first grasping parameter set, the action vector corresponding to the first grasping parameter set; The first set of grabbing parameters is updated according to the action vector to obtain a set of candidate grabbing parameters and grabbing attempts are performed. At the same time, a reward signal is calculated based on the grabbing results. The reward signal is used to comprehensively evaluate the grabbing success rate, stability and efficiency. Based on the reward signal, the candidate crawling parameter set is optimized online to obtain a second crawling parameter set.

6. The adaptive grasping method for robotic arms based on reinforcement learning and imitation learning according to claim 5, characterized in that, The step of optimizing the candidate crawling parameter set online based on the reward signal to obtain a second crawling parameter set includes: The state vector, the action vector, the reward signal, and the new state vector obtained after the grasping attempt are combined to form a quadruple of experience data, and the experience data is stored as historical experience data in the experience replay buffer. A preset number of historical experience data are randomly sampled from the experience replay buffer, and the temporal difference error of the current reinforcement learning policy network is calculated based on the historical experience data. A policy loss function is constructed based on the temporal difference error, and the parameters of the reinforcement learning policy network are updated through the backpropagation algorithm to minimize the policy loss function. The process iteratively executes the operation of constructing a quadruple of experience data from the state vector, the action vector, the reward signal, and the new state vector obtained after the grasping trial, and storing the experience data as historical experience data in the experience replay buffer, until the operation of updating the parameters of the reinforcement learning policy network through the backpropagation algorithm to minimize the policy loss function, until the reward signal corresponding to the new action vector output by the reinforcement learning policy network converges to a stable interval. At this point, the optimal grasping parameter set generated by the reinforcement learning policy network is determined as the second grasping parameter set.

7. The adaptive grasping method for robotic arms based on reinforcement learning and imitation learning according to claim 6, characterized in that, The method further includes: After the robotic arm completes the grasping operation based on the second grasping parameter set, the target object displacement information and the time information are determined. The target object displacement information and the time information correspond to the range of steps from when the robotic arm contacts the target object based on the first grasping parameter set to when the grasping operation is completed. The target object displacement information, the time information, and the force feedback feature parameters are compared with their respective preset thresholds to obtain the comparison results. Based on the comparison results, a policy optimization instruction is generated. The policy optimization instruction is used to add the first feature vector, the second set of grasping parameters, and the corresponding grasping result data corresponding to this grasping task to the training dataset of the imitation learning neural network and / or adjust the network parameters of the reinforcement learning policy network.

8. An adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning, characterized in that, The system includes: The attribute-aware module is used to construct the first feature vector of the target object based on the perception of its multi-dimensional attributes. The imitation learning control module is used to generate a first set of grasping parameters for the robotic arm based on a pre-trained imitation learning neural network and the first feature vector. The first set of grasping parameters includes at least one of grasping position parameters, grasping posture parameters, and grasping force parameters. The force feedback module is used to collect real-time force feedback data between the target object and the robotic arm during the process of the robotic arm contacting the target object based on the first grasping parameter set; An adaptive optimization control module is used to optimize the first grasping parameter set online based on the reinforcement learning policy network and the real-time force feedback data to obtain a second grasping parameter set. The second grasping parameter set is used to control the robotic arm to perform a grasping operation on the target object.

9. An adaptive grasping system for a robotic arm based on reinforcement learning and imitation learning, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the adaptive grasping method for robotic arms based on reinforcement learning and imitation learning as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the adaptive grasping method for a robotic arm based on reinforcement learning and imitation learning as described in any one of claims 1-7.

Images