Small sample-based intelligent robot grasping learning method and system

Abstract

The application relates to the technical field of intelligent robot control, and particularly discloses a small-sample-based intelligent robot grasping learning method and system, wherein the method first collects a small amount of multimodal teaching samples containing RGB images, depth maps and tactile snapshots to construct an initial teaching set; an encoder is pre-trained by using synthetic data, and fine-tuned by using real teaching samples to generate a robust prototype set; a grasping evaluation module generates a pixel-level grasping confidence field and an uncertainty field by calculating the similarity between real-time observation and the prototype set; physically consistent virtual samples are generated based on the uncertainty field to expand the training data; the small sample set after enhancement is used to quickly and adaptively fine-tune the grasping model under a meta-learning framework; finally, the optimal grasping pose is solved by combining physical constraints, and closed-loop learning is realized through online tactile exploration and feedback. The application significantly improves the grasping success rate and robustness of the robot in a small-sample real scene.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot control technology, specifically relating to a grasping learning method and system for intelligent robots based on small sample sizes. Background Technology

[0002] Grasping operations by intelligent robots are a core technology in industrial manufacturing, home services, and other scenarios. The core requirement is the ability to quickly adapt to diverse objects in complex real-world environments, achieving high-precision and robust grasping. However, real-world scenarios present significant challenges due to the wide variation in the geometry and texture of target objects, complex environmental factors such as lighting changes and background interference, and the high cost and long processing time of collecting and labeling real-world grasping data, making it difficult to obtain large-scale, high-quality labeled samples. Therefore, few-shot learning has become a key breakthrough direction for the practical application of robot grasping technology.

[0003] Although existing technologies have made some progress in the field of robotic grasping, some shortcomings still exist under small sample constraints:

[0004] Traditional learning methods either rely on massive amounts of real-world labeled data for supervised training, which is extremely costly and difficult to cover diverse scenarios; or they rely solely on large-scale synthetic data pre-training, but lack an effective adaptation mechanism to real-world scenarios, resulting in significant domain bias in the transfer from simulation to reality, and a sharp decline in generalization performance with a small number of real-world teaching samples. Some methods attempt to directly fine-tune the pre-trained model, but this is prone to catastrophic forgetting and makes it difficult to balance model generality and scenario specificity under small sample conditions.

[0005] To address this issue, this application proposes a grasping learning method for intelligent robots based on small sample sizes. Summary of the Invention

[0006] The purpose of this invention is to provide a grasping learning method and system for intelligent robots based on small sample sizes, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] Few-shot-based intelligent robot grasping learning methods include:

[0009] S1. Multimodal small sample collection and teaching seed construction, used to collect a small number of teaching observations of the target object, and to label and format the teaching observations to obtain the initial teaching set S0;

[0010] The teaching observation includes RGB images, depth maps, and short-term tactile or force snapshots;

[0011] S2. Perform self-supervised or contrastive pre-training on the encoder using the synthetic grasping scene, and iteratively fine-tune the pre-trained encoder using the initial teaching set to map each teaching sample to the prototype embedding space to obtain the prototype set.

[0012] S3. The grasping evaluation module calculates the pixel-level grasping confidence field and corresponding uncertainty field based on the prototype set and the real-time RGB image-depth map to obtain the pixel-level confidence and uncertainty mapping.

[0013] S4. Based on the uncertainty field, key points are sampled in image areas where the uncertainty value is higher than a preset threshold, and physical consistent virtual observations are generated by weighting the uncertainty through lightweight physical rendering or simulation to expand the training samples and obtain an enhanced small sample set.

[0014] S5. Using the enhanced small sample set as task samples, the outer and inner loops of the grasping model composed of the encoder and grasping evaluation module are trained under the meta-learning framework, so that the grasping model can obtain an initialization that can quickly adapt to new teachings, and the grasping model after small sample adaptive fine-tuning is obtained through gradient update.

[0015] S6. Perform pixel-to-3D back-projection on the pixel-level grasping confidence field and the depth map, and solve and optimize the candidate grasping pose set based on shape, collision, and mechanical constraints. Then, perform short-term tactile and force sensing trials on the pixel-level confidence and write the success or failure results along with tactile feedback back as new teaching samples to update the teaching library and use it for the next round of adaptation, obtaining the final grasping execution result and the updated teaching library. Preferably, S1 includes:

[0016] For the same target, N observations (N ≥ 1 and N ≤ 10) are performed from multiple perspectives, and each observation simultaneously records an RGB image, the corresponding depth map, and one or more short-time tactile and force snapshots to describe the contact response of the object surface.

[0017] The initial teaching set S0 = {(Irgb, Id, It, y)}.

[0018] Preferably, S2 includes:

[0019] Pre-training is performed using large-scale self-supervised contrastive learning based on synthetic grasping scenarios, with contrastive loss (e.g., InfoNCE) or its variants as the loss function, to obtain an encoder robust to geometric and material changes. During fine-tuning, the parameter freezing ratio of the encoder backbone is kept below 80% to ensure that the prototype embedding space P remains general and domain-adaptive, resulting in a prototype set.

[0020] Preferably, S3 includes:

[0021] The encoder extracts the pixel embeddings observed in real time;

[0022] The capture evaluation module calculates the similarity between the pixel embedding and the prototype set and normalizes it to a continuous confidence score. Specifically, it calculates the similarity weight between each pixel and all prototype vectors and uses a weighted kernel interpolation or attention weighting mechanism to obtain the pixel-level confidence score. Based on the distribution of the similarity weights, it derives the pixel uncertainty and obtains the pixel-level confidence and uncertainty mapping.

[0023] Preferably, S4 includes:

[0024] Based on the uncertainty field U, M physically consistent virtual observations (M ranging from 5 to 50) are generated in each image region where the uncertainty value is higher than a preset threshold. During the generation process, depth and approximate mechanical parameters are retained so that the virtual observations and real teaching samples are merged according to uncertainty weight to form an enhanced small sample set S1, thereby reducing the domain bias caused by non-physical enhancement.

[0025] Preferably, S5 includes:

[0026] Gradient updates are performed using an inner-loop and outer-loop training strategy based on model-independent meta-learning or its lightweight variants.

[0027] The outer ring optimizes the crawling model across multiple tasks, enabling it to obtain initialization parameters that can quickly adapt to new tasks.

[0028] The learning rate and number of steps of the inner loop are used as adjustable hyperparameters. On the new teaching data provided by the target task, the initial parameters are updated with gradients to obtain the grasping model after small sample adaptive fine-tuning.

[0029] The learning rate and number of steps of the inner loop are used as adjustable hyperparameters to balance rapid adaptation and stability in small sample scenarios, thus obtaining the small sample adaptive model.

[0030] Preferably, S6 includes:

[0031] Based on the depth map and camera intrinsic parameters, geometric transformation is performed to obtain candidate 3D gripping points, thereby achieving pixel-to-3D back projection.

[0032] The solution of the grasping pose set Q is performed by local optimization in the form of constraint minimization. The objective function of local optimization includes minimizing the weighted sum of grasping torque, collision penalty and grasping uncertainty, so as to output executable poses sorted by confidence.

[0033] Short-term tactile and force sensing probes are performed on the candidate grasping poses according to confidence level from high to low. If the probe fails, the failed sample is recorded and added to the teaching library according to priority to enhance the learning of negative samples of the failed state. The judgment threshold of the probe is set based on the force change rate or absolute force value of the tactile and force sensors to determine whether the grasp is successful or slips out.

[0034] This invention also provides a small-sample-based intelligent robot grasping learning system, comprising:

[0035] The data acquisition and processing module is used to acquire multimodal teaching observations of the target object and perform annotation and formatting processing to obtain the initial teaching set;

[0036] The model pre-training and fine-tuning module pre-trains the encoder using a synthetic grasping scenario and iteratively fine-tunes the pre-trained encoder using the initial teaching set to map the teaching samples to the prototype embedding space to obtain a prototype set.

[0037] The capture evaluation module calculates the pixel-level capture confidence field and the corresponding uncertainty field based on the prototype set and real-time RGB-D observations.

[0038] The sample enhancement module samples key points in the high uncertainty region based on the uncertainty field and generates physically consistent virtual observations through physical rendering or simulation to expand the training samples and obtain an enhanced small sample set.

[0039] The meta-learning adaptive module uses the enhanced small sample set as task samples, trains the grasping model in the outer and inner loops under the meta-learning framework, and obtains the grasping model after small sample adaptation through gradient update.

[0040] The grasping execution and feedback module performs a three-dimensional back projection of the pixel-level grasping confidence field and depth map, solves the optimal grasping pose based on multiple constraints, performs short-term tactile / force sensory exploration, and writes the results back as new teaching samples to update the teaching library.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the grasping learning method of the intelligent robot based on few samples.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the grasping learning method of the intelligent robot based on few samples.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] (1) By combining synthetic pre-training with teaching fine-tuning and adopting a representation strategy of prototype conditional interpolation, this invention can generate a highly discriminative and robust embedding representation under the constraint of a small amount of real teaching, thereby reducing the probability of grasping errors caused by representation errors in real-world scenarios.

[0045] (2) In this invention, the continuous pixel-level confidence and uncertainty fields provide fine-grained information for grasping decisions, enabling subsequent sampling, pose optimization, and detection strategies to be made based on risk perception. Incorporating uncertainty penalties into pose optimization helps to proactively avoid high-risk areas, thereby reducing the risk of execution failure and object damage. Attached Figure Description

[0046] Figure 1 This is a flowchart of the grasping learning method for intelligent robots based on few samples according to the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0048] Example 1:

[0049] Please see Figure 1 As shown, the grasping learning method for intelligent robots based on few samples includes:

[0050] S1. Multimodal small sample collection and teaching seed construction, used to collect a small number of teaching observations of the target object, and to label and format the teaching observations to obtain the initial teaching set S0;

[0051] The teaching observation includes RGB images, depth maps, and short-term tactile or force snapshots;

[0052] In step S1, N observations (N ≥ 1 and N ≤ 10) of the same target from multiple perspectives are recorded. For each observation, an RGB image, the corresponding depth map, and one or more short-time tactile and force snapshots are recorded to describe the contact response of the object surface. Each observation constitutes a teaching sample.

[0053] The initial teaching set S0 consists of N teaching samples.

[0054] Each teaching sample is represented as {(Irgb, Id, It, y)};

[0055] Where Irgb represents visible light color observation (RGB), Id represents depth observation (Depth / point cloud), It represents tactile / force snapshot, and y represents teaching annotation (grasp annotation / label);

[0056] Furthermore, the data collected includes, but is not limited to: visual information (RGB images or videos), depth information (depth maps or point clouds), and short-term tactile / force snapshots (force / tactile features recorded at the moment of teaching contact).

[0057] During teaching, typical changes of the target object in a limited viewpoint and relative position are covered to ensure the representativeness of the teaching seed in the local representation space.

[0058] The collected observations are standardized (e.g., coordinate system alignment, time truncation, noise filtering and synchronization) and saved as teaching entries in a unified format to generate the initial teaching set S0.

[0059] Data collection can be conducted through manual teaching, remote operation teaching, or semi-automatic teaching trajectory recording, with the appropriate collection method chosen based on the safety and efficiency constraints of the application scenario.

[0060] S2, a priori synthetic pre-trained and self-supervised fine-tuned encoder, is used to perform self-supervised or contrastive pre-training of the encoder in a large-scale synthetic crawling scenario, and to perform a small number of iterative fine-tunings on the pre-trained encoder using the initial teaching set S0, so as to map each teaching sample to a robust prototype embedding space to obtain a prototype set P;

[0061] In step S2, firstly, a synthetic grasping scene is constructed: the synthetic grasping scene is generated by a computer graphics rendering engine or a physical simulation engine, and includes grasping configurations of various virtual objects under different lighting, viewing angles and backgrounds.

[0062] Secondly, pre-training with synthetic data is performed: a pre-trained encoder is performed using self-supervised contrastive learning based on the above synthetic scenarios (e.g., using the InfoNCE loss function or a variant thereof) to obtain a general visual representation capability that is robust to geometric and material changes.

[0063] Then, fine-tuning is performed using a few samples: the pre-trained encoder is fine-tuned iteratively using the initial teaching set S0 from step S1. A low learning rate or partial parameter freezing strategy is employed during fine-tuning (e.g., keeping no more than 80% of the parameters of the encoder backbone network frozen) to ensure the model maintains its generality while adapting to the real domain.

[0064] Finally, the prototype set is generated: each teaching sample in the initial teaching set S0 is input into the fine-tuned encoder and mapped to the embedding space to obtain the corresponding prototype vector. The set of all these prototype vectors constitutes the prototype set P.

[0065] The encoder serves as the core representation model in this method, and subsequent steps will be used to extract features from real-time observations.

[0066] Furthermore, the pre-training phase employs self-supervised contrastive learning or corresponding unsupervised representation learning paradigms to learn scale / viewpoint / illuminance-invariant features.

[0067] S3. The capture evaluation module generates pixel-level capture confidence field G and uncertainty field U based on prototype condition interpolation. Specifically, the capture evaluation module calculates pixel-level capture confidence field G and corresponding uncertainty field U (or confidence entropy representation) based on the prototype set P and real-time RGB image and depth map observations.

[0068] S3-1: Extracting Real-Time Pixel Embeddings: Input the real-time acquired RGB image and depth map (i.e., RGB-D observation) into the fine-tuned encoder obtained in step S2. Through the forward propagation of this encoder, a high-dimensional feature vector, called the pixel embedding e, is extracted for each pixel position (x, y) in the image. xy .

[0069] S3-2: Calculate the confidence field G based on prototype similarity: for each pixel embedding e xy Calculate its relationship with each prototype vector p in the prototype set P. i The similarity (e.g., cosine similarity) is then used. These similarities are then transformed into a continuous confidence value through a weighted fusion mechanism (e.g., weighted kernel interpolation or attention weighting). Specifically, all similarities are first normalized (e.g., using the softmax function) to obtain a set of weights, which represent the degree of matching between the current pixel and each prototype. Then, these weights are summed with the corresponding similarity values ​​to obtain the final grasp confidence G(x, y) for that pixel. The range of G(x, y) is mapped to [0, 1], with higher values ​​indicating a greater probability of successful grasping at that pixel location.

[0070] S3-3: Calculating the Uncertainty Field U Based on the Similarity Distribution: The normalized weight distribution obtained when calculating the grasp confidence field G directly reflects the certainty of the model's decision. Uncertainty U(x, y) is calculated based on this weight distribution. For example, the information entropy of this distribution can be calculated: if the weights are concentrated on one or a few prototypes (distribution peak), the entropy is low, and the uncertainty is low; if the weights are evenly distributed across multiple prototypes (distribution flat), the entropy is high, and the uncertainty is high. U(x, y) identifies unreliable regions of model prediction, representing the model's confidence level in predicting that pixel.

[0071] S3-4: Output pixel-level mapping: Repeat the above process for all pixels in the image to obtain two two-dimensional fields with the same resolution as the input image: capture confidence field G (continuous probability mapping) and uncertainty field U (used to identify areas that need to be enhanced or handled with care).

[0072] S3-4: Output pixel-level mapping: Repeat the above process for all pixels in the image to obtain two two-dimensional fields with the same resolution as the input image: capture confidence field G (continuous probability mapping) and uncertainty field U (used to identify areas that need to be enhanced or handled with care).

[0073] S4. Uncertainty-driven physical consistency enhancement is used to sample key points in image regions with uncertainty values ​​higher than a preset threshold based on the uncertainty field U, and to expand the training samples by generating physically consistent virtual observations by weighting the uncertainty through lightweight physical rendering or simulation, thereby obtaining an enhanced small sample set S1.

[0074] In step S4, based on the uncertainty field U, for each image sub-region where the uncertainty value is higher than the preset threshold, M physically consistent virtual observations (M ranges from 5 to 50) are generated through lightweight physical rendering or simulation (such as viewpoint transformation, lighting simulation, occlusion addition, or finite deformation simulation). During the generation process, depth and approximate mechanical parameters are preserved so that the virtual observations and real teaching samples are merged according to uncertainty weighting to form an enhanced small sample set S1, thereby reducing the domain bias caused by non-physical enhancement.

[0075] Furthermore, key image sub-regions are selected for enhancement based on uncertainty, prioritizing the generation of virtual samples in areas with high uncertainty and high information value. The lightweight physical rendering or simulation includes, but is not limited to, multi-view transformation of objects, simulation of lighting changes, addition of slight occlusion, or deformation simulation based on a simplified physics engine.

[0076] The generated virtual samples should maintain physical consistency, that is, retain depth and mechanical approximation information as much as possible (e.g., viewpoint changes, occlusion simulation, finite deformation, etc.).

[0077] During training, synthetic samples are assigned weights related to uncertainty to balance the influence of real teaching and synthetic samples.

[0078] S5. Small sample meta-learning adaptive fine-tuning: Using the enhanced small sample set S1 as task samples, the outer loop and inner loop training are performed on the grasping model composed of the encoder and grasping evaluation module obtained by S2 fine-tuning under the meta-learning framework, so that the grasping model can obtain an initialization that can quickly adapt to new teachings, and the small sample adaptive model Θ* is obtained through a small gradient update.

[0079] In step S5, the few-sample meta-learning adaptive fine-tuning uses an inner-loop and outer-loop training strategy based on grasping model-irrelevant meta-learning (MAML) or a lightweight variant thereof to update gradients;

[0080] The outer ring optimizes the grasping model across multiple tasks (different objects / viewpoints) to obtain initialization parameters that can quickly adapt to new tasks.

[0081] On the limited amount of new teaching data provided by the target task, the above initialization parameters are updated in a few steps to achieve rapid adaptation.

[0082] The learning rate and number of steps of the inner loop are used as adjustable hyperparameters to balance rapid adaptation and stability in small sample scenarios, thus obtaining the small sample adaptive fine-tuning grasping model.

[0083] Furthermore, the outer loop training is conducted with a set of tasks in multiple tasks or scenarios to learn initial parameters with rapid adaptability; the inner loop performs small-step updates on the target teaching to achieve online adaptation.

[0084] Synthetic samples and real teachings can be mixed in an uncertainty-weighted manner for outer loop training to improve the robustness of outer loop initialization.

[0085] When deploying in real time, the number of fine-tuning steps and the learning rate of the inner loop should be used as adjustable hyperparameters to balance adaptation speed and stability.

[0086] S6. Geometric back projection and optimal 3D grasping pose calculation and online short-probe playback are used to perform pixel-to-3D back projection of the pixel-level grasping confidence field G and the depth map, and solve and optimize the candidate grasping pose set Q based on shape, collision and mechanical constraints. Then, short-term tactile and force sensing are performed on the pixel-level confidence field, and the success or failure results along with tactile feedback are written back as new teaching samples to update the teaching library and use it for the next round of adaptation, so as to obtain the final grasping execution result and the updated teaching library.

[0087] In step S6, a geometric transformation is performed based on the depth map and camera intrinsic parameters to obtain candidate 3D grab points, thereby achieving pixel-to-3D back projection.

[0088] The solution for the grasping pose set Q is performed through local optimization in the form of constraint minimization. The objective function of the local optimization includes minimizing the weighted sum of the grasping torque, collision penalty, and grasping uncertainty, to output executable poses sorted by confidence. The objective function is expressed as follows:

[0089] min[λ1*Torque(q) + λ2*Collision(q) + λ3*Uncertainty(q)]);

[0090] Where q represents the candidate grasping pose; Torque(q) represents the grasping torque generated by the gripper on the target object at grasping pose q; Collision(q) represents the penalty for collision between the robot (mainly the gripper) and obstacles (or other non-target objects) in the environment at grasping pose q; and Uncertainty(q) represents the grasping uncertainty corresponding to grasping pose q. This value is obtained by sampling (e.g., taking the maximum or average value) the region near the grasping point in the pixel-level uncertainty field U generated by S3. λ1 ​​is the weighting coefficient of the grasping torque, λ2 is the weighting coefficient of the collision penalty, and λ3 is the weighting coefficient of the grasping uncertainty.

[0091] For the candidate grasping poses, short-term tactile and force sensing probes are performed according to confidence levels from high to low. If the probe fails, the failed sample is recorded and added to the teaching library according to priority to enhance the learning of negative samples of the failed state. The judgment threshold of the probe is set based on the force change rate or absolute force value of the tactile and force sensors to determine whether the grasp is successful or slips out.

[0092] Furthermore, the pixel-to-3D inverse projection utilizes known in-camera depth values ​​for geometric transformation to extract candidate 3D points and generate pose candidates.

[0093] Pose determination employs multi-objective constrained optimization (e.g., weighting among torque, collision risk, and uncertainty penalty) to sort the candidate set and perform local optimization to obtain an executable pose.

[0094] Before execution, short-term tactile / force sensory trials are conducted on high-confidence candidates to verify the grasping hypothesis; the trial results and tactile time series are written back as new teaching samples or negative samples for subsequent training or online fine-tuning.

[0095] As shown above, by combining synthetic pre-training with teaching fine-tuning and adopting a representation strategy of prototype conditional interpolation, this method can still generate highly discriminative and robust embedding representations to viewpoint, lighting and some appearance variations under the constraint of a small amount of real teaching, thereby reducing the probability of grasping errors caused by representation errors in real-world scenarios.

[0096] Generating physically consistent virtual observations based on uncertainty avoids non-physical sample biases that may be introduced by blind, arbitrary image augmentation. This strategy enables synthetic samples to be closer to real-world scenarios in both semantic and physical terms, thereby allowing augmented samples to play a positive role in training and improving the model's generalization ability in real-world environments.

[0097] Example 2:

[0098] This embodiment also provides a small-sample-based intelligent robot grasping learning system, including:

[0099] The data acquisition and processing module is used to acquire multimodal teaching observations of the target object and perform annotation and formatting processing to obtain the initial teaching set;

[0100] The model pre-training and fine-tuning module pre-trains the encoder using a synthetic grasping scenario and iteratively fine-tunes the pre-trained encoder using the initial teaching set to map the teaching samples to the prototype embedding space to obtain a prototype set.

[0101] The capture evaluation module calculates the pixel-level capture confidence field and the corresponding uncertainty field based on the prototype set and real-time RGB-D observations.

[0102] The sample enhancement module samples key points in the high uncertainty region based on the uncertainty field and generates physically consistent virtual observations through physical rendering or simulation to expand the training samples and obtain an enhanced small sample set.

[0103] The meta-learning adaptive module uses the enhanced small sample set as task samples, trains the grasping model in the outer and inner loops under the meta-learning framework, and obtains the grasping model after small sample adaptation through gradient update.

[0104] The grasping execution and feedback module performs a three-dimensional back projection of the pixel-level grasping confidence field and depth map, solves the optimal grasping pose based on multiple constraints, performs short-term tactile / force sensory exploration, and writes the results back as new teaching samples to update the teaching library.

[0105] Specifically, the data acquisition and processing module includes:

[0106] An RGB-D camera is used to acquire RGB images and depth maps of the target.

[0107] Tactile / force sensors, integrated into the robot's end effector, are used to record short snapshots of tactile or force sensations at the moment of taught contact;

[0108] The data synchronization and formatting unit is used to perform time synchronization, coordinate system alignment, and unified format storage of multimodal observation data.

[0109] Specifically, the crawling evaluation module includes:

[0110] An encoder network is used to extract pixel-level embedding features from real-time RGB-D observations;

[0111] A similarity calculation unit is used to calculate the similarity between each pixel embedding and each prototype vector in the prototype set;

[0112] The confidence and uncertainty generation unit is used to generate a continuous pixel-level capture confidence field based on similarity weights through weighted interpolation or attention mechanism, and derive the pixel-level uncertainty field based on the information entropy or variance of the similarity distribution.

[0113] Specifically, the crawling, execution, and feedback module includes:

[0114] Geometric backprojection unit is used to backproject pixel-level confidence maps to three-dimensional space based on camera intrinsic parameters and depth map;

[0115] A pose optimization solver is used to locally optimize candidate grasp poses with the goal of minimizing the weighted sum of grasping torque, collision penalty, and uncertainty.

[0116] The online exploration and feedback unit is used to control the robot to perform short-term tactile / force exploration on candidate poses according to confidence level, and generate new teaching samples and update the teaching library based on the success or failure of the exploration.

[0117] Example 3:

[0118] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the grasping learning method for the intelligent robot based on few samples.

[0119] Example 4:

[0120] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the grasping learning method for the intelligent robot based on few samples.

[0121] Example 5:

[0122] Household clutter grabbing, scenario: 10 common object categories;

[0123] 1. Step S2: Experimental equipment and data acquisition;

[0124] Robot hardware: 6-DOF robotic arm (load capacity 3kg), with a two-finger electric gripper connected in parallel at the end effector; the gripper has a built-in 3-axis force sensor (sampling 200Hz); the end effector has a front-mounted RGB-D camera (resolution 640×480, depth accuracy ±3mm); the camera's intrinsic parameters are pre-calibrated.

[0125] Object collection: 10 categories of common household objects (e.g., plastic cups, ceramic bowls, key bundles, sponges, books, small boxes, apples, bananas, coin stacks, metal knives).

[0126] Initial teaching (S0): Take N=5 multi-view teachings for each class (each teaching record: RGB image, depth map, 1 frame short tactile / force snapshot).

[0127] Calculation: 10 categories × 5 = 50 teaching samples.

[0128] Data collection methods: Teaching is conducted remotely by a human (handheld terminal demonstrating grasping at a safe height) or by low-speed automatic trajectory recording; tactile snapshots record 10ms of force-time samples at the moment of contact and take the average as the teaching force value.

[0129] 2. Step S2: Pre-training and fine-tuning parameters;

[0130] Synthetic pre-training: 50k synthetic views were generated using a desktop scene rendering engine (physically realistic rendering, randomized object materials / lighting / viewpoint) for self-supervised contrastive learning (InfoNCE, batch size 256, temperature τ=0.07), and trained for 100 epochs.

[0131] Fine-tuning: Use S0 (50 lines) for comparative fine-tuning for 20 epochs, freeze 70% of the encoder backbone parameters, and only fine-tune the top layer and projection head.

[0132] 3. Step S3: Prototype generation and confidence field calculation;

[0133] The prototype P is composed of: each teaching mapping is pi, so the size of P is |P|=50.

[0134] For a pixel x in the test scene, the similarity between its embedding ex and the three closest prototypes is rawsim=[0.82,0.60,0.40], and the temperature τ=0.1.

[0135] Calculate the softmax weights:

[0136] sim / τ:0.82 / 0.1=8.2; 0.60 / 0.1=6.0; 0.40 / 0.1=4.0.

[0137] exp(8.2)≈3614.923(e^{8.2}=3614.923);

[0138] exp(6.0)=403.429; exp(4.0)=54.598.

[0139] Weight w1 = 3614.923 / (3614.923 + 403.429 + 54.598) ≈ 0.888;

[0140] w2 = 403.429 / 4072.95 ≈ 0.099;

[0141] w3=54.598 / 4072.95≈0.013.

[0142] Confidence level G(x) = 0.888 / 0.82 + 0.099 / 0.60 + 0.013 / 0.40 = 0.79276.

[0143] Mapping G to [0,1]: G(x)≈0.793.

[0144] Entropy U(x)=-[0.888log0.888+0.099log0.099+0.013log0.013]

[0145] ≈0.1048+0.2280+0.0565=0.3893.

[0146] If K=50, normalize by ln(50)≈3.912, then Unorm≈0.3893 / 3.912≈0.0996.

[0147] Note: The confidence level for this pixel is high (G≈0.793), and the normalized uncertainty is low (Unorm≈0.10).

[0148] 4. Step S4: Uncertainty drives physical consistency enhancement;

[0149] Uncertainty region sampling: The average number of image regions with uncertainty values ​​higher than a preset threshold detected in the initial image for each type of object is 8 key sub-regions (empirical value).

[0150] M=30 virtual observations (viewpoint / lighting / occlusion / slight deformation) are generated for each sub-region, and approximate mechanical parameters (friction coefficient, approximate mass) are preserved.

[0151] Calculate the number of generated samples: 8 × 30 = 240 per class.

[0152] The total number of synthetic samples in all 10 categories is 240 × 10 = 2400.

[0153] The size of the enhanced small sample set S1 = S0 + synthesis = 50 + 2400 = 2450.

[0154] The weight w{aug}(x) is calculated using the formula w{aug}=1 / (1+αU), where α=2.

[0155] If a pixel has Unorm=0.10, then w{aug}=1 / (1+2*0.10)≈0.8333.

[0156] Note: Pixels with low uncertainty are given more weight, and the training contribution of virtual samples is adjusted according to this weight.

[0157] 5. Step S5: Small sample learning adaptive fine-tuning;

[0158] Meta-learning configuration: 200 outer loop tasks (combining different objects and different perspectives to form tasks), 50 outer loop epochs; 3 inner loop steps, αin inner learning rate = 0.01.

[0159] Training process: Perform outer loop training on S1 to obtain initial θ; during deployment, perform inner loop 3-step update on the new target teaching.

[0160] 6. Step S6: Pose calculation and online short probe playback;

[0161] Back projection parameters: Use camera intrinsics to obtain pixel-to-point cloud mapping; select topK=50 pixels from G as capture candidates (sorted by G); back project the pixels into 3D to obtain a point set.

[0162] Pose optimization weight settings: λ1=0.5 (Torque), λ2=1.0 (Collision), λ3=1.5 (uncertainty penalty, prioritizing the avoidance of uncertain regions).

[0163] Online short probe: Perform a short tactile probe (contact time 100ms) for each candidate pose. The judgment threshold is: a sudden force change >0.5N is considered a successful contact feature; if it fails, it will backtrack to the next candidate pose.

[0164] Execution log and write-back: Success / failure samples and tactile time series are saved to the teaching library and used as new samples during the next fine-tuning.

[0165] 7. Experimental design and evaluation indicators;

[0166] Test subjects: 10 types of objects, each type was placed at a random location 50 times, for a total of 10 × 50 = 500 trials (10 × 50 = 500).

[0167] Baseline (Comparison Method): A typical single-modal image fine-tuning method (a scaled-down version of CNN-based DenseNet-121 + SMOTE random augmentation) from publicly available literature was adopted, along with traditional pixel binarization confidence annotation and L2 weighted loss. The training data for conventional techniques were also augmented to an equal amount using S0 (2400 samples generated using SMOTE) to ensure comparable sample sizes.

[0168] Evaluation metrics: success rate of grasping, average force deviation per grasp (N), average time per grasp (s), and average number of adaptation steps (inner loop update steps).

[0169] 8. Experimental Results;

[0170] Each item is based on statistics from 500 trials:

[0171] The success rate of conventional data scraping techniques is 78.0% (i.e., 0.78, 500 × 0.78 = 390 successful attempts).

[0172] The success rate of this embodiment (method) is 87.4% (i.e., 0.874, 500 × 0.874 = 437 successful captures).

[0173] The success rate increase was 87.4% - 78.0% = 9.4 percentage points.

[0174] The relative percentage increase = 0.094 / 0.78 ≈ 0.1205 → 12.05%.

[0175] The average grasping time for conventional technology is 3.6 s / time; the average grasping time for this embodiment is 3.9 s / time (slightly increased, mainly due to the short detection time), and the difference is 3.9 - 3.6 = 0.3 s.

[0176] The average tactile force deviation (target force is expected force) for conventional techniques is 0.45N;

[0177] In this embodiment, the value is 0.32N, and the value is reduced to 0.45 - 0.32 = 0.13N (indicating greater stability).

[0178] In this embodiment, the average number of adaptation steps for the inner loop is actually 2 (the inner loop is set to 3 steps, but an average of 2 steps is enough to reach the threshold); the average online fine-tuning time is ≈6.0s (including one short probe).

[0179] Note: Success rate increased by 9.4 percentage points (absolute), relative increase of about 12%, while grasping stability (force deviation) improved by about 28.9% (0.13÷0.45≈0.2889→28.89%).

[0180] As shown above, the meta-learning framework provides the model with the ability to "adapt with minimal teaching," enabling the robot to quickly adjust parameters and output reliable strategies in just a few steps when faced with new objects or new placement states. This capability is particularly important in scenarios requiring frequent handling of new objects, such as industrial line changes, flexible production, and home environments.

[0181] Continuous pixel-level confidence and uncertainty fields provide fine-grained information for grasping decisions, enabling subsequent sampling, pose optimization, and detection strategies to be tailored to risk perception. Incorporating uncertainty penalties into pose optimization helps to proactively avoid high-risk areas, thereby reducing the risk of execution failure and object damage.

[0182] Example 6:

[0183] Industrial flexible parts (fabric / sponge-wrapped parts) gripping:

[0184] Scene and Data Acquisition:

[0185] Application scenario: Handling flexible packages (soft packaging bags, sponge bags) on industrial production lines. These objects undergo significant deformation and exhibit complex friction / contact behavior.

[0186] Equipment: 6-DOF robotic arm (load capacity 5kg), gripper with tactile array (16 channels, sampling 500Hz), high-resolution RGB-D (1280×720).

[0187] Initial teaching S0: Take 5 types of flexible parts, and teach N=3 for each type (emphasizing small sample size), total S0=5×3=15 (5×3=15).

[0188] Haptic snapshot: Record the force timing 50ms before and after contact and extract the peak value and slope as force features.

[0189] Parameters and enhancement strategies:

[0190] Synthetic pre-training: Using physics-based soft body rendering (preserving the deformation of the finite element approximation), 80k synthetic samples are generated for self-supervised pre-training (InfoNCE) for 120 epochs.

[0191] Uncertainty region sampling: An average of 12 high-uncertainty sub-regions are sampled per class, generating M=50 physically consistent virtual samples / sub-regions (more samples are needed due to the greater deformation of flexible components). Calculation of the number of synthesized samples: 12 × 50 = 600 per class; total number of synthesized samples across 5 classes = 600 × 5 = 3000.

[0192] S1 size = S0 + synthesis = 15 + 3000 = 3015.

[0193] Meta-learning and pose optimization:

[0194] The meta-learning outer loop has 300 tasks, an outer loop epoch of 60, an inner loop step count of s of 4, and αin of 0.008.

[0195] Pose optimization weights: λ1=0.6 (Torque), λ2=1.2 (Collision), λ3=2.0 (uncertainty penalty, as uncertainty is more critical for flexible components).

[0196] Test Design and Evaluation:

[0197] Test subjects: 5 types of flexible parts, 60 tests for each type, total number of tests = 5 × 60 = 300.

[0198] Conventional techniques also employ single-modal DenseNet-121 reduction + SMOTE enhancement (synthetic sample size aligned to 3000) and traditional pose calculation.

[0199] Experimental results (quantitative):

[0200] The success rate of conventional data scraping techniques is 70.7% (300 × 0.707 = 212.1 → 212 successful attempts).

[0201] In this example, the success rate of data capture is 78.9% (300 × 0.789 = 236.7 → 236 successful captures).

[0202] Absolute increase = 78.9% − 70.7% = 8.2 percentage points;

[0203] Relative increase = 0.082 / 0.707 ≈ 0.1160 → 11.60%.

[0204] The average grasping time for conventional techniques is 4.2 seconds; in this embodiment, it is 4.6 seconds (an increase of 0.4 seconds due to more short detection times and optimizations).

[0205] The average force deviation for conventional techniques is 0.62 N; in this embodiment, it is 0.44 N.

[0206] Improvement = 0.18N (0.18 ÷ 0.62 ≈ 0.2903 → 29.03% improvement).

[0207] The comprehensive comparison is shown in Table 1 below (Comparison of Examples 1 and 2 with conventional technologies);

[0208] Table 1

[0209]

[0210] In the table, "Success Rate" is the overall success rate of the grasping test; "Force Deviation" is the deviation between the target grasping force and the actual mean (the smaller the better); "Time" is the average execution time for each grasping operation; and "Number of Samples (S1)" is the total number of augmented samples used for training.

[0211] As shown above, writing back the results of short-term tactile / force sensory explorations to the teaching library and using them for subsequent training constitutes a closed-loop learning mechanism, enabling the system to continuously correct and optimize based on real-world performance feedback. This online incremental learning mechanism can gradually accumulate experience, correct model biases, and improve task reliability during long-term deployment.

[0212] Multimodal teaching (vision + depth + tactile) provides direct observation signals for understanding and predicting contact processes. Especially when dealing with flexible, deformable, or frictionally significant objects, tactile information compensates for the shortcomings of vision, making the grasping strategy more considerate of contact mechanics, thereby improving the ability to handle complex objects.

[0213] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A grasping learning method for intelligent robots based on small sample sizes, characterized in that, Includes the following steps: S1. Collect teaching observations of the target object, and label and format the teaching observations to obtain an initial teaching set; The teaching observation includes RGB images, depth maps, and short-term tactile or force snapshots; S2. Perform self-supervised or contrastive pre-training on the encoder using the synthetic grasping scene, and iteratively fine-tune the pre-trained encoder using the initial teaching set to map each teaching sample to the prototype embedding space to obtain the prototype set. S3. The grasping evaluation module calculates the pixel-level grasping confidence field and corresponding uncertainty field based on the prototype set and the real-time RGB image-depth map to obtain the pixel-level confidence and uncertainty mapping. S4. Based on the uncertainty field, key points are sampled in image areas where the uncertainty value is higher than a preset threshold, and physical consistent virtual observations are generated by weighting the uncertainty through lightweight physical rendering or simulation to expand the training samples and obtain an enhanced small sample set. S5. Using the enhanced small sample set as task samples, the outer and inner loops of the grasping model composed of the encoder and grasping evaluation module are trained under the meta-learning framework, so that the grasping model can obtain an initialization that can quickly adapt to new teachings, and the grasping model after small sample adaptive fine-tuning is obtained through gradient update. S6. Perform pixel-to-3D back projection on the pixel-level grasping confidence field and the depth map, and solve and optimize the candidate grasping pose set based on shape, collision and mechanical constraints. Then, perform short-term tactile and force sensing trials on the pixel-level confidence and write the success or failure results along with tactile feedback back as new teaching samples to update the teaching library and use it for the next round of adaptation, so as to obtain the final grasping execution result and the updated teaching library.

2. The grasping learning method for intelligent robots based on few samples according to claim 1, characterized in that, S1 includes: For the same target, N observations are performed from multiple perspectives, where N ≥ 1 and N ≤ 10. Each observation simultaneously records an RGB image, the corresponding depth map, and one or more short-time tactile and force snapshots to describe the contact response of the object's surface.

3. The grasping learning method for intelligent robots based on few samples according to claim 1, characterized in that, S2 includes: Self-supervised contrastive learning based on synthetic grasping scenarios is used for pre-training, with contrastive loss or its variant as the loss function, to obtain an encoder that is robust to geometric and material changes. During fine-tuning, the parameter freezing ratio of the encoder backbone is kept below 80% to ensure that the prototype embedding space remains general and domain-adaptive, resulting in a prototype set.

4. The grasping learning method for intelligent robots based on few samples according to claim 1, characterized in that, S3 includes: The encoder extracts the pixel embeddings observed in real time; The capture evaluation module calculates the similarity between the pixel embedding and the prototype set and normalizes it to a continuous confidence score. Specifically, it calculates the similarity weight between each pixel and all prototype vectors and uses a weighted kernel interpolation or attention weighting mechanism to obtain the pixel-level confidence score. Based on the distribution of the similarity weights, it derives the pixel uncertainty and obtains the pixel-level confidence and uncertainty mapping.

5. The grasping learning method for intelligent robots based on small sample sizes according to claim 1, characterized in that, S4 includes: Based on the uncertainty field, M physically consistent virtual observations are generated in each image region where the uncertainty value is higher than a preset threshold. During the generation process, depth and approximate mechanical parameters are preserved so that the virtual observations and real teaching samples are merged into an enhanced small sample set by weighting according to uncertainty.

6. The grasping learning method for intelligent robots based on few samples according to claim 1, characterized in that, S5 includes: Gradient updates are performed using an inner-loop and outer-loop training strategy based on model-independent meta-learning or its lightweight variants. The outer ring optimizes the crawling model across multiple tasks, enabling it to obtain initialization parameters that can quickly adapt to new tasks. The learning rate and number of steps of the inner loop are used as adjustable hyperparameters. On the new teaching data provided by the target task, the initial parameters are updated with gradients to obtain the grasping model after small sample adaptive fine-tuning. The learning rate and number of steps of the inner loop are used as adjustable hyperparameters to balance rapid adaptation and stability in small sample scenarios, resulting in a grasping model after small sample adaptive fine-tuning.

7. The grasping learning method for intelligent robots based on few samples according to claim 1, characterized in that, S6 includes: Based on the depth map and camera intrinsic parameters, geometric transformation is performed to obtain candidate 3D gripping points, thereby achieving pixel-to-3D back projection. The solution for the set of grasping poses is performed by local optimization in the form of constraint minimization. The objective function of local optimization includes minimizing the weighted sum of grasping torque, collision penalty and grasping uncertainty, so as to output executable poses sorted by confidence. Short-term tactile and force sensing probes are performed on candidate grasping poses in descending order of confidence. If the probe fails, the failed sample is recorded and added to the teaching library according to priority to enhance the learning of negative samples of the failed state. The judgment threshold of the probe is set based on the force change rate or absolute force value of the tactile and force sensors to determine whether the grasp is successful or slips out.

8. A learning system for grasping intelligent robots based on small sample sizes, characterized in that, include: The data acquisition and processing module is used to perform step S1 as described in claim 1, to acquire multimodal teaching observations of the target object and perform annotation and formatting processing to obtain an initial teaching set; The model pre-training and fine-tuning module is used to perform step S2 as described in claim 1, to pre-train the encoder by synthesizing the grasping scene, and to iteratively fine-tune the pre-trained encoder using the initial teaching set, so as to map the teaching samples to the prototype embedding space to obtain the prototype set. The capture evaluation module is used to perform step S3 as described in claim 1, and to calculate the pixel-level capture confidence field and the corresponding uncertainty field based on the prototype set and real-time RGB-D observations. The sample enhancement module is used to perform step S4 as described in claim 1, which samples key points in a high uncertainty region based on the uncertainty field, and generates physically consistent virtual observations through physical rendering or simulation to expand the training samples and obtain an enhanced small sample set. The meta-learning adaptive module is used to execute step S5 as described in claim 1, using the enhanced small sample set as task samples, to train the grasping model in the outer and inner loops under the meta-learning framework, and to obtain the small sample adaptive grasping model through gradient update. The grasping execution and feedback module is used to perform step S6 as described in claim 1, perform three-dimensional back projection of the pixel-level grasping confidence field and depth map, solve the optimal grasping pose based on multiple constraints, perform short-term tactile / force sensory exploration, and write the results back as new teaching samples to update the teaching library.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images