Human-computer interaction-oriented robot lightweight retrieval and grabbing method, medium and equipment

Abstract

The application discloses a kind of robot lightweight retrieval and grabbing methods for human-computer interaction.The method first uses the data collection strategy based on shape center, organizes the demonstration of grabbing according to geometric category instead of specific instance, to construct the compact data set with expansion with a small number of demonstrations.Second, by identifying user pointing gesture, combined with principal component analysis and ray projection to determine target object, intuitive intent recognition is realized.Finally, based on image features and target mask, fast similarity retrieval is carried out, and the existing grabbing contact point is mapped to the new target using rigid transformation, realizing the redirection of grabbing action execution.The application has lightweight, low latency and high generalization, without relying on large-scale training data or complex model, it can achieve high success rate grabbing in multi-category objects and multiple scenes, improve the stability and adaptability of robot in human-computer interaction task.

Description

Technical Field

[0001] This invention belongs to the field of human-computer intelligent interaction, specifically relating to a lightweight robot retrieval and grasping method for human-computer interaction. Background Technology

[0002] Robotic grasping and human-robot interaction are important research areas in service robots, home robots, and industrial robots. Achieving stable and flexible grasping capabilities is key for robots to complete complex tasks and achieve natural interaction. Existing technologies mainly rely on two types of methods: one is based on pose estimation, and the other is based on large-scale deep learning.

[0003] Pose estimation-based methods typically require accurate 3D reconstruction and pose inference of the target object, followed by the generation of a grasping trajectory using a planning algorithm. While these methods can achieve a certain success rate in controlled environments, they are susceptible to changes in lighting, object occlusion, and sensor noise in complex scenes, leading to inaccurate pose estimation and a significantly increased grasping failure rate.

[0004] Large-scale deep learning-based methods rely on massive amounts of labeled data and complex network models to learn the geometric features of objects and grasping strategies. While these methods perform well on some benchmark tasks, their drawbacks are also significant: First, data acquisition and labeling are extremely costly, making it difficult to quickly generalize to new tasks; second, model training requires substantial computational resources, making it unsuitable for real-time applications on lightweight platforms; and finally, when encountering unfamiliar objects, the model's generalization performance is often insufficient, making it difficult to reliably perform grasping in open environments.

[0005] These problems are even more pronounced in human-computer interaction scenarios. Users typically want to quickly specify the target object the robot wants to manipulate using simple and intuitive methods, such as pointing or gestures. However, existing technologies lack effective mechanisms for recognizing human intent, resulting in an unnatural interaction process and difficulty for the robot to accurately understand user needs. Furthermore, existing methods also have significant shortcomings in generalization ability across objects and real-time response, often failing to simultaneously achieve both grasping accuracy and smooth interaction.

[0006] In summary, existing robotic grasping technologies have limitations in data organization, intent recognition, and action execution, making it difficult to meet the comprehensive requirements of real-time performance, lightweight design, and generalization in human-computer interaction scenarios. Therefore, a new method is urgently needed that can reduce data dependence through shape-centered data organization, achieve intent recognition through intuitive human pointing interaction, and improve the accuracy and efficiency of grasping execution by combining lightweight retrieval and redirection strategies. This would provide more reliable technical support for the application of robots in human-computer collaboration. Summary of the Invention

[0007] The purpose of this invention is to provide a lightweight robot retrieval and grasping method for human-computer interaction.

[0008] The technical solution for achieving the objective of this invention is as follows:

[0009] A lightweight retrieval and grasping method for robots oriented towards human-computer interaction includes the following steps:

[0010] Step 1: Based on the shape center-based data collection method, the crawled demonstrations are organized by geometric categories rather than specific object instances, and redundant samples are removed by clustering methods to form a compact and scalable dataset.

[0011] Step 2: During the interaction, the user's pointing gesture is obtained, the principal component analysis is used to extract the principal axis of the gesture direction, and the target object is determined by ray projection to achieve human intention recognition.

[0012] Step 3: Based on the image features of the target object and the mask, perform similarity retrieval, select a matching reference demonstration, and map the reference grasping contact point to the target object through rigid transformation to generate and execute a new grasping trajectory.

[0013] A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method.

[0014] An electronic device includes a memory and one or more processors, the memory being used to store one or more programs; when the one or more programs are executed by the one or more processors, the method is implemented.

[0015] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention adopts a data collection method based on shape center, organizes demonstration data by geometric categories rather than specific object instances, and can build a compact and scalable dataset under the condition of a small number of manual demonstrations, which significantly reduces the cost of data collection and annotation, and improves the generalization ability of the method in new objects and new scenes; (2) The present invention introduces a human intention recognition mechanism based on pointing gestures, extracts the gesture direction through principal component analysis and determines the target object by combining ray projection, so that users can interact with the robot in a natural and intuitive way, which improves the interaction efficiency and the robot's ability to understand the user's intentions; (3) The present invention proposes a lightweight similarity retrieval and rigid transformation redirection strategy, which can quickly retrieve similar demonstrations and map them to new target objects under low latency conditions without relying on large-scale deep networks, thereby realizing real-time and stable grasping operations, which significantly improves the adaptability and practicality of the system in open environments. Attached Figure Description

[0016] Figure 1This is a flowchart of the lightweight robot retrieval and grasping method for human-computer interaction according to the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0018] Combination Figure 1 A lightweight retrieval and grasping method for robots oriented towards human-computer interaction, specifically including the following steps:

[0019] Step 1: Based on the shape center-based data collection method, the crawled demonstrations are organized by geometric categories rather than specific object instances, and redundant samples are removed by clustering methods to form a compact and scalable dataset.

[0020] The crawled data obtained from manual demonstrations was categorized geometrically:

[0021]

[0022] in, Indicates the first Geometric categories, Categories include, but are not limited to, spheres, cylinders, and handle-like structures; for each category Extract its set of grabbing contact points:

[0023]

[0024] in, Indicates the first One grab contact point, ;

[0025] Then, clustering methods were used to divide the contact points into several clusters:

[0026]

[0027] in, Indicates the first In the geometric category, the first One grab contact point, ;

[0028] Calculate the center point of each cluster:

[0029]

[0030] in, express Each individual grasping contact point within a cluster is represented by its coordinates in three-dimensional space. This coordinate is used as the shape center, and objects located less than a threshold distance from the cluster center are discarded. Redundant samples;

[0031] The final shape center database is as follows:

[0032]

[0033] It is stored as a mapping relationship of "geometric category - shape center - crawling contact point", which provides a basis for subsequent retrieval and crawling redirection.

[0034] Step 2: During the interaction, the user's pointing gesture is obtained, the principal component analysis is used to extract the principal axis of the gesture direction, and the target object is determined by ray projection to achieve human intention recognition.

[0035] By using a vision system to collect key points on the user's hand, a set of pointing gesture points can be obtained:

[0036]

[0037] in, w represents the three-dimensional coordinates of key points on the hand. Principal component analysis was performed on the gesture point set to extract the first principal component vector.

[0038]

[0039] in, Represents the mean of a set of points. The main axis indicates the direction of the direction;

[0040] Use your fingertips or hand reference points As the starting point, the direction vector Define a ray:

[0041]

[0042] Combine the ray with the mask of objects in the scene. By intersecting the points, we obtain the intersection point:

[0043]

[0044] Based on the Euclidean distance from the intersection point to the starting point, the nearest object is identified as the target:

[0045]

[0046] This completes the identification of the user's intended meaning.

[0047] Step 3: Based on the image features of the target object and the mask, perform similarity retrieval, select a matching reference demonstration, and map the reference grasping contact point to the target object through rigid transformation to generate and execute a new grasping trajectory.

[0048] Extract target object Image feature vectors This feature is based on visual encoding within the target mask region; and it is applied to various reference demonstrations in the database. Extract the corresponding feature vector ; Calculate the similarity distance between the target object and the reference demonstration:

[0049]

[0050] The smaller the distance, the higher the matching degree; the reference demonstration corresponding to the minimum distance is selected:

[0051]

[0052] Let the set of gripping contact points in the selected reference demonstration be:

[0053]

[0054] The set of points corresponding to the target object is:

[0055]

[0056] in, , Thus, the transformed contact point is obtained:

[0057]

[0058] It generates the grab trajectory of the target object and redirects the grab action.

Claims

1. A lightweight robot retrieval and grasping method for human-computer interaction, characterized in that, Includes the following steps: Step 1: Based on the shape center-based data collection method, the crawled demonstrations are organized according to geometric categories rather than specific object instances, and redundant samples are removed by clustering methods to form a compact and scalable dataset; Step 2: During the interaction, the user's pointing gesture is obtained, the principal component analysis is used to extract the principal axis of the gesture direction, and the target object is determined by ray projection to achieve human intention recognition. Step 3: Based on the image features of the target object and the mask, perform similarity retrieval, select a matching reference demonstration, and map the reference grasping contact point to the target object through rigid transformation to generate and execute a new grasping trajectory; Step 1 organizes the crawling demonstration based on the data collection method of the shape center, specifically including: The crawled data obtained from manual demonstrations was categorized by geometric type: in, Indicates the first Geometric categories, Categories include, but are not limited to, spheres, cylinders, and handle-like structures; for each category Extract its set of grabbing contact points: in, Indicates the first One grab contact point, ; Then, clustering methods were used to divide the contact points into several clusters: in, Indicates the first In the geometric category, the first A cluster of grabbing contact points, ; Calculate the center point of each cluster: in, express Each individual grasping contact point within a cluster is represented by a coordinate point in three-dimensional space; this is used as the shape center, and objects less than a threshold distance from the cluster center are discarded. Redundant samples; The final shape center database is as follows: It is stored as a mapping relationship between geometric category – shape center – crawling contact point, providing a basis for subsequent retrieval and crawling redirection.

2. The lightweight robot retrieval and grasping method for human-computer interaction according to claim 1, characterized in that, Step 2 achieves human intent recognition through pointing gestures, specifically including: By using a vision system to collect key points on the user's hand, a set of pointing gesture points can be obtained: in, w represents the three-dimensional coordinates of key points on the hand. Principal component analysis was performed on the gesture point set to extract the first principal component vector. in, Represents the mean of a set of points. Indicates the direction vector of the principal axis; Use your fingertips or hand reference points As the starting point, the direction vector Define a ray: Combine the ray with the mask of objects in the scene. By intersecting the points, we obtain the intersection point: Based on the Euclidean distance from the intersection point to the starting point, the nearest object is determined as the target object. : This completes the identification of the user's intended meaning.

3. The lightweight robot retrieval and grasping method for human-computer interaction according to claim 1, characterized in that, Step 3 implements crawling redirection based on similarity retrieval and rigid transformation, specifically including: Extract target object Image feature vectors This feature is based on visual encoding within the target mask region; and it is applied to various reference demonstrations in the database. Extract the corresponding feature vector ; Calculate the similarity distance between the target object and the reference demonstration: The smaller the distance, the higher the matching degree; the reference demonstration corresponding to the minimum distance is selected: Let the set of gripping contact points in the selected reference demonstration be: The set of points corresponding to the target object is: Therefore, the transformed contact point is obtained: in, , It generates the grab trajectory of the target object and redirects the grab action.

4. A computer storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 3.

5. An electronic device, characterized in that, The device includes a memory and one or more processors, the memory being used to store one or more programs; when the one or more programs are executed by the one or more processors, they implement the method as described in any one of claims 1 to 3.

Images