Robot control method and device, readable storage medium, and computer program product

Abstract

Disclosed are a control method and device for a robot, a readable storage medium, and a computer program product. The method comprises: performing image segmentation on an RGB image in a current workspace to obtain a mask of each object in the RGB image; obtaining a scene point cloud of the current workspace based on the mask of each object, a depth image of the current workspace, and a target object, wherein the scene point cloud identifies the target object and non-target objects of the current workspace; inputting the scene point cloud, the action sequence of the robot in the current time frame, and latent variables sampled from a latent variable distribution into an action prediction decoder to predict the action sequence of the robot in a first time frame, wherein the latent variable distribution is the action sequence distribution obtained when training the action prediction decoder, and the first time frame includes a first predetermined number of time frames after the current time frame; and mapping the predicted action sequence to the robot to control the robot to grasp the target object.

Description

Technical Field

[0001] The present disclosure generally relates to the field of robotics technology, and more specifically, to a robot control method and device, a readable storage medium, and a computer program product. Background Art

[0002] At present, remote-controlled robots can replace humans to perform operations in dangerous environments, but the human operators who perform remote operations need to stay focused for a long time and perform complex operations, which is prone to operational errors. In order to reduce the burden on human operators in remote operations, human-machine fusion intelligent enhanced control combines the efficient decision-making ability of humans with the accurate execution ability of robots, aiming to improve operational efficiency and reduce errors. However, the current human-machine fusion intelligent enhanced control has the following disadvantages:

[0003] 1. Difficulty in handling cluttered scenes: Many experiments and application scenarios of human-machine fusion intelligent enhanced control methods are limited to single or a few graspable target objects, and it is difficult to handle cluttered scenes with multiple objects randomly placed. This makes it difficult for existing solutions to accurately identify or flexibly grasp target objects in unstructured complex scenes, and it is difficult to avoid collisions with non-target objects.

[0004] 2. Motion trajectory and real-time issues: After clarifying the task objectives, many human-machine fusion intelligent enhanced control methods do not perform continuous real-time trajectory planning in combination with the environment, and are unable to complete the task efficiently and accurately, resulting in incoherent motion trajectories in applications and difficulty in adapting to dynamic environments.

[0005] 3. Inaccurate prediction of user intention: The current human-machine fusion intelligent enhanced control method only infers user intention based on a small amount of information. It is difficult to accurately predict the user's grasping target in a complex environment, resulting in grasping task failure or poor user experience.

[0006] 4. The types of objects that can be grasped are limited: Most human-machine fusion intelligent enhanced control methods can only grasp simple and regular spheres or cubes, and it is difficult to expand to the diverse and irregularly shaped objects in the real world. This problem makes these methods unable to be applied in actual production and life.

[0007] 5. Low dexterity of the end effector: In most solutions, the end effector carried by the robotic arm uses a simple gripper, which makes it difficult to grasp irregularly shaped objects or perform complex operation tasks, resulting in limited capabilities of the human-machine fusion intelligent enhanced control method in practical applications. Summary of the invention

[0008] The embodiments of the present disclosure provide a robot control method and device, a readable storage medium, and a computer program product, which can effectively solve the problem that the prior art cannot accurately and efficiently grasp the target object.

[0009] In a general aspect, a robot control method is provided, comprising: performing image segmentation on an RGB image in a current workspace to obtain a mask for each object in the RGB image; obtaining a scene point cloud of the current workspace based on the mask of each object, a depth image of the current workspace, and a target object, wherein the scene point cloud identifies the target object and non-target objects of the current workspace; inputting the scene point cloud, the action sequence of the robot in a current time frame, and latent variables sampled from a latent variable distribution into an action prediction decoder to predict the action sequence of the robot in a first time frame, wherein the latent variable distribution is a distribution of action sequences obtained when training the action prediction decoder, and the first time frame includes a first predetermined number of time frames after the current time frame; and mapping the predicted action sequence to the robot to control the robot to grasp the target object.

[0010] Optionally, before obtaining the scene point cloud of the current workspace based on the mask of each object, the depth image of the current workspace and the target object, it also includes: selecting a mask with a confidence greater than a first predetermined value from the mask of each object as an initial mask; using masks with an overlap rate greater than a second predetermined value in the initial masks as multiple masks of the same object, and using the mask with the highest confidence among the multiple masks as the final mask of the same object.

[0011] Optionally, the target object is determined in the following manner: inputting the point cloud of each object in the current workspace into an object encoder to obtain feature information of each object; inputting the robot's action sequence in a second time length into a motion encoder to obtain the robot's motion features, wherein the second time length includes a second predetermined number of time frames before the current time frame; inputting the feature information and motion features into an intention decoder to predict probability information of each object interacting with the robot; and determining the target object in the current workspace based on the probability information.

[0012] Optionally, based on the mask of each object, the depth image of the current workspace and the target object, a scene point cloud of the current workspace is obtained, including: multiplying the mask of each object by the depth image, and obtaining the initial point cloud of each object based on the multiplication result and internal parameters of the camera; inputting the initial point cloud of each object into a point cloud completion module to obtain the point cloud of each object; appending a feature channel to the point cloud of each object, and obtaining a scene point cloud based on the point cloud to which the feature channel is appended, wherein the appended feature channel is used to mark the corresponding object as a target object or a non-target object.

[0013] Optionally, before mapping the predicted action sequence to the robot to control the robot to grasp the target object, it also includes: for the current time frame and each time frame in the first time length, performing the following processing: inputting the scene point cloud, the robot's action sequence in the time frame, and the latent variables sampled from the latent variable distribution into the action prediction decoder, predicting the robot's action sequence in a third time length, wherein the third time length is the second predetermined number of time frames after the time frame, and the second predetermined number is less than the first predetermined number; aggregating all action sequences belonging to the same time frame in the third time length to obtain the final action sequence of each time frame.

[0014] Optionally, before aggregating the action sequences belonging to the same time frame in the third duration to obtain the final action sequence for each time frame, it also includes: performing the following processing for each action sequence in all action sequences of the third duration and the first duration: based on the action sequence, obtaining the contact points on the prior contact surface of the robot hand; determining a first distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is the side close to the object when grasping the object; when the distance starts to increase, re-sampling the latent variables from the latent variable distribution and predicting the action sequence; when the distance is less than the third predetermined value and the number of samplings exceeds the preset number, stopping predicting the action sequence.

[0015] Optionally, the scene point cloud, the action sequence of the robot in the current time frame and the latent variables sampled from the latent variable distribution are input into the action prediction decoder to predict the action sequence of the robot in the first time length, including: sampling multiple latent variables from the latent variable distribution; for each of the multiple latent variables, performing the following processing: inputting the scene point cloud, the action sequence of the robot in the current time frame and the latent variables into the action prediction decoder to predict the action sequence of the robot in the first time length; for each action sequence of the first time length, performing the following processing: based on the action sequence, obtaining the contact points on the prior contact surface of the robot's hand, and determining the first distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is the side close to the object when grasping the object; based on the action sequence, obtaining the hand vertices of the robot located inside the target object, determining the second distance from each hand vertex to the nearest object point, and determining the third distance from all the hand vertices of the robot to their respective nearest background object points; based on the first distance, the second distance and the third distance, selecting an action sequence from the action sequences corresponding to each of the multiple latent variables as the final action sequence of the robot in the first time length.

[0016] Optionally, after mapping the predicted action sequence to the robot, it also includes: for the last time frame, performing the following processing: determining the contact points on the prior contact surface among the hand vertices of the robot in the time frame, and obtaining the fourth distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is the side close to the object when grasping the object; obtaining the hand vertices located inside the target object from the hand vertices of the robot in the time frame, and determining the fifth distance from each hand vertex inside the target object to the nearest object point; obtaining the sixth distance of the robot's grasping joint angle before and after adjustment; and adjusting the robot's action based on the fourth distance, the fifth distance and the sixth distance to control the robot to grasp the target object.

[0017] Optionally, the action prediction decoder is trained in the following manner: obtaining training samples, wherein the training samples include a given scene point cloud, an action sequence of a given time frame, and a future action sequence, wherein the future action sequence includes an action sequence of a first predetermined number of time frames after the given time frame; inputting the training samples into an initial action prediction encoder to obtain a latent variable distribution; inputting latent variables sampled from the latent variable distribution, a given scene point cloud, and an action sequence of a given time frame into an initial action prediction decoder to predict an estimated action sequence of the robot at the first time length; adjusting parameters of the initial action prediction encoder and the initial action prediction decoder based on the loss between the estimated action sequence and the future action sequence and the KL loss to obtain an action prediction decoder.

[0018] Optionally, mapping the predicted action sequence to the robot includes: matching the hand posture in the predicted action sequence with the hand of the robot; matching the hand position in the predicted action sequence with the hand of the robot; and regularizing the joint angles of the matched robot's hand.

[0019] In another general aspect, a control device for a robot is provided, comprising: a segmentation unit, configured to perform image segmentation on an RGB image in a current workspace to obtain a mask for each object in the RGB image; a point cloud acquisition unit, configured to acquire a scene point cloud of the current workspace based on the mask of each object, a depth image of the current workspace and a target object, wherein the target object and non-target objects of the current workspace are identified in the scene point cloud; a prediction unit, configured to input the scene point cloud, the action sequence of the robot in a current time frame and latent variables sampled from a latent variable distribution into an action prediction decoder to predict the action sequence of the robot in a first time length, wherein the latent variable distribution is a distribution of action sequences obtained when training the action prediction decoder, and the first time length includes a first predetermined number of time frames after the current time frame; and a mapping unit, configured to map the predicted action sequence to the robot to control the robot to grasp the target object.

[0020] Optionally, the point cloud acquisition unit is also configured to select a mask with a confidence greater than a first predetermined value from the mask of each object as an initial mask before acquiring the scene point cloud of the current workspace based on the mask of each object, the depth image of the current workspace and the target object; use the masks in the initial masks with an overlap rate greater than a second predetermined value as multiple masks of the same object, and use the mask with the highest confidence among the multiple masks as the final mask of the same object.

[0021] Optionally, the target object is determined in the following manner: inputting the point cloud of each object in the current workspace into an object encoder to obtain feature information of each object; inputting the robot's action sequence in a second time length into a motion encoder to obtain the robot's motion features, wherein the second time length includes a second predetermined number of time frames before the current time frame; inputting the feature information and motion features into an intention decoder to predict probability information of each object interacting with the robot; and determining the target object in the current workspace based on the probability information.

[0022] Optionally, the point cloud acquisition unit is further configured to multiply the mask of each object with the depth image, and then obtain the initial point cloud of each object based on the multiplication result and the internal parameters of the camera; input the initial point cloud of each object into the point cloud completion module to obtain the point cloud of each object; add a feature channel to the point cloud of each object, and obtain the scene point cloud based on the point cloud with the feature channel added, wherein the added feature channel is used to mark the corresponding object as a target object or a non-target object.

[0023] Optionally, the mapping unit is further configured to perform the following processing for the current time frame and each time frame in the first time length before mapping the predicted action sequence to the robot to control the robot to grasp the target object: input the scene point cloud, the action sequence of the robot in the time frame, and the latent variables sampled from the latent variable distribution into the action prediction decoder to predict the action sequence of the robot in a third time length, wherein the third time length is a second predetermined number of time frames after the time frame, and the second predetermined number is less than the first predetermined number; aggregate all action sequences belonging to the same time frame in the third time length to obtain the final action sequence of each time frame.

[0024] Optionally, the mapping unit is further configured to perform the following processing on each action sequence in all action sequences of the third time length and the first time length before aggregating action sequences belonging to the same time frame in the third time length to obtain the final action sequence for each time frame: based on the action sequence, obtain the contact points on the prior contact surface of the robot hand; determine the first distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is the side close to the object when grasping the object; when the distance starts to increase, re-sample the latent variables from the latent variable distribution and predict the action sequence; when the distance is less than the third predetermined value and the number of samplings exceeds the preset number, stop predicting the action sequence.

[0025] Optionally, the prediction unit is further configured to sample multiple latent variables from the latent variable distribution; for each of the multiple latent variables, perform the following processing: input the scene point cloud, the robot's action sequence in the current time frame and the latent variables into the action prediction decoder to predict the robot's action sequence in the first time length; for each action sequence of the first time length, perform the following processing: based on the action sequence, obtain the contact points on the prior contact surface of the robot's hand, and determine the first distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is the side close to the object when grasping the object; based on the action sequence, obtain the robot's hand vertices located inside the target object, determine the second distance from each hand vertex to the nearest object point, and determine the third distance from all the robot's hand vertices to their respective nearest background object points; based on the first distance, the second distance and the third distance, select an action sequence from the action sequences corresponding to each of the multiple latent variables as the robot's final action sequence in the first time length.

[0026] Optionally, the mapping unit is further configured to perform the following processing for the last time frame after mapping the predicted action sequence to the robot: determine the contact points on the prior contact surface among the hand vertices of the robot in the time frame, and obtain a fourth distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is a side close to the object when grasping the object; obtain the hand vertices located inside the target object from the hand vertices of the robot in the time frame, and determine the fifth distance from each hand vertex inside the target object to the nearest object point; obtain the sixth distance of the robot's grasping joint angle before and after adjustment; and adjust the robot's action based on the fourth distance, the fifth distance and the sixth distance to control the robot to grasp the target object.

[0027] Optionally, the action prediction decoder is trained in the following manner: obtaining training samples, wherein the training samples include a given scene point cloud, an action sequence of a given time frame, and a future action sequence, wherein the future action sequence includes an action sequence of a first predetermined number of time frames after the given time frame; inputting the training samples into an initial action prediction encoder to obtain a latent variable distribution; inputting latent variables sampled from the latent variable distribution, a given scene point cloud, and an action sequence of a given time frame into an initial action prediction decoder to predict an estimated action sequence of the robot at the first time length; adjusting parameters of the initial action prediction encoder and the initial action prediction decoder based on the loss between the estimated action sequence and the future action sequence and the KL loss to obtain an action prediction decoder.

[0028] Optionally, the mapping unit is further configured to match the hand posture in the predicted action sequence with the hand of the robot; match the hand position in the predicted action sequence with the hand of the robot; and regularize the joint angles of the matched robot's hand.

[0029] In another general aspect, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform any robot control method as described above.

[0030] In another general aspect, a system is provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform any robot control method as described above.

[0031] In another general aspect, a computer program product is provided, comprising computer instructions, which when executed by a processor implement any of the above-mentioned robot control methods.

[0032] According to the control method and device, readable storage medium, and computer program product of the robot of the embodiment of the present disclosure, the target object is marked in the point cloud of the object, and the intention prediction is performed by analyzing the action sequence of the user, which significantly improves the accuracy of grasping the target object in a complex environment, reduces the failure rate of the grasping task, and improves the user experience, so that the messy scene with multiple objects randomly placed can be processed, and the target object can be accurately identified and flexibly grasped; in addition, the present disclosure can predict the action sequence that is continuous and real-time in time, that is, it can plan a continuous and natural trajectory in combination with the environment and ensure real-time performance, so as to ensure efficient and accurate completion of the task; furthermore, the present disclosure also predicts the action sequence of several (first predetermined number) time frames at one time according to the action sequence of the current time frame, that is, predicts an action block, so that the problem of cumulative error in imitation learning is solved by using the action block technology, and the prediction frequency is reduced by action block. Therefore, through the present disclosure, the problem that the existing technology cannot accurately and efficiently grasp the target object can be effectively solved.

[0033] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other objects and features of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings showing the embodiments, in which:

[0035] Figure 1 is a flowchart showing a method for controlling a robot according to an embodiment of the present disclosure;

[0036] Figure 2 is a schematic diagram showing the overall flow of a control system of a robot according to an embodiment of the present disclosure;

[0037] Figure 3 is a schematic diagram showing a scene perception process of an embodiment of the present disclosure;

[0038] Figure 4 is a schematic diagram showing the intention recognition process of an embodiment of the present disclosure;

[0039] Figure 5 is a schematic diagram showing a continuous grasping planning process according to an embodiment of the present disclosure;

[0040] Figure 6 is a schematic diagram showing a control system of a robot according to an embodiment of the present disclosure;

[0041] Figure 7 is a block diagram showing a control device of a robot according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0043] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0044] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0045] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0046] In the specification, when an element (such as a layer, a region, or a substrate) is described as being “on”, “connected to”, or “coupled to” another element, the element may be directly “on”, “connected to”, or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may be no other elements present therebetween.

[0047] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0048] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0049] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0050] The present invention proposes a human-machine fusion intelligent enhanced control framework to realize the control of the robot, so as to accurately predict the user's intention in a cluttered scene and generate a continuous five-finger grasping trajectory, thereby solving the problems of the existing human-machine fusion intelligent enhanced control method, such as limited objects and environment, low dexterity of the end effector, inaccurate prediction of user intention, discontinuous motion trajectory, and poor real-time performance.

[0051] The robot control method and device, readable storage medium, and computer program product disclosed in the present invention are described in detail below with reference to the accompanying drawings.

[0052] The present invention provides a robot control method. Figure 1 is a flowchart showing a control method of a robot according to an embodiment of the present disclosure. Figure 1 , the robot control method comprises the following steps:

[0053] In step S101, the RGB image in the current workspace is segmented to obtain a mask of each object in the RGB image.

[0054] As an example, the RGB image may be acquired through a camera, which is not limited in the present disclosure.

[0055] As an example, in order to improve the reliability of motion planning and object segmentation algorithms, the RGB image obtained from the camera is preprocessed, that is, the image of the current workspace is filtered out from the RGB image, that is, the RGB image of the current workspace in the above step. This step can filter out irrelevant information outside the current workspace and ensure that subsequent algorithms run on accurate data related to the task.

[0056] As an example, the Mask R-CNN model can be used for image segmentation to generate an accurate mask for each object in the image. In order to enhance the generalization ability of the model, BlenderProc can be used to create a synthetic dataset of a cluttered desktop scene, where objects are randomly sampled, placed, and freely fall on the table. When the object reaches a stable state, RGB images (such as RGB-D images) can be captured from random perspectives. It should be noted that the pre-trained Mask R-CNN model can be fine-tuned on the synthetic dataset through the layer freezing method, which is not limited in this disclosure. The fine-tuned Mask R-CNN model can accurately segment the RGB image and generate a mask for each object in the image, thereby identifying and locating each object.

[0057] It should be noted that after obtaining the mask of each object, the mask can be screened and deduplicated to obtain a relatively accurate object. This part will be introduced in detail later and will not be discussed here.

[0058] In step S102, based on the mask of each object, the depth image of the current workspace and the target object, a scene point cloud of the current workspace is acquired, wherein the target object and non-target objects of the current workspace are identified in the scene point cloud.

[0059] As an example, the depth image may be acquired through a camera of a camera, which is not limited in the present disclosure.

[0060] As an example, in order to improve the reliability of motion planning and object segmentation algorithms, the depth image obtained from the camera is preprocessed, that is, the image of the current workspace is filtered out from the depth image, that is, the depth image of the current workspace in the above step. This step can filter out irrelevant information outside the current workspace and ensure that subsequent algorithms run on accurate data related to the task.

[0061] According to an embodiment of the present disclosure, based on the mask of each object, the depth image of the current workspace and the target object, obtaining the scene point cloud of the current workspace may include: after multiplying the mask of each object with the depth image, obtaining the initial point cloud of each object based on the multiplication result and the internal parameters of the camera; inputting the initial point cloud of each object into the point cloud completion module to obtain the point cloud of each object; adding a feature channel to the point cloud of each object, and obtaining the scene point cloud based on the point cloud with the feature channel added, wherein the added feature channel is used to mark the corresponding object as a target object or a non-target object. Through this embodiment, the scene point cloud can be conveniently and quickly obtained based on the point cloud of each object.

[0062] As an example, the depth image of the scene can be multiplied with the mask of each object, and after obtaining the multiplication result, the point cloud of each object is restored using the camera intrinsic parameter based on the multiplication result, and then the restored point cloud is completed by the point cloud completion module (PoinTr), thereby obtaining the complete point cloud of each object. It should be noted that the above-mentioned point cloud completion module can be a module pre-trained based on the point cloud.

[0063] As an example, after obtaining a complete point cloud, a feature channel can be added to the complete point cloud to identify the object to achieve scene recoding. For example, the target object and the non-target object can be marked as 1 and 0 respectively, which is not limited in the present disclosure. It should be noted that the scene point cloud can be a summary of the point clouds of each object.

[0064] According to an embodiment of the present disclosure, before obtaining the scene point cloud of the current workspace based on the mask of each object, the depth image of the current workspace and the target object, a mask with a confidence greater than a first predetermined value can be selected from the mask of each object as an initial mask; the masks with an overlap rate greater than a second predetermined value in the initial mask are used as multiple masks of the same object, and the mask with the highest confidence among the multiple masks is used as the final mask of the same object. Through this embodiment, after obtaining the mask of the object, screening and deduplication processing are performed to obtain an accurate mask for each object, avoiding the situation where multiple masks appear for an object and the object is not recognized.

[0065] As an example, in order to reconstruct the point cloud of an object from a depth image, the mask of the object can be processed to ensure accurate object recognition. First, in order to avoid the situation where the object is not recognized, a mask with a confidence greater than a first predetermined value can be selected from the mask obtained by the segmentation process as the initial mask. Secondly, in order to solve the situation where a single object may be predicted with multiple masks, the masks in the initial masks with an overlap rate exceeding a second preset value can be grouped, and the initial mask with the highest confidence can be selected as the final mask of the object.

[0066] As an example, the first predetermined value may be 0.4, or other suitable values, which are determined according to actual conditions and are not limited in the present disclosure.

[0067] As an example, the second predetermined value may be 80%, or other suitable values, which are determined according to actual conditions and are not limited in the present disclosure.

[0068] According to an embodiment of the present disclosure, the target object can be determined in the following manner: input the point cloud of each object in the current workspace into the object encoder to obtain the feature information of each object; input the action sequence of the robot in the second time length into the motion encoder to obtain the motion features of the robot, wherein the second time length includes the second predetermined number of time frames before the current time frame; input the feature information and the motion features into the intention decoder to predict the probability information of each object interacting with the robot; based on the probability information, determine the target object in the current workspace. Through this embodiment, the feature information in the point cloud of the object and the motion features of the robot are used to predict the probability of interaction between each object and the robot. Based on the interaction probability, the target object can be determined relatively accurately, thereby improving the prediction accuracy of the grasping target in a complex environment, reducing the failure rate of the grasping task, and improving the user experience.

[0069] As an example, the object encoder may use PointNet; the motion encoder may use a bidirectional gated recurrent unit (Bi-GRU) architecture to enhance the capture of bidirectional dependencies and improve performance.

[0070] As an example, after obtaining the point cloud of a single object in a given scene and its corresponding robot motion data, this embodiment can directly predict the current human intention of each object in the given scene, that is, whether it is a target object. Feature extraction is implemented by an object encoder and a motion encoder. Specifically, the point cloud of the object is input into the object encoder to process the point cloud of the object and obtain 256-dimensional feature information; the action sequence of the robot in the past several frames is input into the motion encoder to process the action sequence of the past several time frames and obtain 256-dimensional motion features. It should be noted that the data of each time frame can be represented by the key points of the differentiable MANO model. Then, the feature information and the motion features are connected and input into the intention decoder to obtain the probability information of each object interacting with the robot, wherein the intention decoder, as a binary classifier, can use the Sigmoid activation function to generate a probability score reflecting the human intention to interact with the object, that is, the probability information.

[0071] As an example, after obtaining the probability information of each object interacting with the robot, the object corresponding to the maximum probability information can be used as the target object, that is, the object is most likely to be the object to be grasped by the robot.

[0072] As an example, the second predetermined number is determined according to actual conditions, and the present disclosure does not limit it.

[0073] In step S103, the scene point cloud, the robot's action sequence in the current time frame, and the latent variables sampled from the latent variable distribution are input into the action prediction decoder to predict the robot's action sequence in the first time length, wherein the latent variable distribution is the action sequence distribution obtained when training the action prediction decoder, and the first time length includes the first predetermined number of time frames after the current time frame.

[0074] As an example, during the training of the action prediction decoder, the action sequence distribution, that is, the latent variable distribution in the above step, can be obtained, a latent variable can be randomly sampled from the latent variable distribution, and input into the action prediction decoder together with the scene point cloud and the action sequence of the robot's current time frame to obtain the action sequence of the robot in the first time length. This embodiment ensures that the distribution of the output action sequence adapts to the latent variable distribution through latent variable input.

[0075] As an example, some time-continuous action sequences can be grouped together and executed as a block. In the implementation process, the size of the block is fixed, and each time the action sequence is predicted, the action sequence of a block is predicted. Specifically, each time the prediction is made, the action sequence of the next block is generated based on the scene point cloud, the action sequence of the robot's current time frame, and the randomly sampled latent variables. This embodiment uses the action block technology to solve the problem of cumulative error in imitation learning, and the prediction frequency can also be reduced through action block.

[0076] As an example, the above-mentioned first predetermined number is determined according to actual conditions, and the present disclosure does not limit it.

[0077] According to an embodiment of the present disclosure, a scene point cloud, an action sequence of the robot in the current time frame, and latent variables sampled from a latent variable distribution are input into an action prediction decoder to predict the action sequence of the robot in the first time length, which may include: sampling multiple latent variables from a latent variable distribution; for each of the multiple latent variables, performing the following processing: inputting the scene point cloud, the action sequence of the robot in the current time frame, and the latent variables into an action prediction decoder to predict the action sequence of the robot in the first time length; for each action sequence of the first time length, performing the following processing: based on the action sequence, obtaining contact points on a priori contact surface of the robot's hand, and determining a first distance from each contact point to the nearest object point on the target object, wherein the priori contact surface is a side close to the object when grasping the object; based on the action sequence, obtaining the hand vertices of the robot located inside the target object, determining a second distance from each hand vertex to the nearest object point, and determining a third distance from all hand vertices of the robot to their respective nearest background object points; based on the first distance, the second distance, and the third distance, selecting an action sequence from the action sequences corresponding to each of the multiple latent variables as the final action sequence of the robot in the first time length.

[0078] Through this embodiment, based on the action sequence of one time frame, multiple action sequences of subsequent time frames can be predicted, and the best action sequence can be selected from the multiple predicted ones as the predicted final action sequence, thereby improving the prediction accuracy and further ensuring the accuracy of robot grasping.

[0079] As an example, B latent variables can be sampled from the latent variable distribution each time, and B action sequences of the first duration are generated, each action sequence contains k actions, and then a scoring mechanism is constructed to comprehensively evaluate the B action sequences through three indicators, and select an optimal action sequence as the final action sequence of the first duration. The above three indicators can be the first distance, the second distance and the third distance of the above embodiment, or other indicators, which are not limited by the present disclosure.

[0080] The following is a detailed description of the three indicators of this embodiment;

[0081] 1) The first distance, also known as the interaction loss The goal of this interaction loss is to encourage the action sequence to approach the target object, that is, for each predicted action sequence First, select the points belonging to the prior contact surface (the palm surface in this embodiment) from the action sequence. Then calculate the point V of the prior contact surface P To the target object point cloud Specifically, for each contact point Calculate the distance from it to the nearest object point on the target object Then calculate the interaction loss:

[0082]

[0083] Among them, N1 represents the number of points on the prior contact surface, Represents the jth point of the target object point cloud. For each contact point, find the point of the target object point cloud closest to it. That is, B represents that point, B represents B latent variables corresponding to B trajectories, so A B-dimensional array will be returned.

[0084] 2) The second distance, also known as embedding loss The embedding loss evaluates the embedding of the predicted action sequence with the target object in the scene. The goal is to penalize the embedding of the action sequence into the target object, that is, for each predicted action sequence And scene point cloud Calculate the robot's hand vertices that are inside the target object Recalculate The distance to the nearest object point on the target object Then calculate the embedding loss:

[0085]

[0086] in, represents the i-th hand vertex inside the scene point cloud, represents the jth point of the scene point cloud, B represents B latent variables corresponding to B trajectories, so A B-dimensional array will be returned.

[0087] 3) The third distance, also known as potential background collision loss The potential collision loss aims to penalize action sequences that are close to background objects, that is, for each predicted action sequence and background object point cloud For each hand vertex of the robot, calculate the distance from it to the nearest background object point Then calculate the potential collision loss:

[0088] For all Among them, N2 represents the number of points in the background point cloud, represents the i-th point of the background object point cloud, B represents B latent variables corresponding to B trajectories, so Will return a B-dimensional array, Indicates the threshold value, for example, It means that when the distance is less than 0.05m*0.05m, a potential collision is calculated. Otherwise, it is too far and a collision is unlikely.

[0089] In order to ensure that only valid samples are selected, all action sequences whose embedding loss is not 0 are first excluded, and then the interaction loss and potential collision loss are normalized separately. After weighting, we get the total loss, and then find the index of the minimum value in the total loss. The action sequence corresponding to this index is the optimal action sequence. The total loss is:

[0090]

[0091] Among them, λ inter and λ col Represent the corresponding weights respectively.

[0092] It should be noted that the hand vertices of the robot inside the target object can be found in the following way: given the point cloud of the robot hand, the hand normal (directly calculated by point cloud + triangular surface, each mesh is composed of points and surfaces), and the object point cloud, then find the nearest hand point corresponding to each point on the object. For example, if the object has 3000 points and the hand has 778 points, then the indexes of 3000 hand points (with duplication) will be obtained in the end, and these indexes correspond to the points on the object one by one. Based on 3000 hand points to 3000 object points, 3000 vectors V1 will be obtained, and then the hand normals will be taken out according to the indexes of 3000 hand points to obtain 3000 hand normals (with duplication) V2. The inner product of the obtained V1 and V2 is calculated. The inner product of the points inside will be greater than 0, and the inner product of the points outside will be less than 0. From this, the hand vertices of the robot located inside the target object can be determined.

[0093] According to an embodiment of the present disclosure, the action prediction decoder can be trained in the following manner: obtaining training samples, wherein the training samples include a given scene point cloud, an action sequence of a given time frame, and a future action sequence, wherein the future action sequence includes an action sequence of a first predetermined number of time frames after the given time frame; inputting the training samples into the initial action prediction encoder to obtain a latent variable distribution; inputting the latent variables sampled from the latent variable distribution, the given scene point cloud, and the action sequence of the given time frame into the initial action prediction decoder to predict the robot's estimated action sequence for the first duration; based on the loss between the estimated action sequence and the future action sequence and the KL loss, adjusting the parameters of the initial action prediction encoder and the initial action prediction decoder to obtain an action prediction decoder. Through this embodiment, given the training samples and the loss function, the training of the action prediction decoder can be conveniently and quickly implemented.

[0094] As an example, this embodiment uses a generative model based on a conditional variational autoencoder (CVAE), which uses conditional information to control generation, where the conditional information is the scene point cloud and the action sequence of the current time frame. During the training phase, both the encoder and decoder of the model are used to learn the grasp generation task, taking the scene point cloud, the action sequence of a given time frame, and the future action sequence as input; during testing, only the decoder is used to generate a grasp with only the scene point cloud and the action sequence of the current time frame as input.

[0095] As an example, during the training process, given a scene point cloud, an action sequence and a future action sequence of a given time frame, wherein the action sequence and the future action sequence of a given time frame can be represented by the key points of a differentiable MANO model, and PointNet is used to extract the features of the scene point cloud, and two independent multi-layer perceptrons (MLPs) extract the action sequence and the future action sequence of a given time frame. In the training stage, the features of the scene point cloud and the features of the action sequence of a given time frame can be connected as conditional features, and the conditional features and the features of the future action sequence can be further connected as the input of the encoder in the generative model of CVAE (i.e., the action prediction encoder in the above embodiment), thereby obtaining the mean and variance of the posterior Gaussian distribution, i.e., the latent variable distribution. In order to reconstruct the future action sequence, first a latent variable is sampled from the posterior Gaussian distribution, and the decoder in the generative model of CVAE (i.e., the action prediction decoder in the above embodiment) takes the connection of the latent variable and the conditional features as input to predict the action sequence of several time frames following the given time frame. It should be noted that the encoder and decoder of the conditional variational autoencoder model can both use multi-layer perceptrons, which is not limited by the present disclosure.

[0096] As an example, after predicting the action sequence of several time frames after a given time frame, the parameters of the encoder and decoder can be adjusted based on a given objective function to achieve the training of the encoder and decoder. The given objective function can be composed of two parts, the first part is the reconstruction error, which is defined as the loss between the output and the future action sequence (that is, the parameters of the MANO model), and the second part is the KL loss, which ensures that the distribution of the latent variable is close to the standard Gaussian distribution.

[0097] As an example, in actual testing, only the decoder is used for inference, that is, the features of the scene point cloud, the features of the action sequence of the current time frame, and the latent variables randomly sampled from the Gaussian distribution are used as the input of the decoder, and the decoder will generate a continuous and natural sequence of action frames.

[0098] In step S104, the predicted action sequence is mapped to the robot to control the robot to grasp the target object.

[0099] As an example, the mapping here refers to mapping the predicted action sequence to the robot hand. Specifically, the redirection problem can be formulated as a position-based optimization problem, which is discussed in detail below:

[0100] According to an embodiment of the present disclosure, mapping a predicted action sequence to a robot may include: matching the hand posture in the predicted action sequence with the robot's hand; matching the hand position in the predicted action sequence with the robot's hand; and regularizing the joint angles of the matched robot's hand. Through this embodiment, the first part of the matching is used to make the robot's hand match the predicted hand posture as much as possible, the second part of the matching is used to encourage the robot's hand to be closer to the predicted hand position, and the third part can enhance smoothness and time consistency.

[0101] As an example, the objective function used for the mapping may be as follows:

[0102]

[0103] The first term in the above formula is to encourage the robot hand to match the predicted MANO hand posture as much as possible (i.e., the hand posture in the predicted action sequence in this embodiment). Here, and denote the joint angles of the MANO hand and the robot hand at time t, and represents the vectors obtained by the positive kinematics of the MANO hand and the robot hand in the world coordinate system, determined by the pre-selected key points, n represents the total number of vectors, and in addition, and w1(d i ) represents the weight based on length or distance, s(d i ) represents a weight based on length or distance; the second item is to encourage the robot hand to be closer to the predicted MANO hand position (i.e., the hand position in the predicted action sequence in this embodiment), where, represents the position of the jth key point calculated by forward kinematics. In this embodiment, the wrist and the middle fingertip are selected as the position key points, and w2 represents the weight; the third term is an L2 regularization term, which is intended to enhance smoothness and time consistency, and w3 represents the weight. In the optimization process, the sequential least squares (SLSQP) algorithm can be used to iteratively optimize the above objective function.

[0104] According to an embodiment of the present disclosure, before the predicted action sequence is mapped to the robot to control the robot to grasp the target object, the following processing can also be performed for each time frame in the current time frame and the first time length: the scene point cloud, the action sequence of the robot in the time frame, and the latent variables sampled from the latent variable distribution are input into the action prediction decoder to predict the action sequence of the robot in the third time length, wherein the third time length is the second predetermined number of time frames after the time frame, and the second predetermined number is less than the first predetermined number; all the action sequences belonging to the same time frame in the third time length are aggregated to obtain the final action sequence of each time frame. Through this embodiment, in the prediction process, an action block can be predicted for each time frame, so that the time frames contained in each action block will overlap with each other, and the action sequences belonging to the same time frame in each action block are aggregated, thereby achieving accurate and smooth movement.

[0105] As an example, although simple action segmentation in step S103 can reduce the prediction frequency, it is likely to cause incoherent movement of the robot. Therefore, in order to improve smoothness, this embodiment predicts an action block for each time pointer, so that the time frames contained in each action block will overlap with each other. At this time, a linear weighted time integration method is introduced to aggregate the action sequences belonging to the same time pointer in each action, thereby achieving precise and smooth movement.

[0106] According to an embodiment of the present disclosure, before aggregating the action sequences belonging to the same time frame in the third duration to obtain the final action sequence of each time frame, it also includes: for each action sequence in all the action sequences of the third duration and the first duration, the following processing is performed: based on the action sequence, the contact points on the prior contact surface of the robot hand are obtained; the first distance from each contact point to the nearest object point on the target object is determined, wherein the prior contact surface is the side close to the object when grasping the object; when the distance begins to increase, the latent variables are resampled from the latent variable distribution and the action sequence is predicted; when the distance is less than the third predetermined value and the number of samplings exceeds the preset number, the action sequence is stopped from being predicted. Through this embodiment, inaccurate predictions or too many predictions can be avoided.

[0107] As an example, the present disclosure focuses on grasping rather than subsequent operations, so it is necessary to introduce a stop signal in this embodiment, because the continuous generation of action sequences may cause the robot to approach the target first and then move away, so a clear stop instruction is required to ensure the rationality of the action sequence. This embodiment can evaluate the proximity of the action sequence to the target object by monitoring the interaction loss. When the interaction loss continues to decrease, the action sequence can be continuously generated. Once the interaction loss begins to increase, the latent variables will be resampled to predict the action sequence. If the interaction loss is lower than the third predetermined value and the sampling number exceeds the preset number during the prediction process, the generation of the action sequence is stopped, that is, the prediction of the action sequence is stopped.

[0108] As an example, the third predetermined value and the preset number of times can be determined according to actual conditions, and the present disclosure does not limit them.

[0109] According to an embodiment of the present disclosure, after the predicted action sequence is mapped to the robot, it also includes: for the last time frame, the following processing is performed: determine the contact points on the prior contact surface among the hand vertices of the robot in the time frame, and obtain the fourth distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is the side close to the object when grasping the object; obtain the hand vertices located inside the target object from the hand vertices of the robot in the time frame, and determine the fifth distance from each hand vertex inside the target object to the nearest object point; obtain the sixth distance of the robot's grasping joint angle before and after adjustment; based on the fourth distance, the fifth distance and the sixth distance, adjust the robot's action to control the robot to grasp the target object. Through this embodiment, grasping optimization is performed, and the problem of insufficient close contact between the hand and the object and slight embedding of the hand in the object caused by the generated action sequence is further overcome.

[0110] As an example, although the previous scheme of this embodiment can generate a good action sequence through continuous grasping and grasping mapping, there are still two situations that affect the grasping success rate: 1) the generated action sequence causes the hand-object contact to be not close enough, 2) the generated action sequence causes the hand to be slightly embedded in the object. It should be noted that although the evaluation mechanism can avoid this situation with a high probability, the above problems may still exist due to the essential differences in the structure of the robot's hand and the MANO model with a very small probability. Therefore, this embodiment introduces an optimization-based grasping improvement mechanism. The above problem can be expressed as a nonlinear optimization problem, and its specific objective function consists of three parts, namely the fourth distance (contact loss), the fifth distance (embedding loss) and the sixth distance (smoothing loss), as follows:

[0111]

[0112] Among them, λ Cont ,λ′ pene and λsmoo Represent the corresponding weights respectively.

[0113] 1) Contact loss The goal is to encourage the robot's hand to make closer contact with the target object. Given the mapping, the robot's hand vertices in the last frame First, from the hand vertex, point V that belongs to the prior contact surface RC Sampling is performed in It is worth noting that if the contact points of the five fingers of the robot are uniformly sampled, although optimization can be achieved, it is easy to fall into the dilemma of optimizing the four fingers and ignoring the thumb. For tight grasping, the thumb is in a more important position. Therefore, this embodiment samples more contact points of the thumb. Then calculate V′ RC To the target object point cloud Specifically, for each contact point V i ' RC , calculate the distance from it to the nearest object point on the target object Then calculate the interaction loss:

[0114]

[0115] Among them, N4 represents the number of points on the prior contact surface, Represents the jth point in the point cloud of the target object.

[0116] 2) Embedding Loss Used to calculate the embedding of the optimized robot's hand and the objects in the scene. The goal is to penalize the robot's hand for embedding objects. Given the mapping, the robot's hand vertices in the last frame And scene point cloud Calculate the hand vertices of the robot's hand that are inside the target object Then calculate The distance to the nearest object point on the target object Then calculate the embedding loss:

[0117]

[0118] in, represents the i-th hand vertex, Represents the jth point of the scene point cloud.

[0119] 3) Smoothing loss Defined as the distance between grasps before and after improvement, the grasping joint angle after improvement is encouraged Close to the grip joint angle before improvement

[0120]

[0121] Among them, N5 represents the number of degrees of freedom of the robot, and Represent the joint angles before and after improvement respectively.

[0122] In the optimization solution, the sequential least squares (SLSQP) algorithm can be used to optimize the above objective function in real time, which is not limited in the present disclosure.

[0123] In order to facilitate understanding of the technical solution of the present disclosure, Figures 2 to 5 Discuss in detail.

[0124] Figure 2 is a schematic diagram showing the overall flow of the control system of the robot of the embodiment of the present disclosure, such as Figure 2 As shown, the present disclosure provides a human-machine fusion intelligent enhanced control process, which mainly includes three components, namely a scene perception part, an intention recognition part and a continuous grasping planning part.

[0125] The above scene perception part can be composed of three consecutive steps, namely background filtering, instance segmentation and point cloud generation, such as Figure 3 As shown, first, background filtering is performed: the RGB image and depth image obtained from the camera are preprocessed, that is, the image of the current workspace is screened out from the RGB image and the depth image, that is, the RGB image and the depth image of the current workspace in the above embodiment. This step can filter out irrelevant information outside the current workspace to ensure that subsequent algorithms run on accurate data related to the task; secondly, instance segmentation is performed, and the Mask R-CNN model can be used for image segmentation to generate an accurate mask for each object in the image; finally, point cloud generation is performed, and the depth image of the scene can be multiplied by the mask of each object. After obtaining the multiplication result, the point cloud of each object is restored using the camera intrinsic parameter based on the multiplication result, and then the restored point cloud is completed by the point cloud completion module (PoinTr) to obtain a complete point cloud for each object.

[0126] The above intention recognition part is as follows Figure 4 As shown in Figure 1, the point cloud of the object is input into the object encoder (PointNet) to obtain 256-dimensional feature information; the action sequence of the robot in the past several frames (i.e. Figure 4The past action sequence in the image is input into the motion encoder (Bi-GRU) to obtain 256-dimensional motion features. Then, the intention is predicted, that is, the feature information and the motion feature are connected and input into the intention decoder to obtain the probability information of each object interacting with the robot. The object corresponding to the maximum probability information can be used as the target object. Then, the scene is re-encoded, that is, the scene is re-encoded by adding a feature channel to the complete point cloud to identify the object. For example, the target object and non-target object can be marked as 1 and 0 respectively, and then a labeled point cloud, that is, a scene point cloud, is obtained.

[0127] The above continuous grasp planning part can include three consecutive steps, namely continuous grasp generation, action mapping and grasp optimization. Figure 5 As shown in Figure 1, first, a generative model based on conditional variational autoencoder (CVAE) is adopted, which uses conditional information to control generation, where the conditional information is the scene point cloud and the action sequence of the current time frame (i.e. Figure 5 In the training phase, both the encoder and decoder of the model are used to learn the grasp generation task, taking the scene point cloud, the action sequence of a given time frame, and the future action sequence as input; during testing, only the decoder is used, that is, only the scene point cloud and the action sequence of the current time frame are used as input to predict the robot's action sequence in a certain time. Secondly, the predicted action sequence is mapped to the robot hand. Then, grasp optimization is performed to further overcome the problem that the generated action sequence causes the hand-object contact to be not close enough and the hand is slightly embedded in the object. It should be noted that the specific training process, testing process, mapping process, and grasp optimization process have been introduced in detail above and will not be discussed here.

[0128] According to an embodiment of the present disclosure, the present disclosure also provides a control system of a robot, the control system includes a user end and a robot end, the user end includes a data acquisition device and an inertial measurement device, the robot end includes a camera, a robotic arm and a robotic hand, wherein the data acquisition device is used to collect user hand data; the inertial measurement device is used to measure the user's hand position and hand posture; the camera is used to acquire RGB images and depth images; the robotic arm is used to move according to a predicted action sequence; the robotic hand is used to grasp the target object according to the predicted action sequence, and the robotic hand is a five-finger dexterous hand.

[0129] As an example, this embodiment also provides a control system of a robot. Figure 6As shown in the figure, the system includes two parts: the master end (i.e., the user end) and the slave end (i.e., the robot end). The master end includes: a data glove (i.e., a data acquisition device) for collecting the user's hand joint data; three inertial measurement units for measuring the position and posture of the user's hand. The slave end includes: a camera (i.e., a video camera) for sensing the scene, such as acquiring RGB images and depth images; a robotic arm for moving along the target path indicated by the predicted action sequence; and a CASIA dexterous hand (i.e., a robotic arm) for grasping the target object.

[0130] The advantages brought by the technical solution of the present disclosure are specifically as follows:

[0131] 1) Applicable to a variety of object types: The technical solution disclosed in the present invention enables the robot to grasp common objects in daily life, including objects with complex and irregular shapes, significantly expanding the scope of application and solving the limitation that the existing method can only grasp simple and regular objects, making it more practical in actual production and life.

[0132] 2) Effectively handle cluttered scenes: The design of the present disclosure enables the robot to handle cluttered scenes with multiple objects randomly placed, accurately identify and flexibly grasp the target object, and avoid collision with non-target objects. This feature greatly improves the application ability of the system in unstructured complex environments and overcomes the shortcomings of existing solutions in such scenes.

[0133] 3) High-dexterity end effector: The robot disclosed in the present invention adopts a five-finger dexterous hand as the end effector, which can effectively grasp irregularly shaped objects and perform complex operation tasks, solving the limitation of low dexterity of the end effector in the existing solutions and improving the practical application capability of the system.

[0134] 4) Accurately predict user intentions: The present invention predicts intentions by analyzing the user's action sequence, which significantly improves the prediction accuracy of grasping target objects in complex environments, reduces the failure rate of grasping tasks, and improves user experience.

[0135] 5) Continuous and real-time trajectory planning: The technical solution disclosed in the present invention can plan continuous and natural trajectories in combination with the environment and ensure real-time performance, thereby ensuring that tasks are completed efficiently and accurately.

[0136] Figure 7 is a block diagram showing a control device of a robot according to an embodiment of the present disclosure, Figure 7 As shown, the device includes a segmentation unit 70, a point cloud acquisition unit 72, a prediction unit 74 and a mapping unit 76.

[0137] The segmentation unit 70 is configured to perform image segmentation on the RGB image in the current workspace to obtain the mask of each object in the RGB image; the point cloud acquisition unit 72 is configured to obtain the scene point cloud of the current workspace based on the mask of each object, the depth image of the current workspace and the target object, wherein the target object and non-target object of the current workspace are identified in the scene point cloud; the prediction unit 74 is configured to input the scene point cloud, the action sequence of the robot in the current time frame and the latent variables sampled from the latent variable distribution into the action prediction decoder to predict the action sequence of the robot in the first time length, wherein the latent variable distribution is the action sequence distribution obtained when training the action prediction decoder, and the first time length includes the first predetermined number of time frames after the current time frame; the mapping unit 76 is configured to map the predicted action sequence to the robot to control the robot to grasp the target object.

[0138] According to an embodiment of the present disclosure, the point cloud acquisition unit 72 is also configured to select a mask with a confidence greater than a first predetermined value from the mask of each object as an initial mask before acquiring the scene point cloud of the current workspace based on the mask of each object, the depth image of the current workspace and the target object; use the masks with an overlap rate greater than a second predetermined value in the initial masks as multiple masks of the same object, and use the mask with the highest confidence among the multiple masks as the final mask of the same object.

[0139] Optionally, the target object is determined in the following manner: inputting the point cloud of each object in the current workspace into an object encoder to obtain feature information of each object; inputting the robot's action sequence in a second time length into a motion encoder to obtain the robot's motion features, wherein the second time length includes a second predetermined number of time frames before the current time frame; inputting the feature information and motion features into an intention decoder to predict probability information of each object interacting with the robot; and determining the target object in the current workspace based on the probability information.

[0140] According to an embodiment of the present disclosure, the point cloud acquisition unit 72 is also configured to multiply the mask of each object with the depth image, and then obtain the initial point cloud of each object based on the multiplication result and the internal parameters of the camera; input the initial point cloud of each object into the point cloud completion module to obtain the point cloud of each object; add a feature channel to the point cloud of each object, and obtain the scene point cloud based on the point cloud with the feature channel added, wherein the added feature channel is used to mark the corresponding object as a target object or a non-target object.

[0141] According to an embodiment of the present disclosure, the mapping unit 76 is also configured to perform the following processing for the current time frame and each time frame in the first time length before mapping the predicted action sequence to the robot to control the robot to grasp the target object: input the scene point cloud, the action sequence of the robot in the time frame, and the latent variables sampled from the latent variable distribution into the action prediction decoder to predict the action sequence of the robot in the third time length, wherein the third time length is the second predetermined number of time frames after the time frame, and the second predetermined number is less than the first predetermined number; aggregate all action sequences belonging to the same time frame in the third time length to obtain the final action sequence of each time frame.

[0142] According to an embodiment of the present disclosure, the mapping unit 76 is also configured to perform the following processing on each action sequence in all action sequences of the third time length and the first time length before aggregating the action sequences belonging to the same time frame in the third time length to obtain the final action sequence of each time frame: based on the action sequence, obtain the contact points on the prior contact surface of the robot hand; determine the first distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is the side close to the object when grasping the object; when the distance starts to increase, re-sample the latent variables from the latent variable distribution and predict the action sequence; when the distance is less than the third predetermined value and the number of samplings exceeds the preset number, stop predicting the action sequence.

[0143] According to an embodiment of the present disclosure, the prediction unit 74 is also configured to sample multiple latent variables from the latent variable distribution; for each of the multiple latent variables, the following processing is performed: the scene point cloud, the robot's action sequence in the current time frame and the latent variables are input into the action prediction decoder to predict the robot's action sequence in the first time length; for each action sequence of the first time length, the following processing is performed: based on the action sequence, the contact points on the prior contact surface of the robot's hand are obtained, and a first distance from each contact point to the nearest object point on the target object is determined, wherein the prior contact surface is a side close to the object when grasping the object; based on the action sequence, the hand vertices of the robot located inside the target object are obtained, and the second distance from each hand vertex to the nearest object point is determined, and the third distance from all the hand vertices of the robot to their respective nearest background object points is determined; based on the first distance, the second distance and the third distance, an action sequence is selected from the action sequences corresponding to each of the multiple latent variables as the final action sequence of the robot in the first time length.

[0144] According to an embodiment of the present disclosure, the mapping unit 76 is also configured to perform the following processing for the last time frame after mapping the predicted action sequence to the robot: determine the contact points on the prior contact surface among the hand vertices of the robot in the time frame, and obtain the fourth distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is a side close to the object when grasping the object; obtain the hand vertices located inside the target object from the hand vertices of the robot in the time frame, and determine the fifth distance from each hand vertex inside the target object to the nearest object point; obtain the sixth distance of the robot's grasping joint angle before and after adjustment; and adjust the robot's action based on the fourth distance, the fifth distance, and the sixth distance to control the robot to grasp the target object.

[0145] According to an embodiment of the present disclosure, an action prediction decoder is trained in the following manner: obtaining training samples, wherein the training samples include a given scene point cloud, an action sequence of a given time frame, and a future action sequence, wherein the future action sequence includes an action sequence of a first predetermined number of time frames after the given time frame; inputting the training samples into an initial action prediction encoder to obtain a latent variable distribution; inputting latent variables sampled from the latent variable distribution, a given scene point cloud, and an action sequence of a given time frame into an initial action prediction decoder to predict an estimated action sequence of the robot at the first time length; adjusting parameters of the initial action prediction encoder and the initial action prediction decoder based on the loss between the estimated action sequence and the future action sequence and the KL loss to obtain an action prediction decoder.

[0146] According to an embodiment of the present disclosure, the mapping unit 76 is further configured to match the hand posture in the predicted action sequence with the hand of the robot; match the hand position in the predicted action sequence with the hand of the robot; and regularize the joint angles of the matched robot's hand.

[0147] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute a robot control method as described in any of the above embodiments.

[0148] According to an embodiment of the present disclosure, a system is provided, comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute a robot control method as described in any of the above embodiments.

[0149] According to an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which implement any of the above-mentioned robot control methods when executed by a processor.

[0150] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A robot control method, characterized in that: include: Perform image segmentation on the RGB image in the current workspace to obtain a mask for each object in the RGB image; Based on the mask of each object, the depth image of the current workspace and the target object, a scene point cloud of the current workspace is acquired, wherein the target object and non-target objects of the current workspace are identified in the scene point cloud; Inputting the scene point cloud, the action sequence of the robot in the current time frame and the latent variables sampled from the latent variable distribution into an action prediction decoder, predicting the action sequence of the robot in a first time length, wherein the latent variable distribution is the action sequence distribution obtained when training the action prediction decoder, and the first time length includes a first predetermined number of time frames after the current time frame; The predicted action sequence is mapped to the robot to control the robot to grasp the target object.

2. The control method according to claim 1, characterized in that: The target object is determined by: Inputting the point cloud of each object in the current workspace into an object encoder to obtain feature information of each object; Inputting the motion sequence of the robot in a second time length into a motion encoder to obtain the motion characteristics of the robot, wherein the second time length includes a second predetermined number of time frames before the current time frame; Inputting the feature information and the motion feature into an intention decoder to predict probability information of each object interacting with the robot; Based on the probability information, a target object in the current workspace is determined.

3. The control method according to claim 2, characterized in that: The step of acquiring a scene point cloud of the current workspace based on the mask of each object, the depth image of the current workspace and the target object comprises: After multiplying the mask of each object with the depth image, an initial point cloud of each object is obtained based on the multiplication result and the internal parameters of the camera; Input the initial point cloud of each object into the point cloud completion module to obtain the point cloud of each object; A feature channel is added to the point cloud of each object, and the scene point cloud is obtained based on the point cloud to which the feature channel is added, wherein the added feature channel is used to mark the corresponding object as a target object or a non-target object.

4. The control method according to claim 1, characterized in that: Before mapping the predicted action sequence to the robot to control the robot to grasp the target object, the method further includes: For each time frame in the current time frame and the first duration, the following processing is performed: Inputting the scene point cloud, the action sequence of the robot in the time frame and the latent variables sampled from the latent variable distribution into an action prediction decoder, predicting the action sequence of the robot in a third time length, wherein the third time length is a second predetermined number of time frames after the time frame, and the second predetermined number is less than the first predetermined number; All action sequences belonging to the same time frame in the third duration are aggregated to obtain a final action sequence in each time frame.

5. The control method according to claim 4, characterized in that: Before aggregating the action sequences belonging to the same time frame in the third duration to obtain the final action sequence of each time frame, the method further includes: For each action sequence in all the action sequences of the third duration and the first duration, the following processing is performed: based on the action sequence, the contact points on the prior contact surface of the robot hand are obtained; the first distance from each contact point to the nearest object point on the target object is determined, wherein the prior contact surface is a side close to the object when grasping the object; When the first distance starts to increase, re-sampling latent variables from the latent variable distribution and predicting the action sequence; When the first distance is less than a third predetermined value and the number of sampling times exceeds a preset number, the prediction of the action sequence is stopped.

6. The control method according to claim 1, characterized in that: The step of inputting the scene point cloud, the action sequence of the robot in the current time frame, and the latent variables sampled from the latent variable distribution into the action prediction decoder to predict the action sequence of the robot in the first time frame comprises: sampling a plurality of latent variables from the latent variable distribution; For each of the multiple latent variables, the following processing is performed: the scene point cloud, the action sequence of the robot in the current time frame and the latent variable are input into an action prediction decoder to predict the action sequence of the robot in the first time frame; For each action sequence of the first duration, the following processing is performed: based on the action sequence, the contact points on the prior contact surface of the robot hand are obtained, and a first distance from each contact point to the nearest object point on the target object is determined, wherein the prior contact surface is a side close to the object when grasping the object; based on the action sequence, the hand vertices of the robot located inside the target object are obtained, and a second distance from each hand vertex to the nearest object point is determined, and a third distance from all hand vertices of the robot to their respective nearest background object points is determined; Based on the first distance, the second distance and the third distance, an action sequence is selected from the action sequences corresponding to the multiple latent variables as the final action sequence of the robot in the first duration.

7. The control method according to claim 1, characterized in that: After mapping the predicted action sequence to the robot, it also includes: For the last time frame, the following processing is performed: determine the contact points on the prior contact surface among the hand vertices of the robot in the time frame, and obtain the fourth distance from each contact point to the nearest object point on the target object, wherein the prior contact surface is a side close to the object when grasping the object; obtain the hand vertices located inside the target object from the hand vertices of the robot in the time frame, and determine the fifth distance from each hand vertex inside the target object to the nearest object point; obtain the sixth distance of the robot grasping joint angle before and after adjustment; Based on the fourth distance, the fifth distance and the sixth distance, the action of the robot is adjusted to control the robot to grasp the target object.

8. The control method according to claim 1, characterized in that: The motion prediction decoder is trained as follows: Acquire a training sample, wherein the training sample includes a given scene point cloud, an action sequence of a given time frame, and a future action sequence, wherein the future action sequence includes an action sequence of a first predetermined number of time frames after the given time frame; Inputting the training sample into an initial action prediction encoder to obtain a latent variable distribution; Inputting the latent variables sampled from the latent variable distribution, the given scene point cloud and the action sequence of the given time frame into an initial action prediction decoder to predict the estimated action sequence of the robot in the first time length; Based on the loss between the estimated action sequence and the future action sequence and the KL loss, the parameters of the initial action prediction encoder and the initial action prediction decoder are adjusted to obtain the action prediction decoder.

9. A robot control device, characterized in that: include: A segmentation unit is configured to perform image segmentation on the RGB image in the current workspace to obtain a mask of each object in the RGB image; a point cloud acquisition unit configured to acquire a scene point cloud of the current workspace based on a mask of each object, a depth image of the current workspace, and a target object, wherein the target object and non-target objects of the current workspace are identified in the scene point cloud; A prediction unit is configured to input the scene point cloud, the action sequence of the robot in the current time frame and the latent variables sampled from the latent variable distribution into an action prediction decoder, and predict the action sequence of the robot in a first time length, wherein the latent variable distribution is the action sequence distribution obtained when training the action prediction decoder, and the first time length includes a first predetermined number of time frames after the current time frame; A mapping unit is configured to map the predicted action sequence to the robot to control the robot to grasp the target object.

10. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the robot control method according to any one of claims 1 to 8.

11. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the robot control method according to any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the robot control method according to any one of claims 1 to 8 is implemented.

Images