Methods and systems for controlling robotic arms to grasp objects using policy network models

Abstract

A method for training a policy network model for a robotic arm to grasp objects is provided, comprising: synchronously recording RGB and depth images of the grasping operation scene at each time step using a camera; obtaining the spatial position of the robotic arm's end effector when it approaches and successfully grasps the object at the end of its motion trajectory; selecting a 3D bounding box region centered on the obtained spatial position in the point cloud image; using the point cloud bounding boxes of objects spatially nearest to the 3D bounding box region as a unified bounding mask; applying the unified bounding mask to the point cloud image at each time step to obtain a local mask point cloud; obtaining the robotic arm joint angles at each time step; constructing training samples with the local mask point cloud and robotic arm joint angles at the current time step as input and the robotic arm end effector poses at the next multiple time steps as output; and obtaining a policy network model based on a behavior cloning loss function. This invention is applicable to multi-object scenarios and can obtain an end-to-end model with accurate predictions based on local point clouds.

Description

Technical Field

[0001] This invention relates to controlling a robotic arm to grasp objects using artificial intelligence (AI) neural networks, and more specifically, to a method and system for controlling a robotic arm to grasp objects using a policy network model, as well as a method and system for training the policy network model. Background Technology

[0002] With the development of industrial automation and service robot technology, robotic arm grasping technology has been widely used in intelligent manufacturing, logistics sorting, and home services. In traditional robotic arm control technology, it is usually necessary to locate the target object to be grasped in an image and then plan the grasping path for the target object. Existing technologies, such as the patent CN116529760A entitled "Grasping Control Method, Device, Electronic Device and Storage Medium", focus on how to perform mask positioning and analysis of the grasping area in order to plan the corresponding robotic arm motion path based on the specific actual situation of the grasping target determined from the image (e.g., the position of the object, whether the object is overlapping, the posture and orientation of the object, etc.).

[0003] Recently, artificial intelligence technology has begun to be applied to robotic arm grasping control to replace traditional path planning methods. By allowing the model to continuously learn the optimal strategy through trial and error in interaction with the environment, an end-to-end model for robotic arm grasping control can be obtained. Such a model can learn the grasping strategy directly from the grasping scene. For this purpose, it is usually necessary to acquire a global image of the grasping scene, such as an RGB image, which helps to achieve a full and complete scene understanding at the perception level, so as to learn more detailed decision-making basis from the global image. In addition, in the autonomous grasping task of robots, point clouds are also widely used as an input form in 3D environment perception and operation decision-making. Compared with traditional RGB images, point clouds can provide richer spatial geometric information, making robots more robust when facing unknown objects and complex scenes. However, the end-to-end grasping strategy based on point clouds still faces significant challenges in practical deployment. For example, the model's decision results are often uncertain, but because the model is based on massive training samples, its "black box" nature makes it difficult to accurately locate the problem factors when adjusting or optimizing the model, i.e., it is difficult to find a starting point.

[0004] Furthermore, in multi-object scenarios, the robot's strategy needs to simultaneously face multiple candidate objects that can be grasped. In this case, the input point cloud contains multiple potential targets, whose spatial distributions and features overlap or interfere with each other. The network needs to learn how to implicitly make a "selection" in the high-dimensional feature space—that is, decide which object to grasp. However, due to the lack of explicit target indication information (such as task objectives or human intentions), this learned implicit selection is often unstable, easily leading to ambiguity or inconsistency in the output end-effector pose. For example, the trained model may randomly output grasping poses towards different targets when facing the same scene, exhibiting multimodal uncertainty in the strategy. Summary of the Invention

[0005] According to an exemplary embodiment of the present invention, a method for training a policy network model for a robotic arm to grasp objects is provided, comprising: for each complete object grasping process, while a human operator controls the robotic arm to complete continuous grasping operations using a remote control device, simultaneously recording RGB images and depth images of the grasping operation scene at each moment using a camera, wherein the camera is fixedly mounted at the end of the robotic arm and moves with the end of the robotic arm; parsing the motion trajectory of the end of the robotic arm recorded by the camera during each complete grasping process, obtaining the spatial position of the end of the robotic arm when it approaches the object at the end of the motion trajectory and successfully grasps it, and selecting a three-dimensional bounding box region centered on the obtained spatial position in the point cloud image at the corresponding moment, wherein the point cloud image is generated by depth projection of the depth image based on the intrinsic parameter model of the camera focal length; detecting at least one two-dimensional bounding box of an object from the RGB image at the corresponding moment using an object detection network model, and mapping the detected two-dimensional bounding box to the point cloud image at the corresponding moment to obtain at least one object. The object point cloud is generated, and the object point cloud that is closest to the 3D bounding box region is used as a unified bounding mask. The depth image at each time step is projected into the depth image based on the intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene. The unified bounding mask is applied to the point cloud map at each time step to obtain the local mask point cloud of the grasping target object at each time step. The joint angle of the robotic arm at each time step is obtained from the motor state of each joint of the robotic arm. For each complete object grasping process, a training sample is constructed with the local mask point cloud at the current time step and the joint angle of the robotic arm as input and the end-effector pose at the next multiple time steps as output. The end-effector pose includes the position and orientation of the end-effector, which is determined by the joint angle of the robotic arm. Based on the training sample corresponding to at least one object grasping process, learning is performed iteratively according to the behavior cloning loss function to obtain a policy network model for predicting the end-effector pose at the next multiple time steps based on the local cloud mask at the current time step and the joint angle of the robotic arm.

[0006] According to another exemplary embodiment of the present invention, a method for controlling a robotic arm to grasp an object using a policy network model is provided, comprising: synchronously recording RGB images and depth images of a grasping operation scene using a camera, wherein the camera is fixedly mounted at the end of the robotic arm and moves with the end of the robotic arm; performing depth projection on the depth image based on an intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene; detecting two-dimensional bounding boxes of object regions from the RGB images using an object detection network model, and mapping the two-dimensional bounding boxes onto the point cloud map to extract local mask point clouds of the object; and obtaining mechanical... Arm joint angles: The local mask point cloud and the robot arm joint angles at the current moment are used as prediction samples and input into the trained policy network model to predict the robot arm end-effector pose at multiple next moments. The robot arm end-effector pose includes the position and orientation of the robot arm end-effector, which is determined by the robot arm joint angles. The robot arm end-effector poses predicted by the policy network model up to the current moment are averaged, and the robot arm joint angles are calculated based on the robot arm end-effector pose corresponding to the average result. The motor states of each joint of the robot arm are controlled according to the calculated robot arm joint angles to complete the robot arm end-effector pose at the next moment.

[0007] According to another exemplary embodiment of the present invention, an apparatus for training a policy network model for a robotic arm to grasp objects is provided, comprising: an image acquisition unit, configured to, for each complete object grasping process, simultaneously record RGB images and depth images of the grasping operation scene at each moment using a camera while a human operator controls the robotic arm to complete continuous grasping operations using a remote control device, wherein the camera is fixedly mounted at the end of the robotic arm and moves with the end of the robotic arm; an image data preprocessing unit, configured to parse the motion trajectory of the end of the robotic arm recorded by the camera for each complete grasping process, obtain the spatial position of the end of the robotic arm when it approaches the object at the end of the motion trajectory and successfully grasps it, and select a three-dimensional bounding box region centered on the obtained spatial position in the point cloud image at the corresponding moment, wherein the point cloud image is generated by depth projection of the depth image based on the intrinsic parameter model of the camera focal length; and a target detection network model to detect at least one two-dimensional bounding box of an object from the RGB image at the corresponding moment, and map the detected two-dimensional bounding box to the point cloud image at the corresponding moment to obtain at least one object. The system employs a method to generate a point cloud map of the operation scene. The point cloud map is constructed by using the nearest neighbor object point cloud map in the 3D bounding box region as a unified bounding mask. The depth image at each time step is projected using an intrinsic parameter model based on the camera focal length to generate a point cloud map of the operation scene. The unified bounding mask is then applied to the point cloud map at each time step to obtain a local mask point cloud of the target object at each time step. A robotic arm data acquisition unit is used to acquire the joint angles of the robotic arm at each time step from the motor states of each joint. A model training unit is used to construct training samples for each complete object grasping process, using the local mask point cloud and the joint angles of the robotic arm at the current time step as input and the end-effector pose at multiple subsequent time steps as output. The end-effector pose includes the position and orientation of the end-effector, determined by the joint angles. Based on the training samples corresponding to at least one object grasping process, the system iteratively learns according to a behavior cloning loss function to obtain a policy network model for predicting the end-effector pose at multiple subsequent time steps based on the local cloud mask and the joint angles of the robotic arm at the current time step.

[0008] According to another exemplary embodiment of the present invention, a device for controlling a robotic arm to grasp objects using a policy network model is provided, comprising: an image acquisition unit for simultaneously recording RGB images and depth images of a grasping operation scene using a camera, wherein the camera is fixedly mounted at the end of the robotic arm and moves with the end of the robotic arm; an image data preprocessing unit for performing depth projection on the depth image based on an intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene; detecting two-dimensional bounding boxes of object regions from the RGB images using an object detection network model, and mapping the two-dimensional bounding boxes onto the point cloud map to extract local mask point clouds of the object; and a robotic arm data acquisition unit for acquiring data from various joints of the robotic arm. The system obtains the joint angles of the robotic arm based on motor states. A prediction unit uses the current local mask point cloud and the robotic arm joint angles as prediction samples, inputting them into a trained policy network model to predict the end-effector pose at multiple next moments. The end-effector pose includes the position and orientation of the end-effector, determined by the joint angles. An execution unit averages the predicted end-effector poses for the next moment from the current time step, calculates the joint angles based on the average, and controls the motor states of each joint according to the calculated joint angles to complete the end-effector pose at the next moment.

[0009] As described above, in an exemplary embodiment of the present invention, by effectively masking the area where the object is located in the point cloud map of the operation scene, the samples input into the model for learning carry only relatively important information. Although this results in the loss of some understanding of the scene, it avoids the intrusion of some redundant or harmful information caused by the global feature fusion mechanism of the panoramic image in the high-level semantic representation of the feature space. This indirectly plays a role in strengthening the attention mechanism during the training process, and overall, it is more beneficial to the decision-making effect.

[0010] Furthermore, in scenarios with multiple potential target objects, the process of grasping a single object is naturally calculated back-calculated. Without introducing additional instructions or processing, the process of grasping a single object is effectively learned during the training phase. This "explicit target guidance" design without explicit additional processing allows the policy network to focus only on the geometric structure and spatial position of a single object, thereby significantly improving the stability and generalization of the decision. It avoids the instability caused by the lack of clear target guidance information when the policy network makes implicit selections in a high-dimensional feature space, which leads to ambiguity or inconsistency in the output end pose.

[0011] Furthermore, according to an exemplary embodiment of the present invention, an environment model is provided to virtually simulate the training environment of the policy network without the need for actual robotic arm operation, thereby significantly reducing training costs in terms of hardware wear and execution time.

[0012] In a virtual environment, policy networks can explore new actions, breaking through the limitations of the coverage of teaching data. Reinforcement learning allows policies to try action combinations that did not appear in the teaching and find better execution methods from the feedback of the environment model. Moreover, policies can automatically correct the biases of behavior cloning learning. For example, in the initial training stage, policies may be affected by teaching errors, operational inconsistencies, etc. In the model environment, through repeated trials and reward-driven optimization, policies can gradually get rid of these biases and learn a more consistent and robust policy structure. Accordingly, the fine-tuned policies will have stronger generalization ability and higher success rate, especially in the following scenarios where they show significant improvement: (1) When the initial point cloud pose of the object in the real environment is slightly different from that of the teaching data, the policy can still converge stably to the vicinity of the target object; (2) When the target object is partially occluded and the point cloud is incomplete, the policy can maintain reasonable behavior; (3) Fine-grained adjustments during the grasping process are smoother, and there are no more discrete jumps or obvious pose oscillations; (4) When approaching the boundary of the teaching distribution, the policy can take a more reliable approach path based on the experience of the model environment.

[0013] Furthermore, according to an exemplary embodiment of the present invention, the policy can be redeployed back to the real system after reinforcement learning is completed. Since the point cloud format in the model environment is completely consistent with the real observed point cloud, the input form of the policy remains unchanged, allowing for a seamless transition back to the real robotic arm system. The final policy combines the experience gained from the teaching data with optimizations from a large amount of virtual experience in the environment model, achieving stability and success rate far exceeding that of pure behavioral cloning in grasping tasks.

[0014] Furthermore, according to an exemplary embodiment of the present invention, when performing prediction, by performing coordinate system transformation of the point cloud map and performing single detection of two-dimensional bounding boxes for multi-frame reuse during the image preprocessing stage, the model performance can be further improved while the computational cost can be further reduced. Attached Figure Description

[0015] Figure 1 An example of a policy network model training scenario according to an exemplary embodiment of the present invention is shown;

[0016] Figure 2 A flowchart illustrating a method for training a policy network model for a robotic arm to grasp objects, according to an exemplary embodiment of the present invention;

[0017] Figure 3 An example of imitation learning using a policy network model according to an exemplary embodiment of the present invention is shown;

[0018] Figure 4 An example of fine-tuning a policy network model for reinforcement learning according to an exemplary embodiment of the present invention is shown;

[0019] Figure 5 An example of a policy network model deployment scenario according to an exemplary embodiment of the present invention is shown;

[0020] Figure 6 A flowchart illustrating a method for controlling a robotic arm to grasp an object using a policy network model according to an exemplary implementation of the present invention is shown.

[0021] Figure 7 A block diagram of an apparatus for training a policy network model for a robotic arm to grasp objects, according to an exemplary embodiment of the present invention, is shown; and

[0022] Figure 8 A block diagram of a device for controlling a robotic arm to grasp objects using a policy network model according to an exemplary embodiment of the present invention is shown. Detailed Implementation

[0023] The invention will be more readily understood from the following detailed description with reference to the accompanying drawings, wherein the same reference numerals designate units of the same structure.

[0024] As an automated mechanical device that simulates the movements of a human arm, a robotic arm can perform tasks such as grasping objects through programmable control. It widely integrates multiple disciplines such as mechanical design, drive control, sensor perception, and artificial intelligence, and is a core execution unit in fields such as industrial automation, service robots, and intelligent interaction.

[0025] The essence of a robotic arm is a "programmable multi-degree-of-freedom actuator." Through the coordinated movement of multiple joints, it drives an end effector (such as a gripper) to a designated position in three-dimensional space and completes the action. Its core value lies in replacing humans in performing tasks that are highly repetitive, require high precision, are in hazardous environments (such as high temperature, high pressure, or sterile environments), or are in space-constrained, while also possessing the advantages of flexible adjustment and continuous operation.

[0026] For example, robotic arms that sort parts 24 / 7 in industrial settings and surgical robotic arms with a precision of 0.1mm in medical settings both achieve tasks that are difficult for humans to complete stably through "multi-joint collaboration + precise control".

[0027] When using AI models to learn and provide motion decisions for robotic arms, the high dimensionality of the operational space, the long-range exploration characteristics of the tasks, and the involvement of diverse objects present challenges for technicians, such as low model exploration efficiency and complex training process design. Furthermore, the inherent characteristics of different robotic arms and the diversity of their application scenarios also bring personalized technical difficulties to the training and deployment of decision-making models.

[0028] Figure 1An example of a policy network model training scenario according to an exemplary embodiment of the present invention is shown. Figure 1 The robotic arm shown can be applied to task completion scenarios such as large AI toy robots. For example, AI robots placed in amusement parks and shopping malls, whose appearance can be that of cartoon IP characters, can perform tasks such as grasping objects (e.g., grasping cups during ice cream making) while interacting with children or tourists. To enable the robotic arm to complete the grasping task, a pre-trained policy network model can make decisions on how to control the posture of the robotic arm's end effector in the following time based on the object to be grasped and the current state of each joint of the robotic arm. Here, the number of independently movable joints in the robotic arm can vary, which is the core factor determining flexibility. For example, a 4-axis robotic arm (3 rotary joints + 1 translational joint) can only perform simple actions of "grasping-lifting-translating-placing", suitable for desktop toy robots, simple sorting, etc.; a 6-axis robotic arm (6 rotary joints) can achieve arbitrary posture adjustment of the end effector in three-dimensional space (such as tilting, rotating), suitable for industrial assembly, medical surgery, etc.; a 7-axis robotic arm (1 additional redundant joint) can "avoid obstacles" in narrow spaces, such as bypassing human organs to reach the lesion during surgery. Therefore, the angles of each joint of the robotic arm can reflect the current posture of the robotic arm. Conversely, after determining the pose (e.g., position and orientation) of the end effector of the robotic arm, it can be converted into the angles of each joint of the robotic arm, and control signals can be generated to drive the relevant motors to rotate each joint to the angles.

[0029] To enable the policy network model to learn how to operate the robotic arm end-to-end, the teaching process of a human operator remotely controlling the robotic arm to grasp objects via a remote control device (such as a handle) can be used as the learning model's model. During the operation of the robotic arm controlled by the human operator, information such as environmental information of the operation scene and the robotic arm's operation actions (the state of each joint of the robotic arm, the opening and closing of the robotic arm's end effector, etc.) can be collected simultaneously and used as input to the policy network model.

[0030] like Figure 1 As shown, taking a robotic arm with a fixed base on a desktop as an example, the robotic arm can be controlled by a human operator via a remote control device (such as a handle) to operate (e.g., multi-hand gripping). The handle control signal is mapped to the posture of the robotic arm end effector to control the movement of the robotic arm through the robotic arm interface.

[0031] Specifically, a camera (i.e., a webcam) is fixedly mounted on the end effector of the robotic arm (e.g., the gripper) to simultaneously capture images of the operating scene when a human operates the robotic arm, such as RGB images and depth images. This image data is then sent to a stored data recording program. Furthermore, real-time angle information for each joint can be obtained through the robotic arm interface. This information is also sent to the data recording program, and together with the image data and joint control signals, it is used to train the strategy model in a computing device. The entire training process can be divided into three stages: training data acquisition, image data preprocessing, and strategy model training.

[0032] Figure 2 A flowchart illustrating a method for training a policy network model for a robotic arm to grasp objects, according to an exemplary embodiment of the present invention.

[0033] like Figure 2 As shown, in step S100, image data acquisition from the training data is completed. Here, real-time acquisition of changes in the operating environment (especially the target object) is achieved through synchronous shooting by the camera system, and the acquired data is stored in a recording medium. Specifically, for each complete object grasping process, while the human operator controls the robotic arm to complete continuous grasping operations using a remote control device, the camera synchronously records the RGB image and depth image of the grasping operation scene at each moment. The camera is fixedly set at the end of the robotic arm and moves with the end of the robotic arm. Here, each grasping operation corresponds to a complete teaching trajectory, that is, it includes a sequence of continuous changes in the position and orientation of the robotic arm end during a complete grasping task. The camera is used to synchronously capture the operation scene of the robotic arm grasping the object (including the interaction between the background, the object, and the gripper) in real time. As an example, an RGB-D camera can be used, which not only provides color RGB images but also outputs the corresponding depth map.

[0034] In an exemplary embodiment of the present invention, point cloud data is used as the input image for training the strategy model in the robot's autonomous grasping task, primarily for both 3D environment perception and operational decision-making. Compared to traditional RGB images, point clouds provide richer spatial geometric information, enabling the robot to be more robust when facing unknown objects and complex scenes. However, end-to-end grasping strategies based on point clouds still face significant challenges in practical deployment, especially in the face of decision uncertainty under conditions of multiple targets, complex backgrounds, and noise interference.

[0035] For example, when a point cloud scene contains a large number of non-target regions or background points, the model will be affected by noise during encoding, leading to inaccurate modeling of the shape and position features of the target object, thus affecting the stability of the grasping decision. Background parts in point clouds typically account for the vast majority of points, such as desktops, supporting objects, and walls. While these parts may aid scene understanding at the perception level, they are often redundant or even harmful information for specific grasping decisions. Especially when using deep network structures such as PointNet, PointNet++, Transformer-based, or other point-cloud encoders, the global feature fusion mechanism of the point cloud can cause background interference to be implicitly propagated into the high-level semantic representation of the feature space, thereby reducing the ability to identify the features of the target region.

[0036] To this end, an exemplary embodiment of the present invention employs an innovative image preprocessing method to generate local point cloud maps from RGB images and depth images that are suitable for model training and can ensure the effectiveness of decision-making. These local point cloud maps can not only accurately locate the target object being captured, but also minimize computational overhead during the generation process.

[0037] Specifically, in step S200, the motion trajectory of the robotic arm end effector during each complete grasping process recorded by the camera is analyzed to obtain the spatial position of the robotic arm end effector when it approaches the object at the end of the motion trajectory and successfully grasps it. The three-dimensional bounding box region centered on the obtained spatial position is selected in the point cloud map at the corresponding moment. The point cloud map is generated by depth projection of the depth image based on the intrinsic parameter model of the camera focal length.

[0038] According to an exemplary embodiment of the present invention, the target object that the human operator intends to grasp can be determined by reverse deduction, that is, the robotic arm grasping operation process for a single target object can be naturally extracted without any additional display instructions.

[0039] Specifically, during operation, the operational state of the robotic arm's end effector (including the angles of each joint and whether the end effector is open or closed) can be recorded simultaneously. The continuously recorded operational states form a trajectory of a complete grasping process. Near the end of the trajectory, as the robotic arm's end effector (i.e., the gripper) grasps the object through opening and closing operations, the spatial location of the grasping point can be obtained based on the recorded gripper operational state and the dynamic relationship of the relevant motors. This spatial location can then be used to locate the target object in the point cloud map of the operation scene, thus initially determining the point cloud region where the grasped object is located.

[0040] Therefore, it is necessary to generate a point cloud map of the operation scene corresponding to the capture time. For example, the depth image at the relevant time can be depth-projected based on the intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene.

[0041] Specifically, using the imaging principle of a pinhole camera, for example, each pixel (u, v) at position (u, v) in the depth image and its depth value D(u, v) are converted into three-dimensional coordinates (x, y, z) in the camera coordinate system to obtain the corresponding point cloud data in the camera coordinate system:

[0042] in,

[0043]

[0044]

[0045]

[0046] in , It is the camera intrinsic parameter matrix The focal length in the image, expressed in pixels. , It is the camera intrinsic parameter matrix The coordinates of the principal point in the equation.

[0047] Using the above transformations, the entire depth image can be converted into a dense 3D point cloud representation. Furthermore, to further improve the model's learning performance, the camera coordinates can be further converted into world coordinates, i.e., fixed base coordinates. Specifically: the position of the robotic arm's end effector is recorded as the camera's position; an extrinsic parameter matrix is ​​calculated to represent the spatial relationship between the camera's position and the position of the fixed base on which the robotic arm is located; based on the extrinsic parameter matrix, the point cloud image in camera coordinates is converted into a point cloud image in base coordinates to serve as the point cloud image for the operational scene. For example, an extrinsic parameter matrix can be introduced... , .

[0048] After obtaining the point cloud map of the operation scene as described above, the position of the gripper can be mapped onto the point cloud map, and a three-dimensional bounding box region can be selected around the position with a predetermined range (e.g., predetermined x, y, z offset values) as the point cloud map range corresponding to the target gripping object.

[0049] To further improve the accuracy of the point cloud map range, according to an exemplary embodiment of the present invention, a two-dimensional range verification is also performed, and this verification method can be consistent with the image preprocessing in the prediction stage after model deployment, so as to further improve processing efficiency.

[0050] Specifically, in step S300, at least one two-dimensional bounding box of an object is detected from the RGB image at the corresponding time using an object detection network model. The detected two-dimensional bounding box is mapped to the point cloud image at the corresponding time to obtain the object point cloud box of at least one object. The object point cloud box that is closest to the three-dimensional bounding box region is used as a unified box selection mask.

[0051] Here, depending on the actual situation of the scene and considering requirements such as accuracy, real-time performance, and robustness, a suitable object detection network can be used to detect object regions in the RGB image. Here, the RGB image may contain one or more objects. For example, object detection networks such as YOLOv8 or Mask R-CNN can be used to locate the two-dimensional bounding box of the at least one object in the image space.

[0052] Once the 2D bounding box is determined, it can be mapped onto a point cloud to extract the local mask point cloud corresponding to each object. For example, the pixel set of the region corresponding to the 2D bounding box can be used. Convert to a subset of point cloud:

[0053] ,

[0054] Received This refers to the 3D mask point cloud of the object, which has a corresponding object point cloud bounding box that covers the 3D bounding region (x, y, z) of the target object, and can be represented as:

[0055] .

[0056] The point cloud bounding boxes of each object are compared with the 3D bounding box region determined based on the gripping position, and the point cloud bounding box of the object that is spatially closest is determined to correspond to the gripping target object.

[0057] To further save computational efficiency, the object point cloud bounding box of the target object can be directly and fixedly applied to select local point clouds of the target object from the point cloud map at various times throughout the entire grasping process.

[0058] In step S400, the depth image at each time moment is depth-projected based on the intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene, and the unified bounding mask is applied to the point cloud map at each time moment to obtain the local mask point cloud of the grasping target object at each time moment.

[0059] Specifically, after obtaining the unified bounding box mask of the target object, this 3D region will serve as a fixed mask region for the current capture process. It will be used to filter out local mask point clouds from the point cloud map of each frame, without needing to perform detection network or pixel-level back projection for each frame. Instead, the point cloud will be filtered directly based on the 3D spatial range of the unified bounding box mask.

[0060]

[0061]

[0062] In this approach, the model consistently uses a fixed local point cloud region for the same object as input. This avoids repeated calls to the detection network at high frame rates and eliminates target region jitter caused by fluctuations in detection errors, thereby improving the real-time performance and robustness of the entire grasping pipeline. This "spatial fixed region" approach is particularly suitable for scenarios where the target object is relatively stationary during grasping, and can greatly simplify the point cloud preprocessing workflow.

[0063] Furthermore, in multi-object scenarios, robot policies need to simultaneously face multiple candidate objects that can be grasped. In this case, the input point cloud contains multiple potential targets whose spatial distributions and features overlap or interfere with each other. The network needs to implicitly make a "selection" in the high-dimensional feature space—that is, decide which object to grasp. However, due to the lack of explicit target indication information (such as task objectives or human intentions), this implicit selection is often unstable, easily leading to ambiguity or inconsistency in the output end-effector pose. For example, in the same scene, the model may randomly output grasping poses facing different targets, exhibiting multimodal uncertainty in the policy. To address this, according to an exemplary embodiment of the present invention, an additional improved method for input preprocessing based on point cloud masks is proposed. This method extracts the location region of the target object in the two-dimensional image by performing target detection or instance segmentation on the RGB image, and maps this region to the corresponding three-dimensional point cloud, thereby obtaining a local point cloud subset containing only the target object as input to the policy network. This "explicit target guidance" design allows the network to focus only on the geometry and spatial position of a single object, thus significantly improving the stability and generalization of the decision.

[0064] In addition to acquiring image data and performing corresponding preprocessing, it is also necessary to acquire the continuous state changes of the robotic arm throughout the grasping phase as another part of the training data.

[0065] In step S500, the joint angles of the robotic arm at each moment are obtained from the motor states of each joint. Here, depending on the specific design of the robotic arm and the requirements of the application scenario, the rotational position of the motor in the free movement direction of each joint can be detected by an encoder and converted into a motor rotation angle, which is then used as the joint angle in this invention.

[0066] After the above processing, various training data were collected, and the image data underwent preprocessing before being input into the model, thus obtaining the local mask 3D image of the object corresponding to each time step and the angles of each joint of the robotic arm. Next, the strategy model will be trained based on the collected data.

[0067] Figure 3 An example of imitation learning using a policy network model according to an exemplary embodiment of the present invention is shown. Figure 3 As shown, in the data preprocessing stage, images of the operating scene can be acquired during remote teaching by a human operator. Combined with the object being grasped by the robotic arm, a local mask point cloud of that object is generated for training. In the imitation learning stage, the local mask point cloud and its corresponding robotic arm joint angles are input into the neural network of the policy network model for imitation learning. During this training stage of the imitation learning policy, the original scene point cloud is cropped into a local sub-cloud containing only the target object using the aforementioned mask, and this is used as the input to the policy network. The network output is the target pose at the end of the robotic arm (which can be represented as a position vector and quaternions, or 6D pose parameters). Compared to directly inputting the entire scene point cloud, this masking process makes the network learn target features more concentrated, reducing feature dilution caused by background points. Simultaneously, the network output no longer faces the ambiguity of selecting "multiple targets," because the input data itself implicitly determines a unique grasping object.

[0068] Specifically, in step S600, training samples are first constructed. That is, for each complete object grasping process, training samples are constructed with the local mask point cloud and the robot arm joint angle at the current moment as input and the robot arm end pose at the next multiple moments as output. The robot arm end pose includes the position and orientation of the robot arm end, which is determined by the robot arm joint angle.

[0069] As an example, for each complete object grasping process, a trajectory data can be obtained. ,in, This represents the local mask point cloud at time t. This represents the joint angle of the robotic arm at time t. This represents the end-effector pose applied to the robotic arm at time t+1, expressed as follows: Neural networks as policy network models The input, for the next m time steps As a neural network The output of .

[0070] Here, the training objective adopts behavior cloning to minimize the error between the predicted end pose and the corresponding pose in the human-taught trajectory. In this process, the mask not only serves as a data cleaning method but can also be regarded as a "task conditioning" mechanism, which binds the grasping behavior learned by the policy to a specific target object, thereby possessing strong interpretability.

[0071] If we only predict the end-effector posture at the next moment, the control signal will inevitably be less smooth in the time domain due to perturbations in the neural network output during policy deployment. Therefore, when performing behavior cloning, we predict the end-effector posture over a future period (i.e., action blocks, including m time frames) to facilitate smoothing in the time domain during policy deployment.

[0072] To achieve better model performance, the object grasping process can be executed multiple times, thereby collecting raw training data from multiple taught trajectories. Alternatively, the actual grasping process can be executed only once; exemplary embodiments of the present invention do not limit the number of teaching demonstrations for the object grasping process.

[0073] Next, in step S700, based on training samples corresponding to at least one object grasping process, learning is performed iteratively according to the behavior cloning loss function to obtain a policy network model for predicting the end-effector pose at multiple next time steps based on the local cloud mask and the joint angle of the robotic arm at the current time step. As an example, the policy network model is trained based on the following loss function, fitted using minimizing the regression error: .

[0074] After completing the behavior cloning strategy described above, the policy network model is now able to predict the end-effector pose of the robotic arm based on the local mask point cloud of the object (this prediction result can be used to instruct the joints of the robotic arm to perform subsequent actions).

[0075] To further enhance its predictive capabilities, learning goes beyond simply learning from the taught data; it involves understanding states beyond the taught distribution and proactively exploring potential better action patterns. Alternatively, the policy network model can be optimized. For example, an environment model can be trained to simulate the impact of the current local mask point cloud and robotic arm joint angles on the next local mask point cloud and robotic arm joint angles under the policy network model's decisions. The policy network model can then interact with the trained environment model, allowing it to fine-tune itself through reinforcement learning in a virtual environment provided by the environment model, based on the reward for completing the object grasping task, thus training an enhanced policy network model.

[0076] As mentioned above, by training an environment model based on local mask point clouds, the main goal of the model is to simulate the impact of "current local mask point cloud + current robotic arm action" on the scene state of the next frame (i.e., the local mask point cloud of the next frame + the robotic arm action of the next frame), so that the strategy can be repeatedly tested without the need for real robotic arm interaction.

[0077] Therefore, the input to the environment model is the target point cloud currently passing through the mask and the action commands output by the policy; the output includes at least: the local point cloud changes of the target in the next frame, and a judgment on whether the task has been completed (i.e., a "done" signal). Here, the environment model plays the role of a "virtual world," providing a repeatable and quickly invoked predictor of the grasping process. Compared to traditional physics simulation engines, this point cloud-based model does not require precise CAD models or complex physical parameters, but directly learns the geometric changes of objects when manipulated from the teaching data, thus more closely resembling the observation form of real systems.

[0078] When training the environment model, the changes between two consecutive frames of point clouds in the teaching data can be used as supervision signals: given a teaching trajectory, multiple state transition samples can be extracted. Each sample includes the current point cloud, the current robotic arm joint angle, the next frame of point cloud after the action, the next frame of robotic arm joint angle, and whether the action led to task completion. Through learning from a large number of samples, the environment model gradually masters the rules of "how the point cloud changes with the action," including the geometric changes of the target object during the process of being approached, pushed, and lifted by the gripper, and which states can be judged as "successful grasping." This learning method enables the model to obtain sufficient predictive ability without explicit physical modeling.

[0079] Specifically, the environment model can be trained as follows: for each complete object grasping process, acquire a trajectory data. In this process, a flag bit is added to each frame of data, i.e. This indicates whether the capture task is completed at time t. For the last time point, the value is 1, indicating that the capture task is completed. For other times, the value is 0, indicating that the capture task is not completed.

[0080] The environment model uses a Transformer-based encoder. +decoder This is achieved through a structure, in which, The input is The output is the latent variable modeled as a Gaussian distribution. Gaussian distribution mean and standard deviation ,in, , The input is a latent variable. The output is the predicted local mask point cloud at time t+1. Predicted robot arm joint angles at time t+1 And whether the capture task is completed at the predicted time t. ,in, In other words, The output is a hidden variable. The distribution is generally modeled as a Gaussian distribution, using and They represent The mean and standard deviation, Direct output and , From Gaussian distribution Sampling is performed during this process, i.e. It can also be considered The whole is actually a probability distribution function: .

[0081] Accordingly, the environment model is trained based on the following loss function:

[0082] ,

[0083] in, i represents and Dimensions This is a predetermined weight. The first item above... This is the regression loss; the second term is used for constraints. The distribution, The weights of the second term are used, and the summation symbol indicates that the summation is applied to the second term. and Summing is performed on each dimension.

[0084] From a systems perspective, training the environment model allows the learning of grasping strategies to no longer rely entirely on real data. The point cloud sequences recorded during the teaching phase are often relatively limited, but a learned environment model can generate a large amount of additional "simulated experience" within it. The point cloud changes corresponding to these experiences are entirely predicted by the model; while not physically absolutely accurate, they are sufficient to meet the policy's learning needs at the abstract level, such as better approach paths and more stable posture adjustments. Therefore, training the environment model provides the foundation for the next stage of reinforcement learning, enabling the policy to continuously improve in a cheap, fast, and controllable virtual environment.

[0085] Figure 4 An example of fine-tuning a policy network model for reinforcement learning according to an exemplary embodiment of the present invention is shown. Figure 4 As shown, based on the environment model obtained above, a virtual training environment can be constructed for the policy to improve itself. Unlike simply imitating the taught trajectory, this stage employs a model-based reinforcement learning approach: the policy performs actions in the "model environment," observes the point cloud changes predicted by the environment model, and receives rewards based on the model's assessment of task completion, thereby continuously adjusting its action output. The entire process can be understood as allowing the policy to experiment and learn in a highly lightweight, fully data-driven point cloud simulator.

[0086] Specifically, the enhanced policy network model can be trained as follows: the policy treats the environment model as a virtual environment, and then performs interaction sampling within this virtual environment. The original collected data contains multiple trajectories; from the trajectory data of each complete object grasping process, the local mask point cloud of the first frame at time t=0 is extracted. and robotic arm joint angle This serves as the starting point of the virtual environment and is input into the policy network model to obtain the end-effector poses of the robotic arm for the next m time steps. ;

[0087] Next, the environment model will perform an autoregressive analysis on the local mask point cloud at time t+1 over the next m time steps. Robotic arm joint angles at time t+1 And whether the task is completed at time t. Make a prediction:

[0088]

[0089]

[0090]

[0091] ,

[0092] After obtaining the environmental model Predicted time t=m and Then, the policy network model continues to provide the robot arm's end-effector pose for the next m time steps after time t=m, repeating this process until the predicted pose is obtained. A value of 1 indicates that the grasping task is completed. The policy network model is fine-tuned using reinforcement learning based on a temporal difference algorithm. The Q-network's fitting target... Q function In order to be in Virtual execution action The cumulative reward obtained later, the reward function is: The loss function of the Q-network is The reinforcement learning fine-tuning loss function for the policy network model is: .

[0093] It can be seen that by defining the reward function as In other words, the policy only receives a reward upon completing the task; otherwise, it receives no reward. Based on the principle of maximizing cumulative reward in reinforcement learning, the policy can be guided to learn how to complete tasks. A temporal difference algorithm is used to fine-tune the policy through reinforcement learning. In this virtual environment, the output of the environment model, "whether the task is completed," is directly used as the reward signal for reinforcement learning: when an action leads to the target being correctly grasped and reaching a stable state, the model outputs a "done" signal that is true, and the policy receives a positive reward; if the action does not lead the object to a completed state, or causes the point cloud to present an incorrect state, the reward is zero or low. This design allows the policy to automatically adjust its action sequence, optimizing it towards a higher success rate and more stable grasping behavior.

[0094] Fine-tuning strategies within an environment model offers several key advantages with additional improvements:

[0095] First, the training cost is extremely low. Compared to a real robotic arm, the roll-out cost of an environment model is almost zero, so the policy can generate tens of thousands of trials without worrying about hardware wear and tear or execution time.

[0096] Second, policies can explore new actions. Teaching data has limited coverage, while reinforcement learning allows policies to try combinations of actions not shown in the teaching and find better execution methods from feedback from the environmental model.

[0097] Third, the policy can automatically correct biases in behavioral cloning. In the first stage, the policy may be affected by teaching errors, operational inconsistencies, etc. However, in the model environment, through repeated trials and reward-driven optimization, the policy can gradually get rid of these biases and learn a more consistent and robust policy structure.

[0098] The fine-tuned strategy will have stronger generalization ability and a higher success rate, especially showing significant improvement in the following scenarios:

[0099] (1) When the initial point cloud pose differs slightly from the teaching data, the strategy can still converge stably to the vicinity of the target object;

[0100] (2) The strategy can maintain reasonable behavior when the target object is partially occluded or the point cloud is incomplete;

[0101] (3) Fine-grained adjustments during the grasping process are smoother, and there are no more discrete jumps or obvious posture oscillations;

[0102] (4) When approaching the boundary of the teaching distribution, the strategy can take a more reliable approach path based on the experience of the model environment.

[0103] After reinforcement learning is complete, the policy network model can be redeployed back to the real system. Since the point cloud format in the model environment is identical to the real observed point cloud, and the input format of the policy remains unchanged, a seamless transition back to the real robotic arm system is possible. The final policy combines the experience gained from the teaching data with optimizations from a large amount of virtual experience in the environment model, achieving significantly higher stability and success rate in grasping tasks than pure behavior cloning.

[0104] Figure 5 An example of a policy network model deployment scenario according to an exemplary embodiment of the present invention is shown. Figure 5As shown, any trained policy network model according to an exemplary embodiment of the present invention can be deployed to a real-world object grasping application scenario. In this case, the robotic arm acquires RGB and depth images of the current scene in the workspace using an RGB-D camera, and the depth image is used to obtain a point cloud map through depth projection; the RGB image is detected by an object detection network, and the two-dimensional bounding boxes of all candidate object objects are output; a target object is randomly selected from the detection results (or a specific category is selected according to task requirements), and its bounding box is mapped to the point cloud to extract the corresponding three-dimensional mask of the target object; the masked target point cloud is input into the trained grasping policy network; the policy network outputs the end-effector pose of the robotic arm; the robotic arm performs actions according to the output pose. It can be seen that in multi-object scenarios, since the system explicitly selects a mask region as input, the policy always predicts on a defined target, avoiding the decision uncertainty caused by multiple targets in traditional end-to-end point cloud policies. If the task requires grasping multiple objects continuously, the remaining objects can be re-detected after one grasp and the above process can be repeated, thereby achieving stable multi-object grasping.

[0105] Figure 6 A flowchart illustrating an exemplary implementation of the present invention of a method for controlling a robotic arm to grasp an object using a policy network model, the policy network model being trained as described above.

[0106] Those skilled in the art should know that after the policy network model is deployed, it needs to collect and preprocess image data of the operation scene and operation data of the robotic arm in order to perform prediction. These processes are at least partially the same as those during model training (the main difference is that the determination of the grasping target and its mask in the prediction stage is performed in a forward manner). The repeated parts will not be described in detail below.

[0107] In step S1000, the camera is used to synchronously record RGB images and depth images of the grasping operation scene. The camera is fixedly set at the end of the robotic arm and moves with the end of the robotic arm.

[0108] In step 2000, based on the intrinsic parameter model of the camera focal length, the depth image is depth-projected to generate a point cloud map of the operation scene.

[0109] In step 3000, a two-dimensional bounding box of the object region is detected from the RGB image using an object detection network model, and the two-dimensional bounding box is mapped to the point cloud image to extract the local mask point cloud of the object.

[0110] Here, as an example, when the object detection network model detects a single object in an RGB image, the region surrounding the single object in the RGB image is used as a two-dimensional bounding box; when the object detection network model detects multiple objects in an RGB image, an object is randomly selected from the multiple objects or selected based on the task type, and the region surrounding the selected object is used as a two-dimensional bounding box. Both the detection and selection processes are performed automatically and do not require any human intervention.

[0111] After detecting the 2D bounding box, it can be mapped onto the point cloud to select the local mask point cloud of the object to be captured. However, in a practical system, performing RGB object detection and recalculating the point cloud mask for each frame would result in additional computational overhead, and the inference time of the detection network would become the bottleneck of the entire capture system. To further improve real-time performance and system stability, we adopt a strategy of one-time detection and multi-frame reuse. That is, for each complete object capture process, the object detection network model only performs detection on the RGB image before the operation is performed or on the first frame of the RGB image in the current capture round. The resulting 2D bounding box is mapped onto the corresponding point cloud to obtain the object's point cloud bounding box as a unified selection mask. This unified selection mask is then applied to the point cloud at each time step in the current capture round to obtain the local mask point cloud of the object at each time step.

[0112] Specifically, before the strategy begins execution (e.g., the first frame before the robotic arm prepares to grasp), the object detection network performs object detection on the RGB image, obtaining the 2D bounding box of the target object. Then, through depth map backprojection, the point cloud corresponding to this region is transformed into 3D space, resulting in a 3D bounding region (x, y, z) covering the target object, which can be represented as:

[0113]

[0114] This 3D region will serve as a fixed mask area in subsequent processing. In other words, when a new point cloud is acquired in each subsequent frame, there's no need to execute a detection network or pixel-level backprojection; instead, the point cloud is directly filtered based on the pre-defined 3D spatial range.

[0115]

[0116] .

[0117] In step S4000, the joint angles of the robotic arm are obtained from the motor status of each joint.

[0118] In step S5000, the local mask point cloud and the robot arm joint angle at the current moment are used as prediction samples and input into the trained policy network model to predict the robot arm end pose at multiple subsequent moments. The robot arm end pose includes the position and orientation of the robot arm end, which is determined by the robot arm joint angle.

[0119] In step S6000, the robot arm end-effector pose predicted by the policy network model up to the present moment is averaged, and the robot arm joint angle is calculated based on the average end-effector pose. Here, since the policy network model predicts the end-effector pose for multiple subsequent frames for each current frame, multiple prediction results are accumulated at times other than the very beginning. Averaging these multiple prediction results yields a more stable and smoother execution action.

[0120] In step S7000, the motor states of each joint of the robotic arm are controlled according to the calculated joint angles of the robotic arm in order to complete the end-effector pose of the robotic arm at the next moment.

[0121] It can be seen that the robotic arm grasping scheme according to the exemplary embodiment of the present invention can effectively improve the accuracy, stability and generalization of the end-to-end policy network model, especially for multi-target scenarios, it can also achieve a relatively ideal grasping effect, while reducing computational cost and training cost.

[0122] The above methods can be executed by computer program instructions and recorded on a non-volatile computer-readable medium. Alternatively, the above methods can be executed using appropriate hardware and / or a combination of hardware and software.

[0123] Figure 7 A block diagram of an apparatus for training a policy network model for a robotic arm to grasp objects, according to an exemplary embodiment of the present invention, is shown. The apparatus includes: an image acquisition unit 100, an image data preprocessing unit 200, a robotic arm data acquisition unit 300, and a model training unit 400.

[0124] Specifically, the image acquisition unit 100 is used to record RGB and depth images of the grasping operation scene at each moment during each complete object grasping process. While the human operator controls the robotic arm to complete continuous grasping operations using a remote control device, the camera is used to record the grasping operation scene at each moment. The camera is fixedly set at the end of the robotic arm and moves with the end of the robotic arm.

[0125] The image data preprocessing unit 200 is used to analyze the motion trajectory of the robotic arm's end effector during each complete grasping process recorded by the camera, obtain the spatial position of the robotic arm's end effector when it approaches the object at the end of the motion trajectory and successfully grasps it, and select a three-dimensional bounding box region centered on the obtained spatial position in the point cloud map at the corresponding time. The point cloud map is generated by depth projection of the depth image based on the intrinsic parameter model of the camera focal length. The target detection network model detects at least one two-dimensional bounding box of an object from the RGB image at the corresponding time, and maps the detected two-dimensional bounding boxes to the point cloud map at the corresponding time to obtain the object point cloud box of at least one object. The object point cloud box that is spatially nearest to the three-dimensional bounding box region is used as a unified bounding mask. The depth image at each time is depth projected based on the intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene, and the unified bounding mask is applied to the point cloud map at each time to obtain the local mask point cloud of the grasping target object at each time.

[0126] The robotic arm data acquisition unit 300 is used to obtain the robotic arm joint angle at each moment from the motor status of each joint of the robotic arm.

[0127] The model training unit 400 is used to construct training samples for each complete object grasping process, with the local mask point cloud and the robot arm joint angle as input and the robot arm end-effector pose as output for multiple subsequent time steps. The robot arm end-effector pose includes the position and orientation of the robot arm end-effector, which is determined by the robot arm joint angle. Based on the training samples corresponding to at least one object grasping process, learning is performed iteratively according to the behavior cloning loss function to obtain a policy network model for predicting the robot arm end-effector pose for multiple subsequent time steps based on the local cloud mask and the robot arm joint angle at the current time step.

[0128] Figure 8 A block diagram of a device for controlling a robotic arm to grasp an object using a policy network model according to an exemplary embodiment of the present invention is shown. The device includes an image acquisition unit 1000, an image data preprocessing unit 2000, a robotic arm data acquisition unit 3000, a prediction unit 4000, and an execution unit 5000.

[0129] The image acquisition unit 1000 is used to synchronously record RGB images and depth images of the grasping operation scene using a camera. The camera is fixedly mounted at the end of the robotic arm and moves with the end of the robotic arm.

[0130] The image data preprocessing unit 2000 is used for an intrinsic parameter model based on camera focal length to perform depth projection on the depth image to generate a point cloud map of the operation scene; the object detection network model detects two-dimensional bounding boxes of object regions from the RGB image, and uses the two-dimensional bounding boxes to map onto the point cloud map to extract the local mask point cloud of the object.

[0131] The robotic arm data acquisition unit 3000 is used to obtain the joint angles of the robotic arm from the motor status of each joint.

[0132] The prediction unit 4000 uses the local mask point cloud and the robot arm joint angle at the current moment as prediction samples and inputs them into the trained policy network model to predict the robot arm end pose at multiple next moments. The robot arm end pose includes the position and orientation of the robot arm end, which is determined by the robot arm joint angle.

[0133] The execution unit 5000 is used to average the robot arm end-effector pose predicted by the policy network model up to the present moment, and calculate the robot arm joint angles according to the robot arm end-effector pose corresponding to the average result; and control the motor state of each joint of the robot arm according to the calculated robot arm joint angles to complete the robot arm end-effector pose at the next moment.

[0134] Figure 7 and Figure 8 The individual units or modules (shown or not shown) included in the illustrated device can be configured individually as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these components may correspond to dedicated integrated circuits, pure software code, or units or modules combining software and hardware. Furthermore, one or more functions implemented by these components may also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0135] The foregoing has described various exemplary embodiments of the present invention. It should be understood that the above description is merely exemplary and not exhaustive, and the present invention is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for training a policy network model for a robotic arm to grasp objects, comprising: For each complete object grasping process, while the human operator uses a remote control device to control the robotic arm to complete continuous grasping operations, a camera is used to synchronously record the RGB and depth images of the grasping operation scene at each moment. The camera is fixedly set at the end of the robotic arm and moves with the end of the robotic arm. The motion trajectory of the robotic arm end effector during each complete grasping process recorded by the camera is analyzed to obtain the spatial position of the robotic arm end effector when it approaches the object at the end of the motion trajectory and successfully grasps it. The three-dimensional bounding box region centered on the obtained spatial position is selected in the point cloud map at the corresponding moment. The point cloud map is generated by depth projection of the depth image based on the intrinsic parameter model of the camera focal length. The object detection network model detects at least one two-dimensional bounding box of an object from the RGB image at the corresponding time. The detected two-dimensional bounding box is mapped to the point cloud image at the corresponding time to obtain the object point cloud of at least one object. The object point cloud of the object that is closest to the three-dimensional bounding box region is used as a unified box selection mask. The depth image at each time step is projected based on the intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene. The unified bounding mask is applied to the point cloud map at each time step to obtain the local mask point cloud of the target object at each time step. The robot arm joint angles at each moment are obtained from the motor status of each joint of the robot arm. For each complete object grasping process, training samples are constructed, taking the local mask point cloud and robotic arm joint angles at the current moment as input and the robotic arm end-effector poses at multiple subsequent moments as output. The robotic arm end-effector poses include the position and orientation of the end-effector, determined by the robotic arm joint angles. The training samples are constructed as follows: For each complete object grasping process, a trajectory data is acquired. ,in, This represents the local mask point cloud at time t. This represents the joint angle of the robotic arm at time t. This represents the end-effector pose applied to the robotic arm at time t+1, which is the position of the robotic arm's end effector at time t+1. Based on training samples corresponding to at least one object grasping process, learning is performed iteratively according to the behavior cloning loss function to obtain a policy network model for predicting the end-effector pose of the robotic arm at multiple next time steps based on the local cloud mask and the joint angle of the robotic arm at the current time step. The method further includes: An environment model is trained to simulate the influence of the current local mask point cloud and robot arm joint angles on the next local mask point cloud and robot arm joint angles under the decision of the policy network model. The environment model is trained as follows: for each complete object grasping process, a trajectory data point is acquired. ,in, This indicates whether the grasping task is complete at time t. For the last time step, the value is 1, indicating the grasping task is complete; for other time steps, the value is 0, indicating the grasping task is not complete. The environment model uses a Transformer-based encoder. +decoder This is achieved through a structure, in which, The input is The output is the latent variable modeled as a Gaussian distribution. Gaussian distribution mean and standard deviation ,in, , The input is a latent variable. The output is the predicted local mask point cloud at time t+1. Predicted robot arm joint angles at time t+1 And whether the capture task is completed at the predicted time t. ,in, Furthermore, the environment model is trained based on the following loss function: ,in, i represents and Dimensions The policy network model is pre-defined with weights. It interacts with the trained environment model, allowing the policy network model to be fine-tuned through reinforcement learning in the virtual scene provided by the environment model in response to the completion reward of the object grasping task, so as to train an enhanced policy network model.

2. The method of claim 1, wherein: by Neural networks as policy network models The input, for the next m time steps As a neural network The output, Furthermore, the policy network model is trained based on the following loss function, using the minimization of regression error for fitting: .

3. The method as described in claim 2, wherein, The enhanced policy network model is trained using the following method: Extract the local mask point cloud of the first frame at time t=0 from the trajectory data of each complete object grasping process. and robotic arm joint angle Input the policy network model to obtain the end effector pose of the robotic arm at the next m time steps. ; The environment model performs an autoregressive analysis on the local mask point cloud at time t+1 over the next m time steps. Robotic arm joint angles at time t+1 And whether the task is completed at time t. Make a prediction: ， After obtaining the environmental model Predicted time t=m and Then, the policy network model continues to provide the robot arm's end-effector pose for the next m time steps after time t=m, repeating this process until the predicted pose is obtained. A value of 1 indicates that the grasping task is completed. The policy network model is fine-tuned using reinforcement learning based on a temporal difference algorithm. The Q-network's fitting target... Q function In order to be in Virtual execution action The cumulative reward obtained later, the reward function is: The loss function of the Q-network is The reinforcement learning fine-tuning loss function for the policy network model is: .

4. A method for controlling a robotic arm to grasp objects using a policy network model, comprising: The camera is used to simultaneously record RGB and depth images of the grasping operation scene. The camera is fixedly mounted at the end of the robotic arm and moves with the end of the robotic arm. An intrinsic parameter model based on camera focal length is used to perform depth projection on depth images to generate point cloud maps of the operational scene. Two-dimensional bounding boxes of object regions are detected from RGB images using an object detection network model, and the two-dimensional bounding boxes are mapped to a point cloud image to extract local mask point clouds of the objects. Obtain the joint angles of the robotic arm from the motor status of each joint; The local mask point cloud and the robot arm joint angle at the current moment are used as prediction samples and input into the trained policy network model as described in any one of claims 1 to 3 to predict the robot arm end pose at multiple next moments, wherein the robot arm end pose includes the position and orientation of the robot arm end, which is determined by the robot arm joint angle. The strategy network model averages the robot arm end-effector poses predicted up to the present time step, and the robot arm joint angles are calculated based on the average end-effector poses. The motor states of each joint of the robotic arm are controlled according to the calculated joint angles to complete the end effector pose of the robotic arm at the next moment.

5. The method of claim 4, wherein, Two-dimensional bounding boxes for detecting object regions in RGB images using an object detection network model include: When an object detection network model detects a single object in an RGB image, the region surrounding the single object in the RGB image is used as a two-dimensional bounding box; and When the object detection network model detects multiple objects from an RGB image, it randomly selects one object from the multiple objects or selects one object based on the task type, and uses the region around the selected object as a two-dimensional bounding box.

6. The method of claim 5, wherein, The object detection network model detects two-dimensional bounding boxes of object regions from RGB images and maps these bounding boxes onto point cloud maps to extract local mask point clouds of the object. For each complete object grabbing process, the object detection network model only detects RGB images before any operation is performed or the first frame of the RGB image in this round of grabbing. The obtained two-dimensional bounding boxes are mapped onto the corresponding point cloud maps to obtain the object point cloud bounding boxes as a unified bounding box mask. The unified bounding box mask is applied to the point cloud maps at each time step in this round of grabbing to obtain the local mask point cloud of the object at each time step.

7. An apparatus for training a policy network model for a robotic arm to grasp objects, comprising: The image acquisition unit is used to record RGB and depth images of the grasping operation scene at each moment during each complete object grasping process. While the human operator controls the robotic arm to complete continuous grasping operations using a remote control device, the camera is fixedly set at the end of the robotic arm and moves with the end of the robotic arm. An image data preprocessing unit is used to analyze the motion trajectory of the robotic arm's end effector during each complete grasping process recorded by the camera, obtain the spatial position of the robotic arm's end effector when it approaches the object at the end of the motion trajectory and successfully grasps it, and select a 3D bounding box region centered on the obtained spatial position in the point cloud map at the corresponding time. The point cloud map is generated by depth projection of the depth image based on the intrinsic parameter model of the camera focal length. At least one 2D bounding box of an object is detected from the RGB image at the corresponding time by a target detection network model, and the detected 2D bounding boxes are mapped to the point cloud map at the corresponding time to obtain the object point cloud box of at least one object. The object point cloud box that is spatially nearest to the 3D bounding box region is used as a unified bounding mask. The depth image at each time is depth projected based on the intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene, and the unified bounding mask is applied to the point cloud map at each time to obtain the local mask point cloud of the grasping target object at each time. The robotic arm data acquisition unit is used to obtain the joint angles of the robotic arm at each moment from the motor states of each joint; and The model training unit is used to construct training samples for each complete object grasping process. These samples take the current local mask point cloud and robotic arm joint angles as input, and the robotic arm end-effector poses from multiple subsequent time steps as output. The robotic arm end-effector pose includes the position and orientation of the end-effector, determined by the robotic arm joint angles. The training samples are constructed as follows: For each complete object grasping process, a trajectory data point is acquired. ,in, This represents the local mask point cloud at time t. This represents the joint angle of the robotic arm at time t. Let t represent the end-effector pose applied to the robotic arm at time t+1; based on training samples corresponding to at least one object grasping process, iterative learning is performed according to the behavior cloning loss function to obtain a policy network model for predicting the end-effector pose at multiple next time steps based on the local cloud mask and robotic arm joint angles at the current time step; an environment model is trained to simulate the influence of the local mask point cloud and robotic arm joint angles at the current time step on the local mask point cloud and robotic arm joint angles at the next time step under the decision of the policy network model, wherein the environment model is trained according to the following process: for each complete object grasping process, a trajectory data is acquired. ,in, This indicates whether the grasping task is complete at time t. For the last time step, the value is 1, indicating the grasping task is complete; for other time steps, the value is 0, indicating the grasping task is not complete. The environment model uses a Transformer-based encoder. +decoder This is achieved through a structure, in which, The input is The output is the latent variable modeled as a Gaussian distribution. Gaussian distribution mean and standard deviation ,in, , The input is a latent variable. The output is the predicted local mask point cloud at time t+1. Predicted robot arm joint angles at time t+1 And whether the capture task is completed at the predicted time t. ,in, Furthermore, the environment model is trained based on the following loss function: ,in, i represents and Dimensions The policy network model is pre-defined with weights. It interacts with the trained environment model, allowing the policy network model to be fine-tuned through reinforcement learning in the virtual scene provided by the environment model in response to the completion reward of the object grasping task, so as to train an enhanced policy network model.

8. A device for controlling a robotic arm to grasp objects using a policy network model, comprising: The image acquisition unit is used to simultaneously record RGB and depth images of the grasping operation scene using a camera. The camera is fixedly mounted at the end of the robotic arm and moves with the end of the robotic arm. The image data preprocessing unit is used to perform depth projection on the depth image based on the intrinsic parameter model of the camera focal length to generate a point cloud map of the operation scene; the object detection network model detects the two-dimensional bounding boxes of the object region from the RGB image, and uses the two-dimensional bounding boxes to map onto the point cloud map to extract the local mask point cloud of the object. The robotic arm data acquisition unit is used to obtain the joint angles of the robotic arm from the motor status of each joint. The prediction unit is used to input the local mask point cloud and the robot arm joint angle at the current moment as prediction samples into the trained policy network model as described in any one of claims 1 to 3, so as to predict the robot arm end pose at multiple next moments, wherein the robot arm end pose includes the position and orientation of the robot arm end, which is determined by the robot arm joint angle. The execution unit is used to average the robot arm end-effector pose predicted by the policy network model up to the present moment, and to calculate the robot arm joint angles based on the average robot arm end-effector pose. Furthermore, it controls the motor states of each joint of the robot arm according to the calculated robot arm joint angles to complete the robot arm end-effector pose at the next moment.

Images