Construction of human-computer interaction mechanical arm grabbing and placing system and test method and device thereof

Abstract

The application relates to a construction and testing method and device of a human-computer interaction mechanical arm grabbing and placing system, the human-computer interaction mechanical arm grabbing and placing system is constructed by a target detection model and an object pose estimation model, and the method comprises the following steps: based on a grabbing task and a placing task, the target detection model is used to determine the grabbing position of each object to be grabbed, and the object pose estimation model is used to determine the grabbing action and the placing action of each object to be grabbed; the human-computer interaction mechanical arm grabbing and placing system is controlled to grab the corresponding object to be grabbed based on the grabbing position of each object to be grabbed, so as to obtain a detection success rate; the human-computer interaction mechanical arm grabbing and placing system is controlled to grab and place the corresponding object to be grabbed based on the grabbing action and the placing action of each object to be grabbed, so as to obtain a task success rate; and a testing result of the human-computer interaction mechanical arm grabbing and placing system is obtained according to the detection success rate and the task success rate, which can help the mechanical arm to understand the real-time position and posture of the object and ensure accurate control.

Description

Technical Field

[0001] This application relates to the field of depth estimation technology, and in particular to a method and apparatus for constructing and testing a human-computer interaction robotic arm grasping and placing system. Background Technology

[0002] In real-world applications such as human-machine collaboration, object handover, and human-machine interactive teaching, contact between humans and objects, or between robotic arms and objects, is very common. To ensure precise control and collaboration, accurate object pose estimation is crucial for the robotic arm to understand the real-time position and orientation of the object. Therefore, the impact of hand occlusion on algorithm performance cannot be ignored in these scenarios. However, most mainstream object pose estimation datasets, such as LINEMOD and T-LESS, are prepared by placing objects on a tabletop, which limits the applicability of the models to some extent.

[0003] In recent years, some research has attempted to provide datasets containing interactions between objects and hands, such as LINEMOD and T-LESS. However, in these datasets, objects are relatively close to the camera, making it difficult to generalize well to tasks at longer distances. For models, if the training set lacks information relevant to real-world application scenarios, the model's performance in practical applications will be affected. On the other hand, due to the uniqueness of pose parameters, the cost and difficulty of annotating real data are relatively high. Although synthetic data is relatively cheaper, the domain gap between it and real data leads to poor model generalization ability, which urgently needs to be addressed. Summary of the Invention

[0004] This application provides a method and apparatus for constructing and testing a human-computer interaction robotic arm grasping and placing system, in order to solve the problem that object pose estimation models in related technologies cannot effectively help robotic arms understand the real-time position and posture of objects for precise control.

[0005] The first aspect of this application provides a method for constructing and testing a human-computer interaction robotic arm grasping and placement system. The human-computer interaction robotic arm grasping and placement system is constructed from a target detection model and an object pose estimation model. The method includes the following steps: determining the current test task and at least one object to be grasped by the human-computer interaction robotic arm grasping and placement system, wherein the current test task includes a grasping task and a placement task; based on the grasping task and the placement task, determining the grasping position of each object to be grasped using the target detection model, and determining the grasping action and placement action of each object to be grasped using the object pose estimation model; controlling the human-computer interaction robotic arm grasping and placement system to grasp the corresponding object to be grasped based on the grasping position of each object to be grasped, obtaining a detection success rate, and controlling the human-computer interaction robotic arm grasping and placement system to grasp and place the corresponding object to be grasped based on the grasping action and placement action of each object to be grasped, obtaining a task success rate, and obtaining the test result of the human-computer interaction robotic arm grasping and placement system based on the detection success rate and the task success rate.

[0006] Optionally, before determining the current test task and at least one object to be grasped by the human-computer interaction robotic arm grasping and placement system, the method further includes: acquiring pose data, camera images, and a 3D model of at least one target object; constructing a rigid body of the at least one target object based on the pose data and the camera images; obtaining real labeled data of the at least one target object based on a first coordinate transformation formula between the pose data, the camera images, the rigid body, and the 3D model; fusing the real labeled data and preset synthetic data to obtain a synthetic dataset to be trained; training a preset target detection network using the synthetic dataset to be trained to obtain a target detection model; and training a preset object pose estimation network using the synthetic dataset to be trained to obtain the object pose estimation model.

[0007] Optionally, the first coordinate transformation formula is:

[0008]

[0009] in, This is the time-varying coordinate transformation matrix from the 3D model coordinate system of the target object to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the motion capture system coordinate system to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the rigid body coordinate system to the motion capture system coordinate system. This is the coordinate transformation matrix from the 3D model coordinate system to the rigid body coordinate system of the target object.

[0010] Optionally, determining the coordinate transformation matrix from the three-dimensional model coordinate system to the rigid body coordinate system of the target object further includes: determining the coordinates of the reflective marker on the rigid body based on the second coordinate transformation formula; modeling the reflective marker to obtain the position of the reflective marker on the three-dimensional model; and using a manual coarse matching strategy and the iterative nearest point (ICP) algorithm to obtain the coordinate transformation matrix from the three-dimensional model coordinate system to the rigid body coordinate system of the target object.

[0011] Optionally, the second coordinate transformation formula is:

[0012]

[0013] Where, x rigid The position of the reflective marker in rigid body coordinates. Let x be the inverse rotation matrix of the rigid body in the motion capture system coordinates. mc (t) represents the position of the reflective marker in the motion capture system coordinates. This represents the position of the rigid body in the motion capture system coordinates.

[0014] Optionally, the step of fusing the real labeled data and the preset synthetic data to obtain the synthetic dataset to be trained includes: acquiring synthetic data that meets the preset camera shooting position and viewpoint conditions, and fusing it with the real labeled data to generate the synthetic data to be trained; and rendering the synthetic data to be trained using a preset physical rendering strategy to obtain the synthetic dataset to be trained.

[0015] Optionally, acquiring the pose data, camera image, and 3D model of the target object includes: acquiring the initial pose data of the target object; filtering and segmenting the data in the initial pose data that meets preset invalid data conditions to obtain the pose data.

[0016] A second aspect of this application provides a device for constructing and testing a human-computer interaction robotic arm grasping and placement system. The human-computer interaction robotic arm grasping and placement system is constructed from a target detection model and an object pose estimation model. The device includes: a first determining module, configured to determine the current test task of the human-computer interaction robotic arm grasping and placement system and at least one object to be grasped, wherein the current test task includes a grasping task and a placement task; and a second determining module, configured to determine the grasping position of each object to be grasped based on the grasping task and the placement task, using the target detection model. The system utilizes the object pose estimation model to determine the grasping and placement actions for each object to be grasped. A testing module controls the human-computer interaction robotic arm grasping and placement system to grasp the corresponding object based on its grasping position, obtaining a detection success rate. It also controls the system to grasp and place the corresponding object based on the grasping and placement actions, obtaining a task success rate. Finally, the system obtains the test results of the human-computer interaction robotic arm grasping and placement system based on the detection success rate and the task success rate.

[0017] Optionally, before determining the current test task and at least one object to be grasped by the human-computer interaction robotic arm grasping and placing system, the first determining module is further configured to: acquire pose data, camera images, and a 3D model of at least one target object; construct a rigid body of the at least one target object based on the pose data and the camera images; obtain real labeled data of the at least one target object based on a first coordinate transformation formula between the pose data, the camera images, the rigid body, and the 3D model; fuse the real labeled data and preset synthetic data to obtain a synthetic dataset to be trained; train a preset target detection network using the synthetic dataset to be trained to obtain a target detection model; and train a preset object pose estimation network using the synthetic dataset to be trained to obtain the object pose estimation model.

[0018] Optionally, the first coordinate transformation formula is:

[0019]

[0020] in, This is the time-varying coordinate transformation matrix from the 3D model coordinate system of the target object to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the motion capture system coordinate system to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the rigid body coordinate system to the motion capture system coordinate system. This is the coordinate transformation matrix from the 3D model coordinate system to the rigid body coordinate system of the target object.

[0021] Optionally, the first determining module is further configured to: determine the coordinates of the reflective marker on the rigid body based on the second coordinate transformation formula; model the reflective marker to obtain the position of the reflective marker on the three-dimensional model, and use a manual coarse matching strategy and the iterative nearest point ICP algorithm to obtain the coordinate transformation matrix from the three-dimensional model coordinate system to the rigid body coordinate system of the target object.

[0022] Optionally, the second coordinate transformation formula is:

[0023]

[0024] Where, x rigid The position of the reflective marker in rigid body coordinates. Let x be the inverse rotation matrix of the rigid body in the motion capture system coordinates. mc (t) represents the position of the reflective marker in the motion capture system coordinates. This represents the position of the rigid body in the motion capture system coordinates.

[0025] Optionally, the first determining module is further configured to: acquire synthetic data that meets preset camera shooting position and viewing angle conditions, and fuse it with the real labeled data to generate synthetic data to be trained; render the synthetic data to be trained using a preset physical rendering strategy to obtain the synthetic dataset to be trained.

[0026] Optionally, the first determining module is further configured to: acquire the initial pose data of the target object; filter and segment the data in the initial pose data that meets the preset invalid data conditions to obtain the pose data.

[0027] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the construction and testing method of the human-computer interaction robotic arm grasping and placing system as described in the above embodiments.

[0028] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the construction and testing method of the human-computer interaction robotic arm grasping and placing system as described in the above embodiments.

[0029] This application, based on grasping and placement tasks, utilizes an object detection model to determine the grasping position of each object to be grasped, and an object pose estimation model to determine the grasping and placement actions for each object. The human-computer interactive robotic arm grasping and placement system is then controlled to grasp the corresponding object based on its grasping position, yielding a detection success rate. Furthermore, the system is controlled to grasp and place the corresponding object based on its grasping and placement actions, yielding a task success rate. The test results of the human-computer interactive robotic arm grasping and placement system are obtained based on both the detection success rate and the task success rate. This addresses the problems in related technologies where object pose estimation datasets lack real-world data, failing to adequately reflect complex real-world applications. It also solves the problem of poor model generalization ability due to domain differences between real and synthetic data, hindering the robotic arm's ability to accurately understand the real-time position and orientation of objects for precise manipulation. This approach reduces data annotation costs while maintaining model performance, improving model generalization ability and performance in real-world applications, and ultimately enabling the robotic arm to accurately understand the real-time position and orientation of objects for precise manipulation.

[0030] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0031] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0032] Figure 1 This is a flowchart illustrating the construction and testing method of a human-computer interaction robotic arm grasping and placing system according to an embodiment of this application;

[0033] Figure 2 This is a schematic diagram illustrating the process of obtaining real labeled data according to an embodiment of this application;

[0034] Figure 3 This is a schematic diagram of a data acquisition scenario according to an embodiment of this application;

[0035] Figure 4 This is a schematic diagram illustrating the conversion process in various systems based on an embodiment of this application;

[0036] Figure 5 This is a schematic diagram illustrating the actual pose of an object according to an embodiment of this application;

[0037] Figure 6 This is a schematic diagram of synthetic data according to an embodiment of this application;

[0038] Figure 7This is a schematic diagram showing the test results of a human-computer interaction robotic arm grasping and placing system according to an embodiment of this application;

[0039] Figure 8 This is a block diagram of the construction and testing device for a human-computer interaction robotic arm grasping and placing system according to an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0041] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0042] The following describes, with reference to the accompanying drawings, a method and apparatus for constructing and testing a human-computer interaction robotic arm grasping and placing system according to embodiments of this application. Addressing the issues mentioned in the background section regarding the lack of real-world data in object pose estimation datasets, which fails to adequately reflect complex situations in practical applications, and the poor generalization ability of models due to domain differences between real and synthetic data, thus hindering the robotic arm's ability to accurately understand the real-time position and orientation of objects for precise manipulation, this application provides a method for constructing and testing a human-computer interaction robotic arm grasping and placing system. In this method, based on grasping and placing tasks, a target detection model is used to determine the grasping position of each object to be grasped, and an object pose estimation model is used to determine the grasping and placing actions of each object to be grasped. The human-computer interaction robotic arm grasping and placing system is controlled to grasp the corresponding object based on its grasping position, obtaining a detection success rate. The system is also controlled to grasp and place the corresponding object based on its grasping and placing actions, obtaining a task success rate. Finally, the test results of the human-computer interaction robotic arm grasping and placing system are obtained based on the detection success rate and the task success rate. This addresses the problem that related technologies lack real-world data in object pose estimation datasets, which fails to adequately reflect complex real-world applications. Furthermore, the domain gap between real and synthetic data leads to poor model generalization, hindering robotic arms from accurately understanding the real-time position and orientation of objects for precise control. This approach reduces data annotation costs while maintaining model performance, improves model generalization and performance in real-world scenarios, and helps robotic arms understand the real-time position and orientation of objects to ensure precise control.

[0043] Six-DOF object pose estimation is of significant research value in the fields of robotics and computer vision. Object pose estimation involves six parameters, including three translation parameters and three rotation parameters. Its main purpose is to extract the position and orientation information of an object in three-dimensional space from RGB images or other types of data.

[0044] In recent years, deep learning technology has achieved remarkable results in many computer vision tasks, including the six-DOF object pose estimation problem. Currently, there are many advanced methods based on deep learning. Although deep learning-based methods can achieve better performance, deep networks are highly dependent on the quantity and quality of training data. They usually require a large amount of labeled data for training. Therefore, a training dataset that matches the actual application scenario has a crucial impact on the performance of the model.

[0045] On the other hand, the quality and quantity of data have a significant impact on algorithm performance. In the problem of object pose estimation, how to obtain high-quality annotations remains an unsolved research issue. Since object pose involves localization and reconstruction in three-dimensional space, the annotation difficulty is relatively higher compared to other common computer vision tasks (such as object detection and image classification).

[0046] Current common methods for preparing publicly available object pose estimation datasets primarily rely on auxiliary visual labels or the relationships between image sequences. Additionally, SfM (Structure from Motion) techniques, a 3D reconstruction method based on image sequences, can be used with COLMAP. However, these annotation methods have limited applicability in practical applications, especially for moving objects. Therefore, in the field of six-DOF object pose estimation, researching how to obtain high-quality labeled data and achieve high-performance deep learning methods with limited labeled data remains a crucial problem to be solved.

[0047] Specifically, Figure 1 This is a flowchart illustrating the construction and testing method of a human-computer interaction robotic arm grasping and placing system provided in an embodiment of this application.

[0048] like Figure 1 As shown, the construction and testing method of the human-computer interaction robotic arm grasping and placement system is described. The system is constructed from a target detection model and an object pose estimation model. The method includes the following steps:

[0049] In step S101, the current test task of the human-machine interaction robotic arm grasping and placing system and at least one object to be grasped are determined, wherein the current test task includes a grasping task and a placing task.

[0050] At least one of the objects to be grabbed is a common household object.

[0051] Optionally, in some embodiments, before determining the current test task of the human-computer interaction robotic arm grasping and placing system and at least one object to be grasped, the method further includes: acquiring pose data, camera images, and a 3D model of at least one target object; constructing a rigid body of at least one target object based on the pose data and camera images; obtaining real labeled data of at least one target object based on a first coordinate transformation formula between the pose data, camera images, rigid body, and 3D model; fusing the real labeled data and preset synthetic data to obtain a synthetic dataset to be trained; training a preset target detection network using the synthetic dataset to be trained to obtain a target detection model; and training a preset object pose estimation network using the synthetic dataset to be trained to obtain an object pose estimation model.

[0052] In some embodiments, the first coordinate transformation formula is:

[0053]

[0054] in, This is the time-varying coordinate transformation matrix from the 3D model coordinate system of the target object to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the motion capture system coordinate system to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the rigid body coordinate system to the motion capture system coordinate system. This is the coordinate transformation matrix from the 3D model coordinate system to the rigid body coordinate system of the target object.

[0055] In some embodiments, determining the coordinate transformation matrix from the three-dimensional model coordinate system to the rigid body coordinate system of the target object further includes: determining the coordinates of the reflective marker in the rigid body based on the second coordinate transformation formula; modeling the reflective marker to obtain the position of the reflective marker in the three-dimensional model; and using a manual coarse matching strategy and the iterative nearest point ICP algorithm to obtain the coordinate transformation matrix from the three-dimensional model coordinate system to the rigid body coordinate system of the target object.

[0056] In some embodiments, the second coordinate transformation formula is:

[0057]

[0058] Where, x rigid The position of the reflective marker in rigid body coordinates. Let x be the inverse rotation matrix of the rigid body in the motion capture system coordinates. mc (t) represents the position of the reflective marker in the motion capture system coordinates. This represents the position of the rigid body in the motion capture system coordinates.

[0059] Specifically, before acquiring the pose data of at least one target object, embodiments of this application first perform site setup, such as... Figure 2 As shown, an RGB-D camera is placed in the center of the field, facing the subject holding the object. Motion capture cameras are placed around the field to ensure the object is captured from different angles throughout the space. Motion capture cameras are used to collect motion pose data, while the RGB-D cameras serve as sensors for acquiring dataset images and depth maps. During use, both the motion capture camera and the RGB-D camera are configured to capture color images and depth maps at the same FPS. Figure 3 This is a schematic diagram of the installation site.

[0060] Furthermore, in the process of calibrating camera extrinsic parameters, this embodiment of the application uses an L-shaped optical calibration rod, which is randomly placed in the field and can be captured by the camera angle, to collect multiple sets of extrinsic parameter calibration data. Among them, 2D pixels are manually labeled, and the position of the reflective ball on the motion capture camera is used as 3D point, thereby obtaining the extrinsic parameters of the motion capture camera and the RGB-D camera.

[0061] During the data collection process, this application embodiment acquires multiple object view sequences. Each sequence contains a depth map aligned with color image frames. Several reflective markers are first attached to the object, and a virtual rigid body is established. In addition, before each object data recording, a marker collision action is added as a time soft alignment method. The collision frame is found in both the RGB-D image and the motion capture system as the starting point for system synchronization. Since the motion capture camera and the RGB-D camera set in the previous part operate with the same FPS configuration, the image after this frame can be considered completely aligned.

[0062] Furthermore, in this embodiment, a high-precision 3D scanner is used to scan each object. Each object is scanned and reconstructed into 3D models with and without markings. The former is used to calculate the relationship between the 3D model and the rigid body, while the latter is used as the standard model for the dataset.

[0063] Furthermore, in order to obtain the true labeled data of the target object, that is, to find the relationship of the target object's transformation over time in camera coordinates. The coordinate transformation relationships between pose data, camera images, rigid bodies, and 3D models are obtained by the first coordinate transformation formula. Figure 4 is a schematic diagram of the coordinate transformation of the data's true values ​​in each system. These are the pre-calibrated camera external parameters. This refers to the pose of the rigid body under the motion capture system, collected by the motion capture system. Furthermore, since the rigid body and the 3D model are not directly related, it is necessary to solve the relationship between the reconstructed 3D model and the rigid body under the motion capture system. First, the position of the optical marker in the motion capture system coordinate system is transformed to the coordinate position of the rigid body using the second coordinate transformation formula. Then, the optical marker is modeled using the 3D modeling software Blender, and the position of the reflective ball in the reconstructed 3D model is solved. Finally, a manual coarse matching strategy, i.e., a point-to-point method, is used to find the initial transformation matrix. Afterwards, the ICP algorithm is used for refinement, finally yielding the accurate transformation matrix. The transformation relationship, i.e., the coordinate transformation matrix from the 3D model coordinate system of the target object to the rigid body coordinate system, can be used to calculate the coordinate transformation of each image. That is, to accurately label the data and convert it into a relevant training format for training the model, such as... Figure 5 As shown, Figure 5 (a) is a schematic diagram of the target object. Figure 5 In (b), from left to right are the RGB image frames captured by the camera, the segmentation mask of the object, the bounding box of the object, and the true annotation of the object pose.

[0064] In some embodiments, the synthesis dataset to be trained is obtained by fusing real labeled data and preset synthetic data, including: acquiring synthetic data that meets preset camera shooting position and viewpoint conditions, and fusing it with real labeled data to generate the synthesis dataset to be trained; and rendering the synthesis dataset to be trained using a preset physical rendering strategy to obtain the synthesis dataset to be trained.

[0065] It's important to understand that datasets play a crucial role in deep learning because they directly impact model training and performance. In deep learning, models rely on data to learn features and patterns. To ensure successful training and high performance of deep learning models, it's essential to pay attention to the relevant characteristics, size, diversity, quality, and balance of the dataset. These factors are key to improving model performance.

[0066] This application embodiment conducts a detailed impact analysis on human-computer interaction datasets by comparing training strategies that use only synthetic data, only real data, and a combination of both.

[0067] First, training models using only real data is a common approach. The main advantage of this method is the high realism of the data, thus ensuring good model performance in real-world applications. However, a significant drawback is the high cost of annotation, especially for complex tasks such as object pose estimation. Furthermore, due to the limited availability of real data, models trained solely on real data may exhibit poor generalization ability.

[0068] Secondly, training models with synthetic data can generate a large number of samples without incurring additional labeling costs, thus helping to improve the model's generalization ability. However, there may be a significant domain difference between synthetic and real data, which could impair the model's performance in real-world applications.

[0069] Therefore, combining real and synthetic data to train models aims to leverage the advantages of both, thereby improving model performance. Using both real and synthetic data simultaneously during training increases the amount of data while maintaining data realism, thus enhancing the model's generalization ability. However, this approach still faces the problem of domain disparity, as there can be significant differences between synthetic and real data. To address this, the domain disparity can be narrowed by improving the generation methods of synthetic data, employing higher-quality materials and lighting conditions, and introducing a degree of randomization to simulate the diversity of real-world scenes, thereby enabling more effective training of deep learning models.

[0070] Therefore, to expand the range of object poses not covered in the real dataset and enhance the diversity of the training set, we also included a synthetic dataset for training and testing. To narrow the domain gap between real and synthetic data and generate more realistic synthetic data for training, this embodiment employs a physically-based rendering method to generate the synthetic dataset for training.

[0071] Specifically, a cubic space containing six planes is first constructed. Then, realistic materials are randomly selected using the CC0 texture library and applied to each plane of the space. Based on this, each target object is assigned a uniformly sampled initial pose, along with appropriate lighting. To maintain consistency, the camera's shooting position and viewing angle are set to closely approximate the real-world annotation data. Finally, the composite image is rendered using Blender. Since the Pybullet physics engine records the real-time object pose, both the 3D object measurements and the ground truth object pose of the composite image are obtained.

[0072] During the generation of the synthetic dataset for training, six to seven objects are randomly selected for rendering in each scene. To increase diversity, the same scene is captured with five different camera poses. After shooting, the texture material of the spatial plane is changed to further improve the realism of the generated results. The synthetic image and its result are shown below. Figure 6 As shown.

[0073] Finally, in this embodiment of the application, the generated synthetic dataset to be trained is applied to a preset target detection network (YOLOv4 and YOLOX) and a preset object pose estimation network (CDPN and GDRN) for training, to obtain the target detection model and the object pose estimation model, respectively.

[0074] Therefore, by fusing real labeled data with synthetic data, a synthetic dataset with diversity and high realism was successfully generated, which helps to improve the training effect of deep learning models. It can be trained in a wider range of object poses, thereby improving the model's generalization ability and performance in real-world application scenarios.

[0075] In some embodiments, acquiring the pose data, camera images, and 3D model of the target object includes: acquiring the initial pose data of the target object; filtering and segmenting the data in the initial pose data that meets the preset invalid data conditions to obtain the pose data.

[0076] Based on analysis and several practical preparation experiences, this application embodiment requires filtering out invalid data, including: 1) frames with missing rigid bodies. Due to the human holding the object during the preparation process, severe occlusion may inevitably occur during human activity, resulting in fewer than three reflective markers captured by the motion capture camera, leading to rigid body tracking failure and missing corresponding pose data; 2) abnormal jumping caused by interference. During the acquisition process, due to the instability of the motion capture camera, it is easily affected by environmental interference, causing obvious unreasonable deviations in the pose of the rigid body. For this invalid data, a simple Euclidean distance threshold is used for filtering to obtain pose data. After filtering, the dataset is divided into training and test sets for subsequent testing of various algorithms.

[0077] In step S102, based on the grasping task and the placement task, the grasping position of each object to be grasped is determined by the object detection model, and the grasping action and placement action of each object to be grasped are determined by the object pose estimation model.

[0078] In step S103, the human-computer interaction robotic arm grasping and placing system grasps the corresponding object based on the grasping position of each object to be grasped, and obtains the detection success rate. The human-computer interaction robotic arm grasping and placing system grasps and places the corresponding object based on the grasping and placing actions of each object to be grasped, and obtains the task success rate. The test results of the human-computer interaction robotic arm grasping and placing system are obtained based on the detection success rate and the task success rate.

[0079] Specifically, a human-computer interaction robotic arm grasping and placement system was constructed based on an object detection model and an object pose estimation model. Two test tasks were conducted: placement and handheld grasping. The evaluation metrics are consistent with research in related fields, but will not be detailed here. In these two tests, the grasping posture of the human-computer interaction robotic arm was determined based on the object pose estimated by the object detection model and the object pose estimation model. Five grasping attempts were made for each object, for a total of 100 grasping attempts. Two test metrics were proposed: the first is the detection success rate, which refers to the ability to accurately locate the target object during the experiment; the second is the task success rate, which refers to the ability to successfully complete the grasping and placement operations based on the estimated object pose. Test results show that the success rate for both tasks reached over 90%, which confirms that the constructed vision-guided robotic arm grasping system has strong stability and robustness. By adopting a hybrid dataset training strategy, higher performance and better generalization ability were achieved in practical application scenarios. The test results of the human-computer interaction robotic arm grasping and placement system are as follows: Figure 7 As shown, from left to right, the images are: image frames, the results of object detection and object pose estimation, and images of the grasping and placement frames.

[0080] Therefore, based on the test results, compared to using only one type of data, training with a mixture of real and synthetic data can increase the amount of data while maintaining realism, and reduce the impact of the domain difference between real and synthetic data. This helps to improve the model's generalization ability and performance in real-world application scenarios.

[0081] The construction and testing method of the human-computer interaction robotic arm grasping and placement system proposed in the embodiments of this application involves, based on grasping and placement tasks, determining the grasping position of each object to be grasped using a target detection model, and determining the grasping and placement actions of each object to be grasped using an object pose estimation model. The human-computer interaction robotic arm grasping and placement system is then controlled to grasp the corresponding object based on its grasping position, thus obtaining a detection success rate. Furthermore, the human-computer interaction robotic arm grasping and placement system is controlled to grasp and place the corresponding object based on its grasping and placement actions, thus obtaining a task success rate. Finally, the test results of the human-computer interaction robotic arm grasping and placement system are obtained based on the detection success rate and the task success rate. This solves the problems in related technologies, such as the lack of real data in object pose estimation datasets, which fails to better reflect the complex situations in real-world applications. Furthermore, the domain gap between real and synthetic data leads to poor model generalization ability, which cannot effectively help robotic arms understand the real-time position and orientation of objects for precise control. This approach reduces data annotation costs while ensuring model performance, improves the model's generalization ability and performance in real-world application scenarios, and helps robotic arms understand the real-time position and orientation of objects to ensure precise control.

[0082] Next, referring to the accompanying drawings, we describe the construction and testing apparatus for the human-computer interaction robotic arm grasping and placing system proposed in the embodiments of this application.

[0083] Figure 8 This is a block diagram of the construction and testing device for the human-computer interaction robotic arm grasping and placing system according to an embodiment of this application.

[0084] like Figure 8 As shown, the construction and testing device 10 of the human-computer interaction robotic arm grasping and placement system is constructed from a target detection model and an object pose estimation model. The device includes a first determination module 100, a second determination module 200 and a testing module 300.

[0085] The system comprises the following components: a first determining module 100, used to determine the current test task and at least one object to be grasped by the human-computer interaction robotic arm grasping and placing system; a second determining module 200, used to determine the grasping position of each object to be grasped using a target detection model based on the grasping and placing tasks, and to determine the grasping action and placement action of each object to be grasped using an object pose estimation model; and a testing module 300, used to control the human-computer interaction robotic arm grasping and placing system to grasp the corresponding object to be grasped based on the grasping position of each object to be grasped, to obtain a detection success rate, and to control the human-computer interaction robotic arm grasping and placing system to grasp and place the corresponding object to be grasped based on the grasping action and placement action of each object to be grasped, to obtain a task success rate, and to obtain the test result of the human-computer interaction robotic arm grasping and placing system based on the detection success rate and the task success rate.

[0086] Optionally, in some embodiments, before determining the current test task of the human-computer interaction robotic arm grasping and placing system and at least one object to be grasped, the first determining module 100 is further configured to: acquire pose data, camera images, and a 3D model of at least one target object; construct a rigid body of at least one target object based on the pose data and camera images; obtain real labeled data of at least one target object based on a first coordinate transformation formula between the pose data, camera images, rigid body, and 3D model; fuse the real labeled data and preset synthetic data to obtain a synthetic dataset to be trained; train a preset target detection network using the synthetic dataset to be trained to obtain a target detection model; and train a preset object pose estimation network using the synthetic dataset to be trained to obtain an object pose estimation model.

[0087] Optionally, in some embodiments, the first coordinate transformation formula is:

[0088]

[0089] in, This is the time-varying coordinate transformation matrix from the 3D model coordinate system of the target object to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the motion capture system coordinate system to the RGB-D camera coordinate system. This is the coordinate transformation matrix from the rigid body coordinate system to the motion capture system coordinate system. This is the coordinate transformation matrix from the 3D model coordinate system to the rigid body coordinate system of the target object.

[0090] Optionally, in some embodiments, the first determining module 100 is further configured to: determine the coordinates of the reflective marker in the rigid body based on the second coordinate transformation formula; model the reflective marker to obtain the position of the reflective marker in the three-dimensional model; and use a manual coarse matching strategy and the iterative nearest point ICP algorithm to obtain the coordinate transformation matrix from the three-dimensional model coordinate system to the rigid body coordinate system of the target object.

[0091] Optionally, in some embodiments, the second coordinate transformation formula is:

[0092]

[0093] Where, x rigid The position of the reflective marker in rigid body coordinates. Let x be the inverse rotation matrix of the rigid body in the motion capture system coordinates. mc (t) represents the position of the reflective marker in the motion capture system coordinates. This represents the position of the rigid body in the motion capture system coordinates.

[0094] Optionally, in some embodiments, the first determining module 100 is further configured to: acquire synthetic data that meets preset camera shooting position and viewing angle conditions, and fuse it with real labeled data to generate synthetic data to be trained; render the synthetic data to be trained using a preset physical rendering strategy to obtain the synthetic dataset to be trained.

[0095] Optionally, in some embodiments, the first determining module 100 is further configured to: acquire initial pose data of the target object; filter and segment the data in the initial pose data that meet the preset invalid data conditions to obtain pose data.

[0096] It should be noted that the explanation of the above-described embodiment of the construction and testing method of the human-computer interaction robotic arm grasping and placing system also applies to the construction and testing device of the human-computer interaction robotic arm grasping and placing system in this embodiment, and will not be repeated here.

[0097] The construction and testing device for the human-computer interaction robotic arm grasping and placement system proposed in this application, based on grasping and placement tasks, uses a target detection model to determine the grasping position of each object to be grasped, and uses an object pose estimation model to determine the grasping and placement actions of each object to be grasped. The system controls the human-computer interaction robotic arm grasping and placement system to grasp the corresponding object based on its grasping position, obtaining a detection success rate. It also controls the system to grasp and place the corresponding object based on its grasping and placement actions, obtaining a task success rate. The test results of the human-computer interaction robotic arm grasping and placement system are obtained based on the detection success rate and the task success rate. This solves the problems in related technologies where object pose estimation models cannot effectively help the robotic arm understand the real-time position and orientation of objects for precise control, and where the domain difference between real and synthetic data leads to poor model generalization ability. While ensuring model performance, it reduces data annotation costs, improves the model's generalization ability and performance in practical application scenarios, and helps the robotic arm understand the real-time position and orientation of objects to ensure precise control.

[0098] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0099] The memory 901, the processor 902, and the computer program stored on the memory 901 and capable of running on the processor 902.

[0100] When the processor 902 executes the program, it implements the construction and testing method of the human-computer interaction robotic arm grasping and placing system provided in the above embodiments.

[0101] Furthermore, electronic devices also include:

[0102] Communication interface 903 is used for communication between memory 901 and processor 902.

[0103] The memory 901 is used to store computer programs that can run on the processor 902.

[0104] The memory 901 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0105] If the memory 901, processor 902, and communication interface 903 are implemented independently, then the communication interface 903, memory 901, and processor 902 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0107] The processor 902 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0108] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for constructing and testing a human-computer interaction robotic arm grasping and placing system.

[0109] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0110] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0111] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0112] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0113] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0114] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0116] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for constructing and testing a human-computer interaction robotic arm grasping and placing system, characterized in that, The human-computer interaction mechanical arm grabbing and placing system is constructed by a target detection model and an object pose estimation model, and the method comprises the following steps: determining a current test task and at least one object to be grabbed of the human-computer interaction mechanical arm grabbing and placing system, wherein the current test task comprises a grabbing task and a placing task; based on the grabbing task and the placing task, determining a grabbing position of each object to be grabbed by using the target detection model, and determining a grabbing action and a placing action of each object to be grabbed by using the object pose estimation model; and controlling the human-computer interaction mechanical arm grabbing and placing system to grab a corresponding object to be grabbed based on the grabbing position of each object to be grabbed, obtaining a detection success rate, and controlling the human-computer interaction mechanical arm grabbing and placing system to grab and place a corresponding object to be grabbed based on the grabbing action and the placing action of each object to be grabbed, obtaining a task success rate, and obtaining a test result of the human-computer interaction mechanical arm grabbing and placing system according to the detection success rate and the task success rate.

2. The method of claim 1, wherein, Before determining the current test task and the at least one object to be grabbed of the human-computer interaction mechanical arm grabbing and placing system, the method further comprises: obtaining pose data, camera images and a three-dimensional model of at least one target object; constructing a rigid body of the at least one target object according to the pose data and the camera images, and obtaining real annotation data of the at least one target object based on a first coordinate conversion formula between the pose data, the camera images, the rigid body and the three-dimensional model; fusing the real annotation data and preset synthetic data to obtain a training synthetic data set, training a preset target detection network by using the training synthetic data set to obtain a target detection model, and training a preset object pose estimation network by using the training synthetic data set to obtain the object pose estimation model.

3. The method of claim 2, wherein, The first coordinate conversion formula is: wherein, is a time-varying coordinate transformation matrix from the three-dimensional model coordinate system of the target object to the RGB-D camera coordinate system, is a coordinate transformation matrix from the motion capture system coordinate system to the RGB-D camera coordinate system, is a coordinate transformation matrix from the rigid body coordinate system to the motion capture system coordinate system, is a coordinate transformation matrix from the three-dimensional model coordinate system of the target object to the rigid body coordinate system.

4. The method of claim 3, wherein, determining a coordinate conversion matrix from a three-dimensional model coordinate system of the target object to a rigid body coordinate system, further comprising: determining a coordinate of a retro-reflective marker in the rigid body based on a second coordinate conversion formula; modeling the retro-reflective marker to obtain a position of the retro-reflective marker in the three-dimensional model, and obtaining the coordinate conversion matrix from the three-dimensional model coordinate system of the target object to the rigid body coordinate system by using an artificial coarse matching strategy and an iterative closest point (ICP) algorithm.

5. The method of claim 4, wherein, The second coordinate conversion formula is: where x rigid is the position of the retro-reflective marker in the body coordinate system, is the inverse rotation matrix of the body in the motion capture system coordinate system, x mc (t) is the position of the retro-reflective marker in the motion capture system coordinate system, is the position of the body in the motion capture system coordinate system.

6. The method of claim 2, wherein, The fusing of the real annotation data and the preset synthetic data to obtain the training synthetic data set comprises: obtaining synthetic data satisfying a preset camera shooting position and angle condition, and fusing the synthetic data with the real annotation data to generate a training synthetic data; rendering the training synthetic data by using a preset physical rendering strategy to obtain the training synthetic data set.

7. The method of claim 2, wherein, The obtaining of the pose data, the camera images and the three-dimensional model of the target object comprises: obtaining initial pose data of the target object; filtering and segmenting data satisfying a preset invalid data condition in the initial pose data to obtain the pose data.

8. A construction and testing apparatus for human-robot interaction robotic arm pick and place systems, characterized by, The human-computer interaction robot arm grasping and placing system is constructed by a target detection model and an object pose estimation model, and the device comprises: a first determination module configured to determine a current test task and at least one object to be grasped of the human-computer interaction robot arm grasping and placing system, wherein the current test task comprises a grasping task and a placing task; a second determination module configured to determine a grasping position of each object to be grasped by using the target detection model based on the grasping task and the placing task, and determine a grasping action and a placing action of the each object to be grasped by using the object pose estimation model; and a test module configured to control the human-computer interaction robot arm grasping and placing system to grasp a corresponding object to be grasped based on the grasping position of the each object to be grasped, obtain a detection success rate, and control the human-computer interaction robot arm grasping and placing system to grasp and place the corresponding object to be grasped based on the grasping action and the placing action of the each object to be grasped, obtain a task success rate, and obtain a test result of the human-computer interaction robot arm grasping and placing system according to the detection success rate and the task success rate.

9. An electronic device, comprising: comprises a memory, a processor; wherein the processor runs a program corresponding to executable program codes stored in the memory by reading the executable program codes, so as to implement the construction and test method of the human-computer interaction robot arm grasping and placing system according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The program is executed by the processor to implement the construction and test method of the human-computer interaction robot arm grasping and placing system according to any one of claims 1-7.

Images