Six-degree-of-freedom object sensing system and method
By estimating the plane and edges of objects using RGBD images and depth data, and obtaining six degrees of freedom of objects, the problems of inaccurate object perception and high cost in the prior art are solved, and a high accuracy and low cost object perception system is achieved.
Patent Information
- Application Number
- CN202311826608.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-08-31
- Filing Date
- 2023-12-27
- Publication Date
- 2025-02-28
AI Technical Summary
Existing object perception systems are difficult to accurately obtain the position and size of the box, resulting in failed acquisition tasks and rely on high-cost 3D cameras or scanners.
By using RGBD images and depth data, estimate the plane and corners of the object, obtain the six degrees of freedom of the object, thereby determining the position of the object and reducing the system cost.
It improves the accuracy of the object perception system, reduces system costs, and obtains six degrees of freedom of the object without preset templates.
Smart Images
Figure CN119533270A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an object perception system and method, and more particularly to a six-degree-of-freedom object perception system and method. Background Art
[0002] The object sensing system can be used to pick up boxes. After the system completes the perception of the box (including steps such as identifying and locating the box), the box picking task can be performed by a robot arm or manually based on the perception results, and then the box is moved from the pallet or shelf to the specified position. However, since the position and size of the box are not fixed, it is difficult for the sensing system to obtain complete information about the position and size of the box, which reduces the accuracy of the sensing system and causes the box picking task to fail.
[0003] The existing technology can use a 3D camera to take a 3D photo of the box, or use a 3D scanner to scan the box, so as to identify the position and size of the box to improve the accuracy of the system. However, the above methods must rely on high-pixel 3D cameras or scanner equipment, which are relatively expensive. If a general camera is used to take a photo of the box, the photo only contains the plane information of the box, which is not conducive to improving the perception of the box.
[0004] Therefore, how to develop a six-degree-of-freedom object sensing system and method that can improve the above-mentioned existing technologies is an urgent need at present. Summary of the invention
[0005] The purpose of the present disclosure is to provide a six-degree-of-freedom object perception system and method, which, after generating the outline of the object, uses the RGBD image and depth data of the object to estimate the plane and corners of the object, thereby obtaining the six degrees of freedom of the object with orientation and coordinate information, and then determines the picking position of the object. By obtaining the six degrees of freedom of the object, the present disclosure improves the accuracy of the perception system. In addition, in the process of obtaining the six degrees of freedom of the object, the present disclosure only requires RGBD images, and does not require high-pixel 3D cameras or scanner equipment, thereby reducing the cost of the perception system.
[0006] According to the concept of the present disclosure, the present disclosure provides a six-degree-of-freedom object perception system, comprising: an object judgment device, a contour detection device and a posture estimation device. The object judgment device is configured to receive an RGBD image and determine whether an object exists in the RGBD image, and the RGBD image includes depth data. The contour detection device is connected to the object judgment device, wherein the contour detection device performs contour detection on the RGBD image to obtain the contour of the object. The posture estimation device is connected to the contour detection device, wherein the posture estimation device detects multiple corners of the object based on the contour of the object to obtain coordinate information of the multiple corners, the posture estimation device obtains the plane of the object based on the depth data to obtain direction information of the plane, and the posture estimation device generates six degrees of freedom of the object based on the coordinate information and the direction information. The six degrees of freedom of the object are used to determine at least one picking position for picking up the object.
[0007] According to the concept of the present disclosure, the present disclosure provides a six-degree-of-freedom object perception method, comprising the steps of: (a) acquiring an RGBD image, wherein the RGBD image includes depth data; (b) determining whether there is an object in the RGBD image by an object determination device, executing step (c) when the determination is yes, and executing step (a) again when the determination is no; (c) performing contour detection on the RGBD image by a contour detection device to acquire the contour of the object; (d) detecting multiple corners of the object based on the contour of the object by a posture estimation device to acquire coordinate information of the multiple corners; (e) acquiring the plane of the object based on the depth data of the RGBD image by a posture estimation device to acquire direction information of the plane; (f) generating six degrees of freedom of the object according to the coordinate information of the corners of the object and the direction information of the plane by a posture estimation device; and (g) determining a picking position for picking up the object by using the six degrees of freedom of the object by the posture estimation device. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 4 is a system architecture diagram of a six-degree-of-freedom object perception system according to a preferred embodiment of the present disclosure.
[0009] Figure 2 Schematic diagram of an RGBD image captured by a photographic device.
[0010] Figure 3 This is a system architecture diagram of a six-degree-of-freedom object perception system according to another preferred embodiment of the present invention.
[0011] Figure 4 This is a system architecture diagram of a six-degree-of-freedom object perception system according to another preferred embodiment of the present invention.
[0012] Figure 5 The figure is a flow chart of a six-degree-of-freedom object perception method according to a preferred embodiment of the present disclosure.
[0013] Description of reference numerals:
[0014] 1: Six-degree-of-freedom object perception system
[0015] 2: Photographic Device
[0016] 3: Object judgment device
[0017] 4: Contour detection device
[0018] 5: Pose estimation device
[0019] 51: Corner detection device
[0020] 52: Plane recognition device
[0021] 53: Computing Device
[0022] 6: Object
[0023] 71: First Equipment
[0024] 72: Second device
[0025] S1, S2, S3, S4, S5, S6, S7: Steps DETAILED DESCRIPTION
[0026] Some typical embodiments that embody the features and advantages of the present disclosure will be described in detail in the following description. It should be understood that the present disclosure can have various changes in different implementations without departing from the scope of the present disclosure, and the descriptions and illustrations therein are essentially for illustrative purposes, rather than for limiting the present disclosure.
[0027] Figure 1 1 is a system architecture diagram of a six-degree-of-freedom object perception system 1 according to a preferred embodiment of the present disclosure.
[0028] Figure 2 Schematic diagram of RGBD image captured by the photographic device 2. Figure 1 and Figure 2 As shown, the six-degrees-of-freedom (6DoF) object perception system 1 of the present disclosure includes an object judgment device 3, a contour detection device 4, and a pose estimation device 5. In some embodiments, the six-degree-of-freedom object perception system 1 is connected to a photographic device 2, and the photographic device 2 can be configured to generate and provide an RGBD image to the object judgment device 3. The photographic device 2 can be external to the six-degree-of-freedom object perception system 1, or as a component of the six-degree-of-freedom object perception system 1. The RGBD image includes depth data. Specifically, the RGBD image can be a pair of image frames, and the pair of image frames includes a monocular image frame (such as an RGB image) and a corresponding depth image frame.
[0029] The object determination device 3 is connected to the photographing device 2 to receive the RGBD image and determine whether there is an object 6 in the RGBD image. In some embodiments, the object determination device 3 may use a portion of the RGBD image (e.g., the RGB image) to detect the object 6, wherein the object 6 may be, for example, but not limited to, a box.
[0030] The contour detection device 4 is connected to the object determination device 3, and performs contour detection on the RGBD image when the object 6 exists in the RGBD image to obtain the contour of the object 6. In some embodiments, the contour detection device 4 estimates the contour of the object 6 using a model trained and established by a machine learning algorithm, wherein the model can be stored in the contour detection device 4.
[0031] The posture estimation device 5 is connected to the contour detection device 4, and can detect multiple corners of the object 6 based on the contour of the object 6 to obtain coordinate information of the multiple corners. At the same time, the posture estimation device 5 can obtain the plane of the object 6 based on the RGBD image with depth data to obtain the orientation information of the plane. According to the coordinate information and the orientation information, the posture estimation device 5 can generate six degrees of freedom of the object 6. It should be particularly noted that the six degrees of freedom of the object 6 can be used to determine at least one picking position of the object 6.
[0032] See also Figure 3 In some embodiments, the posture estimation device 5 may include a corner detection device 51, a plane recognition device 52, and a calculation device 53. The corner detection device 51 may be connected to the contour detection device 4, and perform corner detection on the object 6 according to the contour of the object 6 to detect multiple corners of the object 6. The plane recognition device 52 may be connected to the contour detection device 4, and perform plane recognition on the object 6 according to the depth data and the contour of the object 6 to obtain the plane of the object 6. The calculation device 53 may be connected to the corner detection device 51 and the plane recognition device 52, and calculate the coordinate information and direction information according to the multiple corners and planes of the object 6, and then generate six degrees of freedom of the object 6 based on the coordinate information and the direction information. The six degrees of freedom include three translational degrees of freedom and three rotational degrees of freedom, wherein the three translational degrees of freedom include forward and backward movement along the X-axis, left and right movement along the Y-axis, and up and down movement along the Z-axis, and the three rotational degrees of freedom include rotation around the X-axis, rotation around the Y-axis, and rotation around the Z-axis. The computing device 53 may use the six degrees of freedom of the object 6 to determine a picking position for picking up the object 6. In some embodiments, the computing device 53 may calculate a reference center point as the picking position according to the coordinate information.
[0033] Therefore, the present disclosure can use the RGB image (or RGB data) of the RGBD image to identify the object 6, and obtain the corner features of the object 6, and use the depth data corresponding to the RGB image to obtain the plane features of the object 6. The corner features can be used to estimate the translational freedom of the six degrees of freedom, and the plane features can be used to estimate the rotational freedom of the six degrees of freedom. The six degrees of freedom of the object 6 can be obtained through the corner features and the plane features. By obtaining the six degrees of freedom of the object, the present disclosure can improve the accuracy of the system's perception of the object. In addition, in the process of obtaining the six degrees of freedom of the object, the present disclosure only requires RGBD images, and does not require high-pixel 3D cameras or scanner equipment, thereby reducing the cost of the perception system. In addition, compared to the prior art that requires a preset template to identify the position and size of the box, the present disclosure can obtain the six degrees of freedom of the object through RGBD images without the need for a preset template.
[0034] In some embodiments, the object determination device 3 may use a machine learning algorithm to determine whether the object 6 exists in the RGBD image. The machine learning algorithm may be used to train the object determination device 3 on the determination method of the object 6, and the determination method may be stored in the object determination device 3. The object determination device 3 may determine whether the object 6 exists in the RGBD image according to the trained determination method.
[0035] In some embodiments, the object 6 is placed on a pallet or a shelf. When the six-degree-of-freedom object sensing system 1 obtains the picking position of the object 6, the user or the robot arm can pick up the object 6 from the pallet or the shelf according to the picking position and move the object 6 to a designated position. In some embodiments, the six-degree-of-freedom object sensing system 1 can be set in a robot arm, and the robot arm has a moving device. After the robot arm picks up the object 6, it can be moved to a designated position through the moving device, and the robot arm then places the object 6 on a designated pallet or shelf.
[0036] In some embodiments, the RGBD image includes a plurality of objects 6 . After determining the picking positions of the plurality of objects 6 , the computing device 53 may number or encode the plurality of objects 6 respectively, so as to identify different objects 6 .
[0037] In some embodiments, the six-degree-of-freedom object perception system 1 can be connected to ChatGPT (chat generative pre-trained transformer), which can receive a picking instruction for object 6 and convert it into the number (or code) of object 6 after analyzing that the picking instruction contains the object to be picked up as object 6, so that the corresponding object 6 can be picked up by the robot arm according to the number (or code) of object 6.
[0038] In some embodiments, such as Figure 3As shown, the six-degree-of-freedom object sensing system 1 may further include a first device 71 and a second device 72. The object determination device 3 may be disposed in the first device 71, and the contour detection device 4, the corner detection device 51, the plane recognition device 52 and the calculation device 53 may be disposed in the second device 72. In this embodiment, since the first device 71 may apply a machine learning algorithm, the object determination device 3 in the first device 71 may use a machine learning algorithm to determine whether an object 6 exists in the RGBD image. Compared to Figure 3 In another embodiment, as shown in Figure 4 As shown, the contour detection device 4 may be disposed in the first device 71 instead of the second device 72. In this case, the contour detection device 4 in the first device 71 may detect the contour of the object 6 using a machine learning algorithm.
[0039] Figure 5 Flow chart of the six-degree-of-freedom object perception method of the preferred embodiment of the present disclosure. The six-degree-of-freedom object perception method of the present disclosure is applicable to the aforementioned six-degree-of-freedom object perception system 1. Figure 5 As shown, the six-degree-of-freedom perception method disclosed in the present invention includes steps S1, S2, S3, S4, S5, S6 and S7. In step S1, an RGBD image is obtained, wherein the RGBD image includes depth data. In step S2, the object judgment device 3 is used to determine whether there is an object 6 in the RGBD image. When the judgment is yes, step S3 is executed, and when the judgment is no, step S1 is executed again. In step S3, the contour detection device 4 performs contour detection on the RGBD image to obtain the contour of the object 6. In step S4, the posture estimation device 5 detects multiple corners of the object 6 based on the contour of the object 6 to obtain the coordinate information of the multiple corners. In step S5, the posture estimation device 5 obtains the plane of the object 6 based on the depth data of the RGBD image to obtain the direction information of the plane. In step S6, the posture estimation device 5 generates the six degrees of freedom of the object 6 according to the coordinate information of the corners of the object 6 and the direction information of the plane. In step S7 , the posture estimation device 5 uses the six degrees of freedom of the object 6 to determine a picking position for picking up the object 6 .
[0040] In some embodiments, the six-degree-of-freedom object perception method further includes the step of using ChatGPT to receive a grab instruction and convert it into a number corresponding to the object 6 in the RGBD image.
[0041] In summary, the present disclosure provides a six-degree-of-freedom object perception system and method, which improves the accuracy of the perception system by acquiring the six degrees of freedom of the object. In addition, in the process of acquiring the six degrees of freedom of the object, the present disclosure only requires RGBD images, without high-pixel 3D cameras or scanner equipment, thereby reducing the cost of the perception system.
[0042] It should be noted that the above are only preferred embodiments for illustrating the present disclosure, and the present disclosure is not limited to the embodiments described, and the scope of the present disclosure is determined by the claims. The present disclosure may be subject to various changes and modifications by those skilled in the art, but they shall not exceed the scope of protection of the claims.
Claims
1. A six-degree-of-freedom object perception system, comprising: An object determination device is configured to receive an RGBD image and determine whether an object exists in the RGBD image, wherein the RGBD image includes depth data; a contour detection device connected to the object determination device, wherein the contour detection device performs a contour detection on the RGBD image to obtain a contour of the object; and a posture estimation device connected to the contour detection device, wherein the posture estimation device detects multiple corners of the object based on the contour of the object to obtain coordinate information of the multiple corners, the posture estimation device obtains a plane of the object based on the depth data to obtain direction information of the plane, and the posture estimation device generates six degrees of freedom of the object based on the coordinate information and the direction information, in, The six degrees of freedom of the object are used to determine at least one picking position for picking up the object.
2. The six-degree-of-freedom object sensing system according to claim 1, wherein: The posture estimation device comprises: A corner detection device connected to the contour detection device, wherein the corner detection device is configured to perform a corner detection on the object according to the contour of the object to detect the multiple corners of the object; a plane recognition device connected to the contour detection device, wherein the plane recognition device performs a plane recognition on the object according to the depth data and the contour of the object to obtain the plane of the object; and A computing device is connected to the corner detection device and the plane recognition device, wherein the computing device calculates the coordinate information and the direction information according to the multiple corners and the plane of the object, and generates the six degrees of freedom of the object based on the coordinate information and the direction information.
3. The six-degree-of-freedom object sensing system according to claim 2, wherein: The calculation device calculates a reference center point as the picking position according to the coordinate information.
4. The six-degree-of-freedom object sensing system according to claim 1, further comprising: A photographing device is connected to the object judging device and is configured to generate and provide the RGBD image to the object judging device.
5. The six-degree-of-freedom object sensing system according to claim 1, wherein: The contour detection device estimates the contour of the object using a model trained and established by a machine learning algorithm, and the model is stored in the contour detection device.
6. The six-degree-of-freedom object sensing system according to claim 1, wherein: The object determination device uses a machine learning algorithm to determine whether the object exists in the RGBD image. The machine learning algorithm trains a determination method for the object in the object determination device. The object determination device determines whether the object exists in the RGBD image according to the trained determination method.
7. The six-degree-of-freedom object sensing system according to claim 1, wherein: The RGBD image includes a plurality of the objects. After determining the picking positions of the plurality of the objects, the computing device numbers or codes the plurality of the objects respectively.
8. The six-degree-of-freedom object sensing system according to claim 7, wherein: The six-degree-of-freedom object sensing system is connected to a ChatGPT, and the ChatGPT receives a picking instruction and analyzes the picking instruction to pick up the corresponding object according to a code of the object.
9. A six-degree-of-freedom object perception method, comprising the steps of: (a) acquiring an RGBD image, wherein the RGBD image comprises depth data; (b) determining whether an object exists in the RGBD image by an object determination device, and executing step (c) when the determination is yes, and executing step (a) again when the determination is no; (c) performing a contour detection on the RGBD image by a contour detection device to obtain a contour of the object; (d) detecting multiple corners of the object based on the contour of the object by a posture estimation device to obtain coordinate information of the multiple corners; (e) acquiring, by the posture estimation device, a plane of the object based on the depth data of the RGBD image to obtain direction information of the plane; (f) generating, by the posture estimation device, a six-degree-of-freedom of the object according to the coordinate information of the corner of the object and the direction information of the plane; as well as (g) Determining a picking position for picking up the object by using the six degrees of freedom of the object through the posture estimation device.
10. The six-degree-of-freedom object sensing method according to claim 9, wherein: The contour detection device estimates the contour of the object using a model trained and established by a machine learning algorithm, and the model is stored in the contour detection device.
11. The six-degree-of-freedom object sensing method according to claim 9, wherein: The object determination device uses a machine learning algorithm to determine whether the object exists in the RGBD image. The machine learning algorithm trains a determination method for the object in the object determination device. The object determination device determines whether the object exists in the RGBD image according to the trained determination method.
12. The six-degree-of-freedom object sensing method according to claim 9, wherein: The RGBD image includes a plurality of the objects. After determining the picking positions of the plurality of the objects, the computing device numbers or codes the plurality of the objects respectively.
13. The six-degree-of-freedom object sensing method as described in claim 12, wherein the six-degree-of-freedom object sensing method is applicable to a six-degree-of-freedom object sensing system, the six-degree-of-freedom object sensing system is connected to a ChatGPT, the ChatGPT receives a picking instruction and analyzes the picking instruction to pick up the corresponding object according to a code of the object.