Marker-based pose determination method, apparatus, device, medium, and product

Abstract

The marker-based pose determination method, device, equipment, medium and product provided by the present disclosure, the method comprises: acquiring an image captured by a camera and a reference pose of a marker set; the marker set comprises at least one marker, and the marker is fixed on a placement table of an object to be grabbed; determining a real-time pose of the marker set in the image captured by the camera in the current camera coordinate system; if it is determined that the reference pose is different from the real-time pose, determining a transformation relationship between the real-time pose and the reference pose; and performing compensation transformation on an initial pose of the object to be grabbed in the image captured by the camera according to the transformation relationship to obtain an actual pose of the object to be grabbed. The method of the present disclosure can avoid the phenomenon that the change of the camera coordinate system caused by the displacement of the camera eventually leads to the error of the determined pose of the object to be grabbed under the grabbing device, and further leads to the problem of object grabbing failure.

Description

Technical Field

[0001] This disclosure relates to the field of electronics, and more particularly to a method, apparatus, device, medium, and product for determining pose based on markers. Background Technology

[0002] Currently, in the process of automated object grasping, it is first necessary to obtain the pose information of the object to be grasped in the camera coordinate system, and then, based on the correspondence between the camera coordinate system and the grasping device coordinate system (i.e., camera extrinsic parameters), the pose information is converted to the grasping device coordinate system so that the grasping device can be controlled to grasp the object to be grasped through the converted pose information.

[0003] However, when the camera is displaced due to an external impact, the camera coordinate system will also change. If the previous correspondence is still used for conversion, it is easy to fail to accurately grasp the object to be grasped.

[0004] Therefore, a pose determination method is urgently needed to avoid the aforementioned technical problems. Summary of the Invention

[0005] The pose determination method, apparatus, equipment, medium, and product based on markers disclosed herein are intended to solve the problem of object grasping failure caused by camera displacement in related technologies.

[0006] In a first aspect, this disclosure provides a pose determination method based on markers, including:

[0007] The system acquires the reference pose of an image captured by a camera and a set of markers; wherein the image captured by the camera includes the set of markers and an object to be grasped; the set of markers includes at least one marker, which is fixed on a platform for the object to be grasped; the reference pose is the pose of the set of markers in the camera coordinate system when the camera's extrinsic parameters are determined.

[0008] Determine the real-time pose of the set of landmarks in the image captured by the camera in the current camera coordinate system;

[0009] If it is determined that the reference pose is different from the real-time pose, then the transformation relationship between the real-time pose and the reference pose is determined.

[0010] The initial pose of the object to be grasped in the image captured by the camera is compensated and transformed according to the transformation relationship to obtain the actual pose of the object to be grasped; wherein, the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system.

[0011] In one possible implementation, determining the real-time pose of the set of landmarks in the image captured by the camera in the current camera coordinate system includes:

[0012] Edge detection processing is performed on the two-dimensional image captured by the camera to determine the edges contained in the two-dimensional image.

[0013] Edge images of markers in a set of markers; wherein the images captured by the camera include two-dimensional images and 5-degree depth images;

[0014] Based on the edge image, determine the two-dimensional coordinates of the preset marker points in the marker;

[0015] The position of the preset marker point is determined based on the two-dimensional coordinates and the depth image captured by the camera.

[0016] Determine the orientation of the preset marker point; and record the position and orientation of the preset marker point.

[0017] The real-time pose of the markers in the set of markers captured by the camera in the current camera coordinate system is determined.

[0018] In one possible implementation, if the marker type in the marker set is concentric circle and the preset marker point is the center of the concentric circle, then determining the two-dimensional coordinates of the preset marker point in the marker based on the edge image includes:

[0019] The edge image is subjected to image correction processing to obtain a corrected image; wherein, the image correction processing is used to correct the elliptical contour in the edge image to a circular contour; the concentric circle type markers have multiple phases.

[0020] It consists of concentric rings;

[0021] Based on the corrected image, determine the center coordinates of the concentric circles in the corrected image, and use the center coordinates as the two-dimensional coordinates of the preset marker point.

[0022] In one possible implementation, determining the orientation of the preset marker points includes: 0. Among the multiple preset marker points corresponding to the set of markers, determining three non-collinear preset marker points as follows:

[0023] First landmark, second landmark, and third landmark;

[0024] Based on the positions of the first, second, and third marker points, an orthogonal coordinate system with the first marker point as the origin is determined.

[0025] The pose of the first marker point is determined based on the current camera coordinate system and the orthogonal coordinate system.

[0026] 5. In one possible implementation, determining the transformation relationship between the real-time pose and the reference pose includes:

[0027] Based on the least squares method, the real-time pose and the reference pose are fitted to obtain the transformation relationship between the real-time pose and the reference pose.

[0028] In one possible implementation, the set of markers includes multiple markers with different shape parameters;

[0029] Determining the transformation relationship between the real-time pose and the reference pose includes: 0. Determining the correspondence between the real-time pose and the reference pose based on the shape parameters of the marker and the image captured by the camera; the correspondence is used to indicate the relationship between the real-time pose and the reference pose that characterize the pose of the same marker.

[0030] Based on the correspondence, the transformation relationship between the real-time pose and the reference pose is determined.

[0031] In one possible implementation, when the marker is a concentric circle type, the shape parameter is the number of rings contained in the concentric circle and the width of each ring.

[0032] In one possible implementation, the markers in the marker set are of the concentric circle type, and the marker set includes at least three non-collinear markers; the concentric circle type markers are composed of multiple rings with the same center.

[0033] In one possible implementation, obtaining the reference pose of the image captured by the camera and the set of markers includes:

[0034] During the capture process, images captured by the camera are obtained;

[0035] In response to receiving an error message from the grasping device, the reference pose of the set of markers is obtained, wherein the error message is used to indicate that the grasping device has failed to grasp the object to be grasped.

[0036] Secondly, this disclosure provides a pose determination device based on markers, comprising:

[0037] An acquisition unit is used to acquire an image captured by a camera and a reference pose of a set of markers; wherein, the image captured by the camera includes the set of markers and an object to be grasped; the set of markers includes at least one marker, which is fixed on a platform for the object to be grasped; the reference pose is the pose of the set of markers in the camera coordinate system when the extrinsic parameters of the camera are determined.

[0038] The first determining unit is used to determine the real-time pose of the set of landmarks in the image captured by the camera in the current camera coordinate system.

[0039] The second determining unit is used to determine the transformation relationship between the real-time pose and the reference pose if it is determined that the reference pose is different from the real-time pose.

[0040] The compensation unit is used to perform compensation transformation on the initial pose of the object to be grasped in the image captured by the camera according to the transformation relationship, so as to obtain the actual pose of the object to be grasped; wherein, the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system.

[0041] In one possible implementation, the first determining unit includes:

[0042] The detection module is used to perform edge detection processing on the two-dimensional image in the image captured by the camera, and to determine the edge image of the marker in the set of markers contained in the two-dimensional image; wherein, the image captured by the camera includes a two-dimensional image and a depth image;

[0043] The first determining module is used to determine the two-dimensional coordinates of a preset marker point in the marker based on the edge image;

[0044] The second determining module is used to determine the position of the preset marker point based on the two-dimensional coordinates and the depth image of the image captured by the camera;

[0045] The third determining module is used to determine the posture of the preset marker point;

[0046] The fourth determining module is used to determine the position and orientation of the preset marker point as the real-time orientation of the marker in the set of markers in the image captured by the camera in the current camera coordinate system.

[0047] In one possible implementation, if the marker type in the marker set is concentric circle and the preset marker point is the center of the concentric circle, then the first determining module is specifically used for:

[0048] The edge image is subjected to image correction processing to obtain a corrected image; wherein, the image correction processing is used to correct the elliptical contour in the edge image to a circular contour; the concentric circle type marker is composed of multiple rings with the same center;

[0049] Based on the corrected image, determine the center coordinates of the concentric circles in the corrected image, and use the center coordinates as the two-dimensional coordinates of the preset marker point.

[0050] In one possible implementation, the third determining module is specifically used for:

[0051] Among the multiple preset marker points corresponding to the set of markers, three non-collinear preset marker points are identified as the first marker point, the second marker point, and the third marker point.

[0052] Based on the positions of the first, second, and third marker points, an orthogonal coordinate system with the first marker point as the origin is determined.

[0053] The pose of the first marker point is determined based on the current camera coordinate system and the orthogonal coordinate system.

[0054] In one possible implementation, the second determining unit is specifically used for:

[0055] Based on the least squares method, the real-time pose and the reference pose are fitted to obtain the transformation relationship between the real-time pose and the reference pose.

[0056] In one possible implementation, the set of markers includes multiple markers with different shape parameters; the second determining unit includes:

[0057] The fifth determining module is used to determine the correspondence between the real-time pose and the reference pose based on the shape parameters of the marker and the image captured by the camera; the correspondence is used to indicate the relationship between the real-time pose and the reference pose that characterize the pose of the same marker.

[0058] The sixth determining module is used to determine the transformation relationship between the real-time pose and the reference pose based on the correspondence.

[0059] In one possible implementation, when the marker is a concentric circle type, the shape parameter is the number of rings contained in the concentric circle and the width of each ring.

[0060] In one possible implementation, the markers in the marker set are of the concentric circle type, and the marker set includes at least three non-collinear markers; the concentric circle type markers are composed of multiple rings with the same center.

[0061] In one possible implementation, the acquisition unit includes:

[0062] The first acquisition module is used to acquire images captured by the camera during the capture process;

[0063] The second acquisition module is used to acquire the reference pose of the set of markers in response to receiving an error message sent by the grasping device, wherein the error message is used to indicate that the grasping device has failed to grasp the object to be grasped.

[0064] Thirdly, this disclosure provides an electronic device, including: a memory and a processor;

[0065] Memory; memory for storing instructions executable by the processor;

[0066] The processor is configured to execute the method as described in any of the first aspects according to the executable instructions.

[0067] Fourthly, this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0068] Fifthly, this disclosure provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of the first aspects.

[0069] This disclosure provides a method, apparatus, device, medium, and product for determining pose based on markers. The method includes: acquiring an image captured by a camera and a reference pose of a set of markers; wherein the image captured by the camera includes the set of markers and an object to be grasped; the set of markers includes at least one marker, which is fixed on a platform for the object to be grasped; the reference pose is the pose of the set of markers in the camera's camera coordinate system when the camera's extrinsic parameters are determined; determining the real-time pose of the set of markers in the image captured by the camera in the current camera coordinate system; if the reference pose is determined to be different from the real-time pose, determining a transformation relationship between the real-time pose and the reference pose; and performing a compensation transformation on the initial pose of the object to be grasped in the image captured by the camera according to the transformation relationship to obtain the actual pose of the object to be grasped; wherein the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system. In this disclosure, after acquiring the image captured by the camera, the real-time pose of the set of markers in the image is identified and compared with the reference pose of the set of markers. If the real-time pose and the reference pose are determined to be different, the pose of the object to be grasped in the image captured by the camera can be further supplemented based on the transformation relationship between the real-time pose and the reference pose. This avoids the problem of incorrect pose of the object to be grasped under the grasping device due to changes in the camera coordinate system caused by camera displacement, and the problem of object grasping failure that is easily caused. In addition, it can also avoid the problem of errors in the position information of the point cloud in the direction perpendicular to the ground caused by temperature drift of the camera. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0071] Figure 1 A flowchart illustrating a marker-based pose determination method provided in this embodiment of the present disclosure;

[0072] Figure 2 A flowchart illustrating a second marker-based pose determination method provided in this embodiment of the present disclosure;

[0073] Figure 3 A schematic diagram of a concentric circle marker provided in an embodiment of this disclosure;

[0074] Figure 4 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0075] Figure 5 This is a schematic diagram of a marker provided in this disclosure;

[0076] Figure 6 A schematic diagram of yet another type of marker provided in this disclosure;

[0077] Figure 7 This is a schematic diagram of a marker-based pose determination device provided in an embodiment of the present disclosure;

[0078] Figure 8 This is a schematic diagram of the structure of a second marker-based pose determination device provided in an embodiment of this disclosure;

[0079] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0080] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0081] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0082] Currently, to improve object grasping efficiency, automated grasping devices (such as robots and robotic arms) are typically used for object grasping. When grasping an object, the first step is to determine the object's pose in the camera coordinate system using an image of the object captured by a camera. Then, using the coordinate transformation relationship between the camera and grasping device coordinate systems (i.e., camera extrinsic parameters), the pose in the camera coordinate system is converted to the pose in the grasping device coordinate system. Finally, the grasping device is moved based on the object's pose in the grasping device coordinate system, and the grasping device is then controlled to grasp the object.

[0083] However, in practical applications, when the camera's position changes under the action of external forces, the coordinate system corresponding to the camera will change. If the coordinate transformation relationship between the previously determined camera coordinate system and the coordinate system corresponding to the grasping device is still used for pose transformation, the grasping device is likely to fail to grasp the object.

[0084] Alternatively, when the camera experiences temperature drift, the position information of the object in the direction perpendicular to the ground determined based on the image captured by the camera will be incorrect, meaning that the height information of the object cannot be accurately identified. Consequently, if the coordinate transformation relationship between the previously determined camera coordinate system and the coordinate system of the grasping device is still used for pose transformation, it will easily lead to the grasping device failing to grasp the object.

[0085] In related technologies, the gripping device needs to be manually stopped, and the transport of the object on the platform to be gripped needs to cease. At this point, the camera's extrinsic parameters need to be recalibrated to ensure accurate object gripping subsequently. However, this method tends to be time-consuming, reducing gripping efficiency, and requires manual intervention.

[0086] The present disclosure provides a method, apparatus, equipment, medium, and product for determining pose based on markers, which are used to solve the above-mentioned technical problems.

[0087] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0088] Figure 1 This is a flowchart illustrating a marker-based pose determination method provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the method includes the following steps:

[0089] S101. Obtain the reference pose of the image captured by the camera and the set of markers; wherein, the image captured by the camera includes the set of markers and the object to be grasped; the set of markers includes at least one marker, which is fixed on the placement platform of the object to be grasped; the reference pose is the pose of the set of markers in the camera coordinate system when the camera's extrinsic parameters are determined.

[0090] For example, in order to obtain the accurate pose of the object to be grasped in this embodiment, a set of markers is first fixedly set on the placement platform for placing the object to be grasped, wherein the set of markers includes at least one marker. It can be understood that the marker in this embodiment is located within the shooting area of ​​the camera installed on the placement platform, so that the image of the marker in the set of markers can be obtained each time the camera takes an image.

[0091] Furthermore, when the set of markers includes one marker, the reference pose of the set of markers is the reference pose of that marker; when the set of markers includes multiple markers, the reference pose of the set of markers can be the set of reference poses of some of the markers in the set or the set of reference poses of all the markers.

[0092] In this embodiment, the reference pose can be understood as the pose of the set of markers in the coordinate system corresponding to the camera when the camera's external parameters are determined. The camera's external parameters (i.e., extrinsic parameters) can be understood as the transformation relationship between the camera coordinate system and the object grasping device coordinate system.

[0093] When object grasping is required, the system can acquire images captured by the camera, as well as the reference pose of the set of markers obtained when the camera's extrinsic parameters are determined. The images captured by the camera include images of the object to be grasped and images of the set of markers.

[0094] S102. Determine the real-time pose of the set of landmarks in the image captured by the camera in the current camera coordinate system.

[0095] For example, after acquiring the image captured by the camera, the real-time pose of the set of landmarks in the image captured by the camera can be determined based on the image captured by the camera in the current camera coordinate system.

[0096] In one example, when determining the real-time pose of a set of landmarks in an image captured by a camera, the landmarks in the set can first be identified in the image, their positions in the image can be determined, and then, by combining the camera's intrinsic parameters and the depth image captured by the camera, the real-time pose of the set of landmarks can be determined. The method for determining the pose of objects in an image based on a camera image can be found in the description of related technologies.

[0097] S103. If the reference pose is determined to be different from the real-time pose, then determine the transformation relationship between the real-time pose and the reference pose.

[0098] For example, in this embodiment, after obtaining the real-time pose of the set of markers and the reference pose of the set of markers, the two poses are compared. If it is determined that the reference pose and the real-time pose are different, it is considered that the coordinate system corresponding to the pose identified from the image captured by the camera is different from the camera coordinate system corresponding to the reference pose, and the pose of the object to be grasped in the image captured by the camera needs to be compensated and transformed. That is, the transformation relationship between the reference pose and the real-time pose can be determined based on the real-time pose and the reference pose corresponding to the set of markers, so as to compensate for the pose determined by the object to be grasped in the currently acquired image captured by the camera. The transformation relationship between the reference pose and the real-time pose can be used to convert the parameter values ​​in the real-time pose to the parameter values ​​in the reference pose.

[0099] If the reference pose is determined to be the same as the real-time pose, the pose of the object to be grasped, which is determined based on the image captured by the camera, can be directly used as the actual pose of the object to be grasped. Based on the actual pose and the camera extrinsic parameters, the pose under the grasping device can be determined so as to control the grasping device to grasp the object to be grasped.

[0100] In one example, when comparing the reference pose and the real pose, the position information in the reference pose and the position information in the real pose can be compared separately to determine the first comparison result; and the attitude information in the reference pose and the attitude information in the real pose can be compared to determine the second comparison result; and the reference pose and the real pose can be determined to be the same based on the first comparison result and the second comparison result.

[0101] In one example, if the difference between the position information in the reference pose and the position information in the real pose is greater than a first preset value, and / or the difference between the position information in the reference pose and the position information in the real pose is greater than a second preset value, then the reference pose and the real pose are determined to be different.

[0102] S104. Based on the transformation relationship, perform a compensation transformation on the initial pose of the object to be grasped in the image captured by the camera to obtain the actual pose of the object to be grasped; wherein, the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system.

[0103] For example, the initial pose of the object to be grasped in this embodiment can be understood as the pose in the current camera coordinate system identified based on the image captured by the current camera.

[0104] After obtaining the transformation relationship between the reference pose and the real-time pose through step S103, the initial pose of the object to be grasped can be further compensated and transformed according to the transformation relationship to obtain the actual pose of the object to be grasped, thereby realizing pose compensation in the camera coordinate system.

[0105] Then, based on the actual pose and the camera extrinsic parameters obtained during previous camera calibration, the pose of the object to be grasped in the coordinate system of the grasping device can be determined, thereby realizing the grasping of the object to be grasped.

[0106] Understandably, in this embodiment, after acquiring the image captured by the camera, the real-time pose of the set of markers in the image is identified and compared with the reference pose of the set of markers. If the real-time pose and the reference pose are determined to be different, the pose of the object to be grasped in the image captured by the camera can be further supplemented based on the transformation relationship between the real-time pose and the reference pose. This avoids the problem that changes in the camera coordinate system caused by camera displacement ultimately lead to incorrect poses of the object to be grasped under the grasping device, and also avoids the problem of object grasping failure. In addition, it can also avoid the problem of errors in the position information of the point cloud in the direction perpendicular to the ground caused by temperature drift of the camera.

[0107] In some embodiments, the similarity of the shape parameters between the markers in the set of markers set on the placement platform and the object to be grasped is less than a preset threshold. That is, in order to ensure that the markers in the image can be accurately identified when performing object recognition on the image captured by the camera, when setting the markers on the placement platform, markers with a small similarity to the object to be grasped can be selected so that the object to be grasped will not be mistakenly identified as a marker in the future, thereby further improving the accuracy of the pose determined by the object.

[0108] Figure 2 A flowchart illustrating the second marker-based pose determination method provided in this disclosure is shown below. Figure 2 As shown, the method includes the following steps:

[0109] S201. Obtain the reference pose of the image captured by the camera and the set of markers; wherein, the image captured by the camera includes the set of markers and the object to be grasped; the set of markers includes at least one marker, which is fixed on the placement platform of the object to be grasped; the reference pose is the pose of the set of markers in the camera coordinate system when the camera's extrinsic parameters are determined.

[0110] For example, the specific principle of step S201 can be found in step S101, and will not be repeated here.

[0111] In one example, after each image is captured, it can be processed using the above method. Figure 1 The method shown compares the reference pose and the real-time pose to improve the accuracy of object grasping.

[0112] In one example, the marker type of the marker set is concentric circle, and the marker set includes at least three non-collinear markers; the concentric circle type markers are composed of multiple rings with the same center.

[0113] For example, in this embodiment, when selecting a marker from the marker set, a concentric circle type marker can be selected. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of a concentric circle marker provided in an embodiment of this disclosure. The concentric circle marker consists of multiple rings, and the widths of the rings can be different or the same. Furthermore, to improve the accuracy of pose determination, in this embodiment, three or more concentric circle markers can be set, and the centers of these three markers are not collinear. Figure 4 As shown, Figure 4 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure. The diagram shows a top view of a placement table, where three concentric circular markers are placed on the table. Furthermore, when there are more than three markers, it is necessary to ensure that at least three of the markers have non-collinear centers. Therefore, by setting at least three non-collinear markers, the accuracy of determining the pose information in the markers' positions can be improved. Moreover, this embodiment employs... Figure 3 The concentric circle markers shown can be distinguished from the circular objects to be grasped.

[0114] In one example, step S201 includes the following steps: during the grasping process, acquiring an image captured by a camera; in response to receiving an error message sent by the grasping device, acquiring a reference pose of a set of markers, wherein the error message is used to indicate that the grasping device has failed to grasp the object to be grasped.

[0115] For example, in this embodiment, when locating the object to be grasped, the image captured by the camera can be acquired first. If it is determined that the grasping device has not returned any error information indicating a failure to grasp the object, the pose information of the object in the current camera coordinate system can be determined directly based on the image captured by the camera. Then, coordinate transformation is performed based on this pose information and camera extrinsic parameters to complete the object grasping. If, after acquiring the image captured by the camera, it is determined that an error message sent by the grasping device has been received, the real-time pose of the set of markers in the image captured by the camera can be further identified, and the reference pose of the set of markers can be obtained and then... Figure 1The embodiment demonstrates the determination of the transformation relationship and the pose compensation transformation. Furthermore, after this compensation transformation, the transformation relationship can be stored so that pose compensation can be directly performed based on the transformation relationship after acquiring images from the camera later.

[0116] In one example, after storing the transformation relationship, if another error message is received, it can continue according to... Figure 1 The transformation relationship is redefined in the manner shown.

[0117] It is understandable that in this embodiment, the grasping device can proceed only after determining that the grasping has failed. Figure 1 The pose compensation shown is stored for subsequent pose compensation of the object to be grasped. Compared to comparing the reference pose and the real-time pose after each image is captured, this reduces the processing resources required by the device.

[0118] S202. Perform edge detection processing on the two-dimensional image in the image captured by the camera to determine the edge images of the markers in the set of markers contained in the two-dimensional image; wherein, the image captured by the camera includes a two-dimensional image and a depth image.

[0119] For example, in this embodiment, the image captured by the camera includes a two-dimensional image and a depth image. When determining the real-time pose of the set of markers, edge detection can first be performed on the objects contained in the two-dimensional image captured by the camera to determine the edge images corresponding to the markers contained in the two-dimensional image.

[0120] S203. Based on the edge image, determine the two-dimensional coordinates of the preset marker points in the marker.

[0121] For example, in this embodiment, for a marker in the marker set, the pose of a pre-selected marker point in the marker can be used as the pose corresponding to the marker. Furthermore, after determining the edge image corresponding to the marker in the marker set, the two-dimensional coordinates of the preset marker point in the marker can be further determined.

[0122] For example, when the marker is rectangular, you can select each vertex and / or the center point of the marker as its pose.

[0123] In one example, if the marker type in the marker set is concentric circles and the preset marker point is the center of the concentric circle, then step S203 includes the following steps:

[0124] The first step of step S203 is to perform image correction processing on the edge image to obtain the corrected image; wherein, the image correction processing is used to correct the elliptical contour in the edge image to a circular contour; the concentric circle type marker is composed of multiple rings with the same center.

[0125] For example, in this embodiment, a marker is used as... Figure 3 The following explanation uses concentric circles as an example, with the preset marker point corresponding to each circle as its center. After determining... Figure 3 When determining the two-dimensional coordinates corresponding to the center of the concentric circles shown, the first step is to perform image correction processing on the acquired edge image. It is understandable that, due to the influence of the camera's shooting angle, the concentric circles in the captured two-dimensional image will appear as ellipses. Therefore, it is necessary to correct the elliptical contours in the edge image to circular contours to improve the accuracy of the subsequently determined center coordinates.

[0126] The second step of step S203: Based on the corrected image, determine the center coordinates of the concentric circles in the corrected image, and set the center coordinates as the two-dimensional coordinates of the preset marker point.

[0127] For example, after correcting the edge image, the center coordinates of the corresponding circle can be further determined based on the circular outline in the corrected image.

[0128] In one example, when determining the center coordinates, the two-dimensional coordinates of the center can be determined based on the circular outline corresponding to any of the rings in the concentric circles, or the final two-dimensional coordinates can be determined based on multiple center coordinates fitted from the circular outlines of multiple rings.

[0129] It is understandable that in this embodiment, when the marker type in the marker set is concentric circles, and the preset marker point of the marker is the center of the circle, image correction processing can be performed on the edge image when determining the center coordinates. This corrects the elliptical contour in the edge image to a circular contour, thereby improving the accuracy of determining the two-dimensional coordinates of the center. Since the center position of the concentric circle marker is a white area, the problem of the camera being unable to accurately determine the depth information of the point cloud at the black-and-white boundary in the image when acquiring the depth image can be avoided.

[0130] For example, Figure 5 This is a schematic diagram of a marker provided in this disclosure. Figure 5 The marker is square in shape. In related technologies, the marker's pose is determined by identifying the poses of three or four vertices within its square outline. However, because inaccurate pose determination can easily occur when identifying point clouds at the boundary between black and white areas, therefore, [the following is omitted as the text is incomplete and cannot be translated]. Figure 3 Concentric circles can avoid the above problems. For example, Figure 6 This is a schematic diagram of another marker provided in this disclosure. Similarly, when identifying the point cloud pose of a circular outline in the image, because the circular outline is located at the boundary between black and white, it will lead to an error in determining the pose of the outline point cloud, thus affecting the subsequent determination of the center coordinates. Compared to Figure 5 and Figure 6 The concentric circle type of marker shown can avoid the above problems.

[0131] S204. Determine the position of the preset marker point based on the two-dimensional coordinates and the depth image of the image captured by the camera.

[0132] For example, after obtaining the two-dimensional coordinates of a preset marker point within a marker, the depth value corresponding to the preset marker point can be further determined based on the depth information contained in the depth image, thereby determining the position of the preset marker point in the current camera coordinate system. In practical applications, when determining the three-dimensional coordinates of the preset marker point (i.e., the position of the preset marker point) based on the two-dimensional coordinates and the depth image, the position of the preset marker point in the current camera coordinate system can be determined by combining the aforementioned two-dimensional coordinates, the depth image, and preset camera intrinsic parameters. The specific principle behind this can be found in related technologies and will not be elaborated here.

[0133] In one example, when determining the depth value corresponding to the center of concentric circles based on the 2D coordinates of the circle's center and the depth image, multiple point clouds can be selected near the center of the concentric circles, and these multiple point clouds are located in... Figure 3 Within the white area where the center of the circle is located, the depth information of the center is further determined using the depth information of multiple point clouds to improve the accuracy of the center's location.

[0134] S205. Determine the orientation of the preset marker points; and determine the position and orientation of the preset marker points as the real-time orientation of the markers in the set of markers in the image captured by the camera in the current camera coordinate system.

[0135] For example, in this embodiment, when the position of the preset marker point is determined, in order to further accurately describe the pose of the preset marker point, the orientation of the preset marker point is further determined, and then the determined orientation and position of the preset marker point are used as the pose of the preset marker point. The orientation of the preset marker point is used to characterize the angular relationship between the preset marker point and the current camera coordinate system.

[0136] Understandably, in this embodiment, when determining the real-time pose of the set of markers, the pose of a preset marker point corresponding to the marker can be used as the real-time pose of the set of markers, thereby improving the efficiency of pose determination. Furthermore, when determining the real-time pose of the set of markers, edge detection can first be performed on the two-dimensional image to identify the edge images of the markers in the two-dimensional image. Then, the two-dimensional coordinates of the marker points are determined based on the edge images, and further combined with the depth image to convert the two-dimensional coordinates into three-dimensional coordinates, which are then used as the positions of the marker points. Afterward, the pose of the marker points is determined to obtain their position.

[0137] In one example, step S205, "determine the pose of the preset marker point," includes the following steps:

[0138] First step: Among the multiple preset marker points corresponding to the set of markers, determine three non-collinear preset marker points as the first marker point, the second marker point, and the third marker point;

[0139] The second step: Based on the positions of the first, second, and third marker points, determine an orthogonal coordinate system with the first marker point as the origin;

[0140] The third step: Determine the pose of the first marker point based on the current camera coordinate system and orthogonal coordinate system.

[0141] For example, in this embodiment, when determining the pose of a preset marker point, three non-collinear marker points can be selected from among the multiple marker points corresponding to at least one marker in the marker set. Then, the pose of the preset marker point is determined based on the two-dimensional coordinates of the three non-collinear marker points. Specifically, the three non-collinear marker points can be referred to as the first marker point, the second marker point, and the third marker point, respectively. The first marker point can be considered as the marker point whose pose is to be determined. Furthermore, an orthogonal coordinate system with the first marker point as the origin can be constructed based on the three marker points. For example, a first vector is determined based on the positions of the first and second marker points; and a second vector is determined based on the positions of the first and third marker points. The first vector can then be used as one coordinate axis in the orthogonal coordinate system; the third vector obtained by the cross product of the first and second vectors can be used as the second coordinate axis of the orthogonal coordinate system; and finally, the cross product of the first and third vectors can be used as the third coordinate axis. Then, based on the obtained orthogonal coordinate system and the current camera coordinate system, the angular relationship between the corresponding coordinate axes of the two coordinate systems is determined as the attitude of the first marker point.

[0142] It is understood that in this embodiment, when determining the pose of the preset marker point, two other preset marker points can be selected, and then the pose of the preset marker point can be determined based on the orthogonal coordinate system constructed by the three non-collinear preset marker points, so as to determine the pose information of each marker point.

[0143] S206. If it is determined that the reference pose and the real-time pose are different, then determine the transformation relationship between the real-time pose and the reference pose.

[0144] In one example, the step S206, "determining the transformation relationship between the real-time pose and the reference pose", can be achieved through the following steps: based on the least squares method, the real-time pose and the reference pose are fitted to obtain the transformation relationship between the real-time pose and the reference pose.

[0145] For example, in this embodiment, when determining the transformation relationship between the real-time pose and the reference pose, the least squares method can be used to fit the two poses, thereby determining the transformation relationship between the real-time pose and the reference pose.

[0146] It is understandable that in this embodiment, by using the least squares method to fit the two poses, the fitting efficiency can be improved, thereby further improving the efficiency of determining the real-time pose of the object to be grasped.

[0147] In one example, if the set of markers includes multiple markers with different shape parameters, then the "determining the transformation relationship between real-time pose and reference pose" in step S206 may include the following steps: "determining the correspondence between real-time pose and reference pose based on the shape parameters of the markers and the image captured by the camera; the correspondence is used to indicate the relationship between the real-time pose and the reference pose that characterize the pose of the same marker; and determining the transformation relationship between real-time pose and reference pose based on the correspondence."

[0148] For example, in this embodiment, when the set of markers includes multiple markers, when determining the transformation relationship between the real-time pose and the reference pose, it is first necessary to determine which reference pose each of the currently determined multiple markers corresponds to in terms of its real-time pose. That is, for example, when the set of markers includes marker A, marker B, and marker C, it is necessary to first match the three obtained reference poses with the three real-time poses one by one. When establishing the correspondence, the multiple markers in the set of markers in this embodiment have different shape parameters. Therefore, when identifying the markers, the correspondence between the reference pose and the real-time pose can be determined based on the shape parameters of each marker. That is, reference poses and real-time poses with the same shape parameters are determined as having a correspondence. Then, based on the determined correspondence, the transformation relationship between the reference pose and the real-time pose is determined.

[0149] It is understood that in this embodiment, multiple markers with different shape parameters can be set in the marker set so that the corresponding reference pose and real-time pose can be determined in the subsequent process, thereby improving the accuracy of the subsequent transformation relationship determination.

[0150] In one example, based on the example above, when the marker is of the concentric circle type, the shape parameters are the number of rings contained in the concentric circle and the width of each ring.

[0151] For example, in this embodiment, when the set of markers includes multiple concentric circle type markers, the shape parameters corresponding to each marker can be determined according to the number and width of the rings contained in the marker, so as to mark multiple markers.

[0152] S207. Perform a compensation transformation on the initial pose of the object to be grasped in the image captured by the camera according to the transformation relationship to obtain the actual pose of the object to be grasped; wherein, the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system.

[0153] For example, the specific principle of step S207 can be found in step S104, and will not be repeated here.

[0154] In this embodiment, when determining the real-time pose of the set of markers, the pose of the preset marker points corresponding to the markers can be used as the real-time pose of the set of markers, thereby improving the efficiency of pose determination. Furthermore, when determining the real-time pose of the set of markers, edge detection can first be performed on the 2D image to identify the edge images of the markers in the 2D image. Then, the 2D coordinates of the marker points are determined based on the edge images, and further combined with the depth image to convert the 2D coordinates into 3D coordinates, which are then used as the positions of the marker points. Afterwards, the pose of the marker points is determined to obtain their position. Moreover, when the marker type in the set of markers is concentric circles, and the preset marker point of the marker is the center of the circle, image correction processing can be performed on the edge image when determining the center coordinates to correct the elliptical contours in the edge image to circular contours, thereby improving the accuracy of determining the 2D coordinates of the center. Since the center position of the concentric circle marker is a white area, the problem of the camera being unable to accurately determine the depth information of the point cloud at the black-and-white boundary in the image when acquiring the depth image can be avoided.

[0155] Figure 7 This is a schematic diagram of a marker-based pose determination device provided in an embodiment of the present disclosure, as shown below. Figure 7 As shown, the device includes:

[0156] The acquisition unit 701 is used to acquire the image captured by the camera and the reference pose of the set of markers; wherein, the image captured by the camera includes the set of markers and the object to be grasped; the set of markers includes at least one marker, which is fixed on the placement platform of the object to be grasped; the reference pose is the pose of the set of markers in the camera coordinate system when the camera's extrinsic parameters are determined.

[0157] The first determining unit 702 is used to determine the real-time pose of the set of landmarks in the image captured by the camera in the current camera coordinate system.

[0158] The second determining unit 703 is used to determine the transformation relationship between the real-time pose and the reference pose if the determined reference pose is different from the real-time pose.

[0159] The compensation unit 704 is used to perform compensation transformation on the initial pose of the object to be grasped in the image captured by the camera according to the transformation relationship, so as to obtain the actual pose of the object to be grasped; wherein, the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system.

[0160] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0161] Figure 8 This is a schematic diagram of the structure of a second marker-based pose determination device provided in an embodiment of this disclosure. Figure 7 Based on the device structure shown, the first determining unit 702 in this embodiment includes:

[0162] The detection module 7021 is used to perform edge detection processing on the two-dimensional image in the image captured by the camera, and to determine the edge image of the marker in the set of markers contained in the two-dimensional image; wherein, the image captured by the camera includes a two-dimensional image and a depth image;

[0163] The first determining module 7022 is used to determine the two-dimensional coordinates of a preset marker point in the marker based on the edge image;

[0164] The second determining module 7023 is used to determine the position of the preset marker point based on the two-dimensional coordinates and the depth image of the image captured by the camera;

[0165] The third determining module 7024 is used to determine the attitude of the preset marker point;

[0166] The fourth determining module 7025 is used to determine the position and orientation of the preset marker points as the real-time orientation of the markers in the set of markers in the image captured by the camera in the current camera coordinate system.

[0167] In one possible implementation, if the marker type in the marker set is concentric circle and the preset marker point is the center of the concentric circle, then the first determining module 7022 is specifically used for:

[0168] Image correction processing is performed on the edge image to obtain the corrected image; wherein, the image correction processing is used to correct the elliptical contour in the edge image to a circular contour; the concentric circle type marker is composed of multiple rings with the same center;

[0169] Based on the corrected image, determine the center coordinates of the concentric circles in the corrected image, and set the center coordinates as the two-dimensional coordinates of the preset marker points.

[0170] In one possible implementation, the third determining module 7024 is specifically used for:

[0171] Among the multiple preset marker points corresponding to the set of markers, three non-collinear preset marker points are identified as the first marker point, the second marker point, and the third marker point.

[0172] Based on the positions of the first, second, and third marker points, an orthogonal coordinate system with the first marker point as the origin is determined.

[0173] Determine the pose of the first marker point based on the current camera coordinate system and orthogonal coordinate system.

[0174] In one possible implementation, the second determining unit 703 is specifically used for:

[0175] Based on the least squares method, the real-time pose and the reference pose are fitted to obtain the transformation relationship between the real-time pose and the reference pose.

[0176] In one possible implementation, the set of markers includes multiple markers with different shape parameters; the second determining unit 703 includes:

[0177] The fifth determining module 7031 is used to determine the correspondence between the real-time pose and the reference pose based on the shape parameters of the marker and the image captured by the camera; the correspondence is used to indicate the relationship between the real-time pose and the reference pose that represent the pose of the same marker.

[0178] The sixth determining module 7032 is used to determine the transformation relationship between the real-time pose and the reference pose based on the correspondence.

[0179] In one possible implementation, when the marker is a concentric circle type, the shape parameters are the number of rings contained in the concentric circle and the width of each ring.

[0180] In one possible implementation, the marker type of the marker set is concentric circle type, and the marker set includes at least three non-collinear markers; the concentric circle type markers are composed of multiple rings with the same center.

[0181] In one possible implementation, the acquisition unit 701 includes:

[0182] The first acquisition module 7011 is used to acquire images captured by the camera during the grasping process;

[0183] The second acquisition module 7012 is used to acquire the reference pose of the set of markers in response to receiving an error message sent by the grasping device, wherein the error message is used to indicate that the grasping device has failed to grasp the object to be grasped.

[0184] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0185] This disclosure provides an electronic device, including: a memory and a processor;

[0186] Memory; memory used to store processor-executable instructions;

[0187] The processor is used to execute methods according to executable instructions.

[0188] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure, such as... Figure 9 As shown, the electronic device includes:

[0189] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can invoke logical instructions stored in the memory 294 to execute the methods of the above embodiments.

[0190] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0191] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0192] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0193] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods.

[0194] This disclosure provides a computer program product including a computer program that, when executed by a processor, implements any one of the methods.

[0195] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0196] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A pose determination method based on markers, characterized in that, include: The system acquires the reference pose of an image captured by a camera and a set of markers; wherein the image captured by the camera includes the set of markers and an object to be grasped; the set of markers includes at least one marker, which is fixed on a platform for the object to be grasped; the reference pose is the pose of the set of markers in the camera coordinate system when the camera's extrinsic parameters are determined. The position and orientation of the preset marker points are determined as the real-time orientation of the set of markers in the image captured by the camera in the current camera coordinate system. The position of the preset marker points is determined based on the two-dimensional coordinates of the preset marker points in the markers and the depth image of the image captured by the camera. If it is determined that the reference pose is different from the real-time pose, then the transformation relationship between the real-time pose and the reference pose is determined. The initial pose of the object to be grasped in the image captured by the camera is compensated and transformed according to the transformation relationship to obtain the actual pose of the object to be grasped; wherein, the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system. If the marker type in the marker set is concentric circle and the preset marker point is the center of the concentric circle, the two-dimensional coordinates of the preset marker point in the marker are determined as follows: Image correction processing is performed on the edge images of the markers in the marker set contained in the two-dimensional image to obtain a corrected image; wherein, the image correction processing is used to correct the elliptical contours in the edge images to circular contours; the concentric circle type markers are composed of multiple rings with the same center; based on the corrected image, the center coordinates of the concentric circles in the corrected image are determined, and the center coordinates are determined as the two-dimensional coordinates of the preset marker point; The pose of the preset marker point is determined as follows: among the multiple preset marker points corresponding to the set of markers, three non-collinear preset marker points are identified as the first marker point, the second marker point, and the third marker point; based on the positions of the first marker point, the second marker point, and the third marker point, an orthogonal coordinate system with the first marker point as the origin is determined; based on the current camera coordinate system and the orthogonal coordinate system, the pose of the first marker point is determined.

2. The method according to claim 1, characterized in that, Determining the real-time pose of the set of landmarks in the image captured by the camera in the current camera coordinate system includes: Edge detection processing is performed on the two-dimensional image captured by the camera to determine the edge images of the markers in the set of markers contained in the two-dimensional image; wherein, the image captured by the camera includes a two-dimensional image and a depth image; Based on the edge image, determine the two-dimensional coordinates of the preset marker points in the marker; The position of the preset marker point is determined based on the two-dimensional coordinates and the depth image captured by the camera. Determine the orientation of the preset marker point; and determine the position and orientation of the preset marker point as the real-time orientation of the marker in the set of markers in the image captured by the camera in the current camera coordinate system.

3. The method according to claim 1, characterized in that, Determining the transformation relationship between the real-time pose and the reference pose includes: Based on the least squares method, the real-time pose and the reference pose are fitted to obtain the transformation relationship between the real-time pose and the reference pose.

4. The method according to claim 1, characterized in that, The set of markers includes multiple markers with different shape parameters; Determining the transformation relationship between the real-time pose and the reference pose includes: Based on the shape parameters of the marker and the image captured by the camera, the correspondence between the real-time pose and the reference pose is determined; the correspondence is used to indicate the relationship between the real-time pose and the reference pose that characterize the pose of the same marker. Based on the correspondence, the transformation relationship between the real-time pose and the reference pose is determined.

5. The method according to claim 4, characterized in that, When the marker is a concentric circle type, the shape parameters are the number of rings contained in the concentric circle and the width of each ring.

6. The method according to claim 1, characterized in that, The marker set consists of concentric circle markers, and the marker set includes at least three non-collinear markers; the concentric circle markers are composed of multiple rings with the same center.

7. The method according to any one of claims 1-6, characterized in that, Obtain the reference pose of the images and landmarks captured by the camera, including: During the capture process, images captured by the camera are obtained; In response to receiving an error message from the grasping device, the reference pose of the set of markers is obtained, wherein the error message is used to indicate that the grasping device has failed to grasp the object to be grasped.

8. A pose determination device based on markers, characterized in that, The marker-based pose determination device is used to implement the marker-based pose determination method according to any one of claims 1-7, including: An acquisition unit is used to acquire an image captured by a camera and a reference pose of a set of markers; wherein, the image captured by the camera includes the set of markers and an object to be grasped; the set of markers includes at least one marker, which is fixed on a platform for the object to be grasped; the reference pose is the pose of the set of markers in the camera coordinate system when the extrinsic parameters of the camera are determined. The first determining unit is used to determine the real-time pose of the set of landmarks in the image captured by the camera in the current camera coordinate system. The second determining unit is used to determine the transformation relationship between the real-time pose and the reference pose if it is determined that the reference pose is different from the real-time pose. The compensation unit is used to perform compensation transformation on the initial pose of the object to be grasped in the image captured by the camera according to the transformation relationship, so as to obtain the actual pose of the object to be grasped; wherein, the initial pose of the object to be grasped is the initial pose of the object to be grasped in the current camera coordinate system.

9. An electronic device, characterized in that, include: Memory, processor; Memory; Memory used to store the processor's executable instructions; The processor is configured to execute the method as described in any one of claims 1-7 according to the executable instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

Images