Pose determination method, apparatus, system, and computer-readable storage medium

Abstract

This disclosure relates to a pose determination method, apparatus, system, and computer-readable storage medium, and pertains to the field of engineering machinery technology. The pose determination method includes: determining a first pose information of the container in the image domain and a second pose information of the container in the point cloud domain based on acquired image data and first point cloud data of a container; fusing the first pose information and the second pose information based on a first error evaluation result corresponding to the first pose information and a second error evaluation result corresponding to the second pose information; and determining the pose of the container based on the fusion result of the first pose information and the second pose information. The technical solution of this disclosure can improve the calculation accuracy of pose information.

Description

Technical Field

[0001] This disclosure relates to the field of engineering machinery technology, and in particular to a pose determination method, pose determination device, pose determination system, computer-readable storage medium, and computer program product. Background Technology

[0002] In smart ports and automated logistics warehousing scenarios, accurately identifying the spatial orientation of containers is a core prerequisite for automated container loading, unloading, and stacking.

[0003] In related technologies, a single sensor is used to determine the pose information of a container. Summary of the Invention

[0004] The inventors of this disclosure have discovered the following problems in the above-mentioned related technologies: the pose determination method relying on a single sensor has limitations, resulting in low accuracy of pose information calculation.

[0005] In view of this, this disclosure proposes a pose determination technology that can improve the calculation accuracy of pose information.

[0006] According to some embodiments of this disclosure, a pose determination method is provided, comprising: determining a first pose information of the container in the image domain and a second pose information of the container in the point cloud domain based on acquired image data of the container and first point cloud data; fusing the first pose information and the second pose information based on a first error evaluation result corresponding to the first pose information and a second error evaluation result corresponding to the second pose information; and determining the pose of the container based on the fusion result of the first pose information and the second pose information.

[0007] In some embodiments, determining the pose of a container based on the fusion result of the first pose information and the second pose information includes: calibrating the fusion result based on the second point cloud data of the container to determine the pose of the container, wherein the second point cloud data is reference point cloud data.

[0008] In some embodiments, calibrating the fusion result based on the second point cloud data of the container includes: determining the third point cloud data of the container based on the fusion result; and calibrating the fusion result based on the transformation matrix between the second point cloud data and the third point cloud data, wherein the transformation matrix is ​​used to convert the third point cloud data into the second point cloud data.

[0009] In some embodiments, determining the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain based on the acquired image data of the container and the first point cloud data includes: filtering the first feature information and the second feature information based on the feature matching degree between the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain to obtain candidate matching point pairs, wherein the feature matching degree of the candidate matching point pairs is greater than a threshold; and determining the first pose information and the second pose information based on the candidate matching point pairs.

[0010] In some embodiments, the first feature information and the second feature information are filtered based on the feature matching degree between the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain to obtain candidate matching point pairs, including: determining the feature similarity between the first feature information and the second feature information based on the distance between them; and filtering candidate matching point pairs from the first feature information and the second feature information based on the feature similarity.

[0011] In some embodiments, determining the first pose information and the second pose information based on candidate matching point pairs includes: using the shape features of the container as constraints to filter candidate matching point pairs; and determining the first pose information and the second pose information based on the filtering results of the candidate matching point pairs.

[0012] In some embodiments, determining the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain based on the acquired image data of the container and the first point cloud data includes: determining the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain based on the shape features of the container, wherein the shape features include the outline of the lifting components of the container and / or the outline of the lifting mechanism on the lifting components; and determining the first pose information and the second pose information based on the first feature information and the second feature information.

[0013] In some embodiments, determining first feature information of the container in the image domain and second feature information of the container in the point cloud domain based on the shape features of the container includes: determining the first feature information based on the contour of the lifting component and the contour of the lifting mechanism; and determining the second feature information based on the contour of the lifting component.

[0014] In some embodiments, the first feature information includes feature points extracted from the image data of the container. Determining the first pose information and the second pose information based on the first feature information and the second feature information includes: determining a first rotation matrix based on the position coordinates of the feature points in the image domain and the position coordinates of the container in the volume coordinate system, wherein the position coordinates of the container in the volume coordinate system are determined based on the shape features of the lifting component; and determining the first pose information based on the first rotation matrix.

[0015] In some embodiments, determining the first pose information and the second pose information based on the first feature information and the second feature information includes: determining a first direction vector of the container's outline based on the second feature information; determining a second rotation matrix based on the first direction vector and the second direction vector, wherein the second direction vector is a calibration direction vector; and determining the second pose information based on the second rotation matrix.

[0016] In some embodiments, fusing the first pose information and the second pose information based on the first error evaluation result corresponding to the first pose information and the second error evaluation result corresponding to the second pose information includes: fusing the first pose information and the second pose information based on the weighted average of the first pose information and the second pose information, wherein the weight of the first pose information and the weight of the second pose information are determined based on the first error evaluation result and the second error evaluation result.

[0017] According to some other embodiments of this disclosure, a pose determination device is provided, comprising: a first determination module configured to determine a first pose information of the container in the image domain and a second pose information of the container in the point cloud domain based on acquired image data of the container and first point cloud data; a fusion module configured to fuse the first pose information and the second pose information based on a first error evaluation result corresponding to the first pose information and a second error evaluation result corresponding to the second pose information; and a second determination module configured to determine the pose of the container based on the fusion result of the first pose information and the second pose information.

[0018] In some embodiments, the second determining module is configured to calibrate the fusion result based on the second point cloud data of the container to determine the pose of the container, wherein the second point cloud data is reference point cloud data.

[0019] In some embodiments, the second determining module is configured to determine the third point cloud data of the container based on the fusion result; and to calibrate the fusion result based on the transformation matrix between the second point cloud data and the third point cloud data, wherein the transformation matrix is ​​used to convert the third point cloud data into the second point cloud data.

[0020] In some embodiments, the first determining module is configured to filter the first feature information and the second feature information based on the feature matching degree between the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain, so as to obtain candidate matching point pairs, wherein the feature matching degree of the candidate matching point pairs is greater than a threshold; and determine the first pose information and the second pose information based on the candidate matching point pairs.

[0021] In some embodiments, the first determining module is configured to determine the feature similarity between the first feature information and the second feature information based on the distance between the first feature information and the second feature information; and to filter candidate matching point pairs from the first feature information and the second feature information based on the feature similarity.

[0022] In some embodiments, the first determining module is configured to use the shape features of the container as constraints to filter candidate matching point pairs; and based on the filtering results of the candidate matching point pairs, to determine the first pose information and the second pose information.

[0023] In some embodiments, the first determining module is configured to determine first feature information of the container in the image domain and second feature information of the container in the point cloud domain based on the shape features of the container, wherein the shape features include the outline of the lifting components of the container and / or the outline of the lifting mechanism on the lifting components; and to determine first pose information and second pose information based on the first feature information and the second feature information.

[0024] In some embodiments, the first determining module is configured to determine first feature information based on the contour of the lifting component and the contour of the lifting mechanism; and to determine second feature information based on the contour of the lifting component.

[0025] In some embodiments, the first feature information includes feature points extracted from the image data of the container. A first determining module is configured to determine a first rotation matrix based on the position coordinates of the feature points in the image domain and the position coordinates of the container in the volume coordinate system, the position coordinates of the container in the volume coordinate system being determined based on the shape features of the lifting component; and to determine first pose information based on the first rotation matrix.

[0026] In some embodiments, the first determining module is configured to determine a first direction vector of the container's outline based on second feature information; determine a second rotation matrix based on the first direction vector and a second direction vector, wherein the second direction vector is a calibration direction vector; and determine second pose information based on the second rotation matrix.

[0027] In some embodiments, the fusion module is configured to fuse the first pose information and the second pose information based on a weighted average of the first pose information and the second pose information, wherein the weights of the first pose information and the second pose information are determined based on the first error evaluation result and the second error evaluation result.

[0028] According to some embodiments of this disclosure, a pose determination system is provided, including: a pose determination device configured to perform the pose determination method in any of the above embodiments; and a data acquisition device configured to acquire image data of a container and first point cloud data.

[0029] According to further embodiments of the present disclosure, a pose determination apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the pose determination method of any of the above embodiments based on instructions stored in the memory device.

[0030] According to further embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the pose determination method of any of the above embodiments.

[0031] According to further embodiments of this disclosure, a computer program product is also provided, including instructions that, when executed by a processor, cause the processor to perform the pose determination method according to any of the foregoing embodiments.

[0032] In the above embodiments, the parallel computation of the first pose information corresponding to the image data and the second pose information corresponding to the point cloud data can utilize the texture and semantic information provided by the image data itself, as well as the three-dimensional geometric structure information provided by the point cloud data itself, to provide a high-quality computational basis for the pose calculation results. Moreover, based on the error of the pose calculation results, fusing the first pose information and the second pose information can further improve the calculation accuracy of the pose information. Attached Figure Description

[0033] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0034] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 A flowchart illustrating some embodiments of the pose determination method of this disclosure; Figure 2 Flowcharts illustrating other embodiments of the pose determination method of this disclosure; Figure 3 Block diagrams illustrating some embodiments of the pose determination system of this disclosure are shown; Figure 4 Block diagrams illustrating other embodiments of the pose determination system of this disclosure are shown; Figure 5 Block diagrams illustrating some embodiments of the pose determination apparatus of this disclosure; Figure 6 Block diagrams illustrating other embodiments of the pose determination apparatus of this disclosure; Figure 7 Block diagrams showing further embodiments of the pose determination apparatus of this disclosure are presented. Detailed Implementation

[0035] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0040] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0041] As mentioned above, in related technologies, the pose information of a container is determined by relying solely on point cloud data acquired by a single lidar sensor or image data acquired solely by a single vision sensor.

[0042] However, point cloud data lacks texture information, resulting in limited accuracy in recognizing surface features such as corners and edges of containers. Furthermore, two-dimensional image information is easily affected by lighting and occlusion, further impacting container recognition accuracy. Therefore, pose determination methods relying on a single sensor have limitations, leading to low accuracy in pose information calculation.

[0043] To address at least one of the aforementioned problems, this disclosure proposes a pose determination technique that can improve the accuracy of pose information calculation.

[0044] For example, the technical solution of this disclosure can be implemented through the following embodiments.

[0045] Figure 1 Flowcharts illustrating some embodiments of the pose determination method of this disclosure are shown.

[0046] like Figure 1As shown, in step 110, based on the acquired image data of the container and the first point cloud data, the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain are determined.

[0047] For example, RGB (Red, Green, Blue) image data and point cloud data output by a TOF (Time of Flight) camera can be used to acquire image data and first point cloud data of the container. The first pose information of the container in the image domain can be determined based on the image data, and the second pose information of the container in the point cloud domain can be determined based on the first point cloud data, thus obtaining the first pose information and the second pose information in parallel.

[0048] In this way, parallel computing of the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain can not only take advantage of the features of image data and point cloud data, but also improve the computational efficiency and accuracy of pose information.

[0049] For example, before calculating the first and second pose information, the acquired image data and point cloud data can be preprocessed. An adaptive filtering algorithm can be used to dynamically remove image noise and smooth the point cloud data based on interference from the container operation scenario (e.g., port yard lighting, dust interference). Then, through coordinate system mapping, based on sensor calibration parameters, the pixel coordinates of the container in the local coordinate system and the 3D coordinates of the point cloud are unified to the volume coordinate system. The origin of the local coordinate system can be defined at the center of the locking hole on the corner fitting of the container. The volume coordinate system can be the container coordinate system, and the origin of the container coordinate system can be defined at the center point of the line connecting the two locking holes. The two locking holes can be two lifting holes installed on corner fittings at opposite corners of the container.

[0050] In this way, although the pixel coordinates and the point cloud 3D coordinates are aligned to the same keyhole center through the same local coordinate system, the origin of the local coordinate system (the keyhole) is located at the edge corner of the container, which is not conducive to the overall identification and grasping of the container. Therefore, by transforming the coordinate system, the pixel coordinates and point cloud 3D coordinates under the local coordinate system are unified under the container coordinate system. This not only unifies the data quality but also provides data support for the subsequent registration and fusion of image data and point cloud data, thereby improving the accuracy of the container's pose calculation results.

[0051] In some embodiments, based on the shape features of the container, first feature information of the container in the image domain and second feature information of the container in the point cloud domain are determined. The shape features include the outline of the lifting components of the container and / or the outline of the lifting mechanism on the lifting components. Based on the first feature information and the second feature information, first pose information and second pose information are determined. For example, the first pose information can be determined based on the first feature information of the container in the image domain, and the second pose information can be determined based on the second feature information of the container in the point cloud domain.

[0052] For example, a lifting component can be a corner piece located at the top corner of the container body, and the lifting mechanism can be a locking hole on the corner piece. A corner piece can have multiple locking holes, each performing different tasks such as lifting, securing, and connecting. The locking hole used for lifting can be called a lifting hole. The shape features of the inner contour of the lifting hole on the corner piece, the outer contour of the corner piece, and the intersection of the outer contour edges can serve as feature bases for container target recognition and pose calculation.

[0053] In this way, determining the container's feature information based on the contours of corner fittings and / or keyholes can not only avoid the impact of keyhole deformation on pose calculation results, but also improve the recognition accuracy of container surface features by increasing the richness of feature information, thereby improving the calculation accuracy of pose information.

[0054] In some embodiments, first feature information is determined based on the contours of the lifting components and the lifting mechanism; second feature information is determined based on the contours of the lifting components. For example, the first feature information can be determined by performing corner detection and contour extraction on image data of the container. The second feature information can be determined by performing corner detection and geometric feature extraction on point cloud data of the container.

[0055] For example, models such as YOLOv8 can be used to detect key positioning features such as lifting holes on container corner fittings and the intersection of the outer contour lines of the corner fittings, and to extract the outer contour of the container corner fittings. The first feature information can be determined based on the two-dimensional contour features of the corner fittings' outer contour and the inner contour of the lifting holes, as output by the model. When multiple containers are stacked, the first feature information can also include the outer contour features of each container.

[0056] For example, methods such as voxel mesh filtering and curvature analysis, or voxel mesh filtering and PointNet networks, can be used to detect the corner fittings of a container and extract their geometric features. Based on the three-dimensional contour features of the corner fittings obtained using these methods, including the outer contour (e.g., the four edges of the corner fitting) and geometric features in the point cloud dimension, a second feature information can be determined. The geometric features can include three-dimensional structural information such as the distance and angle of the corner fitting relative to the optical center of the TOF camera.

[0057] In step 120, the first pose information and the second pose information are fused based on the first error evaluation result corresponding to the first pose information and the second error evaluation result corresponding to the second pose information.

[0058] In this way, the first and second pose information obtained by parallel computation are fused based on the error, providing high-quality initial pose values ​​for subsequent accurate point cloud registration.

[0059] For example, based on the calculated first pose information, error evaluation can be performed through reprojection to obtain a first error evaluation result. Based on the calculated second pose information, error evaluation can be performed through the orthogonality constraint of the rotation matrix to obtain a second error evaluation result.

[0060] For example, based on the first feature information, the first rotation matrix and the first translation vector obtained by the PnP (Perspective-n-Point) algorithm can be used to determine the first pose information. Then, the first error evaluation result of the first rotation matrix and the first translation vector is calculated using the reprojection method.

[0061] In this way, by using the reprojection method to verify the accuracy of the first rotation matrix and the first translation vector, the calculated first pose information can be closer to the pose information of the container under standard conditions, thereby improving the accuracy of the pose calculation results.

[0062] For example, based on the second feature information, the second rotation matrix R obtained using a rotation matrix solving algorithm based on the line direction vector can be used to determine the second pose information. Error evaluation is then performed using the orthogonality constraint of the rotation matrix, and the second error evaluation result of the second rotation matrix R is calculated. When performing error evaluation using the orthogonality constraint of the rotation matrix, it is possible to set R to satisfy... T The second error evaluation result of the second rotation matrix R, where R=1 and det(R)=1, is less than the threshold.

[0063] This avoids the influence of the bidirectional ambiguity of the line direction vector (e.g., the line direction vector v and the line direction vector -v represent the same line) on the second pose calculation result, thereby improving the calculation accuracy of pose information.

[0064] In some embodiments, the first pose information and the second pose information are fused based on the weighted average of the first pose information and the second pose information. The weights of the first pose information and the second pose information are determined based on the first error evaluation result and the second error evaluation result.

[0065] For example, based on the first error evaluation result and the second error evaluation result, a normalized result of the first error evaluation result can be calculated as the first weight corresponding to the first pose information; based on the first error evaluation result and the second error evaluation result, a normalized result of the second error evaluation result can be calculated as the second weight corresponding to the second pose information. The first pose information and the second pose information can be fused based on the first error evaluation result and the first weight, and the weighted average of the second error evaluation result and the second weight.

[0066] In this way, by flexibly allocating the first weight corresponding to the first error evaluation result and the second weight corresponding to the second error evaluation result based on the error between the first and second pose information, the influence of calculation error on the fusion result can be avoided, thereby improving the accuracy of pose calculation result.

[0067] In step 130, the pose of the container is determined based on the fusion result of the first pose information and the second pose information.

[0068] For example, the first pose information from the image domain and the second pose information from the point cloud domain can be fused to obtain the container's pose information. The pose information can include the container's position (e.g., coordinates) and angle. The calculated pose information is then output to the forklift machine, facilitating the forklift machine's identification and handling of the container.

[0069] In this way, the feature association method based on image and point cloud fusion can determine the pose information of the container, overcome the application limitations of a single sensor, and improve the calculation accuracy of pose information by significantly reducing the influence of factors such as illumination changes, object occlusion and container deformation.

[0070] In some embodiments, the fusion result is calibrated based on the second point cloud data of the container to determine the container's pose, where the second point cloud data is the reference point cloud data. For example, the fusion result can also be used as the container's initial pose, and the initial pose can be calibrated using the reference point cloud data to optimize the initial pose.

[0071] For example, the reference point cloud data could be point cloud data collected from a container in a standard orientation. In this case, the front of the container is parallel to the imaging plane of the TOF camera, the bottom of the container is horizontal, the four corner pieces of the container are symmetrically distributed at the four corners of the front of the container, and the front of the corner pieces is parallel to the imaging plane of the camera. The lifting hole is located at the center of the front of the corner piece, and when the lifting hole is elliptical, its major axis is horizontal.

[0072] In some embodiments, the third point cloud data of the container is determined based on the fusion result; the fusion result is calibrated based on the transformation matrix between the second point cloud data and the third point cloud data, the transformation matrix being used to convert the third point cloud data into the second point cloud data.

[0073] For example, the calculation results of the initial pose can be optimized by using point-to-line ICP (Iterative Closest Point) or Fast ICP accelerated by KD (k-dimensional) trees, based on the initial pose.

[0074] For example, the third point cloud data can be determined based on the initial pose, and the third point cloud data can be used as the source point cloud, while the second point cloud data can be used as the target point cloud. The input of the ICP algorithm is the source point cloud and the target point cloud, and the output is the transformation matrix that aligns the source point cloud to the target point cloud.

[0075] Since the source point cloud data is determined by the initial pose, the process of aligning the source point cloud to the target point cloud to perform fine registration of the point cloud data can achieve calibration and optimization of the initial pose, thereby improving the calculation accuracy of pose information.

[0076] The following examples illustrate how to determine the first pose information and the second pose information in step 110.

[0077] In some embodiments, the first feature information and the second feature information are filtered based on the feature matching degree between the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain to obtain candidate matching point pairs, wherein the feature matching degree of the candidate matching point pairs is greater than a threshold; based on the candidate matching point pairs, the first pose information and the second pose information are determined.

[0078] In this way, by selecting feature point pairs that meet the requirements from image data and point cloud data based on the threshold relationship of feature matching degree, the computational workload of pose results can be reduced, thereby improving the computational efficiency of pose information. Moreover, calculating the feature matching degree between the first feature information and the second feature information is equivalent to performing coarse registration on the first feature information and the second feature information, which can also improve the computational accuracy of pose information.

[0079] In some embodiments, the feature similarity between the first feature information and the second feature information can be determined based on the distance between them; candidate matching point pairs can then be selected from the first feature information and the second feature information based on the feature similarity. For example, a feature association method based on distance metric and graph matching can be used to select candidate matching point pairs from the first feature information and the second feature information.

[0080] For example, the feature similarity between the first and second feature information can be determined by calculating metrics such as the Euclidean distance between the corner features of the image and the point cloud, and the similarity of feature vectors. Based on the similarity between features, a graph matching algorithm is used to select candidate matching point pairs from the first and second feature information, taking into account the geometric constraints between features.

[0081] In this way, by establishing feature associations between point cloud data and image data, the computational load of the pose calculation process can be reduced by filtering matching candidate point pairs. Moreover, the filtered matching candidate points can simultaneously utilize the texture and semantic information provided by image data and the three-dimensional geometric structure provided by point cloud data. By fusing image data and point cloud data, the quality and robustness of the data are improved, thereby increasing the accuracy of the pose calculation results.

[0082] In some embodiments, the shape features of the container are used as constraints to filter candidate matching point pairs; based on the filtering results of the candidate matching point pairs, first pose information and second pose information are determined. For example, the shape features may include the geometric features of the corner fittings of the container. The geometric features of the corner fittings can be used as prior constraints to filter candidate matching pairs.

[0083] For example, prior geometric features such as the standard spacing of the corner fittings and the included angle of the edges of the container can be introduced as flexible constraints, and the effective edges can be screened through the RANSAC (Random Sample Consensus) method.

[0084] In this way, phenomena such as container deformation can be addressed by fitting non-ideal orthogonal relationships, thereby improving the accuracy of pose calculation results.

[0085] In some embodiments, the first feature information includes feature points extracted from the image data of the container. A first rotation matrix is ​​determined based on the position coordinates of the feature points in the image domain and the position coordinates of the container in the volume coordinate system, the position coordinates of the container in the volume coordinate system being determined based on the shape features of the lifting components; based on the first rotation matrix, a first pose information is determined. For example, the first pose information can be determined using a PnP algorithm based on the extracted feature points' position coordinates in the image domain and the position coordinates of these feature points on the container in the volume coordinate system. The position coordinates of these feature points on the container in the volume coordinate system can be determined by the structural dimensions of the corner fittings and lock holes.

[0086] For example, the PnP algorithm's input includes two-dimensional position coordinates in the image pixel coordinate system, three-dimensional position coordinates in the volume coordinate system, and the camera intrinsic parameter matrix. Its output includes a rotation matrix and a translation vector. The rotation matrix output by the PnP algorithm can be used as the first rotation matrix, and the output translation vector as the first translation vector. Based on the first rotation matrix and the first translation vector, the first pose information can be determined.

[0087] For example, the first feature information may include the two-dimensional contour feature information of the corner piece and the lifting hole. Based on the two-dimensional contour feature information, key feature points such as corner points, keyhole center points, intersection points, midpoints, and endpoints can be extracted. The position coordinates of the key feature points included in the extracted first feature information in the image domain can be used as two-dimensional position coordinates in the image pixel coordinate system.

[0088] For example, the position coordinates determined based on the structural parameters of the mounting holes in the corner piece can be used as three-dimensional position coordinates in the volume coordinate system. For instance, the structural parameters may include known dimensions of the distance from the center of the keyhole to the four corners of the corner piece. Based on these structural dimensions, the position coordinates of each corner relative to the center of the keyhole can be determined.

[0089] In some embodiments, a first direction vector of the container's contour is determined based on second feature information; a second rotation matrix is ​​determined based on the first and second direction vectors, where the second direction vector is a calibration direction vector; and second pose information is determined based on the second rotation matrix. For example, the second pose information can be determined based on the first and second direction vectors using a rotation matrix solving algorithm based on straight line direction vectors.

[0090] For example, the input to a rotation matrix solving algorithm based on a line direction vector includes a source direction vector and a target direction vector, and the output is a unique rotation matrix. The calculated first direction vector can be used as the source direction vector, the pre-calibrated second direction vector as the target direction vector, and the rotation matrix output by the algorithm as the second rotation matrix.

[0091] For example, the second feature information may include the three-dimensional contour feature information of the corner piece. Based on the three-dimensional contour feature information, key feature points such as corner points, intersection points, midpoints, and endpoints can be extracted. For the key feature points, the RANSAC algorithm or PCA (principal components analysis) algorithm can be used to calculate the direction vector of the contour line, which is used as the first direction vector.

[0092] For example, the second direction vector can be a pre-calibrated reference direction vector. The container can be placed at a calibration position, and a TOF camera can be used to collect the 3D coordinates of the four sides of the container's corner fittings in the world coordinate system. After fitting, the true direction vectors u1 and u2 of the long and short sides in the world coordinate system are obtained, and u1 and u2 are used as reference direction vectors for subsequent pose calculations. For example, Figure 3 In the pose determination system shown, the direction of the red arrow on the container is the calibrated reference direction.

[0093] For example, a second translation vector can be determined based on the calculated position coordinates of key feature points and their coordinates in standard point cloud data, along with a second rotation matrix. Based on the second rotation matrix and the second translation vector, second pose information can then be determined.

[0094] Figure 2 Flowcharts illustrating some other embodiments of the pose determination method of this disclosure are shown.

[0095] like Figure 2 As shown, in step 210, the input image and point cloud data are used.

[0096] In step 220, the image and point cloud multimodal data are preprocessed.

[0097] In step 230, corner detection and contour extraction are performed through image branching to determine the first feature information.

[0098] In step 240, corner detection and geometric feature extraction are performed through point cloud branches to determine the second feature information.

[0099] In step 250, feature associations are constructed and flexible geometric constraints are introduced. This ensures the compatibility between point cloud data and image data, and improves the accuracy of the basis for calculating pose information.

[0100] In step 260, the initial pose is calculated in parallel. This improves the computational efficiency of the initial pose by parallelly calculating the first pose information of the image branch and the second pose information of the point cloud branch.

[0101] In step 270, the point cloud is finely registered. This fine registration of the point cloud data optimizes the calculation of the initial pose, thereby improving the accuracy of the calculated initial pose.

[0102] In step 280, the precise pose of the container is output.

[0103] Thus, through Figure 2The pose determination method shown can identify the pose of containers, which can overcome the limitations of relying on single data for container identification, thereby achieving efficient and accurate pose identification of containers and providing reliable technical support for automated grasping operations in industry.

[0104] Figure 3 Block diagrams illustrating some embodiments of the pose determination system of this disclosure are shown.

[0105] like Figure 3 As shown, the pose determination system 3 may include two Time-of-Flight (TOF) cameras 31, a container 32, a computing unit 33, and a forklift 34. The TOF cameras 31 can acquire the three-dimensional depth information of the container using the time-of-flight method, capturing the container's outline, corner features, and other characteristics, and outputting RGB image information. Corner fittings 37 with lifting holes 36 can be installed at the eight corners of the container body 32 for lifting tasks. The computing unit 33 can receive and process the data acquired by the TOF cameras 31, calculate the container's pose, and output the pose information to the forklift 34. The forklift 34 can perform operations such as gripping, handling, and stacking of the container based on the received pose information.

[0106] In some embodiments, two Time-of-Flight (TOF) cameras 31 can be arranged on either side above the area to be identified on the container. The TOF cameras 31 emit light signals, measure the time of flight of the light signals after reflection from the container, calculate the distance to the container surface, and acquire the container's three-dimensional depth information. Simultaneously, they output RGB image information, providing raw data for perceiving the container's outline and position, as well as for pose recognition. The two TOF cameras 31 collect data from different angles, covering a more comprehensive recognition range and improving the completeness and accuracy of pose recognition.

[0107] In some embodiments, the TOF camera 31 can also output RGB image information and point cloud data simultaneously.

[0108] In some embodiments, the container 32 serves as the object of pose recognition, and its corner pieces 37, contour, and other features are key bases for the data acquisition by the TOF camera 31 and the analysis and processing by the calculation unit 33. The corner pieces 37 are located at the eight corners of the container's upper surface and have features such as lifting holes 36 and intersections of edge lines. The red lines with arrows represent the container's contour, used to define the coordinate system and calculate the container's angles. The direction of the red arrows is the calibrated reference direction. The lifting holes 36, the intersections of edge lines, and the container's contour can serve as feature bases for identifying the container and performing pose calculations.

[0109] In some embodiments, the computing unit 33 is signal-connected to the TOF camera 31, and can receive depth data collected by the TOF camera. It then uses algorithms to analyze, process, and calculate the received data to determine the container's position coordinates, attitude angle, and other pose information. In some embodiments, the computing unit 33 transmits the processed pose information to the forklift crane 34 control system.

[0110] In some embodiments, the forklift 34 serves as the actuator and is equipped with a spreader twist lock 35. During operation, the forklift 34 grips the container 32 by inserting the spreader twist lock 35 into the lifting hole 36 of the container 32 and rotating the spreader twist lock 35.

[0111] In some embodiments, the stacker 34 receives container pose information output by the computing unit 33 and performs a grabbing action based on this information.

[0112] In some embodiments, the pose determination system 3 can be used in scenarios such as port yards and logistics parks to help automate and intelligentize container operations and improve logistics efficiency and accuracy.

[0113] Figure 4 Block diagrams illustrating other embodiments of the pose determination system of this disclosure are shown.

[0114] like Figure 4 As shown, the pose determination system 4 may include a pose determination device 41 and a data acquisition device 42.

[0115] In some embodiments, the pose determination device 41 is configured to perform the pose determination method in any of the above embodiments. For example, the pose determination device 41 may integrate […]. Figure 3 The calculation unit 33 shown.

[0116] In some embodiments, the data acquisition device 42 is configured to acquire image data and first point cloud data of the container. For example, the data acquisition device 42 integrates... Figure 3 The TOF camera 31 shown.

[0117] Figure 5 Block diagrams illustrating some embodiments of the pose determination apparatus of this disclosure are shown.

[0118] like Figure 5As shown, the pose determination device 5 includes: a first determination module 51, configured to determine the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain based on the acquired image data of the container and the first point cloud data; a fusion module 52, configured to fuse the first pose information and the second pose information based on the first error evaluation result corresponding to the first pose information and the second error evaluation result corresponding to the second pose information; and a second determination module 53, configured to determine the pose of the container based on the fusion result of the first pose information and the second pose information.

[0119] In some embodiments, the second determining module 53 is configured to calibrate the fusion result based on the second point cloud data of the container to determine the pose of the container, wherein the second point cloud data is reference point cloud data.

[0120] In some embodiments, the second determining module 53 is configured to determine the third point cloud data of the container based on the fusion result; and to calibrate the fusion result based on the transformation matrix between the second point cloud data and the third point cloud data, wherein the transformation matrix is ​​used to convert the third point cloud data into the second point cloud data.

[0121] In some embodiments, the first determining module 51 is configured to filter the first feature information and the second feature information based on the feature matching degree between the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain, so as to obtain candidate matching point pairs, wherein the feature matching degree of the candidate matching point pairs is greater than a threshold; and determine the first pose information and the second pose information based on the candidate matching point pairs.

[0122] In some embodiments, the first determining module 51 is configured to determine the feature similarity between the first feature information and the second feature information based on the distance between the first feature information and the second feature information; and to filter candidate matching point pairs from the first feature information and the second feature information based on the feature similarity.

[0123] In some embodiments, the first determining module 51 is configured to use the shape features of the container as constraints to filter candidate matching point pairs; and to determine the first pose information and the second pose information based on the filtering results of the candidate matching point pairs.

[0124] In some embodiments, the first determining module 51 is configured to determine first feature information of the container in the image domain and second feature information of the container in the point cloud domain based on the shape features of the container, wherein the shape features include the outline of the lifting components of the container and / or the outline of the lifting mechanism on the lifting components; and to determine first pose information and second pose information based on the first feature information and the second feature information.

[0125] In some embodiments, the first determining module 51 is configured to determine first feature information based on the contour of the lifting component and the contour of the lifting mechanism; and to determine second feature information based on the contour of the lifting component.

[0126] In some embodiments, the first feature information includes feature points extracted from the image data of the container. The first determining module 51 is configured to determine a first rotation matrix based on the position coordinates of the feature points in the image domain and the position coordinates of the container in the volume coordinate system, the position coordinates of the container in the volume coordinate system being determined based on the shape features of the lifting component; and to determine first pose information based on the first rotation matrix.

[0127] In some embodiments, the first determining module 51 is configured to determine a first direction vector of the container's outline based on second feature information; determine a second rotation matrix based on the first direction vector and a second direction vector, wherein the second direction vector is a calibration direction vector; and determine second pose information based on the second rotation matrix.

[0128] In some embodiments, the fusion module 52 is configured to fuse the first pose information and the second pose information based on a weighted average of the first pose information and the second pose information, wherein the weights of the first pose information and the second pose information are determined based on the first error evaluation result and the second error evaluation result.

[0129] Figure 6 Block diagrams illustrating other embodiments of the pose determination apparatus of this disclosure are shown.

[0130] like Figure 6 As shown, the pose determination device 6 of this embodiment includes a memory 61 and a processor 62 coupled to the memory 61. The processor 62 is configured to execute the pose determination method in any embodiment of this disclosure based on instructions stored in the memory 61.

[0131] The memory 61 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, database, and other programs.

[0132] Figure 7 Block diagrams showing further embodiments of the pose determination apparatus of this disclosure are presented.

[0133] like Figure 7 As shown, the pose determination device 7 of this embodiment includes a memory 71 and a processor 72 coupled to the memory 71. The processor 72 is configured to execute the pose determination method in any of the foregoing embodiments based on instructions stored in the memory 71.

[0134] The memory 71 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, boot loader, and other programs.

[0135] The pose determination device 7 may also include an input / output interface 73, a network interface 74, and a storage interface 75. These interfaces 73, 74, and 75, as well as the memory 71 and processor 72, can be connected via, for example, a bus 76. The input / output interface 73 provides a connection interface for input / output devices such as a monitor, mouse, keyboard, touchscreen, microphone, and speakers. The network interface 74 provides a connection interface for various networked devices. The storage interface 75 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0136] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The pose determination method, apparatus, system, and computer-readable storage medium according to this disclosure have been described in detail above. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0138] The methods and systems of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the specific order described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0139] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure.

Claims

1. A pose determination method, comprising: Based on the acquired image data of the container and the first point cloud data, the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain are determined. Based on the first error evaluation result corresponding to the first pose information and the second error evaluation result corresponding to the second pose information, the first pose information and the second pose information are fused. The pose of the container is determined based on the fusion result of the first pose information and the second pose information.

2. The pose determination method of claim 1, wherein, Determining the pose of the container based on the fusion result of the first pose information and the second pose information includes: The fusion result is calibrated based on the second point cloud data of the container to determine the pose of the container. The second point cloud data is the reference point cloud data.

3. The pose determination method of claim 2, wherein, The calibration of the fusion result based on the second point cloud data of the container includes: The third point cloud data of the container is determined based on the fusion result; The fusion result is calibrated based on the transformation matrix between the second point cloud data and the third point cloud data, wherein the transformation matrix is ​​used to convert the third point cloud data into the second point cloud data.

4. The pose determination method of claim 1, wherein, The determination of the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain based on the acquired image data and first point cloud data includes: Based on the feature matching degree between the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain, the first feature information and the second feature information are filtered to obtain candidate matching point pairs, wherein the feature matching degree of the candidate matching point pairs is greater than a threshold. Based on the candidate matching point pairs, the first pose information and the second pose information are determined.

5. The pose determination method of claim 4, wherein, The step of filtering candidate matching point pairs based on the feature matching degree between the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain includes: Based on the distance between the first feature information and the second feature information, the feature similarity between the first feature information and the second feature information is determined; Based on the feature similarity, the candidate matching point pairs are selected from the first feature information and the second feature information.

6. The pose determination method of claim 4, wherein, The step of determining the first pose information and the second pose information based on the candidate matching point pairs includes: The candidate matching point pairs are filtered using the shape features of the container as constraints; Based on the filtering results of the candidate matching point pairs, the first pose information and the second pose information are determined.

7. The pose determination method of any one of claims 1 to 6, wherein, The determination of the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain based on the acquired image data and first point cloud data includes: Based on the shape features of the container, a first feature information of the container in the image domain and a second feature information of the container in the point cloud domain are determined. The shape features include the outline of the lifting components of the container and / or the outline of the lifting mechanism on the lifting components. Based on the first feature information and the second feature information, the first pose information and the second pose information are determined.

8. The pose determination method of claim 7, wherein, The step of determining the first feature information of the container in the image domain and the second feature information of the container in the point cloud domain based on the shape features of the container includes: The first feature information is determined based on the outline of the hoisting component and the outline of the hoisting mechanism; The second feature information is determined based on the outline of the hoisting component.

9. The pose determination method of claim 7, wherein, The first feature information includes feature points extracted from the image data of the container. The step of determining the first pose information and the second pose information based on the first feature information and the second feature information includes: Based on the position coordinates of the feature points in the image domain and the position coordinates of the container in the volume coordinate system, a first rotation matrix is ​​determined, wherein the position coordinates of the container in the volume coordinate system are determined based on the shape features of the lifting component; The first pose information is determined based on the first rotation matrix.

10. The pose determination method according to claim 7, wherein, The step of determining the first pose information and the second pose information based on the first feature information and the second feature information includes: Based on the second feature information, a first direction vector of the container's outline is determined; Based on the first direction vector and the second direction vector, a second rotation matrix is ​​determined, wherein the second direction vector is a calibration direction vector; The second pose information is determined based on the second rotation matrix.

11. The pose determination method according to any one of claims 1 to 7, wherein, The step of fusing the first pose information and the second pose information based on the first error evaluation result corresponding to the first pose information and the second error evaluation result corresponding to the second pose information includes: Based on the weighted average of the first pose information and the second pose information, the first pose information and the second pose information are fused. The weights of the first pose information and the second pose information are determined based on the first error evaluation result and the second error evaluation result.

12. A pose determination device, comprising: The first determining module is configured to determine the first pose information of the container in the image domain and the second pose information of the container in the point cloud domain based on the acquired image data and first point cloud data of the container. The fusion module is configured to fuse the first pose information and the second pose information based on the first error evaluation result corresponding to the first pose information and the second error evaluation result corresponding to the second pose information. The second determining module is configured to determine the pose of the container based on the fusion result of the first pose information and the second pose information.

13. A pose determination system, comprising: The pose determination apparatus is configured to perform the pose determination method according to any one of claims 1 to 11; The data acquisition device is configured to acquire image data and first point cloud data of the container.

14. A pose determination device, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the pose determination method according to any one of claims 1 to 11 based on instructions stored in the memory.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pose determination method according to any one of claims 1 to 11.

16. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the pose determination method according to any one of claims 1 to 11.

Images