A method, device, equipment and storage medium for shelf pose recognition

Abstract

The present invention discloses a method, device, equipment and storage medium for shelf pose recognition. The method includes: collecting image information of a shelf to be recognized and detecting the overall side surface of the shelf to be recognized; performing point cloud segmentation on the horizontal and vertical connecting rods of the overall side surface of the shelf to be recognized to obtain the connecting rod point cloud of the shelf to be recognized; calculating the real ground or establishing a virtual ground, and according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correcting and performing plane fitting on the connecting rod point cloud to obtain the pose information of the shelf to be recognized; performing linear fitting on the vertical connecting rods in the connecting rod point cloud and calculating the intersection points of the vertical connecting rods and the real ground or the virtual ground, so as to obtain the position information of the shelf to be recognized; thereby realizing the pose recognition of the shelf to be recognized. The present invention can ensure high accuracy, high robustness and high generality in the recognition of the position and pose of the shelf.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and positioning, and in particular, to a method, device, equipment and storage medium for recognizing the pose of a shelf. Background Art

[0002] Autonomous forklifts are widely used in intelligent warehousing, and more and more warehousing centers use autonomous forklifts to replace manual labor for goods handling. During the process of handling goods, the goods are usually loaded into a shelf (trolley or cage) and transported as a whole. Due to some reasons, when the shelf is placed in its corresponding storage location, its position is not ideal relative to the storage location, and there may be a certain deviation.

[0003] Since an autonomous forklift needs to insert and pick up the shelf through an insertion arm before handling the shelf. If the forklift does not know the position and pose of the shelf in advance and does not adjust the optimal insertion position of the insertion arm, it may fail to insert successfully during the insertion process, or the shelf may be unstable on the insertion arm, resulting in the failure of goods handling. There are three challenges in the recognition of the position and pose of the shelf as follows: First, most shelves are assembled based on cylindrical connecting rods, with complex structures and no specific characteristic rules, making it difficult to summarize general characteristics that conform to the shelf; Second, due to the complex factory environment, there will be interference from other uncontrollable factors such as other shelves, equipment, pedestrians, and light, increasing the complexity of recognizing the pose of the shelf. Third, there are a large number of types of shelves, and more types of shelves that cannot be predicted in advance need to have their poses recognized.

[0004] Therefore, there is an urgent need for a method for recognizing the position and pose of a shelf that can improve the accuracy, robustness, and generality of pose recognition. Summary of the Invention

[0005] The present invention provides a method, device, equipment and storage medium for recognizing the pose of a shelf to solve the technical problems of low accuracy, robustness, and generality in recognizing the position and pose of a shelf in the prior art.

[0006] To solve the above technical problems, an embodiment of the present invention provides a method for recognizing the pose of a shelf, including:

[0007] Collecting image information of the shelf to be recognized and detecting the overall side surface of the shelf to be recognized; the overall side surface of the shelf to be recognized includes a horizontal connecting rod and a vertical connecting rod;

[0008] Performing point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the shelf to be recognized to obtain the connecting rod point cloud of the shelf to be recognized;

[0009] Calculate the real ground or establish a virtual ground, and correct and perform plane fitting on the link point cloud according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground to obtain the pose information of the shelf to be recognized;

[0010] Perform linear fitting on the vertical links in the link point cloud of the shelf to be recognized, and calculate the intersection points of the vertical links and the real ground or the virtual ground, so as to obtain the position information of the shelf to be recognized;

[0011] Obtain the pose information of the shelf to be recognized based on the pose information and position information of the shelf to be recognized, thereby completing the pose recognition of the shelf to be recognized.

[0012] As a preferred solution, the detection of the overall side surface of the shelf to be recognized is specifically:

[0013] According to the deep learning recognition algorithm, judge and detect the shelf to be recognized in the image information, thereby identifying the horizontal links and vertical links of the shelf to be recognized, and thus obtaining the overall side surface of the shelf to be recognized.

[0014] As a preferred solution, the point cloud segmentation of the horizontal links and vertical links of the overall side surface of the shelf to be recognized to obtain the link point cloud of the shelf to be recognized is specifically:

[0015] Perform point cloud segmentation on the horizontal links and vertical links of the overall side surface of the shelf to be recognized to obtain the point cloud information corresponding to each horizontal link and the point cloud information corresponding to each vertical link;

[0016] Combine the point cloud information corresponding to each horizontal link and the point cloud information corresponding to each vertical link respectively to obtain the link point cloud of the overall side surface of the shelf to be recognized.

[0017] As a preferred solution, the calculation of the real ground or the establishment of a virtual ground is specifically:

[0018] Collect ground data and identify the real ground;

[0019] Select a number of points in the forklift coordinate system, and according to the external parameters of the preset camera, transform the selected points in the forklift coordinate system into the camera coordinate system, and perform plane fitting on the points transformed into the camera coordinate system to obtain a virtual ground;

[0020] Verify the credibility of the real ground through the virtual ground;

[0021] If the credibility passes, use the real ground for the pose recognition of the shelf;

[0022] If the credibility fails, use the virtual ground for the pose recognition of the shelf.

[0023] As a preferred solution, according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, the link point cloud is corrected and plane-fitted to obtain the pose information of the shelf to be recognized, specifically:

[0024] Perform a first plane-fitting on the link point cloud to obtain the first normal vector of the first fitted plane;

[0025] According to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correct the link point cloud on the overall side of the shelf to be recognized after the first plane-fitting;

[0026] Perform a second plane-fitting on the corrected link point cloud to obtain the second normal vector of the second fitted plane, and then use the second normal vector as the pose information of the shelf to be recognized.

[0027] As a preferred solution, perform a straight-line fitting on the vertical links in the link point cloud and calculate the intersection points of the vertical links and the virtual ground, so as to obtain the position information of the shelf to be recognized, specifically:

[0028] Perform a straight-line fitting on the vertical links in the link point cloud;

[0029] Calculate the intersection point coordinates of the vertical links after straight-line fitting and the real ground or the virtual ground;

[0030] Take the midpoint of the intersection point coordinates as the position information of the shelf to be recognized.

[0031] As a preferred solution, after obtaining the pose information of the shelf to be recognized according to the pose information and position information of the shelf to be recognized, and completing the pose recognition of the shelf to be recognized, it further includes:

[0032] Transform the pose information obtained by the shelf to be recognized in the camera coordinate system into the forklift coordinate system, so that the forklift can perform handling after the pose recognition of the shelf to be recognized.

[0033] Correspondingly, the present invention further provides a shelf pose recognition device, including: an acquisition and detection module, a point cloud segmentation module, a pose information module, a position information module, and a pose information module;

[0034] The acquisition and detection module is used to acquire the image information of the shelf to be recognized and detect the overall side of the shelf to be recognized; the overall side of the shelf to be recognized includes horizontal links and vertical links;

[0035] The point cloud segmentation module is used to perform point cloud segmentation on the horizontal links and vertical links on the overall side of the shelf to be recognized to obtain the link point cloud of the shelf to be recognized;

[0036] The attitude information module is used to calculate the real ground or establish a virtual ground, and correct and perform plane fitting on the link point cloud according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, so as to obtain the attitude information of the shelf to be recognized;

[0037] The position information module is used to perform linear fitting on the vertical links in the link point cloud, and calculate the intersection points of the vertical links and the real ground or the virtual ground, so as to obtain the position information of the shelf to be recognized;

[0038] The pose information module is used to obtain the pose information of the shelf to be recognized according to the attitude information and position information of the shelf to be recognized, so as to complete the pose recognition of the shelf to be recognized.

[0039] As a preferred solution, the detection of the overall side surface of the shelf to be recognized is specifically:

[0040] According to the deep learning recognition algorithm, judge and detect the shelf to be recognized in the image information, so as to recognize the horizontal links and vertical links of the shelf to be recognized, and thus obtain the overall side surface of the shelf to be recognized.

[0041] As a preferred solution, the point cloud segmentation of the horizontal links and vertical links of the overall side surface of the shelf to be recognized to obtain the link point cloud of the shelf to be recognized is specifically:

[0042] Perform point cloud segmentation on the horizontal links and vertical links of the overall side surface of the shelf to be recognized to obtain the point cloud information corresponding to each horizontal link and the point cloud information corresponding to each vertical link;

[0043] Combine the point cloud information corresponding to each horizontal link and the point cloud information corresponding to each vertical link respectively to obtain the link point cloud of the overall side surface of the shelf to be recognized.

[0044] As a preferred solution, the calculation of the real ground or the establishment of a virtual ground is specifically:

[0045] Collect ground data and recognize the real ground;

[0046] Select a number of points in the forklift coordinate system, and according to the external parameters of the preset camera, transform the selected points in the forklift coordinate system into the camera coordinate system, and perform plane fitting on the points transformed into the camera coordinate system to obtain a virtual ground;

[0047] Verify the credibility of the real ground through the virtual ground;

[0048] If the credibility passes, use the real ground for the pose recognition of the shelf;

[0049] If the confidence level fails, virtual ground is used for pose recognition of the shelf.

[0050] As a preferred solution, according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, the link point cloud is corrected and plane fitting is performed to obtain the pose information of the shelf to be recognized. Specifically:

[0051] Perform first plane fitting on the link point cloud to obtain the first normal vector of the first fitting plane;

[0052] According to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correct the link point cloud on the overall side surface of the shelf to be recognized after the first plane fitting;

[0053] Perform second plane fitting on the corrected link point cloud to obtain the second normal vector of the second fitting plane, and then use the second normal vector as the pose information of the shelf to be recognized.

[0054] As a preferred solution, perform straight line fitting on the vertical links in the link point cloud, and calculate the intersection point of the vertical links and the virtual ground, so as to obtain the position information of the shelf to be recognized. Specifically:

[0055] Perform straight line fitting on the vertical links in the link point cloud;

[0056] Calculate the intersection point coordinates of the vertical links after straight line fitting and the real ground or the virtual ground;

[0057] Take the midpoint of the intersection point coordinates as the position information of the shelf to be recognized.

[0058] As a preferred solution, after obtaining the pose information of the shelf to be recognized based on the pose information and position information of the shelf to be recognized, thereby completing the pose recognition of the shelf to be recognized, it further includes:

[0059] Transform the pose information obtained by the shelf to be recognized in the camera coordinate system into the forklift coordinate system, so that the forklift can perform handling after pose recognition of the shelf to be recognized.

[0060] Correspondingly, the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the shelf pose recognition method described in any one of the above.

[0061] Correspondingly, the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the shelf pose recognition method described in any one of the above.

[0062] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0063] The technical solution of the present invention obtains the image information of the shelf to be recognized, detects the horizontal connecting rod and the vertical connecting rod on the overall side of the shelf to be recognized, and then performs point cloud segmentation on the horizontal connecting rod and the vertical connecting rod, avoiding the problem that the shelf cannot be positioned and the pose cannot be recognized due to the complex shelf structure and other unknown problems, improving the recognition versatility and robustness, and by establishing a virtual ground, and then through correction, plane fitting and straight line fitting, to obtain the corresponding pose information and position information respectively, further restricting the calculation process of the shelf pose to be recognized and optimizing the solution result of the pose information, thereby ensuring the accuracy, robustness and versatility of the shelf pose recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 : is a flowchart of the steps of a shelf pose recognition method provided by an embodiment of the present invention;

[0065] Figure 2 : is a schematic diagram of calculating the shelf position information by wheels provided by an embodiment of the present invention;

[0066] Figure 3 : is a specific flowchart of a shelf pose recognition method provided by an embodiment of the present invention;

[0067] Figure 4 : is a schematic structural diagram of a shelf pose recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

[0069] Before the forklift forks and transports the shelf, it needs to automatically adjust its position according to the placement pose of the shelf, so that the forklift's fork arms can fork the bottom of the shelf at the best position and attitude, thereby ensuring the stability and reliability of the goods during transportation. In an exemplary embodiment of the present invention, a method for identifying the position and attitude of a shelf based on an RGBD sensor is proposed. This method combines a deep learning algorithm and can perform pose estimation for shelves composed of any type of connecting rods, while maintaining sufficient accuracy and robustness.

[0070] Most existing shelves are assembled by splicing cylindrical connecting rods. Different combinations of cylindrical connecting rods form different shelves, and there is no specific rule. Coupled with the complex factory environment and the interference of other uncontrollable factors such as other shelves, equipment, pedestrians, and light, a deep learning method is used to detect and segment the shelves to be recognized. Since the basic unit of shelf combination is the cylindrical connecting rod, considering that the cylindrical connecting rod can be used as the basic feature of the shelf for segmentation, different combinations of cylindrical connecting rods will be equivalent to different combinations of the basic feature units of the shelf, and the result after combination is still the shelf. Thus, it well solves the challenges brought by the complex structure, many interference factors, and unknown types to be recognized during the shelf pose recognition process.

[0071] Usually, inside the RGBD sensor, the RGB image and depth information collected by it are registered and aligned, and the aligned RGB image and depth image are output. The data collected by the RGBD sensor described later all refer to the aligned RGB image and depth image output by the RGBD sensor. Assume the transformation relationship between the RGBD camera coordinate system and the body coordinate system of the forklift, that is, the external parameters of the RGBD camera have been calibrated.

[0072] Embodiment 1

[0073] Please refer to Figure 1 , a method for identifying the pose of a shelf provided by an embodiment of the present invention, includes the following steps S101 - S105:

[0074] Step S101: Collect the image information of the shelf to be recognized, and detect the overall side of the shelf to be recognized; the overall side of the shelf to be recognized includes horizontal connecting rods and vertical connecting rods.

[0075] It should be noted that, in this embodiment, exemplarily, the RGBD sensing camera is used to collect the image information of the shelf, and the collected image information is the RGB image and the depth image; further, in this embodiment, the shelf is used to place or hold goods, including but not limited to trolleys, cages, etc. that can be used for placing goods. It can be understood that the shelf pose recognition in this embodiment can be used for most shelves used for placing or holding goods.

[0076] As a preferred solution of this embodiment, the overall side surface of the to-be-recognized shelf detected specifically is as follows:

[0077] According to the deep learning recognition algorithm, judge and detect the to-be-recognized shelf in the image information, thereby recognizing the horizontal connecting rod and the vertical connecting rod of the to-be-recognized shelf, and thus obtaining the overall side surface of the to-be-recognized shelf.

[0078] It can be understood that for the method of shelf recognition, first, the deep learning algorithm is used to perform shelf recognition on the entire image, and then according to the results of the recognition bounding box (bbox), including the aspect ratio and the center position of the bbox, combined with the area that should appear in the image. The setting method of this area is that there are storage locations for the shelf in the actual environment, and the projection of the storage location on the RGB image will result in an aspect ratio and a center position on the RGB. Combining this information to judge the result of the bbox to lock the to-be-recognized shelf.

[0079] It should be noted that the aspect ratio and the center point position of the shelf can be set by pre-setting. Through the aspect ratio and the center point position of the shelf that the forklift needs to fork, and the obtained image information, the shelf in the information image can be roughly obtained. For example: If there is a shelf in an image, and there is an incomplete shelf displayed in the edge information of the image, then directly filter out the detection frame of the incomplete shelf in the edge information, thereby excluding the interference of other shelves to the to-be-detected and recognized shelf, and judging whether there is a to-be-detected shelf in the current image. At the same time, through pre-set deep learning recognition algorithms, including but not limited to Yolo series algorithms, SSD series algorithms, or Fast R-CNN series, etc., to detect the overall side cylindrical connecting rods of the to-be-recognized shelf. Exemplarily, in this embodiment, it includes 3 horizontal connecting rods and 2 vertical connecting rods, so that not only can it be ensured that the detected cylindrical connecting rods completely belong to the to-be-recognized shelf, but also even combinations of different numbers and forms of connecting rods can be completely detected, and the correspondence between the detection frame and the horizontal and vertical connecting rods is judged through the relative position and the aspect ratio of the detection frame.

[0080] Further, when segmenting the shelf side surface, other semantic segmentation deep learning recognition algorithms such as Unet, FCN, deeplabv3+ can also be used but are not limited to them.

[0081] Step S102: Perform point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the to-be-recognized shelf to obtain the connecting rod point cloud of the to-be-recognized shelf.

[0082] As a preferred solution of this embodiment, the performing point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the to-be-recognized shelf to obtain the connecting rod point cloud of the to-be-recognized shelf specifically is as follows:

[0083] Perform point cloud segmentation on the horizontal and vertical connecting rods on the overall side of the shelf to be recognized, obtaining the point cloud information corresponding to each horizontal connecting rod and the point cloud information corresponding to each vertical connecting rod; respectively combine the point cloud information corresponding to each horizontal connecting rod and the point cloud information corresponding to each vertical connecting rod to obtain the connecting rod point cloud on the overall side of the shelf to be recognized.

[0084] It should be noted that in this embodiment, based on the detection box results of the cylindrical connecting rods on the side of the shelf in the RGB image, after restoring the depth map synchronously transmitted by the RGBD sensor into a point cloud, point cloud segmentation is performed to obtain the point cloud of each cylindrical connecting rod, and then these point clouds are combined to obtain the point cloud on the side of the shelf that only contains cylindrical connecting rods, thus removing the interference of the goods loaded inside the shelf on the point cloud segmentation of the side of the shelf.

[0085] Step S103: Calculate the real ground or establish a virtual ground, and according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correct the connecting rod point cloud and perform plane fitting to obtain the pose information of the shelf to be recognized.

[0086] In this embodiment, since the side of the shelf is perpendicular to the ground, this prior information can be used to estimate the pose of the shelf better and more accurately.

[0087] As a preferred solution of this embodiment, the calculation of the real ground or the establishment of a virtual ground is specifically:

[0088] Collect ground data and identify the real ground;

[0089] Select several points in the forklift coordinate system, and according to the external parameters of the preset camera, transform the selected points in the forklift coordinate system into the camera coordinate system, and perform plane fitting on the points transformed into the camera coordinate system to obtain a virtual ground; verify the credibility of the real ground through the virtual ground; if the credibility passes, use the real ground for the pose recognition of the shelf; if the credibility fails, use the virtual ground for the pose recognition of the shelf.

[0090] It should be noted that the real ground is identified through the data collected by RGBD, and at the same time, a virtual ground is established according to the external parameters of the camera, and the credibility of the calculated real ground is verified by the virtual ground. If the credibility passes the verification, the subsequent calculations use the real ground, otherwise the virtual ground is used instead. The point cloud equation of the ground Ax + By + Cz + D = 0 can be directly obtained by the camera acquiring the ground point cloud data. However, since the RGBD sensing camera mainly acquires images of the shelf, it is easy to cause problems such as insufficient acquisition of ground point cloud data or fitting failure. Therefore, by establishing a virtual ground in the RGBD camera coordinate system, the problem of insufficient acquisition of ground point cloud data can be avoided.

[0091] In this embodiment, the virtual ground is obtained by selecting multiple fixed points in the forklift coordinate system, transforming the ground points in the forklift coordinate system to the camera coordinate system according to the calibrated external parameters of the RGBD camera, and performing plane fitting.

[0092] As a preferred solution of this embodiment, the method of correcting the link point cloud and performing plane fitting according to the vertical relationship between the shelf to be recognized and the virtual ground to obtain the pose information of the shelf to be recognized is specifically as follows:

[0093] Perform a first plane fitting on the link point cloud to obtain the first normal vector of the first fitting plane; correct the link point cloud on the overall side surface of the shelf to be recognized after the first plane fitting according to the vertical relationship between the shelf to be recognized and the virtual ground; perform a second plane fitting on the corrected link point cloud to obtain the second normal vector of the second fitting plane, and then use the second normal vector as the pose information of the shelf to be recognized.

[0094] It can be understood that by performing plane fitting on the link point cloud of the side surface of the shelf composed of shelf cylindrical links, the RANSAC algorithm can be used during the fitting process to reduce the interference of noise on the fitting result. In this embodiment, based on the prior information that the ground is perpendicular to the side surface of the shelf, the normal vector of the fitting plane of the side surface of the shelf is corrected, and at the same time, the points outside a distance of 3 from the fitting plane are removed to achieve a refined adjustment of the point cloud on the side surface of the shelf. Through the above operations, a relatively perfect point cloud of the side surface of the shelf can be obtained, and by performing plane fitting on it again, the normal vector of the plane is the pose information of the shelf in the RGBD camera coordinate system.

[0095] Step S104: Perform a straight line fitting on the vertical links in the link point cloud, and calculate the intersection points of the vertical links and the real ground or the virtual ground, so as to obtain the position information of the shelf to be recognized.

[0096] As a preferred solution of this embodiment, the method of performing a straight line fitting on the vertical links in the link point cloud, calculating the intersection points of the vertical links and the real ground or the virtual ground, and thus obtaining the position information of the shelf to be recognized is specifically as follows:

[0097] Perform a straight line fitting on the vertical links in the link point cloud; calculate the intersection point coordinates of the vertical links after straight line fitting and the real ground or the virtual ground; use the midpoint of the intersection point coordinates as the position information of the shelf to be recognized.

[0098] In this embodiment, when calculating the position of the shelf, two vertical links on the left and right of the point cloud on the side surface of the shelf perpendicular to the ground are considered. Among them, perform a straight line fitting on these two links, respectively and The coordinates of the intersection points of the two connecting rods with the plane of the ground (point cloud equation) Ax + By + Cz + D = 0 are (m1k1 + x 01 , n1k1 + y 01 , p1k1 + z 01 ) and (m2k2 + x 02 , n2k2 + y 02 , p2k2 + z 02 ); where, and

[0099] As another preferred solution of this embodiment, when calculating the position of the shelf, two wheels of the shelf can also be considered to replace the vertical connecting rods on the left and right sides. Calculate the centers of the point clouds of the wheels respectively, and then calculate the midpoint of the centers of the two wheels. However, there is a large calculation error in this method because the wheels at the bottom of the shelf are universal wheels and the swinging positions cannot be determined. As shown in Figure 2 , the two extreme cases of the universal wheel swinging on the basis of the alignment of the side of the shelf. The dotted line between P1 and P2 represents the midline position of the left and right vertical rods of the shelf, which is the ideal representation of the shelf position. The left and right dotted lines of P1 and P2 are the positions of the shelf calculated based on the midpoint of the wheel centers respectively. Therefore, the error is between the two dotted lines P1 and P2.

[0100] Step S105: Obtain the pose information of the shelf to be recognized according to the pose information and position information of the shelf to be recognized, so as to complete the pose recognition of the shelf to be recognized. The midpoint of the intersection coordinates is the position of the shelf in the RGBD camera coordinate system, that is:

[0101] As a preferred solution of this embodiment, after obtaining the pose information of the shelf to be recognized according to the pose information and position information of the shelf to be recognized, so as to complete the pose recognition of the shelf to be recognized, it further includes:

[0102] Transform the pose information obtained by the shelf to be recognized in the camera coordinate system into the forklift coordinate system, so that the forklift can perform the handling after the pose recognition of the shelf to be recognized.

[0103] It should be noted that based on the transformation relationship between the RGBD camera coordinate system and the forklift body coordinate system, transforming the pose recognized by the shelf in the camera coordinate system into the forklift coordinate system can be used for the subsequent work process of the forklift.

[0104] In this embodiment, please refer to Figure 3, which is a flow chart of the trolley pose recognition solution in another embodiment of the present invention. After initializing the surrounding environment information through the RGBD sensing camera, the external parameters of the camera coordinate system based on the forklift coordinate system are obtained. Then, the RGBD sensor collects RGB images and aligned depth data. Based on the deep learning recognition algorithm, the sides and connecting rods of the trolley are detected in the RGB image, and it is determined whether there is a trolley in the current storage location based on the detection results.

[0105] Using the external parameters of the camera coordinate system based on the forklift coordinate system, any four points are selected to fit the virtual plane of the ground in the camera coordinate system, that is, the virtual ground, and its corresponding point cloud plane; or directly calculate the point cloud plane of the real ground, and obtain the ground equation by fitting the virtual ground point cloud plane or the point cloud plane of the real ground.

[0106] Perform rough point cloud segmentation on the trolley, segment the point cloud of the trolley connecting rod on the depth image, and based on the segmented point cloud of the trolley connecting rod, fit the equation of the trolley side. At the same time, correct the equation of the trolley side based on the ground equation, refine the point cloud of the trolley side, remove the points outside 3σ of the side equation, and fit the trolley side again to obtain the pose of the trolley.

[0107] Fit the point cloud of the trolley connecting rod segmented on the depth image to obtain the straight line equations of the left and right connecting rods, and obtain the position of the trolley by calculating the intersection points of the straight line equations and the ground equation.

[0108] According to the obtained pose and position of the trolley, obtain the pose of the trolley in the camera coordinate system. Based on the external parameters of the camera coordinate system based on the forklift coordinate system, further obtain the pose of the trolley in the forklift coordinate system, and determine whether the result is reasonable, so as to complete the recognition of the trolley pose.

[0109] Implementing the above embodiments has the following effects:

[0110] The technical solution of the present invention detects the horizontal connecting rods and vertical connecting rods on the overall side of the shelf to be recognized by obtaining the image information of the shelf to be recognized, and then performs point cloud segmentation on the horizontal connecting rods and vertical connecting rods, avoiding problems such as complex shelf structures and other unknown problems that prevent the shelf from being positioned and its pose from being recognized, improving the recognition versatility and robustness. By establishing a virtual ground, and then through correction, plane fitting, and straight line fitting, the corresponding pose information and position information are obtained respectively, further restricting the calculation process of the pose of the shelf to be recognized and optimizing the solution result of the pose information, thus ensuring the accuracy, robustness, and versatility of the recognition of the shelf pose.

[0111] Embodiment 2

[0112] Please refer to Figure 4, which is a shelf pose recognition device provided by another embodiment of the present invention, including: an acquisition and detection module 201, a point cloud segmentation module 202, an attitude information module 203, a position information module 204, and a pose information module 205.

[0113] The acquisition and detection module 201 is used to acquire the image information of the shelf to be recognized and detect the overall side surface of the shelf to be recognized; the overall side surface of the shelf to be recognized includes a horizontal connecting rod and a vertical connecting rod.

[0114] The point cloud segmentation module 202 is used to perform point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the shelf to be recognized to obtain the connecting rod point cloud of the shelf to be recognized.

[0115] The attitude information module 203 is used to calculate the real ground or establish a virtual ground, and perform correction and plane fitting on the connecting rod point cloud according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground to obtain the attitude information of the shelf to be recognized.

[0116] The position information module 204 is used to perform linear fitting on the vertical connecting rod in the connecting rod point cloud and calculate the intersection point of the vertical connecting rod and the real ground or the virtual ground, so as to obtain the position information of the shelf to be recognized.

[0117] The pose information module 205 is used to obtain the pose information of the shelf to be recognized according to the attitude information and position information of the shelf to be recognized, so as to complete the pose recognition of the shelf to be recognized.

[0118] As a preferred solution of this embodiment, the detection of the overall side surface of the shelf to be recognized is specifically:

[0119] As a preferred solution, the detection of the overall side surface of the shelf to be recognized is specifically:

[0120] According to the deep learning recognition algorithm, judge and detect the shelf to be recognized in the image information, so as to recognize the horizontal connecting rod and the vertical connecting rod of the shelf to be recognized, and thus obtain the overall side surface of the shelf to be recognized.

[0121] As a preferred solution, the point cloud segmentation of the horizontal connecting rod and the vertical connecting rod of the overall side surface of the shelf to be recognized to obtain the connecting rod point cloud of the shelf to be recognized is specifically:

[0122] Perform point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the shelf to be recognized to obtain the point cloud information corresponding to each horizontal connecting rod and the point cloud information corresponding to each vertical connecting rod;

[0123] Combine the point cloud information corresponding to each horizontal link and the point cloud information corresponding to each vertical link respectively to obtain the link point cloud of the overall side of the shelf to be recognized.

[0124] As a preferred solution, the calculation of the real ground or the establishment of the virtual ground is specifically as follows:

[0125] Collect ground data and identify the real ground;

[0126] Select a number of points in the forklift coordinate system, and according to the external parameters of the preset camera, transform the selected points in the forklift coordinate system into the camera coordinate system, and perform plane fitting on the points transformed into the camera coordinate system to obtain the virtual ground;

[0127] Verify the credibility of the real ground through the virtual ground;

[0128] If the credibility passes, use the real ground for the pose recognition of the shelf;

[0129] If the credibility does not pass, use the virtual ground for the pose recognition of the shelf.

[0130] As a preferred solution, according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correct the link point cloud and perform plane fitting to obtain the pose information of the shelf to be recognized, specifically as follows:

[0131] Perform the first plane fitting on the link point cloud to obtain the first normal vector of the first fitting plane;

[0132] According to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correct the link point cloud of the overall side of the shelf after the first plane fitting;

[0133] Perform the second plane fitting on the corrected link point cloud to obtain the second normal vector of the second fitting plane, and then use the second normal vector as the pose information of the shelf to be recognized.

[0134] As a preferred solution, perform linear fitting on the vertical links in the link point cloud and calculate the intersection points of the vertical links and the virtual ground, so as to obtain the position information of the shelf to be recognized, specifically as follows:

[0135] Perform linear fitting on the vertical links in the link point cloud;

[0136] Calculate the intersection coordinates of the vertical links after linear fitting and the real ground or the virtual ground;

[0137] Take the midpoint of the intersection coordinates as the position information of the shelf to be recognized.

[0138] As a preferred solution, after obtaining the pose information of the shelf to be recognized based on the pose information and position information of the shelf to be recognized, thereby completing the pose recognition of the shelf to be recognized, it further includes:

[0139] Transform the pose information obtained for the shelf to be recognized in the camera coordinate system into the forklift coordinate system, so that the forklift can perform handling after the pose recognition of the shelf to be recognized.

[0140] Implementing the above embodiments has the following effects:

[0141] The technical solution of the present invention obtains the image information of the shelf to be recognized to detect the horizontal connecting rod and vertical connecting rod on the overall side of the shelf to be recognized, and then performs point cloud segmentation on the horizontal connecting rod and vertical connecting rod, avoiding problems such as complex shelf structures and other unknown issues that prevent the shelf from being positioned and its pose from being recognized, improving the recognition versatility and robustness. By establishing a virtual ground, and then through correction, plane fitting, and line fitting, the corresponding pose information and position information are obtained respectively, further constraining the pose calculation process of the shelf to be recognized and optimizing the solution result of the pose information, thereby ensuring the accuracy, robustness, and versatility of the shelf pose recognition.

[0142] Embodiment III

[0143] Correspondingly, the present invention further provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the shelf pose recognition method described in any one of the above embodiments.

[0144] The terminal device of this embodiment includes: a processor, a memory, and a computer program and computer instructions stored in the memory and executable on the processor. When the processor executes the computer program, it implements each step in Embodiment 1 above, such as Figure 1 Steps S101 to S105 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above device embodiment, such as the pose information module 203.

[0145] Exemplarily, the computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device. For example, the attitude information module 203 is configured to calculate the real ground or establish a virtual ground, and correct and perform plane fitting on the link point cloud according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, so as to obtain the attitude information of the shelf to be recognized.

[0146] The terminal device may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of the terminal device, and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine some components, or different components. For example, the terminal device may further include an input / output device, a network access device, a bus, etc.

[0147] The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the terminal device, and connects various parts of the entire terminal device through various interfaces and lines.

[0148] The memory can be used to store the computer programs and / or modules. By running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, the processor realizes various functions of the terminal device. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

[0149] Among them, if the modules / units integrated in the terminal device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be realized. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0150] Embodiment 4

[0151] Correspondingly, the present invention also provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program. Among them, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the shelf pose recognition method described in any one of the above embodiments.

[0152] The specific embodiments described above further elaborate on the objective, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only for the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. In particular, it is pointed out that for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for identifying the pose of a shelf, characterized in that, Including: Collecting image information of the shelf to be recognized and detecting the overall side surface of the shelf to be recognized; the overall side surface of the shelf to be recognized includes a horizontal connecting rod and a vertical connecting rod; Performing point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the shelf to be recognized to obtain the connecting rod point cloud of the shelf to be recognized; Calculating the real ground or establishing a virtual ground, and according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correcting and performing plane fitting on the connecting rod point cloud to obtain the pose information of the shelf to be recognized; Performing linear fitting on the vertical connecting rods in the connecting rod point cloud and calculating the intersection points of the vertical connecting rods and the real ground or the virtual ground, so as to obtain the position information of the shelf to be recognized; According to the pose information and the position information of the shelf to be recognized, obtaining the pose information of the shelf to be recognized, thereby completing the pose recognition of the shelf to be recognized.

2. The shelf pose recognition method according to claim 1, characterized in that, The detecting the overall side surface of the shelf to be recognized specifically is: According to the deep learning recognition algorithm, judging and detecting the shelf to be recognized in the image information, thereby recognizing the horizontal connecting rod and the vertical connecting rod of the shelf to be recognized, and thus obtaining the overall side surface of the shelf to be recognized.

3. The method for identifying the pose of a shelf according to claim 1, wherein Performing point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the shelf to be recognized to obtain the connecting rod point cloud of the shelf to be recognized, specifically is: Performing point cloud segmentation on the horizontal connecting rod and the vertical connecting rod of the overall side surface of the shelf to be recognized to obtain the point cloud information corresponding to each horizontal connecting rod and the point cloud information corresponding to each vertical connecting rod; Combining the point cloud information corresponding to each horizontal connecting rod and the point cloud information corresponding to each vertical connecting rod respectively to obtain the connecting rod point cloud of the overall side surface of the shelf to be recognized.

4. The shelf pose recognition method according to claim 1, characterized in that, The calculating the real ground or establishing a virtual ground specifically is: Collecting ground data and recognizing the real ground; Selecting a number of points in the forklift coordinate system, and according to the external parameters of the preset camera, transforming the selected points in the forklift coordinate system into the camera coordinate system, and performing plane fitting on the points transformed into the camera coordinate system to obtain a virtual ground; Verifying the credibility of the real ground through the virtual ground; If the credibility passes, the real ground is used for the pose recognition of the shelf; If the credibility fails, the virtual ground is used for the pose recognition of the shelf.

5. The shelf pose recognition method according to claim 4, characterized in that, The correcting and performing plane fitting on the connecting rod point cloud according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground to obtain the pose information of the shelf to be recognized, specifically is: Performing a first plane fitting on the connecting rod point cloud to obtain a first normal vector of the first fitting plane; According to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground, correcting the connecting rod point cloud of the overall side surface of the shelf to be recognized after the first plane fitting; Performing a second plane fitting on the corrected connecting rod point cloud to obtain a second normal vector of the second fitting plane, and then taking the second normal vector as the pose information of the shelf to be recognized.

6. The shelf pose recognition method according to claim 4, wherein The performing linear fitting on the vertical connecting rods in the connecting rod point cloud and calculating the intersection points of the vertical connecting rods and the virtual ground to obtain the position information of the shelf to be recognized, specifically is: Perform linear fitting on the vertical connecting rods in the connecting rod point cloud of the shelf to be recognized; Calculate the intersection coordinates of the vertical connecting rods after linear fitting and the real ground or the virtual ground; Use the midpoint of the intersection coordinates as the position information of the shelf to be recognized.

7. The method for identifying the pose of a shelf according to claim 4, wherein After obtaining the pose information of the shelf to be recognized based on the pose information and position information of the shelf to be recognized, thereby completing the pose recognition of the shelf to be recognized, the method further includes: Transform the pose information obtained by the shelf to be recognized in the camera coordinate system into the forklift coordinate system, so that the forklift can carry out handling after pose recognition of the shelf to be recognized.

8. A shelf pose recognition device, characterized in that, It includes: An acquisition and detection module, a point cloud segmentation module, a pose information module, a position information module, and a pose information module; The acquisition and detection module is used to acquire the image information of the shelf to be recognized and detect the overall side surface of the shelf to be recognized; the overall side surface of the shelf to be recognized includes a horizontal connecting rod and a vertical connecting rod; The point cloud segmentation module is used to perform point cloud segmentation on the horizontal connecting rod and the vertical connecting rod on the overall side surface of the shelf to be recognized to obtain the connecting rod point cloud of the shelf to be recognized; The pose information module is used to calculate the real ground or establish a virtual ground, and perform correction and plane fitting on the connecting rod point cloud according to the vertical relationship between the shelf to be recognized and the real ground or the virtual ground to obtain the pose information of the shelf to be recognized; The position information module is used to perform linear fitting on the vertical connecting rods in the connecting rod point cloud and calculate the intersection points of the vertical connecting rods and the real ground or the virtual ground, so as to obtain the position information of the shelf to be recognized; The pose information module is used to obtain the pose information of the shelf to be recognized based on the pose information and position information of the shelf to be recognized, thereby completing the pose recognition of the shelf to be recognized.

9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the shelf pose recognition method according to any one of claims 1 - 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the shelf pose recognition method according to any one of claims 1 - 7.

Images