Methods, devices, electronic equipment, and palletizing robotic arms for object positioning

Abstract

This application discloses a method, apparatus, electronic device, and palletizing robot arm for object localization. The method includes scanning a target object at preset scanning points and different scanning angles to obtain multiple sets of point cloud data; selecting target point cloud data that meets preset target surface conditions from the multiple sets of point cloud data; fitting each set of target point cloud data into a line segment; determining the target height of the target object as the difference between the vertical distance from the scanning point to the line segment and the height of the scanning point; determining the two-dimensional pose information of the target object based on the length of the line segment, the scanning angle, and the perpendicular from the scanning point to the line segment; and obtaining the three-dimensional pose information of the target object based on the target height and the two-dimensional pose information. According to the embodiments of this application, the target height and two-dimensional pose information of the target object can be determined by multiple scans of the target object, thereby obtaining the three-dimensional position information of the target object.

Description

Technical Field

[0001] This application belongs to the technical field of warehouse automation, and particularly relates to a method, apparatus, electronic device and palletizing robot arm for object positioning. Background Technology

[0002] In the palletizing methods of relevant robotic arms, the target object needs to be positioned when grasping or placing it. Therefore, it is often necessary to add mechanical limiters to determine the grasping point and the placement point, or to use some high-cost visual recognition methods to locate the three-dimensional pose of the target object.

[0003] First, when using mechanical limiters to determine the gripping and stacking points, the mechanical limiters will push the target object to be gripped to the target position, so that the robotic arm can grip the target object at the same target position each time it grips the target object. Alternatively, when stacking items, the robotic arm will stack the gripped items in the same fixed position, and the mechanical limiters will push the items stacked by the robotic arm to the expected target object position.

[0004] Therefore, mechanical limiters will exert hard pressure on the target object during the pushing process, causing irreversible damage to the target object. Without the assistance of mechanical limiters, the robotic arm cannot accurately determine the three-dimensional pose information of the target object, so it cannot accurately grasp the target object at the predetermined target position, and it is also difficult to accurately place the grasped item on the target object in the expected stacking position.

[0005] Furthermore, mechanical limiting methods are often inflexible. If different target objects are changed, the mechanical structure of the entire production line using the robotic arm will need to be restructured, affecting changeover speed and production efficiency. At the same time, it is also necessary to ensure that the target objects are all of the same size, otherwise the robotic arm will be unable to position itself when grasping.

[0006] Secondly, existing visual recognition methods are often expensive, with industrial 3D cameras costing tens of thousands of yuan, and ordinary color depth cameras facing the problem of inaccuracy. Thirdly, visual recognition methods require the shooting distance to be proportional to the size of the target object. In some scenarios where it is not convenient to suspend objects at high places or where the work space is limited, the shooting distance will greatly limit the ability of visual recognition to locate the target object.

[0007] On the other hand, during the palletizing process of the robotic arm, the three-dimensional pose information of the target object is also obtained through visual recognition.

[0008] However, visual recognition relies on images captured by a camera, requiring a predetermined ratio between the shooting distance and the size of the target object in order to obtain relatively accurate visual recognition results. Therefore, in some scenarios where it is not convenient to suspend the camera at a high place, or in scenarios with limited workspace, the shooting distance will greatly limit the camera's recognition ability.

[0009] Therefore, without the assistance of devices such as mechanical limiters or visual recognition methods, the robotic arm has difficulty accurately determining the pose of the target object. Summary of the Invention

[0010] This application provides a method, apparatus, electronic device, and palletizing robot arm for object positioning, which can accurately obtain the three-dimensional pose information of the target object through laser scanning.

[0011] In a first aspect, embodiments of this application provide a method for object positioning, applied to a palletizing robotic arm, the palletizing robotic arm including a LiDAR; the method includes:

[0012] The target object is scanned at preset scanning points and at different scanning angles using a lidar to obtain multiple sets of target point cloud data;

[0013] Determine the target height of the scan point relative to each group of target point cloud data;

[0014] Based on the coverage area of ​​each set of target point cloud data and the vertical projection position of the scanning point and each set of target point cloud data, the two-dimensional pose information of the target object is determined.

[0015] The three-dimensional pose information of the target object is obtained based on the target height and two-dimensional pose information.

[0016] Furthermore, the target object is scanned at preset scanning points and at different scanning angles using a lidar system, resulting in multiple sets of target point cloud data, including:

[0017] The scanning point is controlled to rotate around the rotation axis according to the scanning angle, and a laser surface is emitted towards the target object to scan the target object at different scanning angles and obtain multiple sets of point cloud data; the rotation axis is the perpendicular line between the scanning point and the target surface of the target object.

[0018] Target point cloud data that meets the preset target surface conditions are selected from multiple sets of point cloud data.

[0019] Further, the target height of the scan point relative to each set of target point cloud data is determined, including:

[0020] Each set of target point cloud data is fitted into a line segment;

[0021] The difference between the vertical distance from the scan point to the line segment and the height of the scan point is determined as the target height of the target object.

[0022] Furthermore, by scanning the target object from different scanning angles, multiple sets of point cloud data are obtained, including:

[0023] In the polar coordinate system, the target object is scanned at different scanning angles at the scanning point to obtain the polar coordinates of multiple point clouds; the polar coordinates of the point cloud include the rotation angle of the scanning point and the distance from the scanning point to the point cloud.

[0024] The polar coordinates of each point cloud are converted to point cloud data in the Cartesian coordinate system to obtain multiple sets of point cloud data.

[0025] Furthermore, target point cloud data that meets the preset target surface conditions are selected from multiple sets of point cloud data, including:

[0026] For each set of point cloud data, compare the distance between each point cloud data and the scanned point;

[0027] Determine the shortest distance among all distances;

[0028] The target point cloud data is determined based on the distance and the shortest distance, wherein the target point cloud data is the point cloud data in which the difference between the distance and the shortest distance is less than a preset filtering value.

[0029] Furthermore, target point cloud data that meets the preset target surface conditions are selected from multiple sets of point cloud data, including:

[0030] Determine the ratio of the difference in the vertical coordinate to the difference in the horizontal coordinate of the first and second point cloud data of two adjacent point clouds in each of the multiple sets of point cloud data;

[0031] If the ratio is equal to the tangent of the rotation angle corresponding to the second point cloud data, then the two first point cloud data in each group of point cloud data are determined to be the data of the breakpoint point cloud.

[0032] The target point cloud data is obtained by selecting the target point cloud that is arranged between the two breakpoint point clouds from each group of point clouds.

[0033] In this context, the coverage area of ​​each group of target point cloud data is the length of the corresponding fitted line segment; the vertical projection position is the position of the foot of the perpendicular from the scan point to the line segment.

[0034] Furthermore, based on the coverage area of ​​each set of target point cloud data and the vertical projection position of the scan point with each set of target point cloud data, the two-dimensional pose information of the target object is determined, including:

[0035] The position of the endpoint of each line segment is determined by each line segment, the position of the perpendicular foot corresponding to each line segment, the scanning angle corresponding to each line segment, and the preset trigonometric functions. The trigonometric functions represent the calculation relationship between the position of the line segment, the position of the perpendicular foot, the scanning angle, and the position of the endpoint.

[0036] Fit the endpoint coordinates of each line segment to the target shape of the target surface;

[0037] Determine the center position of the target shape;

[0038] The angle between the center position, any side of the target shape, and the target coordinate axis in the Cartesian coordinate system is determined as the two-dimensional pose information of the target object.

[0039] Secondly, embodiments of this application provide an object positioning device, the device comprising:

[0040] The scanning module is used to scan the target object at preset scanning points and different scanning angles using the lidar to obtain multiple sets of target point cloud data;

[0041] The height determination module is used to determine the target height of the scan point relative to each group of target point cloud data;

[0042] The two-dimensional pose determination module is used to determine the two-dimensional pose information of the target object based on the coverage area of ​​each set of target point cloud data and the vertical projection position of the scanning point and each set of target point cloud data.

[0043] A three-dimensional pose determination module is used to obtain the three-dimensional pose information of the target object based on the target height and the two-dimensional pose information.

[0044] Thirdly, embodiments of this application provide an electronic device, the device comprising:

[0045] Processor and memory storing computer program instructions;

[0046] The method for locating objects as described above when the processor executes computer program instructions.

[0047] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the object location method as described above is implemented.

[0048] Fifthly, embodiments of this application provide a method for object location as described above, whereby instructions in a computer program product are executed by a processor of an electronic device, causing the electronic device to perform any of the preceding methods.

[0049] Sixthly, embodiments of this application provide a palletizing robotic arm, which includes an electronic device, wherein the electronic device, when in operation, is used to perform the object positioning method as described in any of the preceding embodiments.

[0050] The object positioning method, apparatus, electronic device, and palletizing robot arm of this application embodiment scan the target object at preset scanning points and different scanning angles using a lidar on the palletizing robot arm. This allows for the acquisition of multiple sets of target point cloud data without using visual recognition or image acquisition. The target height and two-dimensional pose information are then determined based on each set of target point cloud data. Furthermore, the three-dimensional pose information of the target object is obtained by fusing the data without relying on visual algorithms or image algorithms. It is evident that this method achieves accurate identification of the three-dimensional position of the target object without relying on visual recognition or image acquisition methods. This avoids the dependence of traditional vision systems on shooting distance, ambient lighting, and equipment cost. At the same time, since the target point cloud data is acquired through lidar, there is no need to set up mechanical limiting structures, thus avoiding physical damage caused by pushing the target object. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating a method for object location provided in an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of point cloud data fitting provided in an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of a robotic arm palletizing scenario provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of a process for a robotic arm to identify a pallet, provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a process for a robotic arm to identify a cargo box, provided in an embodiment of this application;

[0057] Figure 6 This is another schematic diagram of point cloud data fitting provided in an embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the structure of an object positioning device provided in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0061] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0062] To address the problems in the prior art, embodiments of this application provide a method, apparatus, electronic device, and palletizing robotic arm for object positioning.

[0063] The object positioning method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0064] Figure 1 A flowchart illustrating an object positioning method provided in another embodiment of this application is shown.

[0065] refer to Figure 1 One embodiment of the object positioning method of this application is executed by a robotic arm equipped with a lidar at its end, and specifically includes the following steps S101-S104.

[0066] S101. Using a lidar, the target object is scanned at preset scanning points at different scanning angles to obtain multiple sets of target point cloud data.

[0067] The lidar has a laser emitter that emits laser lines. In this embodiment, when implementing this method, the lidar and the laser emitter can be regarded as the same point in space, and the preset scanning point can be the position where the lidar or the laser emitter at the end of the robotic arm emits laser lines.

[0068] In this embodiment, the robotic arm can scan the target object by controlling a lidar or laser emitter to project a laser surface or laser line onto the target object at the scanning point, and after scanning the target object, multiple sets of point cloud data can be obtained.

[0069] Each set of point cloud data specifically includes the coordinate data of multiple point clouds, and each point cloud can be regarded as a location point in space.

[0070] Specifically, in the process of obtaining multiple sets of point cloud data, the robotic arm can control the lidar and laser emitter to scan the target object multiple times, and control the lidar to rotate at different scanning angles during each scan, thereby obtaining a set of point cloud data corresponding to that scan.

[0071] In each scan, multiple laser lines can be projected onto the target object and combined into a laser surface. Alternatively, a single laser surface can be projected onto the target object during each scan, so that the laser surface rotates according to the corresponding scanning angle each time the lidar rotates.

[0072] Based on this, each set of point cloud data can include the coordinates of each point cloud in a two-dimensional plane coordinate system, which is the two-dimensional plane coordinate system of the plane where the laser surface projected in this scan is located.

[0073] Based on the identified multiple sets of point cloud data, the target point cloud data can be selected from these sets.

[0074] The target point cloud data can be the data of the target point cloud that falls on the target object from all the point clouds obtained, specifically the data of the point cloud that falls on the preset target surface among multiple faces of the target object.

[0075] For each set of point cloud data, during the process of obtaining target point cloud data, the target point cloud that falls on the target object or the target surface of the target object can be selected from multiple point cloud data using preset target surface conditions, and the corresponding target point cloud data can be determined. By taking the target point cloud data in each set of point cloud data as a set of target point cloud data, the target point cloud data corresponding to each set of point cloud data can be obtained.

[0076] Based on this, by scanning the target object multiple times from different angles, point cloud data obtained from different scanning angles can be used for identification during the target object identification process, thereby improving the accuracy of the identification results. During the target object identification process, by using the target surface condition to filter multiple sets of point cloud data, point cloud data that does not fall on the target object can be removed, so that the filtered target point cloud data are all point clouds that fall on the target object, thus making the identification more accurate when using the target point cloud that falls on the target object.

[0077] S102. Determine the target height of the scanning point relative to each group of target point cloud data.

[0078] Based on the multiple sets of target point cloud data determined in step 101 above, for each set of target point cloud data, due to the jitter of the scanning points and other reasons, the target point cloud obtained in each scan will have an error with the expected position, which will cause the target point cloud to not be accurately aligned along the same straight line. Therefore, it is difficult to determine the intersection line between the laser surface projected in this scan and the target surface. Therefore, the target point cloud data can be fitted into a line segment by straight line fitting and used as the intersection line between the laser surface projected in this scan and the target surface.

[0079] When fitting a line segment, based on the current arrangement of each target point cloud, the goal can be to make the distances from each point cloud to the line segment equal or approximately equal, or to fit the line segment according to the distances from each point cloud to the line segment being less than a pre-set fitting threshold.

[0080] Since the line segment is fitted from various target point cloud data, and each target point cloud represents the point cloud data falling on the target surface, the two endpoints of the line segment are the first and last target point cloud data in a row of target point cloud data, or points near the first and last target point cloud data. Therefore, the two endpoints of the line segment are located at the two edges of the target surface.

[0081] Based on this, if the line segment belongs to both the plane of the laser surface and the plane of the target surface, and its two endpoints are located at the two edges of the target surface, then the line segment fitted by the target point cloud data is the intersection line of the projected laser surface and the target surface.

[0082] Figure 2 A specific example of point cloud data fitting is shown.

[0083] exist Figure 2 In the example, during the scan, the scanning point 201 projects multiple laser lines 203 at different angles onto the target object 202, thereby forming a set of multiple point cloud data in the target object and the area around the target object.

[0084] The point cloud data includes multiple target point cloud data that are on the target object 202, and multiple non-target point cloud data 205 that are not on the target object.

[0085] Furthermore, by utilizing preset target surface conditions, target point cloud data located on target object 202 can be filtered out from this set of point cloud data.

[0086] Among them, the target point cloud data falling on the target object 202 form a row. Due to the jitter of the lidar or laser emitter, this row of target point cloud data is difficult to be accurately arranged into a straight line. Therefore, it is necessary to fit each target point cloud data into a line segment 204 that is equidistant or approximately equidistant from each target point cloud data.

[0087] like Figure 2 As shown, since the line segment 204 is obtained by fitting the point cloud data of each target, the two endpoints of the line segment 204 are located at the edge of the target object 202, so that the line segment can be regarded as the intersection line between the surface where the laser line 203 is located and the target surface of the target object 202.

[0088] Based on the fitted line segment, the height of the line segment in the two-dimensional plane coordinate system where the laser surface is located can be regarded as the height of the target surface, and the height of the line segment can be used as the target height of the target object.

[0089] When determining the target height, you can first determine the height of each line segment.

[0090] When determining the height of each line segment, the origin and horizontal axis of the two-dimensional plane coordinate system where the laser surface is located can be placed in the plane used to place the target object.

[0091] Therefore, the difference between the height of the scanning point in the two-dimensional plane coordinate system where the laser surface is located and the vertical distance from the scanning point to the line segment can be used as the height of the line segment.

[0092] Furthermore, after determining the height of each line segment, the determined height of any line segment can be used as the target height of the target object.

[0093] In some cases, since there may be some error in the determined heights of each line segment, the average value of the determined heights of each line segment can be used as the target height of the target object.

[0094] As can be seen, in the process of identifying target objects, by decomposing the three-dimensional pose information of the target object into two two-dimensional planes and identifying the two-dimensional pose information and the target height respectively, the difficulty and computational load of directly identifying the three-dimensional pose information are reduced. In particular, since each set of target point clouds is on the target object, by fitting each set of target point clouds into a line segment, the line segment can represent the intersection of the two decomposed two-dimensional planes. One of the two-dimensional planes contains both the line segment and the target height. Therefore, in the process of identifying the target height, based on the line segment representing the intersection line, the target height can be determined in the two-dimensional plane where the line segment and the target height are located by the difference between the vertical distance from the scanning point to the line segment and the height of the scanning point. This ensures that the identified target height belongs to one of the decomposed two-dimensional planes.

[0095] S103. Based on the coverage area of ​​each group of target point cloud data and the vertical projection position of the scanning point and each group of target point cloud data, determine the two-dimensional pose information of the target object.

[0096] Based on each line segment obtained from each scan, the length of the line segment can be determined by the coordinates of the two endpoints of each line segment in the two-dimensional plane coordinate system where the laser surface is located.

[0097] For each line segment obtained from a scan, the coordinates of the two endpoints of the line segment in the two-dimensional plane coordinate system of the target surface can be calculated using the line segment length, the scanning angle of the scan, and the foot of the perpendicular from the scanning point to the line segment during the scan, that is, the intersection of the perpendicular from the scanning point to the line segment and the line segment.

[0098] Furthermore, the two-dimensional pose information of the target object can be obtained by fitting the coordinates of the two endpoints of each line segment in the two-dimensional plane coordinate system where the target surface is located.

[0099] Specifically, the two-dimensional pose information of the target object can be the two-dimensional pose information of the target surface in its two-dimensional plane coordinate system.

[0100] S104. Obtain the three-dimensional pose information of the target object based on the target height and two-dimensional pose information.

[0101] Based on the obtained two-dimensional pose information of the target object and the target height of the target object, the two-dimensional pose information and the target height can be combined to obtain the three-dimensional pose information of the target object in three-dimensional space.

[0102] Specifically, the three-dimensional space can be a pre-set three-dimensional coordinate system in the robotic arm, such as a three-dimensional coordinate system with the lidar or laser transmitter as the origin.

[0103] Based on this, this embodiment can scan the target object at different scanning angles at preset scanning points using a lidar on the palletizing robot arm. This allows for the acquisition of multiple sets of target point cloud data without the use of visual recognition or image acquisition. The target height and two-dimensional pose information can then be determined based on each set of target point cloud data. Furthermore, the three-dimensional pose information of the target object can be obtained by fusing the data without relying on visual or image algorithms. As can be seen, this method achieves accurate identification of the three-dimensional position of the target object without relying on visual recognition or image acquisition methods. This avoids the dependence of traditional vision systems on shooting distance, ambient lighting, and equipment costs. At the same time, since the target point cloud data is acquired using lidar, there is no need to set up mechanical limiting structures, thus avoiding physical damage caused by pushing the target object.

[0104] Figure 3 This illustrates a palletizing scenario using a robotic arm, such as... Figure 3 As shown, in each palletizing operation, the robotic arm 301 grabs the box 302, which is the target object, and places the box 302 on the pallet 303, which is another target object.

[0105] The robotic arm 301 is equipped with a lidar 3011 at its end for locating the three-dimensional pose information of a target object. This lidar 3011 can perform 0-degree positioning within a plane. Up to 360 The rotation trajectory of the lidar 3011 can be parallel to the upper surface of the cargo box 302 and the upper surface of the pallet 303. In this embodiment, different rotation angles of the lidar 3011 can be used as the scanning angles of the lidar 3011.

[0106] The lidar 3011 can be, for example, a single-line lidar equipped with a laser emitter that can project a laser line.

[0107] When the lidar 3011 is at any scanning angle, the laser emitter can perform 0... in a plane. Up to 360 The rotation causes the projected laser line to form a 0° shape within that plane. Up to 360 In this embodiment, the rotation angle of the laser line can be defined as the different angles at which the laser line rotates.

[0108] When the lidar 3011 is at any scanning angle, the trajectory of the rotating laser line forms a laser surface. This laser surface is perpendicular to the rotation trajectory of the lidar 3011 and can be perpendicular to the upper surface of the cargo box 302 and the upper surface of the pallet 303.

[0109] In this embodiment, during the process of identifying the cargo box 302 and the pallet 303, it is necessary to scan the cargo box 302 and the pallet 303 to be identified multiple times using different scanning angles.

[0110] In each scan, the starting rotation angle is set to a preset 0. The laser emitter emits a laser line, which is projected onto the upper surface of the cargo box 302 or the pallet 303 to form a laser point, and this laser point is used as a point cloud obtained in this scan.

[0111] Furthermore, after controlling the laser emitter to rotate according to the preset rotation angle, the laser emitter is restarted, and the laser line is projected onto the upper surface of the cargo box 302 or the pallet 303 to form the next laser point, thus obtaining the next point cloud of this scan.

[0112] This continues until the laser emitter rotates a predetermined number of times, or until the rotation angle of the laser emitter changes from 0 to 1. Rotate until the preset target rotation angle is reached to complete this scan.

[0113] After completing the current scan, the lidar 3011 is rotated according to the preset scanning angle to perform the next scan, until the lidar 3011 has rotated a predetermined number of times, or the scanning angle of the lidar 3011 starts from 0. Rotate to the preset target scanning angle to complete the scanning of the cargo box 302 or pallet 303 and obtain the corresponding three-dimensional pose information.

[0114] Based on this, in this embodiment, the robotic arm scans the cargo box and pallet using the LiDAR 3011, thereby determining the three-dimensional pose information of the cargo box 302 and the pallet 303. This enables the robotic arm to accurately grasp the cargo box 320 according to the three-dimensional pose information of the cargo box 302 without the aid of a limiting device, and accurately place the grasped cargo box 302 onto the corresponding pallet 303 according to the three-dimensional pose information of the pallet 303, thus completing the precise palletizing work.

[0115] In another embodiment of this application, Figure 4 A flowchart of the robotic arm positioning pallet is shown, including the following steps S401-S405.

[0116] like Figure 4 As shown, when the robotic arm identifies the three-dimensional pose information of the pallet as the target object, it can first execute S401, and the robotic arm extends to reach above the pallet.

[0117] In this step, the top surface of the pallet can be used as the target surface.

[0118] When the robotic arm reaches above the pallet, the upper surface of the pallet is aligned with the lidar at the end of the robotic arm to satisfy the condition for the lidar to scan the upper surface.

[0119] In addition to requiring the robotic arm's end to be parallel to the upper surface of the pallet for the LiDAR to scan the upper surface of the pallet, the robotic arm also needs to be within a predetermined range above the pallet, such as within a 500mm x 500mm range above the pallet.

[0120] Furthermore, such as Figure 4 As shown, when the robotic arm is on the upper surface of the pallet, S402 can be executed, and the robotic arm drives the end-effector LiDAR to rotate, scanning the pallet multiple times.

[0121] In this step, the robotic arm can control the LiDAR at its end to rotate, thereby performing multiple scans of the pallet.

[0122] Based on multiple scans of the stack, further execution is possible. Figure 4 S403 extracts point cloud data from the upper surface of the stack.

[0123] In this process, multiple point clouds can be obtained based on each scan of the surface of the stack, and these can be used as a set of point clouds for that scan. Multiple sets of point clouds can be obtained based on multiple scans of the surface of the stack.

[0124] In this step, based on the obtained multiple sets of point clouds, the point cloud data of each point cloud can be extracted.

[0125] Based on this, S404 can be further executed to calculate the three-dimensional pose information of the stack.

[0126] In this step, the point cloud data of each point cloud obtained by S403 can be used to calculate the three-dimensional pose information of the pallet in three-dimensional space.

[0127] The three-dimensional space can be a three-dimensional space based on the world coordinate system.

[0128] Based on this, S405 can be executed to obtain the three-dimensional pose information in the base coordinate system.

[0129] The base coordinate system can be a three-dimensional Cartesian coordinate system established with the lidar at the end of the robotic arm or other positions on the robotic arm as the origin.

[0130] In this step, based on the three-dimensional pose information obtained in S404, the three-dimensional pose information in the world coordinate system can be converted into three-dimensional pose information in the base coordinate system, thus completing the pose recognition of the pallet.

[0131] Based on this, in this embodiment, when performing pose recognition on the pallet, the robotic arm completes multiple scans of the pallet by rotating the lidar at its end above the pallet. By calculating the point cloud data on the upper surface of the pallet obtained from the scans, the three-dimensional pose information of the pallet is obtained. This enables the robotic arm to accurately place the boxes on the pallet according to the three-dimensional pose information when stacking the boxes.

[0132] In another embodiment of this application, Figure 5 A flowchart illustrating the process of a robotic arm positioning a cargo box is shown, including the following steps S501-S505.

[0133] like Figure 5 As shown, when the robotic arm recognizes the three-dimensional pose information of the cargo box as the target object, it can first execute S501, and the robotic arm extends to reach above the cargo box.

[0134] In this step, the top surface of the cargo box can be used as the target surface.

[0135] When the robotic arm reaches above the cargo box, the upper surface of the cargo box is aligned with the lidar at the end of the robotic arm to satisfy the condition for the lidar to scan the upper surface.

[0136] In addition to requiring the robotic arm's end to be parallel to the upper surface of the cargo box, the conditions for scanning the upper surface with the robotic arm's lidar also require the robotic arm to be within a predetermined range above the cargo box, such as within a 500mm x 500mm range above the cargo box.

[0137] Furthermore, such as Figure 5 As shown, when the robotic arm is on the upper surface of the cargo box, S502 can be executed, and the robotic arm drives the end-effector LiDAR to rotate, scanning the cargo box multiple times.

[0138] In this step, the robotic arm can control the LiDAR at its end to rotate, thereby performing multiple scans of the cargo box.

[0139] Based on multiple scans of the cargo container, further processing can be performed. Figure 5 S503 extracts point cloud data from the upper surface of the cargo box.

[0140] In this process, multiple point clouds can be obtained based on each scan of the upper surface of the cargo box, and these can be used as a set of point clouds for that scan. Multiple sets of point clouds can be obtained based on multiple scans of the upper surface of the cargo box.

[0141] In this step, based on the obtained multiple sets of point clouds, the point cloud data of each point cloud can be extracted.

[0142] Based on this, S504 can be further executed to calculate the three-dimensional pose information of the cargo box.

[0143] In this step, the point cloud data of each point cloud obtained by S503 can be used to calculate the three-dimensional pose information of the cargo box in three-dimensional space.

[0144] The three-dimensional space can be a three-dimensional space based on the world coordinate system.

[0145] Based on this, S505 can be executed to obtain the three-dimensional pose information in the base coordinate system.

[0146] The base coordinate system can be a three-dimensional Cartesian coordinate system established with the lidar at the end of the robotic arm or other positions on the robotic arm as the origin.

[0147] In this step, based on the three-dimensional pose information obtained in S504, the three-dimensional pose information in the world coordinate system can be converted into three-dimensional pose information in the base coordinate system to complete the pose recognition of the cargo box.

[0148] Based on this, in this embodiment, when performing pose recognition on the cargo box, the robotic arm completes multiple scans of the cargo box by rotating the lidar at its end above the pallet. By calculating the point cloud data on the upper surface of the cargo box obtained from the scans, the three-dimensional pose information of the cargo box is obtained, so that the robotic arm can accurately grasp the cargo box according to the three-dimensional pose information of the cargo box when it grabs the cargo box.

[0149] In this embodiment, the three-dimensional pose information of the cargo box and pallet is obtained by LiDAR scanning, which is applicable to... Figure 3 In addition to the scenarios shown, it can also be used in scenarios where it is inconvenient to suspend the camera at a high position, and is not limited by workspace and shooting distance.

[0150] In another embodiment of this application, during the process of scanning the target object multiple times at different scanning angles, the scanning point can be controlled to rotate around a predetermined rotation axis, wherein the rotation axis can be a perpendicular line from the scanning point to the target surface of the target object.

[0151] In this embodiment, before scanning, the rotation amplitude of the scanning angle can be set within a predetermined angle range. After obtaining point cloud data during each scan, the scanning point is controlled to rotate with this rotation amplitude, so that scanning can be performed at different scanning angles each time.

[0152] exist Figure 3 In the example, when the lidar is used as the scanning point, the vertical line from the lidar to the upper surface of the cargo box can be used as the rotation axis of the lidar, so that the planar trajectory formed by the lidar through rotation is parallel to the target surface of the target object.

[0153] Before the lidar rotates, the angle range and rotation amplitude can be set for the lidar's scanning angle. For example, the scanning angle range can be set to 0° to 180°, the rotation amplitude can be set to 10°, and the predetermined position can be set to 0°.

[0154] Based on this, the lidar will perform the first scan when the scanning angle is 0°, and after the first scan, it will rotate once with a rotation amplitude of 10°. The second scan will be performed when the scanning angle is 10°. After the second scan, it will still rotate once with a rotation amplitude of 10° to reach a scanning angle of 20°, and then perform the third scan. The final scan will be performed when the lidar's scanning angle reaches 180°, thus completing 19 scans.

[0155] Based on this, this embodiment controls the scanning point to rotate around the rotation axis, so that the laser surface perpendicular to the target surface also rotates at different scanning angles. After multiple scans, multiple laser surfaces perpendicular to the target surface are obtained, and each laser surface intersects the adjacent laser surface at the same angle to the same line of intersection. When each laser surface is projected onto the target surface, multiple line segments intersecting at the same intersection point are formed on the target surface, and each line segment intersects the adjacent line segment at the same angle.

[0156] In another embodiment of this application, before scanning the target object multiple times at different scanning angles, different rotation amplitudes can be set for the scanning angle according to the different volumes of different target objects and the inverse proportional relationship between the rotation amplitude and the volume of the target object.

[0157] In this embodiment, when identifying the three-dimensional pose information of the target object, since it is necessary to use fitted line segments for identification, the larger the area of ​​the object covered by the fitted line segments, the more accurate the identification result will be.

[0158] Ideally, each fitted line segment is the intersection of the laser surface and the target surface. In reality, it can also be fitted by points on the intersection of the laser surface and the target surface. Therefore, when the angle range is fixed, the smaller the rotation amplitude when the laser surface rotates according to the scanning angle, the more scans are performed, the more intersections are formed between the laser surface and the target surface, and the more fitted line segments are obtained, thus making the identified three-dimensional pose information more accurate.

[0159] Based on this, for target objects of different volumes, a small number of line segments are sufficient to cover a large area of ​​the target object at the intersection of the laser surface and the target surface. However, for another target object with a larger volume, a larger number of line segments are needed to cover a large area of ​​the target object.

[0160] Therefore, a smaller rotation amplitude can be set for a larger target object, which can fit more line segments, while a larger rotation amplitude can be set for a smaller target object. This ensures accurate identification of 3D pose information without wasting computing power and maintaining work efficiency, thereby reducing the number of fitted line segments.

[0161] In other cases, for target objects with regular shapes and small volumes, since some areas of the target object have the same shape as the remaining areas, a small range of angles can be set for the scanning angle. This allows for the identification of some areas of the target object by fitting a small number of line segments, thereby inferring the three-dimensional pose information of the remaining areas.

[0162] Based on this, in this embodiment, before scanning, based on the relative size of the target object, a smaller rotation range is set for the scanning angle when the volume is larger, and a larger rotation range is set for the scanning angle when the volume is smaller. This ensures accurate identification of 3D pose information while avoiding excessive waste of computing power. Furthermore, by setting a relatively small angle range for target objects with regular shapes and small volumes, computing power can be further saved while ensuring accurate identification of 3D pose information.

[0163] In another embodiment of this application, when the scanning point scans the target object at different scanning angles, for each set of point clouds obtained by scanning, point cloud data in the Cartesian coordinate system can be obtained by coordinate transformation.

[0164] In this embodiment, during each scan of the scanning point, a laser line can be projected onto the target surface of the target object by the laser emitter in the lidar, and a point cloud can be formed on the target surface. During each scan, the laser line can be rotated by multiple rotation angles to form a set of multiple point clouds on the target surface.

[0165] Therefore, the scanning point can scan the target object in the polar coordinate system. The coordinates of each point cloud can be the polar coordinates of the point cloud in the polar coordinate system. Specifically, the polar coordinates of the point cloud can include the current rotation angle of the scanning point and the distance from the scanning point to the point cloud.

[0166] Furthermore, before performing calculations using point cloud data in a two-dimensional coordinate system, the polar coordinates of each point cloud can be converted into point cloud coordinates in a two-dimensional Cartesian coordinate system and used as the point cloud data, thereby making the obtained point cloud data applicable to calculations in a two-dimensional Cartesian coordinate system.

[0167] The two-dimensional Cartesian coordinate system can be, for example, the two-dimensional coordinate system of the laser surface formed by the rotation trajectory of the laser line.

[0168] Based on this, in this embodiment, based on the polar coordinates of each point cloud obtained during the scanning process, the point cloud coordinates are converted into a point cloud coordinate system in a two-dimensional coordinate system, so that each set of point cloud data can be used to calculate the target height and two-dimensional pose information of the target object in the two-dimensional coordinate system.

[0169] In another embodiment of this application, for each set of point cloud data, in the process of filtering out the target point cloud data, the target point cloud data can be determined based on the distance between each point cloud data and the scanning point.

[0170] In this embodiment, the distance between each point cloud data point and the scanned point can be compared based on the polar coordinates of each point cloud data point.

[0171] Furthermore, after comparison, the point cloud data with the shortest distance to the scanned point can be determined from each point cloud data.

[0172] Among them, the point cloud data with the shortest distance should be the point cloud data that falls on the target surface.

[0173] In some cases, if there is a connection line perpendicular to the target surface among the connection lines between each point cloud data and the scan point, then the connection line perpendicular to the target surface is the connection line between the point cloud data and the scan point with the shortest distance.

[0174] In other cases, if there is no connection line perpendicular to the target surface among the connection lines between each point cloud data and the scan point, then the angle between the connection line between the shortest point cloud data and the scan point and the target surface is the largest, that is, the angle between this connection line and the target surface is closest to the perpendicular angle.

[0175] Understandably, for other point cloud data that also fall on the target surface, the distance between the point cloud data and the scan point should be slightly greater than the shortest distance; for other point cloud data that do not fall on the target surface, the distance between the point cloud data and the scan point should be much greater than the shortest distance.

[0176] In other words, for any point cloud data other than the one with the shortest distance to the scan point, the distance between it and the scan point should be greater than the shortest distance. However, the difference between this distance and the shortest distance should be kept within a predetermined reasonable range. If the difference exceeds this reasonable range, it can be considered that the point cloud data is much larger than the shortest distance and falls outside the target object. It can be considered that the distance between the point cloud data falling outside the target object and the scan point is too large due to the height of the target object itself, thus causing the difference between it and the shortest distance to exceed the reasonable range.

[0177] Based on this, a filter value can be preset as a reasonable range. If the difference between the distance between any point cloud data and the scan point and the shortest distance is greater than the reasonable range, it is considered that the distance is much greater than the shortest distance, and the point cloud data is determined to fall outside the target object. If the difference between the distance between any point cloud data and the scan point and the shortest distance is less than or equal to the reasonable range, it is considered that the distance is slightly greater than the shortest distance, and the point cloud data is determined to fall on the target object.

[0178] In one example of this embodiment, the filter value can be set to 20cm.

[0179] Furthermore, after determining the shortest distance from each point cloud to the scan point, point cloud data with a difference between the distance between the point cloud and the scan point and the shortest distance greater than or equal to 20cm are selected from each point cloud data and retained as the target point cloud.

[0180] Based on this, in this embodiment, the point cloud data with the shortest distance to the scanning point is determined from each point cloud data. This point cloud data can be used as a reference, that is, the connecting line perpendicular to the target surface or closest to the perpendicular target surface is used as the reference for judging the position of each point cloud data. Thus, point cloud data whose connecting line exceeds the shortest distance but is still within a reasonable filtering range can be used as target point cloud data falling on the target object.

[0181] In another embodiment of this application, when the target object is relatively tall, for example, the height of the cargo box is higher than the height of the pallet, or the height is relatively higher, when identifying the target point cloud data on the surface of the cargo box, the coordinate difference between two adjacent point cloud data in the Cartesian coordinate system can be compared to determine whether there is a breakpoint point cloud in these two point cloud data. After determining the breakpoint point cloud, the point cloud arranged between the two breakpoint point clouds is determined as the target point cloud, and its data is used as the target point cloud data, so that the selected target point cloud data can be more accurate.

[0182] Among them, in a set of arranged point clouds, the breakpoint point cloud is a non-target point cloud that falls outside the target surface and is adjacent to the point cloud that falls on the target surface. In other words, the breakpoint point cloud can specifically represent the boundary between the target point cloud and the non-target point cloud. The point cloud on one side of the breakpoint point cloud is the target point cloud, and the point cloud on the other side is the non-target point cloud.

[0183] In this embodiment, each scan yields a set of point clouds: during the scan, the laser emitter projects laser lines onto the target object at different rotation angles, thereby forming a row of laser points on the target object and in the nearby area outside the target object, and these points are used as a set of point clouds.

[0184] Therefore, it can be determined that in a set of arranged point clouds, the arrangement of each point cloud should be as follows: first, arrange one or more point clouds that do not fall on the target object; then, arrange multiple point clouds that fall on the target object; then, arrange one or more point clouds that do not fall on the target object; and it can be determined that there are two breakpoint point clouds in a set of point clouds.

[0185] In this embodiment, Figure 6 This is another concrete example of point cloud data fitting.

[0186] Among them, such as Figure 6 As shown, box L represents the target surface 615 of the target object. In one scan, based on the laser emitter without rotation angle, a total of 7 point clouds are obtained: point cloud 601, point cloud 602, point cloud 603, point cloud 604, point cloud 605, point cloud 606 and point cloud 607.

[0187] The seven point clouds are arranged as follows:

[0188] Starting with point cloud 601, point cloud 601 that does not fall on target surface 615 is arranged first. Then, five point clouds that fall on target surface 615 are arranged: point clouds 602 to point cloud 606. Finally, point cloud 607 that does not fall on target surface 615 is arranged.

[0189] In this embodiment, the two-dimensional coordinate system of the plane where the scanning point and each point cloud are located is taken as the first two-dimensional Cartesian coordinate system. Based on the above arrangement, it can be found that since the target object is relatively high, for two adjacent point clouds, if one point cloud is on the target surface 615 and the other point cloud is not on the target surface 615, then the difference between these two adjacent point cloud data is numerically correlated with the tangent value of the corresponding rotation angle.

[0190] Specifically, the numerical correlation can be expressed as follows: if the coordinates of two point clouds in the first two-dimensional Cartesian coordinate system are respectively taken as the first point cloud data and the second point cloud data, then the ratio of the difference between the vertical coordinates and the difference between the horizontal coordinates of these two point clouds is equal to or approximately equal to the tangent of the rotation angle corresponding to one of the point cloud data. The approximate equality can be considered as the difference between the ratio and the tangent being within a preset error range.

[0191] In a specific example, the following formula (1) can be used to show that there is a numerical correlation between the differences between point cloud data and the tangent of the corresponding rotation angle:

[0192] (1)

[0193] Where k represents the order of the point cloud, y k+1Let y represent the ordinate of the (k+1)th point cloud in the first two-dimensional Cartesian coordinate system. k Let x represent the ordinate of the k-th point cloud in the first two-dimensional Cartesian coordinate system. k+1 Let x represent the x-coordinate of the (k+1)th point cloud in the first two-dimensional Cartesian coordinate system. k This represents the x-coordinate of the k-th point cloud in the first two-dimensional Cartesian coordinate system. This represents the rotation angle corresponding to the (k+1)th point cloud.

[0194] Based on this, it can be verified whether the point cloud data between any two adjacent point clouds can make the above formula (1) hold. In some cases, if the formula (1) and If they are completely equal, then the (k+1)th point cloud is considered to be the breakpoint point cloud that falls exactly on the edge of the target object. and If they are approximately equal, then the (k+1)th point cloud is considered to be the breakpoint point cloud adjacent to the edge of the target object.

[0195] Furthermore, based on the two determined breakpoint point clouds, the point cloud between the two breakpoint point clouds can be regarded as the target point cloud falling on the target object, and thus the coordinates of the target point cloud in the first two-dimensional Cartesian coordinate system can be used as the target point cloud data.

[0196] Based on this, in this embodiment, for two adjacent point clouds, by comparing the point cloud data of the two point clouds, it can be verified whether there is a breakpoint point cloud in the two adjacent point clouds based on whether the ratio of the difference between the vertical coordinate and the difference between the horizontal coordinate of the first point cloud data and the second point cloud data is equal to or approximately equal to the tangent of the rotation angle corresponding to the second point cloud data. Then, two breakpoint point clouds are selected from multiple point clouds, thereby obtaining the target point cloud between the two breakpoint point clouds and the corresponding target point cloud data.

[0197] In another embodiment of this application, when determining the two-dimensional pose information of the target object, based on the line segments fitted from each set of target point cloud data, the coordinates of the two endpoints of the line segment in the two-dimensional plane where the target surface is located can be calculated using the length of the line segment, the scanning angle, and the scanning point in a preset trigonometric function. Thus, the endpoints of each line segment are fitted into the target shape in the two-dimensional plane, and the two-dimensional pose information of the target object is determined based on the target shape.

[0198] In this embodiment, based on the line segment fitted in the first two-dimensional Cartesian coordinate system, the length of the line segment can be determined, and the foot of the perpendicular from the scanning point to the line segment can be determined in the first two-dimensional Cartesian coordinate system.

[0199] Furthermore, a second two-dimensional Cartesian coordinate system can be set up for the two-dimensional plane where the target surface is located, and the coordinates of the perpendicular foot can be determined in the second two-dimensional Cartesian coordinate system.

[0200] Based on this, the positions of the two endpoints, i.e., the endpoint coordinates of each endpoint, can be calculated using the line segment length, the perpendicular foot coordinates, and the scanning angle when fitting the line segment from the preset trigonometric functions.

[0201] Specifically, the following formula (2) is used as the trigonometric function for calculating the endpoints of a line segment:

[0202] (2)

[0203] in,( , ) represents the coordinates of one of the two endpoints of the line segment. , ( ) represents the coordinates of the other endpoint of the line segment. The x-coordinate represents the foot of the perpendicular. This represents the scanning angle at which the scanning radar fits the line segment after the current nth rotation; This is the length of the line segment.

[0204] Furthermore, after determining the endpoint coordinates of the line segments based on the second two-dimensional Cartesian coordinate system, the target shape can be fitted in the second two-dimensional Cartesian coordinate system using the endpoint coordinates of each line segment, with the shape of the target surface as the target.

[0205] Based on the target shape fitted in the second two-dimensional Cartesian coordinate system, the coordinates of the center of the target shape in the second two-dimensional Cartesian coordinate system, that is, the center position, can be determined.

[0206] Furthermore, the angle between any side of the target shape and a preset target coordinate system in the second two-dimensional Cartesian coordinate system can be determined.

[0207] Based on this, the center position and included angle of the target shape are used as the two-dimensional pose information of the target surface of the target object.

[0208] exist Figure 6 In the example, based on the determined target point clouds: point cloud 602, point cloud 603, point cloud 604, point cloud 605, and point cloud 606, line segment 614 can be fitted.

[0209] Furthermore, based on line segment 614, using the above formula (2), the length of line segment 614, the foot of the perpendicular from the scanning point to line segment 614 617, and the current scanning angle, the two endpoints of line segment 614 can be determined. Figure 6 The endpoint coordinates in the second two-dimensional Cartesian coordinate system, that is, the coordinates of point cloud 602 and point cloud 606 in... Figure 6 The coordinates in the second two-dimensional Cartesian coordinate system.

[0210] exist Figure 6 In the diagram, point cloud 613 and point cloud 609 are the two endpoints of the fitted second line segment; point cloud 610 and point cloud 611 are the two endpoints of the fitted third line segment; and point cloud 612 and point cloud 608 are the two endpoints of the fitted fourth line segment.

[0211] Furthermore, based on the rectangular target on the target surface 615, the target shape can be fitted using point clouds 613, 609, 610, 611, 612, 608, 602, and 606, that is... Figure 6 Rectangle 616 in the middle.

[0212] Based on the fitted rectangle 616, the center coordinates of the center position of the rectangle can be determined. , And determine the angle between any long side of rectangle 616 and the x-axis. , center coordinates ( , ) and included angle Two-dimensional pose information of the target object.

[0213] Based on this, in this embodiment, for each line segment determined by the first two-dimensional Cartesian coordinate system, the coordinates of the two endpoints of each line segment in the second two-dimensional Cartesian coordinate system can be obtained by performing trigonometric function operations in the second two-dimensional Cartesian coordinate system. Since each endpoint is the point cloud closest to the edge of the target surface in the point cloud falling on the target surface, the target shape of the target surface can be fitted in the second two-dimensional Cartesian coordinate system using the coordinates of each endpoint. This allows the angle between the center coordinates of the target shape and the target coordinate axis to be obtained based on the second two-dimensional Cartesian coordinate system, which is the two-dimensional pose information of the target shape. Since the target shape is the shape of the target surface, this two-dimensional pose information can be used as the two-dimensional pose information of the target object on one side of the target surface.

[0214] In another embodiment of this application, based on the determined target height and two-dimensional pose information of the target object, the target height can be combined with the two-dimensional pose information of the target object during the process of obtaining the three-dimensional pose information of the target object, thereby obtaining the three-dimensional pose information of the target object in three-dimensional space.

[0215] In this embodiment, when calculating the target height, based on a preset mutually perpendicular first two-dimensional Cartesian coordinate system and a second two-dimensional Cartesian coordinate system, the height of each line segment can be calculated in the first two-dimensional Cartesian coordinate system, and the average value of the heights of each line segment is determined as the target height.

[0216] Furthermore, based on the two-dimensional pose information determined in the second two-dimensional Cartesian coordinate system, the target height in the first two-dimensional Cartesian coordinate system is combined with the two-dimensional pose information in the second two-dimensional Cartesian coordinate system to obtain the three-dimensional pose information of the target object in the three-dimensional Cartesian coordinate system.

[0217] In some scenarios, the three-dimensional Cartesian coordinate system can be a preset three-dimensional base coordinate system in the LiDAR at the end of the robotic arm.

[0218] In other scenarios, the 3D pose information obtained in the 3D Cartesian coordinate system can be converted into a preset 3D base coordinate system in the lidar of the robotic arm end effector.

[0219] In some specific examples, based on the obtained two-dimensional pose information center coordinates ( , , ) and target height After combining the two, the resulting three-dimensional pose information can be represented as: ( , , , ).

[0220] Based on this, in this embodiment, in order to obtain the three-dimensional pose information in the three-dimensional Cartesian coordinate system, the three-dimensional Cartesian coordinate system is split into two planar two-dimensional Cartesian coordinate systems by using a preset mutually perpendicular first two-dimensional Cartesian coordinate system and a second two-dimensional Cartesian coordinate system. This allows the target height in the first two-dimensional Cartesian coordinate system to be combined with the two-dimensional pose information in the second two-dimensional Cartesian coordinate system to obtain three-dimensional pose information suitable for the three-dimensional Cartesian coordinate system. This enables the lidar at the end of the robotic arm to correctly identify the pose of the target object according to the three-dimensional pose information.

[0221] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide an object positioning device.

[0222] refer to Figure 7 The device for locating the object includes:

[0223] The scanning module 701 is used to scan the target object at preset scanning points and different scanning angles using a lidar to obtain multiple sets of target point cloud data.

[0224] The height determination module 702 is used to determine the target height of the scanning point relative to each group of target point cloud data;

[0225] The two-dimensional pose determination module 703 is used to determine the two-dimensional pose information of the target object based on the coverage area of ​​each group of target point cloud data and the vertical projection position of the scanning point and each group of target point cloud data.

[0226] The 3D pose determination module 704 is used to obtain the 3D pose information of the target object based on the target height and 2D pose information.

[0227] In one embodiment, the scanning module 701 is specifically used to perform:

[0228] The scanning point is controlled to rotate around the rotation axis according to the scanning angle, and a laser surface is emitted towards the target object to scan the target object at different scanning angles and obtain multiple sets of point cloud data; the rotation axis is the perpendicular line between the scanning point and the target surface of the target object;

[0229] Target point cloud data that meets the preset target surface conditions are selected from multiple sets of point cloud data.

[0230] The process involves scanning the target object from different angles to obtain multiple sets of point cloud data, including:

[0231] In the polar coordinate system, the target object is scanned at different scanning angles at the scanning point to obtain the polar coordinates of multiple point clouds; the polar coordinates of the point cloud include the rotation angle of the scanning point and the distance from the scanning point to the point cloud.

[0232] The polar coordinates of each point cloud are converted to point cloud data in the Cartesian coordinate system to obtain multiple sets of point cloud data.

[0233] Target point cloud data that meets the preset target surface conditions are selected from multiple sets of point cloud data, including:

[0234] For each set of point cloud data, compare the distance between each point cloud data and the scanned point;

[0235] Determine the shortest distance among all distances;

[0236] The target point cloud data is determined based on the distance and the shortest distance, wherein the target point cloud data is the point cloud data in which the difference between the distance and the shortest distance is less than a preset filtering value.

[0237] Target point cloud data that meets the preset target surface conditions is selected from multiple sets of point cloud data, including:

[0238] Determine the ratio of the difference in the vertical coordinate to the difference in the horizontal coordinate of the first and second point cloud data of two adjacent point clouds in each of the multiple sets of point cloud data;

[0239] If the ratio is equal to the tangent of the rotation angle corresponding to the second point cloud data, then the two first point cloud data in each group of point cloud data are determined to be the data of the breakpoint point cloud.

[0240] The target point cloud data is obtained by selecting the target point cloud that is arranged between the two breakpoint point clouds from each group of point clouds.

[0241] In another embodiment, the height determination module 702 is specifically used to perform:

[0242] Each set of target point cloud data is fitted into a line segment;

[0243] The difference between the vertical distance from the scan point to the line segment and the height of the scan point is determined as the target height of the target object.

[0244] In another embodiment, the two-dimensional pose determination module 703 is specifically used to perform:

[0245] The position of the endpoint of each line segment is determined by each line segment, the position of the perpendicular foot corresponding to each line segment, the scanning angle corresponding to each line segment, and the preset trigonometric functions. The trigonometric functions represent the calculation relationship between the position of the line segment, the position of the perpendicular foot, the scanning angle, and the position of the endpoint.

[0246] Fit the endpoint coordinates of each line segment to the target shape of the target surface;

[0247] Determine the center position of the target shape;

[0248] The angle between the center position, any side of the target shape, and the target coordinate axis in the Cartesian coordinate system is determined as the two-dimensional pose information of the target object.

[0249] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0250] The apparatus described above is used to implement the corresponding object positioning method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0251] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the object positioning method of any of the above embodiments.

[0252] Figure 8 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0253] An electronic device may include a processor 801 and a memory 802 storing computer program instructions.

[0254] Specifically, the processor 801 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0255] Memory 802 may include mass storage for data or instructions. For example, and not limitingly, memory 802 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 802 may include removable or non-removable (or fixed) media. Where appropriate, memory 802 may be internal or external to an electronic device. In a particular embodiment, memory 802 is a non-volatile solid-state memory.

[0256] Memory 802 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0257] The processor 801 implements any of the object positioning methods in the above embodiments by reading and executing computer program instructions stored in the memory 802.

[0258] In one example, the electronic device may also include a communication interface 803 and a bus 810. For example, Figure 8 As shown, the processor 801, memory 802, and communication interface 803 are connected through bus 810 and complete communication with each other.

[0259] The communication interface 803 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0260] Bus 810 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 810 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0261] This electronic device can perform the object localization method in the embodiments of this application based on the recognition of the two-dimensional pose information of the target object, thereby achieving a combination of Figure 4 The method described for locating objects.

[0262] Furthermore, in conjunction with the object location methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the object location methods in the above embodiments.

[0263] This application also provides a computer program product, including a computer program, which, when executed, implements any of the object location methods described in the above embodiments.

[0264] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0265] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0266] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a palletizing robot arm, which includes an object positioning device and / or electronic device of any of the foregoing embodiments, wherein the electronic device performs the object positioning method of any of the above.

[0267] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0268] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0269] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method of object localization, characterized by, The application is applied to a palletizing robot, the palletizing robot comprises a laser radar; the method comprises: scanning a target object at different scanning angles at a preset scanning point by using the laser radar to obtain a plurality of sets of target point cloud data; determining a target height of the scanning point relative to each set of target point cloud data; determining two-dimensional pose information of the target object based on a coverage interval of each set of target point cloud data and a vertical projection position of the scanning point and each set of target point cloud data; obtaining three-dimensional pose information of the target object based on the target height and the two-dimensional pose information; the coverage interval of each set of target point cloud data is the length of a corresponding fitted line segment; the vertical projection position is the position of the foot of the perpendicular from the scanning point to the line segment; the determination of the two-dimensional pose information of the target object based on the coverage interval of each set of target point cloud data and the vertical projection position of the scanning point and each set of target point cloud data comprises: determining the position of the end point of each line segment by using each line segment, the position of the foot of each line segment, the scanning angle corresponding to each line segment and a preset trigonometric function, the trigonometric function representing the calculation relationship between the line segment, the position of the foot, the scanning angle and the position of the end point; fitting the end point coordinates of each line segment into a target shape of a target surface of the target object; determining the center position of the target shape; determining the included angle between the center position, any side of the target shape and a target coordinate axis of a Cartesian coordinate system as the two-dimensional pose information of the target object.

2. The method of object localization of claim 1, wherein, the scanning of the target object at different scanning angles at a preset scanning point by using the laser radar to obtain a plurality of sets of target point cloud data comprises: controlling the scanning point to rotate around a rotation axis according to the scanning angle and emitting a laser plane to the target object to scan the target object at different scanning angles to obtain a plurality of sets of point cloud data; the rotation axis is a perpendicular line from the scanning point to a target surface of the target object; screening target point cloud data meeting a preset target surface condition from the plurality of sets of point cloud data.

3. The method of object localization of claim 1, wherein, the determination of the target height of the scanning point relative to each set of target point cloud data comprises: fitting each set of target point cloud data into a line segment; determining the difference between the vertical distance from the scanning point to the line segment and the height of the scanning point as the target height of the target object.

4. The method of object localization of claim 2, wherein, the scanning of the target object at different scanning angles to obtain a plurality of sets of point cloud data comprises: scanning the target object at different scanning angles at the scanning point in a polar coordinate system to obtain a plurality of polar coordinates of point clouds; wherein the polar coordinates of the point clouds comprise a rotation angle of the scanning point and a distance from the scanning point to the point cloud; converting the polar coordinates of each point cloud into point cloud data in a Cartesian coordinate system to obtain the plurality of sets of point cloud data.

5. The method of object localization of claim 2, wherein, the screening of target point cloud data meeting a preset target surface condition from the plurality of sets of point cloud data comprises: for each set of point cloud data, comparing the distances between each point cloud data and the scanning point; determining the shortest distance among the distances; The target point cloud data is determined according to the distance and the shortest distance, wherein the target point cloud data is point cloud data with a difference between the distance and the shortest distance less than a preset screening value.

6. The method of object localization of claim 2, wherein, The target point cloud data meeting the preset target surface condition is screened from each of the plurality of groups of point cloud data, including: A ratio of a vertical coordinate difference and a horizontal coordinate difference of first point cloud data and second point cloud data of two adjacent point clouds in each group of point cloud data is determined respectively; In a case where the ratio is equal to a tangent value of a rotation angle corresponding to the second point cloud data, the two first point cloud data in each group of point cloud data are determined as breakpoint point cloud data; Target point cloud data of target point cloud arranged between two breakpoint point clouds is screened from each group of point cloud data, to obtain the target point cloud data of the target point cloud.

7. An apparatus for object localization, characterized by The device is applied to a palletizing mechanical arm including a laser radar, and includes: A scanning module configured to scan a target object at different scanning angles at a preset scanning point by using the laser radar, to obtain a plurality of groups of target point cloud data; A height determination module configured to determine a target height of the scanning point relative to each group of target point cloud data; A two-dimensional pose determination module configured to determine two-dimensional pose information of the target object based on a coverage interval of each group of target point cloud data and a vertical projection position of the scanning point and each group of target point cloud data; A three-dimensional pose determination module configured to obtain three-dimensional pose information of the target object based on the target height and the two-dimensional pose information; The coverage interval of each group of target point cloud data is a length of a corresponding fitted line segment; the vertical projection position is a position of a foot of a perpendicular line from the scanning point to the line segment; the two-dimensional pose determination module is configured to determine a position of an end point of each line segment by using each line segment, a position of a foot corresponding to each line segment, a scanning angle corresponding to each line segment, and a preset trigonometric function representing a calculation relationship between the line segment, the position of the foot, the scanning angle, and the position of the end point; fit the end point coordinates of each line segment into a target shape of a target surface of the target object; determine a center position of the target shape; and determine an included angle between any side of the target shape and a target coordinate axis of a Cartesian coordinate system as the two-dimensional pose information of the target object.

8. An electronic device, comprising: The device includes a processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the object positioning method of any one of claims 1-6.

9. A palletizing robot comprising electronics, characterized in that The electronic device is configured to implement the object positioning method of any one of claims 1-6.

Images