Material box goods taking method, electronic equipment and storage medium

Abstract

The invention discloses a material box goods taking method, electronic equipment and a storage medium. The method comprises the steps of obtaining a first point cloud of a target material box and a second point cloud of an adjacent object; based on the first point cloud, determining a goods taking safety area of the target material box; based on the second point cloud, the edge area of the adjacent object is determined, and the goods taking safety area is formed by outward expansion of the area where the target material box is located; and whether the target material box can be directly forked or not is determined based on the intersection condition of the edge area and the goods taking safety area. In this way, the potential safety hazard of goods taking caused by skewing of the material box can be avoided.

Description

Technical Field

[0001] This application relates to the field of automated handling technology, and in particular to a bin picking method, electronic equipment, and storage medium. Background Technology

[0002] Automated Guided Vehicles (AGVs) can be used in logistics and warehousing to replace manual management of goods. They can be controlled by system commands to retrieve and store goods. However, due to shelf vibrations or human operation, the position of goods on the shelves can shift, reducing the distance between adjacent objects and creating safety hazards during the AGV retrieval process. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a method, electronic device, and storage medium for retrieving goods from a tin can, which can avoid the safety hazards of retrieving goods caused by a tilted tin can.

[0004] To address the aforementioned technical problems, the first aspect of this application provides a method for retrieving a bin, the method comprising: acquiring a first point cloud of a target bin and a second point cloud of adjacent objects; determining a safe retrieval area for the target bin based on the first point cloud; and determining an edge region of adjacent objects based on the second point cloud, wherein the safe retrieval area is formed by expanding outward from the region where the target bin is located; and determining whether the target bin can be directly retrieved based on the intersection of the edge region and the safe retrieval area.

[0005] Specifically, determining whether the target bin can be directly picked up based on the intersection of the edge area and the safe picking area includes: determining that the target bin can be directly picked up if the edge area and the safe picking area do not intersect; and / or determining that the target bin cannot be directly picked up if the edge area and the safe picking area intersect, and if the adjacent objects are adjacent bins, determining whether the adjacent bins can be moved before picking up the target bin based on the positional relationship between the target bin and the adjacent bins.

[0006] The process of determining whether the target bin can be moved before the target bin can be picked up, based on the positional relationship between the target bin and the adjacent bins, includes: determining whether the target bin is located in the deep part of the shelf and whether the adjacent bin is located in the shallow part of the shelf; wherein, in the same direction, the distance between the deep part of the shelf and the forklift device is greater than the distance between the shallow part of the shelf and the forklift device, and the forklift device is used to pick up the target bin and the adjacent bin; in response to the target bin being located in the deep part of the shelf, the adjacent bin being located in the shallow part of the shelf, and the adjacent bin meeting the picking conditions, it is determined that the adjacent bin can be moved before the target bin can be picked up.

[0007] In the case where it is determined that the target bin can be directly picked up, the method further includes: determining the actual pose of the target bin based on the first point cloud; using the actual pose, determining the moving distance of the picking device and the rotation angle of the telescopic fork on the picking device, so that the picking device moves to the target position according to the moving distance and the telescopic fork rotates to the target direction according to the rotation angle before picking up the target bin.

[0008] Among them, adjacent objects are adjacent material boxes, and the edge area includes the edge line segments of adjacent material boxes.

[0009] The step of determining the edge region of adjacent objects based on the second point cloud includes: using the second point cloud to fit at least one face of an adjacent bin; obtaining a third point cloud belonging to the face; using the third point cloud of the face to determine the coordinates of the two endpoints of the face boundary; and using the coordinates of the two endpoints to fit an edge line segment.

[0010] The process of determining the coordinates of the two endpoints of the surface boundary using the third point cloud of the surface includes: determining the boundary center point of the surface boundary using the third point cloud of the surface; and calculating the coordinates of the two endpoints based on the target orientation angle of the surface, the boundary center point, and the dimensions of the adjacent bins.

[0011] The at least one face includes a first face and a second face; after fitting at least one face of an adjacent bin, the method further includes: calculating the normal vectors of the first face and the second face respectively to obtain a first orientation angle of the first face and a second orientation angle of the second face; converting the second orientation angle into a third orientation angle, wherein the third orientation angle has the same orientation as the first orientation angle; and weighting the first orientation angle and the third orientation angle to obtain a target orientation angle.

[0012] The process of weighting the first and third orientation angles to obtain the target orientation angle includes: using the ratio of the first number of point clouds belonging to the first face to the total number as the first weight of the first orientation angle, and using the ratio of the second number of point clouds belonging to the second face to the total number as the second weight of the third orientation angle; wherein the total number is equal to the sum of the first and second numbers; and using the first and second weights, the first and third orientation angles are weighted and summed to obtain the target orientation angle.

[0013] The safe pick-up area is a rectangular area.

[0014] The process of determining the safe picking area for the target bin based on the first point cloud includes: using the first point cloud to fit the target surface of the target bin, and calculating the target center point and the actual orientation angle of the target surface; the target surface is the surface of the target bin that occupies the largest area within the field of view of the depth camera, and the actual orientation angle represents the angle between the actual orientation of the target surface and the first axis of the telescopic fork coordinate system; obtaining the intersection point of the target center point with the second axis of the telescopic fork coordinate system in the actual orientation; using the point at a first preset distance from the intersection point in the actual orientation as the first center point of the first boundary line of the safe picking area; and using the point at a second preset distance from the intersection point in the actual orientation as the second center point of the second boundary line of the safe picking area. The target distance is defined as follows: The first and second boundary lines are perpendicular to the actual orientation and parallel to each other; the second preset distance is greater than the sum of the first preset distance and the length of the target bin; two points on the first boundary line are determined as the target distance from the first center point, serving as the two endpoints of the first boundary line; and two points on the second boundary line are determined as the target distance from the second center point, serving as the two endpoints of the second boundary line; the target distance is the sum of half the width of the target bin and the third preset distance; based on the two endpoints of the first and second boundary lines, a safe picking area is obtained.

[0015] The process includes, after fitting the target surface of the target bin using the first point cloud, determining whether each target point cloud in the target surface is noise, removing the target point clouds that are noise, and obtaining a new target point cloud that belongs to the target surface; calculating the actual size of the target bin based on the new target point cloud; wherein the actual size includes at least one of the actual width and the actual height; and comparing the actual size with the reference size to verify whether the target bin is a bin to be picked up by forks.

[0016] The determination of whether each target point cloud in the target surface belongs to noise includes: for any target point cloud in the target surface, determining whether the target angle is within a preset angle range and / or whether the target distance is within a preset distance range, so as to determine whether the target point cloud belongs to noise; wherein, the target angle is the angle between the first normal vector of the target surface and the second normal vector of the target point cloud, and the target distance is the distance from the target point cloud to the target surface.

[0017] The adjacent objects are adjacent bins. Before acquiring the first point cloud of the target bin and the second point cloud of the adjacent objects, the process includes: determining whether the target bin and the adjacent bins exist based on the initial point cloud acquired by the depth camera, and obtaining a determination result; in response to the determination result that both the target bin and the adjacent bins exist, acquiring the first point cloud of the target bin and the second point cloud of the adjacent bins and subsequent steps; in response to the determination result that the target bin exists and the adjacent bins do not exist, determining that the target picking method for the target bin is direct picking, acquiring the first point cloud of the target bin, determining the actual pose of the target bin based on the first point cloud, so that the forklift device can pick up the bin based on the actual pose; in response to the determination result that the target bin does not exist, ending the picking task.

[0018] The determination of whether the target bin and adjacent bins exist includes: determining a first preset search range and a second preset search range based on the reference position and reference size of the target bin; wherein the second preset search range is greater than the first preset search range; acquiring a first initial point cloud located within the first preset search range and a second initial point cloud located within the second preset search range; determining that the target bin exists in response to the number of the first initial point cloud being greater than a first preset number; determining that adjacent bins exist in response to the number of the second initial point cloud being greater than a second preset number; wherein the second preset number is greater than the first preset number.

[0019] The process of acquiring the first point cloud of the target bin and the second point cloud of adjacent objects includes: cropping the initial point cloud acquired by the depth camera to obtain the cropped point cloud; clustering the cropped point cloud to obtain the clustering result; and combining the clustering result and the judgment result to obtain the first point cloud of the target bin and the second point cloud of adjacent objects.

[0020] Among them, adjacent objects are adjacent bins. Combining the clustering results and judgment results, the first point cloud of the target bin and the second point cloud of the adjacent objects are obtained, including: Based on the judgment result, determine the number of existing material boxes; select a number of cluster blocks from the clustering results; based on the point clouds in the number of cluster blocks, obtain the first point cloud and the second point cloud of the adjacent material boxes.

[0021] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising a memory and a processor. The memory stores program instructions, and the processor executes the program instructions to implement the method provided in the first aspect.

[0022] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program instructions that can be executed by a processor to implement the method provided in the first aspect.

[0023] The beneficial effects of this application are as follows: Unlike existing technologies, this application obtains a first point cloud of the target bin and a second point cloud of adjacent objects; based on the first point cloud, it determines the safe picking area of ​​the target bin, which is formed by expanding outward from the area where the target bin is located; and based on the second point cloud, it determines the edge areas of adjacent objects; based on the intersection of the edge areas and the safe picking area, it determines whether the target bin can be directly picked up. Since the edge areas of adjacent objects are determined by the point clouds of adjacent objects, information such as the pose and size of adjacent objects are taken into account during this process, resulting in highly accurate edge areas; and by combining whether the edge areas and the safe picking area intersect, it can accurately determine whether the target bin can be directly picked up, thus avoiding affecting adjacent objects during the picking up of the target bin, and also avoiding the impact of the tilt of adjacent objects on the picking up of the target bin, thereby avoiding the safety hazards of picking up the bin caused by the tilt of the bin. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of one embodiment of the forklift device and shelf provided in this application; Figure 2 This is a flowchart illustrating one embodiment of the bin retrieval method provided in this application; Figure 3 This is a schematic diagram of one embodiment of the coordinate system provided in this application; Figure 4 This is a schematic diagram of an implementation of the intersection of the edge area and the safe pickup area provided in this application; Figure 5 This is a schematic diagram of an embodiment of the moving distance of the fork-lifting device and the rotation angle of the telescopic fork on the fork-lifting device in this application; Figure 6 This is a flowchart illustrating an implementation method for determining the edge region of adjacent objects based on a second point cloud in this application. Figure 7 This is a schematic diagram of the projection of adjacent bins and the target bin onto the XY plane of the telescopic fork coordinate system in this application; Figure 8 This is a schematic diagram illustrating one embodiment of the positional relationship between the target bin and adjacent bins in this application; Figure 9 This is a schematic diagram of another embodiment of the positional relationship between the target bin and adjacent bins in this application; Figure 10This is a schematic diagram of one embodiment of the secure pickup area in this application; Figure 11 This is a schematic diagram of one embodiment of the bending area of ​​the target surface in this application; Figure 12 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application; Figure 13 This is a schematic diagram of the framework structure of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] It should be noted that the embodiments of this application contain descriptions involving "first," "second," etc., which are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0027] In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the figures). If the specific posture changes, the directional indication will also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] This application provides a method for retrieving a toy bin. This method can be executed by a forklift device or controlled by other electronic equipment. The forklift device can be an automated guided vehicle (AGV), a forklift with forks, etc. The forklift device is equipped with an image acquisition device to acquire information within a preset range. The toy bin can be placed on a shelf. Upon receiving a forklift command, the forklift device moves to the front of the toy bin according to its position, thereby controlling the image acquisition device to acquire information within the preset range. In one implementation scenario, the forklift device is an AGV, and the forklift device and shelf are as follows... Figure 1 As shown, the forklift device may include the AGV body, a clamping device (i.e., a telescopic fork), an image acquisition device (i.e., a depth camera), storage compartments, and a lifting device. The telescopic fork includes a hook and a fork arm. When it is determined that the target bin can be picked up, the forklift device controls the fork arm to move towards and pick up the target bin. After picking up the bin, it can be placed in the storage compartment. The shelf includes shelf codes to identify the location within the shelf. Each shelf can hold two rows of bins; the row of bins closer to the forklift device is the shallow bin, and the row further away is the deep bin. In one implementation scenario, for ease of calculation, when the image acquisition device acquires the bin point cloud data, the bottom surface of the telescopic fork is controlled to be flush with the top edge of the shelf, where the top edge of the shelf is adjacent to the bottom surface of the bin.

[0031] Please see Figure 2 , Figure 2 This is a flowchart illustrating one embodiment of the bin retrieval method provided in this application. The method includes: S21: Obtain the first point cloud of the target bin and the second point cloud of the adjacent objects.

[0032] In one embodiment, the target bin is the bin to be picked up by forks, and the adjacent bins are one or more bins adjacent to the target bin. The adjacent bins can be bins located in front of, behind, to the left of, or to the right of the target bin. There can be one or two adjacent bins located to the left and right of the target bin. Adjacent objects can be shelves and / or bins; this application focuses on describing adjacent objects as adjacent bins.

[0033] The first and second point clouds can be obtained through an image acquisition device. In one implementation scenario, the image acquisition device is a depth camera, which acquires the first and second point clouds. Understandably, the initial point cloud belonging to the target bin can be directly obtained from all the initial point clouds acquired by the depth camera as the first point cloud, and the initial point clouds belonging to adjacent objects can be obtained from all the initial point clouds as the second point cloud. Alternatively, the initial point clouds belonging to the target bin and the initial point clouds belonging to adjacent objects can be processed to obtain the corresponding first and second point clouds. These processing methods include, but are not limited to, cropping and filtering.

[0034] In one specific implementation, the initial point cloud acquired by the depth camera is cropped to obtain a cropped point cloud. Here, the initial point cloud includes all point clouds. The cropped point cloud is then clustered to obtain clustering results. Combining the clustering results and the judgment results, the first point cloud of the target bin and the second point clouds of adjacent objects are obtained. When the adjacent objects are adjacent bins, the first point cloud of the target bin and the second point cloud of the adjacent bins are obtained by combining the clustering results and the judgment results.

[0035] This section details the process of using adjacent objects as adjacent bins. Considering that the depth camera's acquisition range may include point clouds of other items besides the target bin and adjacent bins, the initial point cloud can be cropped. In one implementation scenario, a certain area can be expanded based on the reference position of the target bin, and the point cloud within the expanded area can be used as the point cloud of the cropped target bin. Similarly, the point clouds of cropped adjacent bins can also be expanded based on the reference positions of the adjacent bins, and the point cloud within the expanded area can be used as the point cloud of the cropped adjacent bins. The reference position is input by the user and represents the theoretical position of the bin. However, in practice, the bin position may shift, causing the actual position to differ from the theoretical position. Therefore, expanding the area based on the reference position avoids omitting point clouds belonging to adjacent / target bins.

[0036] In other implementation scenarios, the point cloud of the target bin area (i.e., the cropped point cloud) can also be obtained directly from the reference position of the target bin. The point cloud of the bin area includes the point clouds of the target bin and adjacent bins. In this case, the reference position of the target bin can be the position of the target bin in the telescopic fork coordinate system. Since the depth camera acquires the point cloud while facing the target bin, the acquired point cloud of the target bin is generally only the point cloud of the face of the target bin facing the depth camera. Therefore, the reference position can be represented by the center point of the face of the target bin. If the acquired point cloud of the target bin contains the point clouds of multiple faces of the target bin, the center point of the face with the most point clouds can be used to represent the reference position.

[0037] The coordinate system of the telescoping fork is as follows Figure 3 As shown, the direction in which the telescopic fork moves towards the material box is the x-axis of the telescopic fork coordinate system, the width of the material box is the y-axis, and the height of the material box is the z-axis. When all point clouds belonging to the material box are transformed into the telescopic fork coordinate system, the width of the material box can be obtained from the y-axis coordinate of the point cloud in the telescopic fork coordinate system, and the height of the material box can be obtained from the z-axis coordinate of the point cloud in the telescopic fork coordinate system.

[0038] In a specific implementation scenario, the user-provided bin position is in the world coordinate system. Therefore, it is necessary to transform the world coordinate system position to the telescopic fork coordinate system to obtain a reference position. Subsequent calculations are based on this reference position in the telescopic fork coordinate system. A camera coordinate system and a vehicle coordinate system are also provided, as shown below. Figure 3 As shown, the x-axis of the camera coordinate system is the same as the y-axis of the telescopic fork coordinate system, the y-axis of the camera coordinate system is the same as the z-axis of the telescopic fork coordinate system, and the z-axis of the camera coordinate system is the same as the x-axis of the telescopic fork coordinate system. The x-axis of the vehicle body coordinate system is the same as the y-axis of the telescopic fork coordinate system, the y-axis of the vehicle body coordinate system is opposite in sign to the x-axis of the telescopic fork coordinate system, and the z-axis of the vehicle body coordinate system is the same as the z-axis of the telescopic fork coordinate system.

[0039] The extrinsic parameter for transforming the telescopic fork coordinate system to the vehicle coordinate system is denoted as `fork_to_car`. The current vehicle pose, `car_pose`, can be obtained from the odometer information on the vehicle. The position of the hopper is represented by the center point of the hopper surface, and the initial pose of the center point of the hopper surface in the world coordinate system is denoted as... The initial pose of the center point of the material box surface in the telescopic fork coordinate system is denoted as... The following transformation relationship exists:

[0040] Based on this transformation relationship, the initial pose of the center point of the material box surface in the telescopic fork coordinate system can be calculated:

[0041] Among these, the external parameters and the current attitude of the vehicle body are ensured to be reversible. Includes Pose x Pose y Pose z Pose yaw Among them, Pose x This represents the x-direction distance from the center point of the material box surface to the center of the telescopic fork coordinate system; it can also represent the x-axis coordinate of the center point of the material box surface in the telescopic fork coordinate system. Pose y This represents the y-direction distance between the center point of the material box surface and the center of the telescopic fork coordinate system; it can also represent the y-axis coordinate of the center point of the material box surface in the telescopic fork coordinate system. Pose z This represents the z-axis distance from the center point of the material box surface to the center of the telescopic fork coordinate system; it can also represent the z-axis coordinate of the center point of the material box surface in the telescopic fork coordinate system. Pose yaw This represents the angular offset of the center point of the material box surface from the center of the telescopic fork coordinate system. In a specific implementation scenario, considering the need to project the center point of the material box surface onto the XY plane of the telescopic fork coordinate system during subsequent applications, the z-axis coordinate of the center point of the material box surface in the telescopic fork coordinate system is not required. Therefore, the initial pose may not include a pose. z .

[0042] Based on the initial pose obtained from the above transformation, the reference position of the bin in the telescopic fork coordinate system can be obtained. In the telescopic fork coordinate system, the point cloud of the bin area is the point cloud located within the clipping range. The clipping range is extended along the positive or negative axis in the x-axis direction of the telescopic fork coordinate system with the x-axis coordinate of the target bin's reference position as the reference. The y-axis direction needs to consider the interference of adjacent bins and make a large-range selection. The z-axis direction is obtained by directly superimposing the bin height with the threshold.

[0043] In one embodiment, when the hopper is a shallow hopper, the cutting range corresponding to the shallow hopper can be determined by the following formula, where x, y, and z represent the coordinates of points located within the cutting range. By extracting the point cloud that satisfies the cutting range from the initial point cloud, the point cloud of the hopper area can be obtained.

[0044]

[0045] When the bin is a deep bin, the cutting range in the x-direction needs to be expanded in combination with the width of the storage space. The cutting range of the deep bin is as follows: extract the point cloud that meets the cutting range from the initial point cloud to obtain the point cloud of the bin area.

[0046]

[0047] in, This indicates the theoretical length of the hopper. This indicates the theoretical width of the hopper. This indicates the theoretical height of the hopper. This indicates the preset safe distance. This represents the threshold for superposition in the z-direction. The threshold representing the expansion in the x-direction. This represents the width of the storage space; the theoretical length, theoretical width, theoretical height, and threshold are all input by the user.

[0048] Clustering is performed on the cropped point cloud, using any existing clustering method. For example, a depth-first search can be used, starting point by point from the current valid point cloud (point cloud with coordinates) index list, checking its neighboring points in the four directions (up, down, left, right). If a neighboring point has not been visited, is valid, and meets the clustering conditions (by judging distance, etc.), it is added to the current clustering result. By continuously expanding neighboring points, a complete cluster is eventually formed. This process is repeated for all valid points until all points have been visited or assigned to a cluster, thus achieving the clustering and segmentation of the point cloud.

[0049] In one implementation scenario, the first point cloud of the target bin and the second point cloud of the adjacent bins can be obtained directly based on the above clustering results.

[0050] In other implementation scenarios, a judgment result can also be obtained, which indicates whether adjacent bins and / or the target bin actually exist. Combining the clustering result and the judgment result, a first point cloud of the target bin and a second point cloud of adjacent bins are obtained. Specifically, the judgment result may include whether adjacent bins and / or the target bin actually exist, and the number of adjacent bins that actually exist. Therefore, the number of existing bins can be determined based on the judgment result. A number of cluster blocks are selected from the clustering result. Based on the point clouds in the cluster blocks, the first point cloud and the second point cloud are obtained. The process of obtaining the judgment result is described below and will not be described here.

[0051] For example, if the determination result is that the target bin exists, and the adjacent bins on the first and second sides of the target bin also exist (the first and second sides being opposite sides, such as the left and right sides of the target bin), then the three clusters with the most points in the clustering results are retained. The relationship between the y-values ​​of the point clouds in these three clusters and 0 is determined. If all are greater than 0, the cluster is determined to be the left bin; if the y-values ​​of all points in the cluster are less than 0, the cluster is determined to be the right bin. For the remaining cluster, the product of the y-values ​​of its left and right boundary points is calculated; if it is less than 0, it is determined to be the target bin. In this embodiment, the target bin is the center point, and the left side of the target bin is the positive y-axis. Therefore, when the y-values ​​of the point clouds belonging to an adjacent bin are all greater than 0, the adjacent bin is the left bin; when the y-values ​​of the point clouds belonging to an adjacent bin are all less than 0, the adjacent bin is the right bin. In other embodiments, the right side of the target bin can be taken as the positive y-axis. In this case, if the y-values ​​of the point clouds belonging to the adjacent bins are all less than 0, the adjacent bin is the left bin; if the y-values ​​of the point clouds belonging to the adjacent bins are all greater than 0, the adjacent bin is the right bin.

[0052] For example, if the result is that the target bin exists, the bin on the first side (left side) of the target bin also exists, and the bin on the second side (right side) of the target bin does not exist, then two cluster blocks with more points are retained. When the y-values ​​of the point clouds in the cluster block are all greater than 0, the cluster block is determined to be the left bin. The remaining cluster block is used to calculate the product of the y-values ​​of its left boundary point and the y-values ​​of its right boundary point. If the product is less than 0, it is determined to be the target bin.

[0053] For example, if the result is that the target bin exists, the bin on the first side (left side) of the target bin does not exist, and the bin on the second side (right side) of the target bin exists, and the y-values ​​of the point cloud in the cluster block are all less than 0, then the cluster block is determined to be an adjacent bin. The remaining cluster block calculates the product of the y-values ​​of its left boundary point and the y-values ​​of its right boundary point. If it is less than 0, then it is determined to be the target bin.

[0054] For example, if the result is that the target bin exists, but the bins on the first side (left side) and the second side (right side) of the target bin do not exist, then only one cluster block with the most points is retained, and the product of the y-value of its left boundary point and the y-value of its right boundary point is calculated. If it is less than 0, then it is determined to be the target bin.

[0055] By combining the judgment results, it can be determined that the clustering result belongs to the first point cloud of the target bin and the second point cloud of the adjacent bin.

[0056] S22: Based on the first point cloud, determine the safe picking area of ​​the target bin; and based on the second point cloud, determine the edge area of ​​adjacent objects.

[0057] In one embodiment, the retrieval safety zone is formed by expanding outward from the area where the target bin is located. The retrieval safety zone can be of any shape, and the expansion area can be determined as needed. In a specific implementation scenario, the retrieval safety zone can have the same shape as the target bin. For example, if the target bin is rectangular, the retrieval safety zone is also rectangular. The range of the retrieval safety zone can be expanded as needed, as long as it completely covers the area where the target bin is located. In one implementation scenario, this application first uses a first point cloud to fit the target surface of the target bin and calculates the target center point and the actual orientation angle of the target surface; the target surface is the surface of the target bin that occupies the largest area within the field of view of the depth camera. Then, based on the target center point, the actual orientation angle, the size of the target bin, and the extension length of the telescopic fork for picking up the target bin, the four vertices of the retrieval safety zone are determined; based on the four vertices, the retrieval safety zone is obtained. The retrieval safety zone determined at this time takes into account the actual pose of the target bin.

[0058] The edge regions of adjacent objects can be represented by the edge segments of the adjacent objects. Since the edge segments of adjacent objects intersect with the safe picking area of ​​the target bin, the adjacent objects will affect the picking of the target bin; therefore, the edge regions can be represented by the edge segments of adjacent objects. The edge segments of adjacent bins can include one or more.

[0059] In one embodiment, the edge segment can be directly determined by the coordinates of each second point cloud; or, the two endpoints of the edge segment can be determined based on the coordinates of each second point cloud and the size of adjacent objects, and the edge segment can be obtained by fitting the coordinates of the two endpoints.

[0060] S23: Based on the intersection of the edge area and the safe picking area, determine whether the target bin can be directly picked up by forklift.

[0061] In one embodiment, when the presence of an adjacent object is detected, it can be determined whether there is an intersection between the edge area of ​​the adjacent object and the safe picking area. If there is no intersection between the edge area and the safe picking area, it is determined that the target box can be directly picked up. If there is an intersection between the edge area and the safe picking area, it is determined that the target box cannot be directly picked up. At this time, a prompt message can be issued to prompt the manager to move the adjacent box or the target box. If the adjacent object is an adjacent box, it can also be further determined based on the positional relationship between the target box and the adjacent box whether the adjacent box can be moved before picking up the target box.

[0062] In one implementation scenario, the neighboring object is an adjacent bin. The edge region of the adjacent bin includes the edge segments of the adjacent bin, which include the edge segment closest to the target bin. This edge segment can be one or multiple segments. For example... Figure 4As shown, the safe picking area of ​​the target bin is rectangular. The edge segments of adjacent bins consist of two line segments. By solving the equations of the four line segments of the rectangle and the equations of the two line segments simultaneously, we can use the intersection point method to determine whether the coordinates of the intersection point are within the domain of the two line segments. If there is an intersection, it indicates that the adjacent bin has encroached on the safe picking area of ​​the target bin; otherwise, it indicates that there is no intersection, meaning that the adjacent bin has not encroached on the safe picking area of ​​the target bin. It is understandable that only when all line segments in the rectangle have no intersection with all edge segments of the adjacent bins can it be considered that the adjacent bins will not interfere with the picking of the target bin.

[0063] Continue to refer to Figure 4 , Figure 4 Several common positional relationships between adjacent bins and the target bin are listed. Figure 4 (a), (b), and (c) represent cases where the target bin and the adjacent bin are located on the same shelf in the same row, for example, both the target bin and the adjacent bin are shallow bins. Figure 4 In (a), the two edge segments of adjacent bins do not intersect with the safe picking area of ​​the target bin, so normal picking is allowed. Figure 4 In cases (b) and (c), the edge segments of adjacent bins intersect with the safe retrieval area of ​​the target bin, making direct or indirect retrieval currently impossible, requiring manual intervention. Similarly, if both the target bin and adjacent bins are deep bins, the judgment logic is similar.

[0064] Figure 4 (d), (e), and (f) refer to cases where the target bin and the adjacent bin are located in different rows on the same shelf level. For example, the target bin is a deep bin, and the adjacent bin is a shallow bin. Figure 4 In (d), the two edge segments of adjacent bins do not intersect with the safe picking area of ​​the target bin, so normal picking is allowed. Figure 4In (e) and (f), the edge segments of adjacent bins intersect with the safe retrieval area of ​​the target bin, making direct retrieval impossible. Essentially, the adjacent bin on the right rear side of the target bin obstructs retrieval, although there may be a spatial distance between them. Removing the adjacent bin on the right rear side of the target bin before retrieving the target bin becomes a potential solution (secondary retrieval). Therefore, in one implementation scenario, after determining that the target bin cannot be directly retrieved, it is possible to further determine whether the target bin is located in the deep shelf position and whether the adjacent bin is located in the shallow shelf position. Specifically, in the same direction, the distance between the deep shelf position and the forklift device is greater than the distance between the shallow shelf position and the forklift device. The forklift device is used to retrieve both the target bin and the adjacent bin. In response to the target bin being located in the deep shelf position, the adjacent bin being located in the shallow shelf position, and the adjacent bin meeting the retrieval conditions, it is determined that the adjacent bin can be moved before retrieving the target bin. The condition for picking up goods is that the edge area of ​​the adjacent bins does not intersect with the safe picking area of ​​the adjacent bins. The method of judgment is the same as whether the edge area of ​​the target bin intersects with the safe picking area of ​​the adjacent bins.

[0065] Specifically, the AGV moves to the designated storage location preparation point corresponding to the right rear bin and begins the identification and verification process. It's important to note that there may still be bins around the adjacent bin. If the retrieval conditions are not met, manual intervention is initiated immediately. If the retrieval conditions are met, the adjacent bin is temporarily stored in a storage compartment. The AGV then moves back to the target bin's storage location preparation point, and identification is restarted. If the current status meets the retrieval conditions, the target bin is retrieved and placed in the storage compartment. Finally, the temporarily stored bin is returned to its original storage location, thus achieving safe retrieval.

[0066] The above method acquires the first point cloud of the target bin and the second point cloud of adjacent objects. Based on the first point cloud, the safe picking area of ​​the target bin is determined, which is formed by expanding outward from the area where the target bin is located. Based on the second point cloud, the edge areas of adjacent objects are determined. Based on the intersection of the edge areas and the safe picking area, it is determined whether the target bin can be directly picked up. Since the edge areas of adjacent objects are determined by the point clouds of adjacent bins, information such as the pose and size of adjacent objects are taken into account during this process, resulting in highly accurate edge areas. Combining the intersection of the edge areas and the safe picking area, it can accurately determine whether the target bin can be directly picked up, which to a certain extent avoids affecting adjacent objects during the picking up of the target bin and also avoids the impact of the tilt of adjacent objects on the picking up of the target bin, thus avoiding the safety hazards of picking up the target bin caused by the tilt of the bin.

[0067] If it is determined that the target bin can be directly picked up, the actual pose of the target bin can be determined based on the first point cloud. Using the actual pose, the moving distance of the picking device and the rotation angle of the telescopic fork on the picking device can be determined, so that the picking device moves to the target position according to the moving distance and the telescopic fork rotates to the target direction according to the rotation angle before picking up the target bin.

[0068] In one implementation, the target surface of the target bin is first fitted based on a first point cloud. The target surface is the surface of the target bin that occupies the largest area within the field of view of the depth camera. Specifically, RANSAC (Random Sample Consensus Algorithm) can be used to fit the target surface, calculate the normal vector of the target surface, and obtain the actual orientation angle and the target center point of the target surface based on the normal vector information. The actual orientation angle and the target center point of the target surface are used as the actual pose of the target bin. The actual orientation angle is used as the rotation angle of the telescopic fork. After rotation, the target direction of the telescopic fork is consistent with the target bin. The movement distance is calculated based on the coordinates of the target center point and the actual orientation angle.

[0069] For example, such as Figure 5 As shown, Figure 5 This is a schematic diagram of one embodiment of the moving distance of the fork-lifting device and the rotation angle of the telescopic fork on the fork-lifting device in this application. Figure 5 This is achieved by projecting the target material box onto the XY plane of the telescopic fork coordinate system. The positive y-axis of the telescopic fork coordinate system points to the left, the positive x-axis points forward, and the origin of the telescopic fork coordinate system is point O. When the target material box shifts to the left, the actual orientation angle is recorded as yaw0, where yaw0 is greater than 0. The coordinates of the target center point are marked as [x0, y0], and the distance moved is... It is obtained through the following formula.

[0070]

[0071] If the distance moved If the value is greater than 0, the forklift device is moved to the left by that distance; if the distance is less than 0, the forklift device is moved to the left by that distance. If the value is less than 0, the forklift device moves that distance to the right.

[0072] When the target bin shifts to the right, the actual orientation angle is yaw1, where yaw1 is less than 0. The target center point is marked as [x1, y1], and the distance traveled is... It is obtained through the following formula.

[0073]

[0074] If the distance moved If the value is greater than 0, the forklift device is moved to the left by that distance; if the distance is less than 0, the forklift device is moved to the left by that distance. If the value is less than 0, the forklift device moves that distance to the right.

[0075] The rotation angle of the telescopic fork is the actual orientation angle of the target hopper.

[0076] Please see Figure 6 , Figure 6 This is a flowchart illustrating an embodiment of determining the edge region of adjacent objects based on a second point cloud in this application. This embodiment uses adjacent material bins as an example to determine the edge region of adjacent objects based on the second point cloud, including: S621: Using the second point cloud, fit at least one face of the adjacent bins.

[0077] Since the depth camera is not directly facing the adjacent bins, considering its positional relationship with the adjacent bins, the point cloud imaging may show point distributions on both the front and side of the adjacent bins. Therefore, the second point cloud is generally composed of the point cloud from the front and the point cloud from the side of the bin, presenting a vertical relationship. However, the angle at which the adjacent bins are placed will affect the number of point clouds on the front and side. At this time, the edge segments on the front and the edge segments on the side may both affect the picking of the target bin, so two edge segments can be identified.

[0078] In an implementation scenario, such as Figure 7 As shown, Figure 7 This is a schematic diagram showing the projection of the adjacent bin and the target bin onto the XY plane of the telescopic fork coordinate system in this application. It is assumed that the adjacent bin is located to the right of the target bin, and both the adjacent bin and the target bin are rectangular, and their sizes can be the same. In this case, Figure 7 Line segments AB and BC in the diagram represent the edge segments of adjacent material boxes.

[0079] To accurately determine edge segments, at least one face of adjacent bins can be fitted using the second point cloud. In one implementation, RANSAC (Random Sample Consensus) can be used to fit the first principal plane (denoted as Plane A), which is usually the plane with the largest area or the most points (possibly the front or side). Then, it is determined whether the remaining number of points meets the requirements for a second fitting. If it does, RANSAC is used again to fit the point set outside the principal plane in the previous step to obtain a second plane (denoted as Plane B), which is usually approximately perpendicular to Plane A. If a second plane is not found, that is, the remaining number of points after the first plane fitting does not meet the fitting conditions or the orientation of Plane B cannot satisfy the approximately perpendicular relationship with Plane A, then only Plane A is used as the point cloud of the front of the adjacent bin. If Plane A and Plane B are extracted, the indexes of the point clouds that meet the requirements when fitting the plane are saved simultaneously. In order to distinguish between the front and the side, the orientation angle of Plane A and Plane B is generally defined as the angle between the normal vector of the face along the positive x-axis of the stretching fork coordinate system and the x-axis. The absolute value of the angle is used to determine whether the fitted face belongs to the front or the side of the material box. Generally speaking, the absolute value of the angle between the normal vector of the front of the adjacent material box and the positive x-axis is closer to 0 degrees, and the absolute value of the angle between the normal vector of the side of the adjacent material box and the positive x-axis is closer to 90 degrees.

[0080] S622: Get the third point cloud belonging to the face.

[0081] In one embodiment, the point cloud belonging to the fitted surface is obtained and used as the third point cloud.

[0082] S623: Use the third point cloud of the surface to determine the coordinates of the two endpoints of the surface boundary.

[0083] S624: Use the coordinates of the two endpoints to fit the edge line segment.

[0084] In one embodiment, since point clouds have coordinates, endpoints can be directly determined based on point cloud coordinates.

[0085] In another embodiment, to avoid the point cloud missing at the endpoints, which would lead to low accuracy of the determined endpoints, the third point cloud of the surface can be used to determine the boundary center point of the surface boundary; then, based on the target orientation angle of the surface, the boundary center point, and the length of the adjacent bins, the coordinates of the two endpoints can be calculated.

[0086] Understandably, regardless of whether the fitted result is one surface or two surfaces, these two surfaces share a common boundary. Therefore, the boundary center point can be determined using the third point cloud of any surface. In one implementation scenario, the third point cloud of the surface with the most point clouds is obtained. The total number of rows and columns corresponding to the third point cloud is counted. Each column is traversed from left to right until the column index with more than half the total number of rows is obtained. The center point of the left boundary of the material box is calculated based on this column. Then, each column is traversed from right to left until the column index with more than half the total number of rows is obtained. The center point of the right boundary of the material box is calculated based on this column. Only one of the right or left boundary center points needs to be obtained. Based on the positional relationship between the adjacent material box and the target material box, if the adjacent material box is to the left of the target material box, its right boundary center point is obtained; if the adjacent material box is to the right of the target material box, its left boundary center point is obtained.

[0087] Furthermore, the coordinates of the two endpoints are calculated using geometric relationships based on the target orientation angle of the surface, the boundary center point, and the dimensions of adjacent bins.

[0088] When a surface is fitted, the normal vector of the surface is calculated using the point cloud belonging to that surface, and the orientation angle of the surface is obtained. At this time, the orientation angle of the surface is the target orientation angle.

[0089] When two surfaces are obtained through fitting (the first surface and the second surface, respectively), the normal vectors of the first surface and the second surface are calculated to obtain the first orientation angle of the first surface and the second orientation angle of the second surface. The second orientation angle is converted into a third orientation angle, where the third orientation angle has the same orientation as the first orientation angle. The first orientation angle and the third orientation angle are weighted to obtain the target orientation angle.

[0090] In one embodiment, the ratio of the first number of point clouds belonging to the first face to the total number is used as the first weight of the first orientation angle, and the ratio of the second number of point clouds belonging to the second face to the total number is used as the second weight of the third orientation angle; wherein, the total number is equal to the sum of the first number and the second number; using the first weight and the second weight, the first orientation angle and the third orientation angle are weighted and summed to obtain the target orientation angle.

[0091] For example, such as Figure 7 As shown, the side orientation angle is obtained by fitting. and the positive orientation angle According to the following formula, the sides of adjacent bins should be oriented at an angle... Turn to positive orientation angle .

[0092]

[0093] The total number of point clouds belonging to the front of adjacent bins is obtained by summing the first number of point clouds belonging to the sides of adjacent bins. The ratio of the first number to the total number is used as the first weight, and the ratio of the second number to the total number is used as the second weight. The first weight is then compared with the frontal orientation angle of the adjacent bins. Multiply to obtain the first product; then multiply the second weight by the positive orientation angle. Multiply them to get the second product. Summing the first and second products gives the target's orientation angle.

[0094] In an implementation scenario, such as Figure 7 As shown, the adjacent bin is located to the right of the target bin. The center point B of the left boundary of the adjacent bin is obtained. ), Target orientation angle of adjacent bins Length L of adjacent bins AB According to the formula , The coordinates of endpoint A can be calculated. The coordinates of endpoint A are ( The projection of the center point of the left boundary onto the xy-plane coincides with one endpoint of the edge segment. Therefore, the x and y coordinates of the center point of the left boundary are the same as the x and y coordinates of the endpoint of the other edge segment. By combining these coordinates, the coordinates of the two endpoints are obtained. Based on these coordinates, the edge segment AB is fitted. Similarly, segment BC is calculated using the same method. At this point, the length of the adjacent bins can be changed to the width of the adjacent bins.

[0095] Since the target bin and / or adjacent bins may be moved due to human error or other reasons after a picking instruction is given to the forklift device, and the forklift device will still execute the picking process, in order to avoid the forklift device performing invalid operations, in one embodiment, this application further includes verifying whether there are bins in the storage locations where the target bin and adjacent bins are located before acquiring the first point cloud of the target bin and the second point cloud of the adjacent bins. Specifically: based on the initial point cloud acquired by the depth camera, it is determined whether the target bin and adjacent bins exist, and a judgment result is obtained; in response to the judgment result that both the target bin and adjacent bins exist, the first point cloud of the target bin and the second point cloud of the adjacent bins and subsequent steps are executed; in response to the judgment result that the target bin exists and the adjacent bins do not exist, the target picking method of the target bin is determined to be direct picking, and the first point cloud of the target bin is acquired, and the actual pose of the target bin is determined based on the first point cloud so that the forklift device picks up the bin based on the actual pose, wherein the calculation of the actual pose is described above; in response to the judgment result that the target bin does not exist, the picking task ends.

[0096] In one implementation scenario, determining the existence of a target bin and adjacent bins based on an initial point cloud acquired by a depth camera includes: determining a first preset search range and a second preset search range based on the reference position and reference size of the target bin; wherein the second preset search range is larger than the first preset search range, and the second preset search range should include the search range of the target bin and at least some of the adjacent bins; acquiring a first initial point cloud located within the first preset search range and a second initial point cloud located within the second preset search range; determining the existence of the target bin in response to the number of the first initial point clouds being greater than a first preset number; determining the existence of adjacent bins in response to the number of the second initial point clouds being greater than a second preset number; wherein the second preset number is greater than the first preset number.

[0097] In this embodiment, the first preset search range can be understood as including only the area where the target bin is located, and the first preset quantity can be set to the minimum quantity when the target bin exists. Therefore, when the first initial point cloud within the first preset search range is greater than the first preset quantity, the existence of the target bin can be determined. The second preset quantity can be set to the maximum quantity when the target bin exists. Therefore, as long as the second initial point cloud within the second preset search range is greater than the second preset quantity, the existence of adjacent bins can be determined. Simultaneously, the positional relationship between adjacent bins and the target bin can also be determined based on the coordinates of the point cloud.

[0098] In other implementation scenarios, a first preset search range can be determined based on the reference position and reference dimensions of the target bin; a second preset search range can be determined based on the reference positions and reference dimensions of adjacent bins. Specifically, the area where the target bin is located can be obtained based on its reference position and reference dimensions, and this area can be expanded by a certain range to obtain the first preset search range. Similarly, the area where adjacent bins are located can be obtained based on their reference positions and reference dimensions, and this area can be expanded by a certain range to obtain the second preset search range.

[0099] In a specific implementation scenario, if the target bin is in a shallow storage location, such as Figure 8 As shown, Figure 8 This is a schematic diagram illustrating one embodiment of the positional relationship between the target bin and adjacent bins in this application. The first preset search range of the target bin, the second preset search range of the bin adjacent to the left of the target bin, and the second preset search range of the bin adjacent to the right of the target bin can be determined using the following formulas. The target bin and adjacent bins can have the same dimensions.

[0100]

[0101]

[0102]

[0103] If the target bin is in a deep storage location, such as Figure 9 As shown, Figure 9 This is a schematic diagram illustrating another embodiment of the positional relationship between the target bin and adjacent bins in this application. The location of the target bin, the location behind the target bin, the deep and shallow locations to the left of the target bin, and the deep and shallow locations to the right of the target bin are each determined to identify whether a bin exists in these locations.

[0104] First, it's necessary to determine if there's a storage location behind the target storage bin. If a storage bin exists, blocking the retrieval of the target bin, it can be directly determined that the target storage bin cannot be retrieved directly. The second preset search range for adjacent storage bins behind the target storage bin can be determined using the following formula.

[0105]

[0106] If no bin is found within the second preset search range of the adjacent bin behind the target bin, further determine whether the target bin exists. The determination method is the same as that for the target bin being in a shallow storage location. If the target bin exists, determine the second preset search range of the adjacent bin to the left of the target bin according to the following formula to determine whether a bin exists in the deep or shallow storage location to the left of the target bin.

[0107]

[0108] The second preset search range of the adjacent bins to the right of the target bin is determined according to the following formula, so as to determine whether there are bins in the deep and shallow storage locations to the right of the target bin.

[0109]

[0110] in, , , This represents the coordinates of points within the first preset search range. , , This indicates the coordinates of a point within the second preset search range of the adjacent bin on the left. , , This indicates the coordinates of a point within the second preset search range of the adjacent bin on the right. , , This represents the coordinates of a point within the second preset search range of the adjacent material bin. This represents the additional threshold in the x-direction. , and The length, width, and height of the target bin are represented in that order. This indicates the configured fixed safety distance. Indicates the width of the storage space.

[0111] In one embodiment, the safe picking area of ​​the target bin can be determined as follows: First, the target surface of the target bin is fitted using a first point cloud, and the target center point and the actual orientation angle of the target surface are calculated. The target surface is the surface of the target bin that occupies the largest area within the field of view of the depth camera, and the actual orientation angle represents the angle between the actual orientation of the target surface and the first axis of the telescopic fork coordinate system. Specifically, the target surface can be fitted based on RANSAC+CERES, the normal vector of the target surface can be obtained, and the actual orientation angle can be solved based on the normal vector information. The actual orientation angle can be defined as positive to the left and negative to the right, and the actual orientation angle can be represented by the angle between the normal vector and the x-axis of the telescopic fork coordinate system. Obtain the row and column indices corresponding to the point cloud belonging to the target surface; count the total number of rows and columns of all point clouds, traverse each column from left to right until a column index with more than half the number of points in that column direction is obtained, and calculate the left boundary center point of the target surface based on that column. Then traverse each column from right to left until a column index with more than half the number of points in that column direction is obtained, and calculate the right boundary center point of the target surface based on that column. The target center point can be obtained based on the left and right boundary center points; for example, the midpoint between the left and right boundary center points can be used as the target center point. In other implementations, the interior points output during the plane fitting process can also be obtained, and the average coordinates of the interior points can be calculated to obtain the target center point.

[0112] Further, the intersection point of the target center point with the second axis of the telescopic fork coordinate system in the actual orientation is obtained; the point at a distance of a first preset distance from the intersection point in the actual orientation is taken as the first center point of the first boundary line of the retrieval safety area; and the point at a distance of a second preset distance from the intersection point in the actual orientation is taken as the second center point of the second boundary line of the retrieval safety area; wherein, the first boundary line and the second boundary line are perpendicular to the actual orientation, and the first boundary line and the second boundary line are parallel; the second preset distance is greater than the sum of the first preset distance and the length of the target bin; two points at a distance of the target distance from the first center point of the first boundary line are determined on the first boundary line and taken as the two endpoints of the first boundary line; and two points at a distance of the target distance from the second center point of the second boundary line are determined on the second boundary line and taken as the two endpoints of the second boundary line; wherein, the target distance is the sum of 1 / 2 width of the target bin and a third preset distance; based on the two endpoints of the first boundary line and the two endpoints of the second boundary line, the retrieval safety area is obtained.

[0113] The first axis can be the x-axis, the second axis can be the y-axis, and the first preset distance can be half the length of the telescopic fork. Along the actual orientation and towards the target bin, a point half the length of the telescopic fork is obtained, which is the first center point. The second preset distance can be the sum of the distance from the intersection point to the target center point, the length of the target bin, and a fourth preset distance. Along the actual orientation and towards the target bin, a point the second preset distance is obtained, which is the second center point. The target distance can be the sum of half the width of the target bin and a third preset distance. The third and fourth preset distances can be set as needed. After obtaining the four endpoints, they can be directly connected to obtain the safe picking area; alternatively, four line segments can be fitted based on the four endpoints, and then connected to obtain the safe picking area.

[0114] In a specific implementation scenario, such as Figure 10 As shown, point o is the origin of the telescopic fork coordinate system, point c is the target center point of the target surface of the target box in the telescopic fork coordinate system, yaw is the actual orientation angle of the target surface, point b is the projection point of the target center point c on the y-axis direction of the telescopic fork coordinate system, point a is the intersection of the target center point c in the actual orientation and the y-axis, that is, point a is the intersection of the trajectory of point c along the actual orientation angle yaw and the y-axis, the distance oa is the lateral movement distance of the fork lifting device, point o will gradually coincide with point a as the fork lifting device moves laterally, and then the telescopic fork is controlled to rotate according to the actual orientation angle so that the direction of the telescopic fork is consistent with the actual orientation of the target box, and then the fork arm is controlled to complete the picking along this direction.

[0115] Figure 10 In the diagram, the length of ad is the first preset distance, which can be set to half the length of the telescopic fork; point d is the first center point of the first boundary line, and the line segment... This is the first boundary line. The coordinates of point a are ( ), the coordinates of point d are ( ), Due to line segments Perpendicular to the ad direction, therefore Angle with the x-axis direction It can be easily calculated that when yaw < 0, ,Right now >0, Along the positive x-axis, Along the negative x-axis; when yaw > 0, ,Right now <0, Along the negative x-axis, Along the positive x-axis.

[0116] by Figure 10 For example, in the case of yaw > 0, >0, Along the positive x-axis, Along the negative x-axis, and The lengths are respectively from and express, coordinates ( The following can be calculated:

[0117] coordinates ( The following can be calculated:

[0118] Similarly, based on the coordinates of point a, the length of ae (i.e., the second preset distance), and the actual orientation angle, the coordinates of point e (i.e., the second center point) can be calculated, and the following can be calculated simultaneously: and The coordinates.

[0119] When the four vertices After the coordinates are solved, the four line segments can be calculated based on the vertex distribution. The equation is expressed as follows: For example, it is known coordinate( )and coordinate( ), expressed using the point-slope form, when ≠ When, slope Represented as:

[0120] The equation of a line segment (with domain constraints) can be expressed as:

[0121] when = When the line segment equation is given, it can be expressed as:

[0122] Similarly, the equations for the other three line segments can be derived, and the safe pickup area can be determined based on the derived equations for the four line segments.

[0123] In one embodiment, after fitting the target surface of the target bin using the first point cloud, the actual size of the target bin can be calculated based on the target point cloud belonging to the target surface; wherein the actual size includes at least one of the actual width and the actual height; the actual size is compared with the reference size to verify whether the target bin is the bin to be picked up by the forklift. If the difference between the actual size and the reference size is within the error range, the target bin is determined to be the bin to be picked up by the forklift, and the subsequent picking process is executed; otherwise, the subsequent picking process is not executed, and a prompt is issued.

[0124] However, considering that the target surface for fitting may contain bending regions, such as Figure 11 As shown, the extracted target point cloud belonging to the target surface may contain errors, leading to low accuracy in the calculated actual size. Therefore, the point cloud of the bending area can be filtered as noise. That is, after fitting the target surface of the target bin, first determine whether each target point cloud in the target surface belongs to noise, and remove the target point clouds belonging to noise to obtain a new target point cloud belonging to the target surface; based on the new target point cloud, calculate the actual size of the target bin; compare the actual size with the reference size to verify whether the target bin is the bin to be picked up by the forklift.

[0125] In a specific implementation scenario, the actual size of the target bin can be calculated based on the target point cloud. First, the center points of the left and right boundaries of the target surface can be determined as described above. The actual width of the target bin is then obtained based on the distance between these two center points. For the actual height of the target bin, each row of the point cloud can be traversed from top to bottom until the first row index with more than half the total number of columns is found. The average z-axis coordinate of all point clouds corresponding to that row index is then calculated to obtain the actual height.

[0126] In a specific implementation scenario, for any target point cloud in the target surface, it can be determined whether the target angle is within a preset angle range and / or whether the target distance is within a preset distance range, so as to determine whether the target point cloud belongs to noise; where the target angle is the angle between the first normal vector of the target surface and the second normal vector of the target point cloud, and the target distance is the distance from the target point cloud to the target surface.

[0127] Specifically, for any target point cloud Due to the ordered nature of point clouds, points within a 5×5 neighborhood can be extracted. To form a point set Calculate the center point of the neighborhood points :

[0128] Constructing the covariance matrix :

[0129] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain eigenvalues ​​and their corresponding eigenvectors. The eigenvector corresponding to the smallest eigenvalue is the local normal vector of the target point cloud. After normalization, the second normal vector of the target point cloud is obtained.

[0130] The first normal vector of the target surface has already been determined and will not be repeated here. The angle between the first normal vector of the target surface and the second normal vector of the target point cloud is calculated to obtain the target angle. Then, the distance from the target point cloud to the target surface is calculated using the point-to-surface distance formula to obtain the target distance.

[0131] In one implementation scenario, the target point cloud is determined to be noise simply because the target angle is outside the preset angle range. Alternatively, the target point cloud is also determined to be noise simply because the target distance is outside the preset distance range. In another implementation scenario, the target point cloud is determined to be noise only when both the target angle and the target distance are outside the preset angle range.

[0132] It is understood that in the above method of specific implementation, the order in which each step is written does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0133] In one specific embodiment, the overall process of this application is as follows: after the forklift device reaches the identification preparation point, it lifts and rotates the clamping device to the designated storage location, identifies the material box placed on the shelf through the depth camera, calculates the center point and orientation of the material box surface, the forklift device adjusts the path according to the identification result and controls the rotation of the telescopic fork, and uses the fork arm and flip hook to complete the picking.

[0134] The forklift device first obtains the initial pose of the target surface of the target box in the world coordinate system. Based on the initial pose of the target surface in the world coordinate system, the current pose of the vehicle body in the world coordinate system, and the extrinsic parameters of the telescopic fork coordinate system to the vehicle body coordinate system, the initial pose of the target surface in the world coordinate system is converted into the target surface, thus obtaining the initial pose of the target surface in the telescopic fork coordinate system. The reference position of the target box can be obtained from the initial pose of the target surface in the telescopic fork coordinate system.

[0135] In the bin recognition scenario, bins are densely stored, meaning each shelf level corresponds to multiple storage locations, and each storage location can hold one bin. When recognizing bins, considering safe retrieval, it's necessary to verify the adjacent storage locations of the target bin. The first step is to determine if the target bin exists in the target storage location and whether there are any adjacent bins interfering with the target bin's movement. Specifically, based on the initial point cloud acquired by the depth camera, it's determined whether the target bin and adjacent bins exist. If the target bin exists but the adjacent bins do not, the first point cloud of the target bin is acquired, and its actual pose is determined based on this first point cloud, allowing the forklift device to retrieve the bin based on its actual pose. If the target bin does not exist, the retrieval task ends. If both the target bin and adjacent bins exist, it's further determined whether the adjacent bins will affect the retrieval of the target bin.

[0136] The initial point cloud acquired by the depth camera is cropped to obtain the point cloud of the material box region. The cropped point cloud of the material box region is clustered. The clustering results and the judgment results of whether the target material box and adjacent material boxes exist are combined to obtain the first point cloud of the target material box and the second point cloud of the adjacent material boxes.

[0137] The target surface of the target bin is fitted based on the first point cloud of the target bin. The actual orientation angle and target center point of the target surface are calculated, and noise points in the target surface are filtered out to obtain a new target point cloud belonging to the target surface. The actual size of the target bin is obtained from the new target point cloud. The actual size is compared with the reference size to verify whether the target bin is the bin to be picked up by the forklift. If the target bin is the bin to be picked up by the forklift, the moving distance of the forklift device and the rotation angle of the telescopic fork on the forklift device are determined based on the actual orientation angle and target center point. This ensures that the forklift device moves to the target position according to the moving distance and the telescopic fork rotates to the target direction according to the rotation angle before picking up the target bin.

[0138] Before the forklift device picks up the target bin, if the existence of an adjacent bin is confirmed, it is necessary to determine the safe picking area of ​​the target bin and whether the adjacent bin has encroached upon the safe picking area. If it has not encroached, the forklift device can be controlled to pick up the target bin; if it has encroached, a prompt message can be issued to the management personnel, or based on the positional relationship between the target bin and the adjacent bin, it can be determined whether the adjacent bin can be moved before picking up the target bin. The safe picking area of ​​the target bin is based on the first point cloud of the target bin. This first point cloud can be a point cloud directly obtained from clustering results, a point cloud belonging to the target surface of the target bin, or a point cloud remaining after noise filtering of the point cloud in the fitted target surface. Whether an adjacent bin has encroached upon the safe picking area can be determined by the intersection of the edge region of the adjacent bin and the safe picking area, based on the second point cloud of the adjacent bin. If the edge region intersects with the safe picking area, the adjacent bin is considered to have encroached upon the safe picking area; otherwise, it is considered that the adjacent bin has not encroached upon the safe picking area.

[0139] If the adjacent bins do not encroach on the safe picking area, the forklift device is controlled to pick up the target bin. The steps of the forklift device moving to the target position according to the travel distance and the telescopic fork rotating to the target direction according to the rotation angle can occur before or after determining whether the adjacent bins have encroached on the safe picking area.

[0140] In real-world scenarios, the object next to the target bin might be a shelf instead of a bin. Therefore, in other implementations, the point cloud of the shelf can also be extracted. In this case, it's necessary to determine whether the adjacent object is a shelf or a bin. This can be done based on the number and width of the point cloud data; the number and width of the shelf's point cloud are smaller than those of the bin. When determining whether a shelf intrudes into the target bin's safe retrieval area, the shelf's edge segments are calculated based on its point cloud data. The length and width information during this calculation can be based on the provided shelf length and width, similar to the calculation method when the object is a bin. If the shelf's edge segments intrude into the target bin's safe retrieval area, obstructing retrieval, a notification is sent to management for intervention.

[0141] Please see Figure 12 , Figure 12 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application.

[0142] Electronic device 120 includes a memory 121 and a processor 122, which are coupled to each other. The processor 122 executes program instructions stored in the memory 121 to implement the steps in any of the above method embodiments. In a specific implementation scenario, electronic device 120 may include, but is not limited to, a microcomputer or a server. Furthermore, electronic device 120 may also include mobile devices such as laptops and tablets, without limitation. Electronic device 120 may be mounted on a forklift device or issue control commands to the forklift device.

[0143] Specifically, processor 122 controls itself and memory 121 to implement the steps in any of the above method embodiments. Processor 122 may also be referred to as a CPU (Central Processing Unit). Processor 122 may be an integrated circuit chip with signal processing capabilities. Processor 122 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 122 may be implemented using integrated circuit chips.

[0144] Please see Figure 13 , Figure 13 This is a schematic diagram of the framework structure of one embodiment of the computer-readable storage medium of this application.

[0145] The computer-readable storage medium 130 stores program instructions 131 that can be executed by a processor, the program instructions 131 being used to implement the steps in any of the above method embodiments.

[0146] The storage media mentioned in this article include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media that can store program code.

[0147] This application also provides a computer program product comprising a computer program that, when executed by a processor, can implement the steps of the methods described in any of the foregoing embodiments. Specifically, the computer program product can be a software or program product containing a computer program, capable of running on a computing device or stored on any available medium.

[0148] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0149] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0150] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for retrieving goods from a material bin, characterized in that, include: Acquire the first point cloud of the target bin and the second point cloud of adjacent objects; Based on the first point cloud, the safe area for picking up the target bin is determined; Based on the second point cloud, the edge region of the adjacent object is determined, and the safe picking area is formed by expanding outward from the area where the target bin is located; Based on the intersection of the edge area and the safe picking area, it is determined whether the target bin can be directly picked up by forklift.

2. The method according to claim 1, characterized in that, The step of determining whether the target bin can be directly forked based on the intersection of the edge region and the safe picking area includes: In response to the fact that the edge region and the safe picking area do not intersect, it is determined that the target bin can be directly picked up by forklift; and / or, In response to the intersection of the edge area and the safe picking area, it is determined that the target bin cannot be directly picked up by forklift. If the adjacent object is an adjacent bin, it is determined, based on the positional relationship between the target bin and the adjacent bin, whether the adjacent bin can be moved before picking up the target bin.

3. The method according to claim 2, characterized in that, The step of determining whether the adjacent bins can be moved before the target bin is picked up based on the positional relationship between the target bin and the adjacent bins includes: Determine whether the target bin is located in the deep part of the shelf and whether the adjacent bin is located in the shallow part of the shelf; wherein, in the same direction, the distance between the deep part of the shelf and the forklift device is greater than the distance between the shallow part of the shelf and the forklift device, and the forklift device is used to pick up the target bin and the adjacent bin; In response to the target bin being located in a deep position on the shelf, the adjacent bin being located in a shallow position on the shelf, and the adjacent bin meeting the retrieval conditions, it is determined that the adjacent bin can be moved before the target bin can be picked up by forklift.

4. The method according to claim 2, characterized in that, If it is determined that the target bin can be directly forked, the method further includes: Based on the first point cloud, the actual pose of the target bin is determined; Using the actual position and orientation, the moving distance of the forklift device and the rotation angle of the telescopic fork on the forklift device are determined, so that the forklift device moves to the target position according to the moving distance and the telescopic fork rotates to the target direction according to the rotation angle to pick up the target material box.

5. The method according to claim 1, characterized in that, The adjacent objects are adjacent material boxes, and the edge region includes the edge line segments of the adjacent material boxes; The step of determining the edge regions of the adjacent objects based on the second point cloud includes: Using the second point cloud, at least one face of the adjacent bins is fitted; Obtain the third point cloud belonging to the surface; Using the third point cloud of the surface, determine the coordinates of the two endpoints of the boundary of the surface; The edge line segment is obtained by fitting the coordinates of the two endpoints.

6. The method according to claim 5, characterized in that, Determining the coordinates of the two endpoints of the boundary of the surface using the third point cloud of the surface includes: Using the third point cloud of the surface, determine the boundary center point of the surface's boundary; The coordinates of the two endpoints are calculated based on the target orientation angle of the surface, the center point of the boundary, and the dimensions of the adjacent bins.

7. The method according to claim 6, characterized in that, The at least one surface includes a first surface and a second surface; after fitting at least one surface of the adjacent bins, the method further includes: Calculate the normal vectors of the first face and the second face respectively to obtain the first orientation angle of the first face and the second orientation angle of the second face; The second orientation angle is converted into a third orientation angle, wherein the third orientation angle has the same orientation as the first orientation angle; The target orientation angle is obtained by weighting the first orientation angle and the third orientation angle.

8. The method according to claim 7, characterized in that, The step of weighting the first orientation angle and the third orientation angle to obtain the target orientation angle includes: The ratio of the first number of point clouds belonging to the first face to the total number is used as the first weight of the first orientation angle, and the ratio of the second number of point clouds belonging to the second face to the total number is used as the second weight of the third orientation angle; wherein, the total number is equal to the sum of the first number and the second number; Using the first weight and the second weight, the first orientation angle and the third orientation angle are weighted and summed to obtain the target orientation angle.

9. The method according to claim 1, characterized in that, The safe area for picking up goods is a rectangular area; The step of determining the safe picking area for the target bin based on the first point cloud includes: Using the first point cloud, the target surface of the target bin is fitted, and the target center point and the actual orientation angle of the target surface are calculated; the target surface is the surface of the target bin that occupies the largest area within the field of view of the depth camera, and the actual orientation angle represents the angle between the actual orientation of the target surface and the first axis of the telescopic fork coordinate system; Obtain the intersection point of the target center point with the second axis of the telescopic fork coordinate system in the actual orientation; The point at which the intersection point is at a first preset distance along the actual orientation is designated as the first center point of the first boundary line of the retrieval safety area; and the point at which the intersection point is at a second preset distance along the actual orientation is designated as the second center point of the second boundary line of the retrieval safety area; wherein the first boundary line and the second boundary line are perpendicular to the actual orientation, and the first boundary line and the second boundary line are parallel; the second preset distance is greater than the sum of the first preset distance and the length of the target bin; Two points are determined on the first boundary line with a target distance from the first center point of the first boundary line, serving as the two endpoints of the first boundary line; and two points are determined on the second boundary line with a target distance from the second center point of the second boundary line, serving as the two endpoints of the second boundary line; wherein, the target distance is the sum of 1 / 2 the width of the target hopper and a third preset distance; The safe picking area is obtained based on the two endpoints of the first boundary line and the two endpoints of the second boundary line.

10. The method according to claim 9, characterized in that, After fitting the target surface of the target hopper using the first point cloud, the method further includes: Determine whether each target point cloud in the target surface belongs to noise, remove the target point cloud that belongs to noise, and obtain a new target point cloud belonging to the target surface. Based on the new target point cloud, the actual size of the target bin is calculated; wherein the actual size includes at least one of the actual width and the actual height; Compare the actual dimensions with the reference dimensions to verify whether the target bin is the bin to be picked up by forks.

11. The method according to claim 10, characterized in that, The step of determining whether each target point cloud in the target surface belongs to noise includes: For any target point cloud in the target surface, determine whether the target angle is within a preset angle range and / or whether the target distance is within a preset distance range, so as to determine whether the target point cloud belongs to noise. Wherein, the target angle is the angle between the first normal vector of the target surface and the second normal vector of the target point cloud, and the target distance is the distance from the target point cloud to the target surface.

12. The method according to claim 1, characterized in that, The adjacent objects are adjacent material boxes. Before acquiring the first point cloud of the target material box and the second point cloud of the adjacent objects, the method further includes: Based on the initial point cloud acquired by the depth camera, it is determined whether the target bin and the adjacent bins exist, and the determination result is obtained. In response to the determination result that both the target bin and the adjacent bin exist, the steps of obtaining the first point cloud of the target bin and the second point cloud of the adjacent bin, and subsequent steps, are executed. In response to the judgment result that the target bin exists and the adjacent bin does not exist, the target picking method of the target bin is determined to be direct picking, and the first point cloud of the target bin is obtained. Based on the first point cloud, the actual pose of the target bin is determined so that the forklift device picks up the bin based on the actual pose. If the determination result indicates that the target bin does not exist, the picking task ends.

13. The method according to claim 12, characterized in that, The step of determining whether the target bin and the adjacent bins exist includes: Based on the reference position and reference size of the target bin, a first preset search range and a second preset search range are determined; wherein the second preset search range is larger than the first preset search range. Obtain a first initial point cloud located within the first preset search range and a second initial point cloud located within the second preset search range; In response to the first initial point cloud having a number greater than a first preset number, the existence of the target bin is determined; in response to the second initial point cloud having a number greater than a second preset number, the existence of the adjacent bin is determined; wherein the second preset number is greater than the first preset number.

14. The method according to claim 1, characterized in that, The acquisition of the first point cloud of the target bin and the second point cloud of adjacent objects includes: The initial point cloud acquired by the depth camera is cropped to obtain the cropped point cloud. Cluster the cropped point cloud to obtain the clustering results; By combining the clustering results and the judgment results, the first point cloud of the target bin and the second point cloud of the adjacent objects are obtained.

15. The method according to claim 14, characterized in that, The adjacent objects are adjacent bins. The process of combining the clustering results and the judgment results to obtain the first point cloud of the target bin and the second point cloud of the adjacent objects includes: Based on the judgment result, the number of existing material boxes is determined; Select the specified number of cluster blocks from the clustering results; Based on the point clouds in the clustered blocks of the specified number of bins, the first point cloud and the second point cloud of the adjacent bins are obtained.

16. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores program instructions and the processor executes the program instructions to implement the method according to any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the method of any one of claims 1 to 15.

Images