Alignment state confirmation method, control system and carrying equipment

Abstract

The embodiment of the invention discloses an alignment state confirmation method, a control system and carrying equipment. According to the main technical scheme, the method comprises the steps that a controller obtains target point clouds of a first stacking object and a second stacking object through a sensor; the controller determines a first pseudo graph corresponding to the first stacking object according to the target point cloud of the first stacking object; the controller determines a second pseudo graph corresponding to the second stacking object according to the target point cloud of the second stacking object; the controller determines relative pose data between the first stacking object and the second stacking object according to the first pseudo-graph and the second pseudo-graph; the controller compares the relative pose data with a threshold value to confirm the alignment state of the first stacked object and the second stacked object. The method and the device are used for controlling the first stacking object and the second stacking object to be aligned during stacking.

Description

Technical Field

[0001] The present application relates to the fields of warehousing logistics technology and machine vision, and in particular to an alignment status confirmation method, a control system and a handling device. Background Art

[0002] Systems that use handling equipment such as AGV (automated guided vehicle) have the advantages of being highly unmanned, automated, and intelligent, which improves production efficiency and operational levels for industries such as warehousing, manufacturing, and logistics. As one of the more typical scenarios, handling equipment is often responsible for the handling of various goods. In the process of handling, it is inevitable to stack goods. Goods are usually packaged in cartons, etc., or stored in cages, wooden boxes, plastic boxes, etc.

[0003] Considering the space utilization, the handling equipment will involve stacking objects such as cartons, cages, wooden boxes, etc. in the process of handling goods. Considering the stability, the handling equipment needs to accurately stack one stacking object on top of another stacking object. In this process, if the two stacking objects cannot be aligned, it may affect the safety of the operation. Summary of the invention

[0004] The present application provides an alignment state confirmation method, a control system and a handling device, which are used to control the alignment of a first stacking object and a second stacking object during stacking.

[0005] This application provides the following solutions:

[0006] According to a first aspect, a method for confirming an alignment state is provided, the method comprising: a controller acquiring a target point cloud of a first stacking object and a second stacking object through a sensor; the controller determining a first pseudo-image corresponding to the first stacking object based on the target point cloud of the first stacking object; the controller determining a second pseudo-image corresponding to the second stacking object based on the target point cloud of the second stacking object; the controller determining relative posture data between the first stacking object and the second stacking object based on the first pseudo-image and the second pseudo-image; the controller comparing the relative posture data with a threshold value to confirm the alignment state of the first stacking object and the second stacking object.

[0007] Optionally, the first pseudo image is a first grayscale image, and the second pseudo image is a second grayscale image.

[0008] Optionally, the method further comprises:

[0009] Before confirming the alignment state, the controller controls the transport device to transport the first stacked object to the preparation position to complete a pre-alignment action relative to the second stacked object.

[0010] Optionally, the controller determines the relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image, including:

[0011] The controller determines a boundary line of the first pseudo image;

[0012] The controller determines a target point of the first pseudo image according to a boundary line of the first pseudo image;

[0013] The controller determines a boundary line of the second pseudo image;

[0014] The controller determines a target point of the second pseudo image according to a boundary line of the second pseudo image;

[0015] The controller determines relative position data between the first stacking object and the second stacking object according to the boundary line of the first pseudo image and the target point of the first pseudo image, and the boundary line of the second pseudo image and the target point of the second pseudo image.

[0016] Optionally, the controller determines the target point of the first pseudo image according to the boundary line of the first pseudo image, including:

[0017] The controller determines the center line of the first pseudo image according to the third boundary line and the fifth boundary line of the first pseudo image; and uses the intersection of the center line of the first pseudo image and the first boundary line of the first pseudo image as the target point of the first pseudo image;

[0018] The controller determines the target point of the second pseudo image according to the boundary line of the second pseudo image, and the target point of the second pseudo image includes:

[0019] The controller determines the center line of the second pseudo image according to the fourth boundary line and the sixth boundary line of the second pseudo image; and uses the intersection of the center line of the second pseudo image and the second boundary line of the second pseudo image as the target point of the second pseudo image.

[0020] Optionally, the third boundary line and the fifth boundary line in the first pseudo image are parallel to each other;

[0021] The second boundary line and the fourth boundary line in the second pseudo image are parallel to each other;

[0022] The third boundary line in the first pseudo image is perpendicular to the first boundary line of the first pseudo image, and the fifth boundary line of the first pseudo image is perpendicular to the first boundary line of the first pseudo image;

[0023] The fourth boundary line in the second pseudo image is perpendicular to the second boundary line of the second pseudo image, and the sixth boundary line of the second pseudo image is perpendicular to the second boundary line of the second pseudo image.

[0024] Optionally, the target point of the first pseudo image is a midpoint of a first boundary line of the first pseudo image;

[0025] The target point of the second pseudo image is the midpoint of the second boundary line of the second pseudo image.

[0026] Optionally, the controller determines the relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image, including:

[0027] The controller determines a boundary line of the first pseudo image;

[0028] The controller determines a boundary point cloud of the first stacking object corresponding to a boundary line of the first pseudo image from the target point cloud of the first stacking object;

[0029] The controller determines a first target point of the first stacking object according to the boundary point cloud of the first stacking object;

[0030] The controller determines a boundary line of the second pseudo image;

[0031] The controller determines, from the target point cloud of the second stacked object, a boundary point cloud of the second stacked object corresponding to a boundary line of the second pseudo image;

[0032] The controller determines a second target point of the second stacking object according to the boundary point cloud of the second stacking object;

[0033] The controller determines a relative angle difference between the first stacking object and the second stacking object according to the boundary point cloud of the first stacking object and the boundary point cloud of the second stacking object;

[0034] The controller determines a relative angle difference between the first stacking object and the second stacking object according to the boundary point cloud of the first stacking object and the boundary point cloud of the second stacking object;

[0035] The controller determines relative position data between the first stacking object and the second stacking object according to a relative position difference and a relative angle difference between the first target point and the second target point.

[0036] Optionally, the controller determines the first target point of the first stacking object according to the boundary point cloud of the first stacking object, including:

[0037] The controller determines a first edge line of the first stacking object according to the first boundary point cloud;

[0038] The controller determines a third edge line of the first stacked object according to the third boundary point cloud;

[0039] The controller determines a fifth edge line of the first stacked object according to the fifth boundary point cloud;

[0040] The controller determines a first center line according to the third edge line and the fifth edge line;

[0041] The controller determines a first target point according to the first center line and the first edge line;

[0042] The controller determines a second target point of the second stacking object according to the boundary point cloud of the second stacking object, including:

[0043] The controller determines a second edge line of the second stacked object according to the second boundary point cloud;

[0044] The controller determines a fourth edge line of the second stacked object according to the fourth boundary point cloud;

[0045] The controller determines a sixth edge line of the second stacked object according to the sixth boundary point cloud;

[0046] The controller determines a second center line according to the second edge line and the sixth edge line;

[0047] The controller determines a second target point according to the second center line and the second edge line.

[0048] Optionally, the first target point is the intersection of the first center line and the first edge line; the second target point is the intersection of the second center line and the second edge line.

[0049] Optionally, the controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, including:

[0050] The controller determines a relative angle difference between the first stacking object and the second stacking object according to a first angle between the first edge line and the second edge line and the first angle.

[0051] Optionally, the controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, including:

[0052] The controller calculates a second angle between the first center line and the second center line, and determines a relative angle difference between the first stacking object and the second stacking object according to the second angle.

[0053] Optionally, the controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, including:

[0054] The controller calculates a second angle between the first center line and the second center line;

[0055] The controller calculates a first angle between the first edge line and the second edge line;

[0056] The controller obtains a relative angle difference between the first stacking object and the second stacking object by weighting the first angle and the second angle.

[0057] Optionally, the controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, including:

[0058] The controller determines a first angle value of a first stacked object;

[0059] The controller determines a second angle value of a second stacked object;

[0060] The controller determines a relative angle difference between the first stacking object and the second stacking object according to a difference between the first angle value and the second angle value.

[0061] Optionally, the controller determines a first angle value of the first stacked object, including:

[0062] The controller determines a first angle value of the first stacking object according to an angle of the first edge line; or

[0063] The controller determines a first angle value of the first stacking object according to an angle of the first center line; or

[0064] The controller obtains a first angle value of the first stacked object by weighting the angle of the first center line and the angle of the first edge line.

[0065] Optionally, the controller determines a second angle value of the second stacked object, including:

[0066] The controller determines a second angle value of the second stacking object according to the angle of the second edge line; or

[0067] The controller determines a second angle value of the second stacked object according to the angle of the second center line; or

[0068] The controller obtains a second angle value of the second stacked object by weighting the angle of the second center line and the angle of the second edge line.

[0069] Optionally, the controller determines a first pseudo image corresponding to the first stacking object according to the target point cloud of the first stacking object, including:

[0070] The controller projects the target point cloud of the first stacking object onto a horizontal plane to generate a first pseudo image;

[0071] The controller determines a second pseudo image corresponding to the second stacking object according to the target point cloud of the second stacking object, including:

[0072] The controller projects the target point cloud of the second stacking object onto a horizontal plane to generate a second pseudo image.

[0073] Optionally, the controller projects the target point cloud of the first stacking object onto a horizontal plane to generate a first pseudo image, comprising:

[0074] The controller projects the target point cloud of the first stacking object onto a horizontal plane using at least two projection resolutions to obtain pseudo images of at least two projection resolutions corresponding to the target point cloud of the first stacking object; the pseudo images of at least two projection resolutions corresponding to the target point cloud of the first stacking object are scaled to a uniform size and superimposed to obtain a first pseudo image;

[0075] The controller projects the target point cloud of the second stacking object onto a horizontal plane to generate a second pseudo image, including:

[0076] The controller projects the target point cloud of the second stacking object onto a horizontal plane using at least two projection resolutions to obtain pseudo images of at least two projection resolutions corresponding to the target point cloud of the second stacking object; and scales the pseudo images of at least two projection resolutions corresponding to the target point cloud of the second stacking object to a uniform size and superimposes them to obtain a second pseudo image.

[0077] Optionally, the controller determines a first pseudo image corresponding to the first stacking object according to the target point cloud of the first stacking object, including:

[0078] The controller projects the target point cloud of the first stacking object onto a horizontal plane using at least two projection resolutions to obtain images of at least two projection resolutions corresponding to the target point cloud of the first stacking object; scales the images of at least two projection resolutions corresponding to the target point cloud of the first stacking object to a uniform size and superimposes them, and converts the superimposed images into a first grayscale image;

[0079] The controller determines a second pseudo image corresponding to the second stacking object according to the target point cloud of the second stacking object, including:

[0080] The controller projects the target point cloud of the second stacking object onto a horizontal plane using at least two projection resolutions to obtain images of at least two projection resolutions corresponding to the target point cloud of the first stacking object; scales the images of at least two projection resolutions corresponding to the target point cloud of the first stacking object to a uniform size and superimposes them, and converts the superimposed images into a second grayscale image.

[0081] Optionally, the boundary line of the first grayscale image is obtained in the following manner:

[0082] The controller extracts straight lines from the first grayscale image according to a probabilistic Hough transform straight line detection algorithm, and determines a boundary line of the first grayscale image from the extracted straight lines according to a size of the first stacked object;

[0083] The boundary line of the second grayscale image is obtained as follows:

[0084] The controller extracts straight lines from the second grayscale image according to a probabilistic Hough transform straight line detection algorithm, and determines a boundary line of the second grayscale image from the extracted straight lines according to a size of the second stacked object.

[0085] Optionally, the controller extracts a straight line from the grayscale image according to a probabilistic Hough transform straight line detection algorithm, including:

[0086] The controller sorts the pixels in the grayscale image according to the grayscale values ​​to obtain a pixel sequence set, wherein the pixel sequence set includes a plurality of pixel sequences, and each pixel sequence includes pixels corresponding to the same grayscale value;

[0087] The controller takes out the pixel sequence in order as the current pixel sequence according to the sorting, and randomly selects pixel points in the current pixel sequence to find the straight line with the highest probability under the angle range, and the number of pixel points on the straight line meets the straight line length requirement and the sum of grayscale values ​​is the largest;

[0088] If the distance between the found straight line and the extracted straight line is greater than or equal to the maximum straight line distance threshold, the controller extracts the straight line, deletes the current pixel sequence from the pixel sequence set, and continues to use the next pixel sequence as the current pixel sequence until the pixel sequence set is empty.

[0089] Optionally, the controller compares the relative posture data with a threshold value to confirm an alignment state of the first stacked object and the second stacked object, including:

[0090] If the relative posture data is greater than or equal to the threshold, the controller confirms that the alignment state is misaligned; if the relative posture data is less than the threshold, the controller confirms that the alignment state is aligned.

[0091] Optionally, the method further comprises:

[0092] When the alignment state is misaligned, the controller controls the handling device to adjust the posture;

[0093] The controller reacquires the target point cloud of the first stacked object and the second stacked object through the sensor;

[0094] The controller re-determines the relative posture data;

[0095] The controller reconfirms the alignment status based on the relative pose data until the relative pose data is less than the threshold.

[0096] Optionally, the method further comprises:

[0097] When the alignment state is aligned, the controller controls the transport device to place the first stacking object on the second stacking object to complete the stacking.

[0098] According to a second aspect, a control system is provided, comprising a controller and a memory, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement any one of the methods described in the first aspect.

[0099] According to a third aspect, a handling device is provided, comprising a controller and a memory, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement any one of the methods described in the first aspect.

[0100] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0101] 1) The controller of the present application determines the first pseudo-image of the first stacking object according to the target point cloud of the first stacking object acquired by the sensor, and determines the second pseudo-image of the second stacking object according to the target point cloud of the second stacking object acquired by the sensor; Next, the controller determines the relative posture data between the first stacking object and the second stacking object according to the first pseudo-image and the second pseudo-image, and compares the relative posture data with the threshold value to confirm the alignment state of the first stacking object and the second stacking object. In this way, the relative posture data between the first stacking object and the second stacking object can be determined during the stacking process, and the alignment state of the first stacking object and the second stacking object can be confirmed based on the relative posture data and the threshold value. This solution can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the posture data of the first stacking object and the second stacking object, and then determine the alignment state of the two.

[0102] 2) The controller of the present application determines the relative posture data between the first stacking object and the second stacking object based on the target point of the first pseudo-image determined by the boundary line of the first pseudo-image, the target point of the second pseudo-image determined by the boundary line of the second pseudo-image, and the boundary line of the first pseudo-image and the target point of the second pseudo-image. In this way, the relative posture data between the first stacking object and the second stacking object can be determined during the stacking process, and the alignment state of the first stacking object and the second stacking object can be confirmed based on the relative posture data and the threshold. This solution can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the posture data of the first stacking object and the second stacking object, and then determine the alignment state of the two.

[0103] 3) The controller of the present application determines the boundary point cloud from the target point cloud based on the boundary line of the pseudo-image, and then determines the first target point of the first stacking object and the target point of the second stacking object based on the boundary point cloud, and then determines the relative posture data between the first stacking object and the second stacking object based on the first target point and the boundary point cloud of the first stacking object, and the second target point and the boundary point cloud of the second stacking object. In this way, the relative posture data between the first stacking object and the second stacking object can be determined during the stacking process, and the alignment state of the first stacking object and the second stacking object can be confirmed based on the relative posture data and the threshold. This solution can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the posture data of the first stacking object and the second stacking object, and then determine the alignment state of the two.

[0104] 4) The controller of the present application determines the third edge line and the fifth edge line from the target point cloud of the first stacking object according to the third edge line and the fifth edge line that are parallel to each other in the first pseudo-image, and then determines the first center line of the first stacking object according to the third edge line and the fifth edge line; then determines the first target point according to the intersection of the first center line of the first stacking object and the first edge line corresponding to the first boundary point cloud; the present application determines the fourth edge line and the sixth edge line from the target point cloud of the second stacking object according to the fourth edge line and the sixth edge line that are parallel to each other in the second pseudo-image, and then determines the second center line of the second stacking object according to the fourth edge line and the sixth edge line; then determines the second target point according to the intersection of the second center line of the second stacking object and the second edge line corresponding to the second boundary point cloud. The above method can quickly determine the first target point and the second target point with a small amount of calculation.

[0105] 5) The controller of the present application projects the target point cloud of the first stacking object and the target point cloud of the second stacking object onto the horizontal plane using at least two projection resolutions, and scales the pseudo-images of at least two projection resolutions corresponding to the target point cloud of the first stacking object and the target point cloud of the second stacking object to a uniform size and superimposes them respectively, to obtain a first pseudo-image corresponding to the first stacking object and a second pseudo-image corresponding to the second stacking object. On the one hand, converting the point cloud into a pseudo-image and superimposing it can reduce the dimension and complexity of the data, thereby improving processing efficiency. On the other hand, by projecting at least two projection resolutions, the features of the point cloud at different scales can be captured, and the pseudo-images of these different scales can be superimposed. The superimposed pseudo-image can further enhance the feature information in the pseudo-image on the basis of retaining the features of the original point cloud, and improve the recognition and information content of the pseudo-image. On the third hand, scaling multiple pseudo-images to a uniform size and superimposing them can average the noise and errors in each pseudo-image, thereby improving the quality of the pseudo-image to a certain extent.

[0106] 6) The controller of the present application dynamically adjusts the posture of the transport equipment based on the size of the relative posture data and the threshold; when the relative posture data is greater than or equal to the threshold, the posture of the transport equipment is adjusted based on the relative posture data, and the target point cloud collected by the sensor carried by the transport equipment is re-acquired to redetermine the relative posture data based on the target point cloud; the alignment state is reconfirmed according to the relative posture data until the relative posture data is less than the threshold, and then the controller controls the transport equipment to place the first stacking object on the second stacking object to complete the stacking, forming a closed-loop servo detection process, and the operation of the transport equipment does not need to be stopped during the servo detection process.

[0107] Of course, any invention of the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0109] Figure 1 A system schematic diagram of a handling device applicable to an embodiment of the present application;

[0110] Figure 2 An application scenario diagram provided for an embodiment of the present application;

[0111] Figure 3 for Figure 2 A schematic diagram of a region where the first stacking object and the second stacking object are parallel to each other when aligned;

[0112] Figure 4 A flowchart of an alignment status confirmation method provided in an embodiment of the present application;

[0113] Figure 5 A schematic diagram of generating a first grayscale image provided in an embodiment of the present application;

[0114] Figure 6 A schematic diagram of generating a second grayscale image provided in an embodiment of the present application;

[0115] Figure 7 A flowchart for determining relative position data between a first stacking object and a second stacking object provided in an embodiment of the present application;

[0116] Figure 8 A schematic diagram of a first grayscale image provided in an embodiment of the present application;

[0117] Fig. 9A schematic diagram of a second grayscale image provided in an embodiment of the present application;

[0118] Fig.10 A flowchart for determining relative position data between a first stacking object and a second stacking object provided in an embodiment of the present application;

[0119] Fig.11 A schematic diagram of calculating a first target point and a second target point provided in an embodiment of the present application;

[0120] Fig.12 A schematic diagram of the direction indicated by an arrow at a first target point and the direction indicated by an arrow at a second target point at different viewing angles provided in an embodiment of the present application;

[0121] Fig.13 A flowchart of an alignment status confirmation method provided in an embodiment of the present application;

[0122] Fig.14 A flowchart of an alignment status confirmation method provided in an embodiment of the present application.

[0123] Among them, 100-handling equipment; 101-handling equipment body; 102-sensor; 103-stacking execution component; 104-controller; 105-memory; A-first stacking object; B-second stacking object; A1-first target image area; B1-second target image area; TC1-first boundary line of the first grayscale image; TC3-third boundary line of the first grayscale image; TC5-fifth boundary line of the first grayscale image; TC7-seventh boundary line of the first grayscale image; Q1-target point of the first grayscale image; Q2-target point of the second grayscale image; L1-center line of the first grayscale image; L2-center line of the second grayscale image ;TB2-the second boundary line of the second grayscale image;TB4-the fourth boundary line of the second grayscale image;TB6-the sixth boundary line of the second grayscale image;TB8-the eighth boundary line of the second grayscale image;M1-the first center line;M2-the second center line;P1-the first target point;P2-the second target point;S1-the first boundary point cloud;S3-the third boundary point cloud;S5-the fifth boundary point cloud;T2-the second boundary point cloud;T4-the fourth boundary point cloud;T6-the sixth boundary point cloud;W1-the first edge line;W3-the third edge line;W5-the fifth edge line;V2-the second edge line;V4-the fourth edge line;V6-the sixth edge line. DETAILED DESCRIPTION

[0124] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0125] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0126] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, the term "according to" used in this article is not limited to being based on a certain object. For example, determining B based on A can mean: determining B based only on A, or determining B based partially on A.

[0127] In the related art, the stacking of upper and lower stacking objects is completed based on the posture of the lower stacking object relative to the handling equipment, without paying attention to the influence of various factors such as inaccurate posture of the upper stacking object, uneven ground, cumulative error of the odometer, and error of the handling equipment itself, which causes the upper and lower stacking objects to be misaligned during stacking, affecting operation safety.

[0128] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system schematic diagram of the handling equipment on which the present application is based is first described. Figure 1 A schematic diagram of a system of a handling device to which the present application embodiment can be applied is shown in FIG. Figure 1 As shown in FIG. 1 , the transport device 100 includes a transport device body 101 , a sensor 102 , a stacking execution component 103 , a controller 104 and a memory 105 .

[0129] The handling equipment 100 involved in the embodiment of the present application may be an unmanned forklift, a pallet truck, a crane, an AGV (Automated Guided Vehicle), an AMR (Automatic Mobile Robot), a humanoid robot, etc. The corresponding stacking execution component 103 may be a fork, a robotic arm, etc.

[0130] The controller 104 is used to control the stacking execution component 103 of the transport device 100 to execute the stacking of the first stacking object and the second stacking object. The controller 104 can be a control mainboard, a control box, a control unit, a vehicle-mounted computer, a computing platform, a tablet computer, a computer, etc. on the transport device body 101, or a system or device that performs a computing or controlling function in a local server or a cloud server, or can be a handheld controller, a remote controller, etc., etc. This is not limited in the embodiments of the present application.

[0131] The sensor 102 may be in the form of a sensor module, which at least includes a sensor for collecting point cloud data.

[0132] The stacking objects involved in the embodiments of the present application can be goods with outer packaging, or containers containing goods, such as cages, wooden boxes, plastic boxes, etc. The embodiments of the present application are described by taking cages as examples. Cages are also called material frames, storage cages, warehouse cages or iron cages. They are an important logistics equipment and are widely used because of their strong structure, stackability, durability, environmental protection and other advantages.

[0133] First, the concepts of terms involved in the embodiments of the present application are introduced.

[0134] Stacking refers to arranging and stacking several objects up and down according to certain rules.

[0135] Stacking objects refer to the objects involved in stacking, which can be the goods themselves, or goods with simple packaging such as wrapping, or containers that can hold and carry goods, such as cages, wooden boxes, plastic boxes, pallets, etc.

[0136] Stacking process: refers to the handling device lifting the first stacking object, so that the first stacking object moves and approaches the second stacking object, and then aligns the first stacking object with the second stacking object by adjusting the posture of the handling device, and then places the first stacking object on the second stacking object to complete the stacking.

[0137] The first stacking object refers to the stacking object located at the top, and the second stacking object refers to the stacking object located at the bottom.

[0138] Alignment state: refers to the state in which two or more stacked objects are arranged in a vertical direction, and at least part of the border lines between the stacked objects are parallel or overlapped. The vertical direction refers to the Z axis in the coordinate system of the handling equipment (such as Figure 1 , Figure 2 and Figure 3 The Z axis in the Figure 2 and Figure 3In the figure, the first target image area A1 of the first stacking object A and the second target image area B1 of the second stacking object B are arranged along a straight line in the vertical direction, and the border lines of A1 and A2 are parallel to each other.

[0139] Figure 3 The three-dimensional coordinate system in is composed of an origin O, an X-axis, a Y-axis, and a Z-axis. The coordinate axes intersect each other perpendicularly. Optionally, the geometric center of the transport equipment is point O, the front and rear travel direction of the transport equipment (i.e., the longitudinal direction of the transport equipment body 101) is the X-axis, wherein the positive direction of the X-axis is the direction away from the transport equipment attachment (such as a fork), the lateral direction of the transport equipment body 101 is the Y-axis, and the height direction of the transport equipment is the Z-axis.

[0140] in, Figure 1 and Figure 2 "X" corresponds to the X-axis in the above three-dimensional coordinate system, "O" corresponds to the origin O in the above three-dimensional coordinate system; "Z" corresponds to the Z-axis in the above three-dimensional coordinate system, and "Y" corresponds to the Y-axis in the above three-dimensional coordinate system. The positive direction of the Y-axis is perpendicular to the paper and outward (not shown in the figure).

[0141] Target point cloud: refers to point cloud data of a target area acquired by a sensor, wherein the target area refers to a specific area on the first stacked object and the second stacked object for detection and analysis, such as key structural areas such as boundaries or edge target points of the first stacked object and the second stacked object.

[0142] Pseudo-image: refers to images generated by algorithms or processed by some special techniques in the field of computer image processing or computer vision. They may not represent actual image data or have some virtualized or approximate relationship with real images. In some cases, pseudo-images can also refer to simulated images or images used to represent some non-real world scenes.

[0143] Grayscale Image: A single-channel image format in which each pixel is represented by only one value, usually ranging from 0 to 255 (for 8-bit depth images). A grayscale value of 0 represents pure black, 255 represents pure white, and the values ​​in between represent varying degrees of gray.

[0144] Grayscale value: The pixel value in the grayscale image is usually used to represent the brightness of the image, that is, the grayscale value. The larger the grayscale value, the brighter the pixel; conversely, the smaller the grayscale value, the darker the pixel.

[0145] Posture: refers to the position and direction of an object in space, generally including coordinates and angles.

[0146] Relative posture data: generally used to indicate the relative position relationship between two objects, specifically refers to the difference between two posture data.

[0147] Figure 4 This is a flow chart of a method for confirming an alignment state provided in an embodiment of the present application, which method can be executed by an unmanned forklift. Figure 4 As shown in , the method may include the following steps:

[0148] Step 401: The controller obtains target point clouds of the first stacking object and the second stacking object through a sensor.

[0149] Step 403: The controller determines a first pseudo image corresponding to the first stacking object according to the target point cloud of the first stacking object; the controller determines a second pseudo image corresponding to the second stacking object according to the target point cloud of the second stacking object.

[0150] Step 405: The controller determines relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image.

[0151] Step 407: The controller compares the relative posture data with a threshold value to confirm the alignment state of the first stacking object and the second stacking object.

[0152] It can be seen from the above process that the controller of the present application determines the first pseudo-image of the first stacking object according to the target point cloud of the first stacking object acquired by the sensor, and determines the second pseudo-image of the second stacking object according to the target point cloud of the second stacking object acquired by the sensor; Next, the controller determines the relative posture data between the first stacking object and the second stacking object according to the first pseudo-image and the second pseudo-image, and compares the relative posture data with the threshold value to confirm the alignment state of the first stacking object and the second stacking object. In this way, the relative posture data between the first stacking object and the second stacking object can be determined during the stacking process, and the alignment state of the first stacking object and the second stacking object can be confirmed based on the relative posture data and the threshold value. This solution can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the posture data of the first stacking object and the second stacking object, and then determine the alignment state of the two.

[0153] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the "first" and "second" and other limitations involved in the present disclosure do not have limitations in terms of size, order, and quantity, and are only used to distinguish in name, such as "first stacking object" and "second stacking object" are used to distinguish two stacking objects. For another example, "first pseudo-image" and "second pseudo-image" are used to distinguish two pseudo-images.

[0154] First, the above step 401, namely "the controller obtains the target point cloud of the first stacking object and the second stacking object through the sensor" is described in detail in conjunction with the embodiment.

[0155] First, a brief introduction to the stacking scenarios involved in the embodiments of the present application is given. Figure 2 As shown, when the unmanned forklift receives a task of stacking the first stacking object A onto the second stacking object B, the controller (such as Figure 1 The controller 104 shown controls the fork of the unmanned forklift to pick up the first stacking object A, and then controls the unmanned forklift to move to the vicinity of the second stacking object B, and then aligns the first target image area A1 of the first stacking object and the second target image area B1 of the second stacking object by controlling the posture of the unmanned forklift, and then stacks the first stacking object A on the second stacking object B. When the controller controls the unmanned forklift to stack the first stacking object A on the second stacking object B, the first stacking object A and the second stacking object B are not aligned due to the external environment (such as uneven ground) and the error of the unmanned forklift itself. The embodiment of the present application is a solution proposed to solve this problem.

[0156] In the embodiment of the present application, multiple sensors can be installed according to different types of sensors to simultaneously scan the first stacked object and the second stacked object.

[0157] Among them, the odometer estimates the distance the unmanned forklift has moved by measuring its movement, and is usually combined with sensor data (i.e., data collected by sensors) to calculate physical quantities such as the position, speed, and posture of the unmanned forklift.

[0158] The sensor can be set at a preset distance below the midpoint between the fork arms of the unmanned forklift (for example, on the vehicle body or on the fork arm structure), or can be set at other locations according to actual conditions. The odometer can be set near the wheels of the unmanned forklift to record the number of rotations of the wheels and estimate the distance moved; the odometer can also be set at the center of the chassis of the unmanned forklift.

[0159] For example, the above sensors and odometers can be controlled by a built-in controller (such as Figure 1 The controller 104 in the embodiment is controlled, for example, based on a SoC (System-on-a-Chip), which is not specifically limited in the embodiments of the present application.

[0160] Before determining the posture adjustment information of the unmanned forklift, it is also necessary to synchronize the odometer and the sensor in time. The synchronization method can adopt commonly used synchronization methods (such as hardware synchronization methods, software synchronization methods (such as timestamp alignment, interpolation synchronization, etc.)).

[0161] In the embodiment of the present application, the controller controls the unmanned forklift to lift the first stacking object, and when the sensor scans the first stacking object and the second stacking object at the same time, the target point clouds of the first stacking object and the second stacking object are collected, specifically:

[0162] The unmanned forklift carries the first stacking object to the stacking operation position, lifts the first stacking object, and when the sensor carried by the unmanned forklift scans the first stacking object and the second stacking object, the sensor obtains the target point cloud of the first stacking object and the second stacking object.

[0163] Here, the stacking operation position may be a position where the sensor on the handling device can simultaneously obtain the target point cloud of the first stacking object and the second stacking object, and at this position, the fork of the unmanned forklift can stack the first stacking object and the second stacking object within the variable posture range of the fork. For example, according to the position of the second stacking object (such as the lower stacking object), a position within a preset distance range in front of the second stacking object is used as the stacking operation position.

[0164] Considering that there is distortion in the target point cloud collected when the first stacked object and the second stacked object are scanned by the sensor at the same time, in the embodiment of the present application, the sensor obtains the target point cloud of the first stacked object and the second stacked object, and further includes: the sensor carried by the unmanned forklift collects the original point cloud of the first stacked object; converts the original point cloud of the first stacked object from the coordinate system where the sensor is located to the coordinate system where the unmanned forklift is located, and dedistorts the original point cloud of the first stacked object according to the odometer information collected by the unmanned forklift to obtain the target point cloud; and

[0165] The sensor carried by the unmanned forklift collects the original point cloud of the second stacking object; the original point cloud of the second stacking object is converted from the coordinate system where the sensor is located to the coordinate system where the unmanned forklift is located, and the original point cloud of the second stacking object is dedistorted according to the odometer information collected by the unmanned forklift to obtain the target point cloud.

[0166] The distortion in the original point clouds of the first stacked object and the second stacked object refers to the shape distortion problem caused by sensor movement or external factors. The odometer information provides the movement information of the sensor when collecting the original point cloud, which usually includes position, speed, acceleration, etc.; the compensation transformation matrix can be determined according to the odometer information; and then the compensation transformation matrix is ​​applied to each point in the original point cloud to obtain the de-distorted point cloud (that is, the target point cloud of the first stacked object and the target point cloud of the second stacked object).

[0167] This application converts the original point cloud collected by the sensor from the coordinate system where the sensor is located to the coordinate system where the unmanned forklift is located, and dedistorts the original point cloud according to the odometer information to obtain a distortion-free target point cloud.

[0168] The sensor involved in the embodiment of the present application may include a radar module, wherein the radar module may include one or more radars. The radar may be installed at a preset distance below the midpoint of the root of the stacking execution component (e.g., fork) of the unmanned forklift, so that the overall field of view of the radar (e.g., Figure 2 The fan-shaped area surrounded by the dotted arrow in the figure can cover the target area of ​​the first pair of stacked objects and the second pair of stacked objects. Therefore, the radar can collect the target point cloud of the first stacked object and the second stacked object. Optionally, the radar is a laser radar (Lidar). The laser radar may include a three-dimensional laser radar.

[0169] It should be noted that the embodiments of the present application can be applied to scenarios with multiple stacked objects.

[0170] The above step 403, i.e., "the controller determines a first pseudo-image corresponding to the first stacking object based on the target point cloud of the first stacking object; the controller determines a second pseudo-image corresponding to the second stacking object based on the target point cloud of the second stacking object" is described in detail below in conjunction with the embodiments.

[0171] In the embodiment of the present application, the controller determines the projection points of each point in the target point cloud of the first stacking object and the target point cloud of the second stacking object on the same horizontal plane (such as the XOY plane), which usually involves converting the point cloud coordinates from the original coordinate system to the coordinate system where the XOY plane is located, and retaining the two-dimensional position information (such as the X and Y coordinates). Optionally, the controller projects the target point cloud of the first stacking object on the XOY plane as a top-down projection to generate a first projection diagram (such as Figure 5 The first projection image is used as the first pseudo image of the first stacking object, and the target point cloud of the second stacking object is projected onto the XOY plane to generate a second projection image (as shown in the upper or lower figure of (a)). Figure 6 As shown in the upper or lower figure of (a), the second projection image is used as the second pseudo image of the second stacking object. Wherein, "X" in the XOY plane corresponds to the X-axis in the above three-dimensional coordinate system; "O" corresponds to the origin O in the above three-dimensional coordinate system; and "Y" corresponds to the Y-axis in the above three-dimensional coordinate system.

[0172] like Figure 5 In some embodiments, the controller projects the target point cloud of the first stacked object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (such as Figure 5 In (a), at least two projection images of the projection resolution corresponding to the target point cloud of the first stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Figure 5In (b), the superimposed projection image is used as the first pseudo image of the first stacking object; in the same way, the controller projects the target point cloud of the second stacking object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (such as Figure 6 In (a), the projection images of at least two projection resolutions corresponding to the target point cloud of the second stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Figure 6 In (b), the superimposed projection image is used as the second pseudo image of the second stacking object.

[0173] In some embodiments, the controller projects the target point cloud of the first stacked object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (eg, Figure 5 In (a), at least two projection images of the projection resolution corresponding to the target point cloud of the first stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Figure 5 (b)), and then convert the superimposed projection image into the first grayscale image (such as Figure 5 In (c), the first grayscale image is used as the first pseudo image of the first stacking object; in the same way, the controller projects the target point cloud of the second stacking object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (such as Figure 6 In (a), the projection images of at least two projection resolutions corresponding to the target point cloud of the second stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Figure 6 (b)), and then convert the superimposed projection image into a second grayscale image (such as Figure 6 In (c), the second grayscale image is used as the second pseudo image of the second stacking object. The purpose of scaling to a uniform size is to ensure that at least two projection images of the projection resolution are correctly aligned before superposition to avoid misalignment or ghosting after superposition.

[0174] The unification of the sizes of the pseudo images of at least two projection resolutions may include scaling the sizes of the pseudo images of the respective projection resolutions to a specified size.

[0175] Optionally, unifying the sizes of pseudo images of at least two projection resolutions may also include first determining the largest pseudo image among the pseudo images of at least two projection resolutions, and unifying the sizes of pseudo images of other projection resolutions to the size of the largest pseudo image.

[0176] It should be noted that the pixel values ​​in the above grayscale images (i.e., the first grayscale image and the second grayscale image) are usually used to represent the brightness of the image, i.e., the grayscale value. The larger the grayscale value, the brighter the pixel; conversely, the smaller the grayscale value, the darker the pixel. When we project the point cloud into the pixel, we can adjust the grayscale value of the pixel according to the attributes of each point in the point cloud.

[0177] During the process in which the controller projects the target point cloud of the first stacking object and the target point cloud of the second stacking object onto the XOY plane respectively and converts them into grayscale images, the grayscale value in the corresponding grayscale image can be determined based on the attributes of each point in the point cloud projected into the pixel (such as the number of points, the intensity of the points, the distance between the points, the angle of the points, and at least one of the elevation difference between the points in the point cloud).

[0178] After the controller determines the grayscale value, the corresponding pixel position is found on the grayscale image according to the two-dimensional position information (X and Y coordinates) of each projection point, and the determined grayscale value is used as the pixel value at the pixel position. The above process is repeated until all projection points are processed, thereby generating a complete grayscale image.

[0179] Optionally, the projection resolution used by the target point cloud of the first stacked object may be the same as the projection resolution used by the target point cloud of the second stacked object, such as the target point cloud of the first stacked object and the target point cloud of the second stacked object both use projection resolutions of 5mm (millimeter) / pixel and 1cm (centimeter) / pixel.

[0180] For example, projecting the target point cloud of the first stacking object onto the XOY plane using 5 mm / pixel and 1 cm / pixel, and generating a first grayscale image corresponding to the first stacking object may include the following steps:

[0181] Step 1: The controller projects the target point cloud of the first stacked object onto the XOY plane at 5 mm / pixel and 1 cm / pixel, and generates a pseudo image corresponding to 5 mm / pixel and a pseudo image corresponding to 1 cm / pixel (corresponding to Figure 5 The higher the projection resolution, the clearer the image. Figure 5 The lines in the upper and lower figures of (a) generally reflect the contour lines or characteristic lines of the first stacked object at a certain viewing angle (ie, the viewing angle corresponding to the top view).

[0182] Step 2: The controller scales the projection image corresponding to 5 mm / pixel and the projection image corresponding to 1 cm / pixel to a uniform size and superimposes them to obtain a superimposed projection image (such as Figure 5 The superimposed projection image is then converted into a grayscale image to obtain a first grayscale image corresponding to the first stacked object (as shown in (b)). Figure 5 (as shown in (c)).

[0183] The second grayscale image is generated in a similar manner to the first grayscale image.

[0184] It should be noted that Figure 5The "O" in corresponds to the origin O in the above three-dimensional coordinate system, the "X" corresponds to the X-axis in the above three-dimensional coordinate system, and the "Y" corresponds to the Y-axis in the above three-dimensional coordinate system.

[0185] It should be noted that, in some embodiments, the projection image of each projection resolution can be divided into three channels, wherein the first channel uses the number of points in the point cloud projected into the pixel, and the more the number of point clouds, the higher the pixel value; the second channel uses the relative posture data of the distance between the point cloud and the target (i.e., the theoretical position of the stacked object) to generate, and the larger the relative posture data, the lower the pixel value; the third channel uses the elevation difference between the points in the point cloud, and within a certain height range, the larger the elevation difference, the higher the pixel value.

[0186] The present application determines the pixel value in the grayscale image based on at least one of the number of points in the point cloud projected into the pixel, the intensity of the points, the distance between points, the angle of the points, and the elevation difference between the points in the point cloud, so that the pixel value in the grayscale image reflects the multi-dimensional properties of the point cloud, so as to improve the accuracy of determining the boundary line from the grayscale image.

[0187] The above step 405, namely "the controller determines the relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image" is described in detail below in conjunction with the embodiments.

[0188] In an embodiment of the present application, key structures such as boundaries and target points are extracted from a first pseudo-image generated in a horizontal plane by projecting a target point cloud of the first stacked object to determine the posture of the first stacked object; and key features such as boundaries and target points are extracted from a second pseudo-image generated in a horizontal plane by projecting a target point cloud of the second stacked object to determine the posture of the second stacked object; relative posture data is determined based on the posture of the first stacked object and the posture of the second stacked object.

[0189] In the embodiment of the present application, the controller determines the relative posture data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image, including:

[0190] The controller determines a boundary line of the first pseudo image; the controller determines a target point of the first pseudo image based on the boundary line of the first pseudo image; the controller determines a boundary line of the second pseudo image; the controller determines a target point of the second pseudo image based on the boundary line of the second pseudo image; the controller determines relative posture data between the first stacking object and the second stacking object based on the boundary line of the first pseudo image and the target point of the first pseudo image, as well as the boundary line of the second pseudo image and the target point of the second pseudo image.

[0191] The boundary line of the first pseudo image in the embodiment of the present application may be: the boundary line of the first projection image generated by projecting the target point cloud of the first stacking object on the XOY plane from top view, such as Figure 5The boundary lines in the upper or lower figure of (a) (corresponding to the edges of the rectangular box); or, the boundary lines of the superimposed projection images obtained by scaling the projection images of at least two projection resolutions corresponding to the target point cloud of the first stacking object to a uniform size and superimposing them, such as Figure 5 The boundary lines in the figure shown in (b) (corresponding to the edges of the rectangular frame); or, the boundary lines of the first grayscale image, Figure 5 The boundary lines in the figure shown in (c) (corresponding to the edges of the rectangular box).

[0192] The boundary line of the second pseudo image in the embodiment of the present application may be: the boundary line of the second projection image generated by projecting the target point cloud of the second stacking object on the XOY plane in a top-down manner, Figure 6 The boundary lines in the upper or lower figure of (a) (corresponding to the edges of the rectangular frame); or, the boundary lines of the superimposed projection images obtained by scaling the projection images of at least two projection resolutions corresponding to the target point cloud of the second stacking object to a uniform size and superimposing them, such as Figure 6 The boundary lines in the figure shown in (b) (corresponding to the edges of the rectangular frame); or, the boundary lines of the second grayscale image, such as Figure 6 The boundary lines in the figure shown in (c) (corresponding to the edges of the rectangular box).

[0193] In the embodiment of the present application, a probabilistic Hough transform straight line detection algorithm is used to extract a straight line from the first pseudo image and the second pseudo image, and then the boundary line between the first pseudo image and the second pseudo image is determined. This can improve the detection speed and reduce the consumption of computing resources while ensuring the detection accuracy.

[0194] Optionally, the boundary line of the first pseudo image includes: a first boundary line of the first pseudo image, a third boundary line of the first pseudo image, a fifth boundary line of the first pseudo image, and a seventh boundary line of the first pseudo image;

[0195] The boundary lines of the second dummy image include: a second boundary line of the second dummy image, a fourth boundary line of the second dummy image, a sixth boundary line of the second dummy image, and an eighth boundary line of the second dummy image.

[0196] In the embodiment of the present application, the controller determines the target point of the first pseudo image according to the boundary line of the first pseudo image, including:

[0197] The controller determines the center line of the first pseudo image according to the third boundary line of the first pseudo image and the fifth boundary line of the first pseudo image; and uses the intersection of the center line of the first pseudo image and the first boundary line of the first pseudo image as the target point of the first pseudo image;

[0198] The controller determines the target point of the second pseudo image according to the boundary line of the second pseudo image, including: the controller determines the center line of the second pseudo image according to the fourth boundary line of the second pseudo image and the sixth boundary line of the second pseudo image; and uses the intersection of the center line of the second pseudo image and the second boundary line of the second pseudo image as the target point of the second pseudo image.

[0199] In the embodiment of the present application, stacking objects with regular shapes or irregular shapes can be used to complete the stacking of stacking objects. The following takes stacking objects with regular shapes as an example.

[0200] Continuing from the above, the boundary lines of the regularly shaped stacked objects are parallel or perpendicular to each other in the pseudo-image; based on this, the third boundary line of the first pseudo-image and the fifth boundary line of the first pseudo-image in the embodiment of the present application are parallel to each other; the second boundary line of the second pseudo-image and the fourth boundary line of the second pseudo-image are parallel to each other; the third boundary line of the first pseudo-image is perpendicular to the first boundary line of the first pseudo-image, and the fifth boundary line of the first pseudo-image is perpendicular to the first boundary line of the first pseudo-image; the fourth boundary line of the second pseudo-image is perpendicular to the second boundary line of the second pseudo-image, and the sixth boundary line of the second pseudo-image is perpendicular to the second boundary line of the second pseudo-image.

[0201] It should be noted that the target point of the first pseudo image is the midpoint of the first boundary line of the first pseudo image; and the target point of the second pseudo image is the midpoint of the second boundary line of the second pseudo image.

[0202] The following is a detailed description using an example in which the first pseudo image is a first grayscale image and the second pseudo image is a second grayscale image.

[0203] exist Figure 7 In the method, determining the relative position data between the first stacking object and the second stacking object according to the first grayscale image and the second grayscale image may include the following steps:

[0204] Step 701: The controller determines a boundary line of a first grayscale image;

[0205] Step 702: The controller determines a target point of the first grayscale image according to a boundary line of the first grayscale image;

[0206] Step 703: The controller determines a boundary line of the second grayscale image;

[0207] Step 704: the controller determines a target point of the second grayscale image according to a boundary line of the second grayscale image;

[0208] Step 705: The controller determines the relative posture data between the first stacking object and the second stacking object according to the boundary line and the target point of the first grayscale image, and the boundary line and the target point of the second grayscale image.

[0209] In one example, the controller extracts boundary lines of a first grayscale image through a probabilistic Hough transform straight line detection algorithm, wherein the boundary lines of the first grayscale image include at least a first boundary line of the first grayscale image, a third boundary line of the first grayscale image, and a fifth boundary line of the first grayscale image; the boundary lines of the second grayscale image include at least a second boundary line of the second grayscale image, a fourth boundary line of the second grayscale image, and a sixth boundary line of the second grayscale image.

[0210] like Figure 8 In some embodiments, the boundary lines of the first grayscale image include a first boundary line TC1 of the first grayscale image, a third boundary line TC3 of the first grayscale image, a fifth boundary line TC5 of the first grayscale image, and a seventh boundary line TC7 of the first grayscale image.

[0211] In some embodiments, the third boundary line TC3 of the first grayscale image is parallel to the fifth boundary line TC5 of the first grayscale image, and the first boundary line TC1 of the first grayscale image is parallel to the seventh boundary line TC7 of the first grayscale image.

[0212] like Fig. 9 In some embodiments, the boundary lines of the second grayscale image include a second boundary line TB2 of the second grayscale image, a fourth boundary line TB4 of the second grayscale image, a sixth boundary line TB6 of the second grayscale image, and an eighth boundary line TB8 of the second grayscale image.

[0213] In some embodiments, the fourth boundary line TB4 of the second grayscale image is parallel to the sixth boundary line TB6 of the second grayscale image, and the second boundary line TB2 of the second grayscale image is parallel to the eighth boundary line TB8 of the second grayscale image.

[0214] In one example, the controller determines a target point of the first grayscale image according to a boundary line of the first grayscale image, including:

[0215] exist Figure 8 In the figure, the controller determines the center line L1 (yellow dotted line) of the first grayscale image according to the third boundary line TC3 of the first grayscale image and the fifth boundary line TC5 of the first grayscale image; and takes the intersection of the center line L1 of the first grayscale image and the first boundary line TC1 of the first grayscale image as the target point Q1 of the first grayscale image.

[0216] The controller determines the target point of the second grayscale image according to the boundary line of the second grayscale image, and the target point of the second grayscale image includes:

[0217] exist Fig. 9 In the figure, the controller determines the center line L2 (yellow dotted line) of the second grayscale image according to the fourth boundary line TB4 of the second grayscale image and the sixth boundary line TB6 of the second grayscale image; and takes the intersection of the center line L2 of the second grayscale image and the second boundary line TB2 of the second grayscale image as the target point Q2 of the second grayscale image.

[0218] In the embodiment of the present application, stacking objects with regular shapes or irregular shapes can be used to complete the stacking of stacking objects. The following takes stacking objects with regular shapes as an example.

[0219] Continuing from the above, the boundary lines of the regularly shaped stacked objects are parallel or perpendicular in the grayscale image; based on this, the third boundary line TC3 of the first grayscale image and the fifth boundary line TC5 of the first grayscale image in the embodiment of the present application are parallel to each other; the second boundary line TB2 of the second grayscale image and the fourth boundary line TB4 of the second grayscale image are parallel to each other; the third boundary line TC3 of the first grayscale image is perpendicular to the first boundary line TC1 of the first grayscale image, and the fifth boundary line TC5 of the first grayscale image is perpendicular to the first boundary line TC1 of the first grayscale image; the fourth boundary line TB4 of the second grayscale image is perpendicular to the second boundary line TB2 of the second grayscale image, and the sixth boundary line TB6 of the second grayscale image is perpendicular to the second boundary line TB2 of the second grayscale image.

[0220] In one example, the target point of the first grayscale image may be any point on the first grayscale image, such as the midpoint on the first boundary line TC1 of the first grayscale image (eg Figure 8 The target point of the second grayscale image can be any point on the second grayscale image, such as the midpoint of the second boundary line TB2 of the second grayscale image (such as Fig. 9 The corresponding figures of this application take the midpoint as an example.

[0221] It should be noted that steps 701 to 702 are executed in parallel with steps 703 to 704; or steps 703 to 704 are executed first, and then steps 701 to 702; or steps 701 to 702 are executed first, and then steps 703 to 704.

[0222] In some embodiments, the controller determines the posture data of the first stacked object based on the position information of the target point Q1 of the first grayscale image and the angle information of the midline L1 of the first grayscale image, and determines the posture data of the second stacked object based on the position information of the target point Q2 of the second grayscale image and the angle information of the midline L2 of the second grayscale image, and then performs a difference calculation on the posture data of the first stacked object and the posture data of the second stacked object to obtain the relative posture data of the first stacked object and the second stacked object.

[0223] In some embodiments, a method for determining relative pose data between a first stacking object and a second stacking object is disclosed, the specific steps being:

[0224] The controller determines the boundary line of the first pseudo image; the controller determines the boundary point cloud of the first stacking object corresponding to the boundary line of the first pseudo image from the target point cloud of the first stacking object; the controller determines the first target point of the first stacking object based on the boundary point cloud of the first stacking object; the controller determines the boundary line of the second pseudo image; the controller determines the boundary point cloud of the second stacking object corresponding to the boundary line of the second pseudo image from the target point cloud of the second stacking object; the controller determines the second target point of the second stacking object based on the boundary point cloud of the second stacking object; the controller determines the relative angle difference between the first stacking object and the second stacking object based on the boundary point cloud of the first stacking object and the boundary point cloud of the second stacking object; the controller determines the relative posture data between the first stacking object and the second stacking object based on the relative position difference and the relative angle difference between the first target point and the second target point.

[0225] The following is a specific description using the first pseudo image being the first grayscale image and the second pseudo image being the second grayscale image as an example. Fig.10 , determining the relative position data between the first stacking object and the second stacking object may include the following steps:

[0226] Step 1001: The controller determines a boundary line of a first grayscale image;

[0227] Step 1002: The controller determines a boundary point cloud of the first stacking object corresponding to a boundary line of the first grayscale image from a target point cloud of the first stacking object;

[0228] Step 1003: The controller determines a first target point of the first stacking object according to the boundary point cloud of the first stacking object;

[0229] Step 1004: The controller determines a boundary line of the second grayscale image;

[0230] Step 1005: The controller determines a boundary point cloud of the second stacking object corresponding to a boundary line of the second grayscale image from the target point cloud of the second stacking object;

[0231] Step 1006: The controller determines a second target point of the second stacking object according to the boundary point cloud of the second stacking object;

[0232] Step 1007: The controller determines a relative angle difference between the first stacking object and the second stacking object according to the boundary point cloud of the first stacking object and the boundary point cloud of the second stacking object;

[0233] Step 1008: The controller determines relative position data between the first stacking object and the second stacking object according to the relative position difference and the relative angle difference between the first target point and the second target point.

[0234] It should be noted that steps 1001 to 1003 are executed in parallel with steps 1004 to 1006; or steps 1004 to 1006 are executed first, and then steps 1001 to 1003; or steps 1001 to 1003 are executed first, and then steps 1004 to 1006.

[0235] Regarding the above step 1002: the controller may determine, according to the first projection relationship, from the target point cloud of the first stacking object, a boundary point cloud of the first stacking object corresponding to the boundary line of the first grayscale image.

[0236] Here, the first projection relationship refers to the relationship between the target point cloud of the first stacking object and the first grayscale image.

[0237] Regarding the above step 1005: the controller may determine, according to the second projection relationship, from the target point cloud of the second stacking object, a boundary point cloud of the second stacking object corresponding to the boundary line of the second grayscale image.

[0238] Here, the second projection relationship refers to the relationship between the target point cloud of the second stacking object and the second grayscale image.

[0239] The following takes the first boundary line TC1 of the first grayscale image, the third boundary line TC3 of the first grayscale image, and the fifth boundary line TC5 of the first grayscale image as examples, and describes the corresponding first boundary point cloud S1, third boundary point cloud S3, and fifth boundary point cloud S5 based on the first projection relationship.

[0240] The first boundary line TC1 of the first grayscale image (eg Figure 8 TC1 in the first grayscale image), the third boundary line TC3 of the first grayscale image (such as Figure 8 TC3 in the first grayscale image), the fifth boundary line TC5 in the first grayscale image (such as Figure 8 TC5 in the first grayscale image), respectively, based on the first projection relationship, a first boundary point cloud S1 (such as Fig.11 S1 in the first grayscale image), and a third boundary point cloud S3 corresponding to the third boundary line TC3 of the first grayscale image (such as Fig.11 S3 in ), and a fifth boundary point cloud S5 corresponding to the fifth boundary line TC5 of the first grayscale image (such as Fig.11 in S5).

[0241] The following takes the second boundary line TB2 of the second grayscale image, the fourth boundary line TB4 of the second grayscale image, and the sixth boundary line TB6 of the second grayscale image as examples, and describes the corresponding second boundary point cloud T2, fourth boundary point cloud T4, and sixth boundary point cloud T6 based on the second projection relationship.

[0242] The second boundary line TB2 of the second grayscale image (eg Fig. 9 TB2 in the second grayscale image), the fourth boundary line TB4 of the second grayscale image (such as Fig. 9 TB4 in the second grayscale image), the sixth boundary line TB6 in the second grayscale image (such as Fig. 9 TB6 in the figure), based on the above second projection relationship, the second boundary point cloud T2 (such as Fig.11 T2 in the second grayscale image), and a fourth boundary point cloud T4 (such as Fig.11 T4 in ), and the sixth boundary point cloud T6 corresponding to the sixth boundary line TB6 of the second grayscale image (such as Fig.11 T6 in ).

[0243] The distribution of each boundary point cloud in the embodiment of the present application directly reflects the geometric shape of the stacked object. For example, a stacked object of a cube will generate a point cloud with six rectangular faces.

[0244] exist Fig.11 In the figure, according to the geometric shape of the first stacking object (such as a rectangle), the first boundary point cloud S1, the third boundary point cloud S3 and the fifth boundary point cloud S5 are distributed on the rectangular surface corresponding to the first stacking object; according to the geometric shape of the second stacking object, the second boundary point cloud T2, the fourth boundary point cloud T4 and the sixth boundary point cloud T6 are distributed on the rectangular surface corresponding to the second stacking object.

[0245] exist Fig.11 In the figure, the least squares method is used to fit the first boundary point cloud S1 distributed on the rectangular surface to obtain the first edge line W1; the least squares method is used to fit the third boundary point cloud S3 distributed on the rectangular surface to obtain the third edge line W3; the least squares method is used to fit the fifth boundary point cloud S5 distributed on the rectangular surface to obtain the fifth edge line W5.

[0246] exist Fig.11 In the figure, the least squares method is used to fit the second boundary point cloud T2 distributed on the rectangular surface to obtain the second edge line V2; the least squares method is used to fit the fourth boundary point cloud T4 distributed on the rectangular surface to obtain the fourth edge line V4; the least squares method is used to fit the sixth boundary point cloud T6 distributed on the rectangular surface to obtain the sixth edge line V6.

[0247] Taking the third boundary point cloud S3 as an example, the least square method is used to fit the third boundary point cloud S3 distributed on the rectangular surface to obtain the third edge line W3 (at Fig.11 The red line in the middle).

[0248] For example, find a straight line y=mx+b on the rectangular surface, where m is the slope and b is the intercept, so that the straight line is as close as possible to the third boundary point cloud S3 distributed on the rectangular surface. The goal of the least squares method is to find the values ​​of m and b so that the sum of the squares of the perpendicular distances (i.e., errors) from all points (i.e., all points in the third boundary point cloud S3 distributed on the rectangular surface) to the straight line is minimized; the sum of the squares of the perpendicular distances (i.e., errors) from all points to the straight line is minimized, and the determined straight line is taken as the third edge line W3.

[0249] It should be noted that, for other boundary point clouds, the edge lines thereof may be determined in the same manner as the third boundary point cloud, which will not be described in detail herein.

[0250] The first target point in the embodiment of the present application may be any point on the first stacking object, such as the midpoint on the first edge line W1 of the first stacking object (eg Fig.11 The second target point may be any point on the second stacked object, such as the midpoint of the second edge line V2 of the second stacked object (e.g. Fig.11 The corresponding figures of this application take the midpoint as an example.

[0251] In one example, the embodiment of the present application may determine the first target point (eg Fig.11 The first target point P1 in , and the second target point (such as Fig.11 The second target point P2 in ( ), specifically:

[0252] exist Fig.11 In the process, the controller determines the first edge line W1 of the first stacking object according to the first boundary point cloud S1; the controller determines the third edge line W3 of the first stacking object according to the third boundary point cloud S3; the controller determines the fifth edge line W5 of the first stacking object according to the fifth boundary point cloud S5; the controller determines the first center line M1 according to the third edge line W3 and the fifth edge line W5; the controller determines the first target point P1 according to the first center line M1 and the first edge line W1.

[0253] exist Fig.11 In the process, the controller determines the second edge line V2 of the second stacking object according to the second boundary point cloud T2; the controller determines the fourth edge line V4 of the second stacking object according to the fourth boundary point cloud T4; the controller determines the sixth edge line V6 of the second stacking object according to the sixth boundary point cloud T6; the controller determines the second center line M2 according to the fourth edge line V4 and the sixth edge line V6; the controller determines the second target point according to the second center line M2 and the second edge line V2.

[0254] The relative position data involved in this embodiment includes a relative position difference and a relative angle difference. The relative position difference can be determined according to the position difference between the first target point P1 and the second target point P2.

[0255] The following describes in detail several ways to determine the relative angle difference:

[0256] First way:

[0257] The controller determines a relative angle difference between the first stacking object and the second stacking object according to a first angle between the first edge line W1 and the second edge line V2 and the first angle.

[0258] exist Fig.11 In the figure, the first angle is an angle on the XOY plane in the three-dimensional coordinate system, and the first angle is an angle formed by projecting the first edge line W1 and the second edge line V2 on the XOY plane.

[0259] Optionally, the first angle may also be: the angle between the third edge line W3 and the fourth edge line V4; or, the angle between the fifth edge line W5 and the sixth edge line V6; or, the angle between the seventh edge line and the eighth edge line.

[0260] Second way:

[0261] The controller calculates a second angle between the first center line M1 and the second center line M2, and determines a relative angle difference between the first stacking object and the second stacking object according to the second angle.

[0262] exist Fig.11 In the figure, the second angle is an angle on the XOY plane in the three-dimensional coordinate system, and the second angle is an angle formed by projecting the first midline M1 and the second midline M2 onto the XOY plane.

[0263] The third way:

[0264] The controller calculates a second angle between the first center line M1 and the second center line M2;

[0265] The controller calculates a first angle between the first edge line W1 and the second edge line V2;

[0266] The controller obtains a relative angle difference between the first stacking object and the second stacking object by weighting the first angle and the second angle.

[0267] exist Fig.11 In , the relative angle difference is obtained by weighting the second angle between the first center line M1 and the second center line M2, and the first angle between the first edge line W1 and the second edge line V2; wherein the weighted proportion distribution can be adjusted according to the actual situation, for example, when the boundary point cloud corresponding to the first edge line W1 and the second edge line V2 has a higher clarity, the weighted proportion of the first angle can be increased. Fig.11In the example, the angle difference can be obtained by the difference between the direction indicated by the arrow at the upper arrow root, i.e., the first target point P1, and the direction indicated by the arrow at the lower arrow root, i.e., the second target point P2. Fig.12 (a) to (c) adopt the Fig.11 The direction indicated by the arrow of the first target point and the direction indicated by the arrow of the second target point are represented by different perspectives.

[0268] Fourth way:

[0269] The controller determines a first angle value of a first stacked object;

[0270] The controller determines a second angle value of a second stacked object;

[0271] The controller determines a relative angle difference between the first stacking object and the second stacking object according to a difference between the first angle value and the second angle value.

[0272] The controller determines the first angle value of the first stacking object, which may be:

[0273] The controller determines a first angle value of the first stacking object according to the angle of the first edge line W1; or

[0274] The controller determines a first angle value of the first stacking object according to the angle of the first midline M1; or

[0275] The controller obtains the first angle value of the first stacking object according to the weighted angle of the first center line M1 and the angle of the first edge line W1, where the weighted ratio can be adjusted according to actual conditions. For example, when the boundary point cloud data corresponding to the first edge line W1 is relatively clear, the angle weighted ratio of the first edge line W1 can be appropriately increased.

[0276] Similarly, the controller determines a second angle value of the second stacked object, which may be:

[0277] The controller determines a second angle value of the second stacking object according to the angle of the second edge line V2; or

[0278] The controller determines a second angle value of the second stacking object according to the angle of the second midline M2; or

[0279] The controller obtains the second angle value of the second stacking object by weighting the angle of the second center line M2 and the angle of the second edge line V2. The weighting ratio can be adjusted according to the actual situation. For example, when the boundary point cloud data corresponding to the second edge line V2 is relatively clear, the weighting ratio of the angle of the second edge line V2 can be appropriately increased.

[0280] It should also be noted that the first angle value of the first stacking object and the second angle value of the second stacking object can be flexibly selected according to actual conditions. For example, when the first angle value of the first stacking object is determined according to the angle of the first edge line W1, the second angle value of the second stacking object can be determined according to the angle of the second center line M2.

[0281] In the embodiment of the present application, the boundary line of the first grayscale image is obtained in the following manner: the controller extracts a straight line from the first grayscale image according to the probabilistic Hough transform straight line detection algorithm, and determines the boundary line of the first grayscale image from the extracted straight lines according to the size of the first stacked object; the boundary line of the second grayscale image is obtained in the following manner: the controller extracts a straight line from the second grayscale image according to the probabilistic Hough transform straight line detection algorithm, and determines the boundary line of the second grayscale image from the extracted straight lines according to the size of the second stacked object.

[0282] Here, the Probabilistic Hough Transform line detection algorithm reduces the amount of calculation by randomly selecting parameters of edge points in the parameter space.

[0283] In the rectangular coordinate system, the straight line of the first grayscale image can be expressed in polar coordinates as r=xcosθ+ysinθ, where r is the distance from the straight line to the origin, and θ is the direction angle of the straight line. Each edge point of the first grayscale image corresponds to a sine curve in the parameter space, and the intersection of these curves represents the straight line passing through these points. The probabilistic Hough transform determines the straight line parameters by randomly selecting edge points and calculating their corresponding parameter space curves, and then finding the intersection of these curves.

[0284] Here, the size of the first stacked object may include length, width and height, and these sizes are used as a basis for screening and matching straight lines from the extracted straight lines to determine the boundary line of the first grayscale image.

[0285] In the embodiment of the present application, a traditional probabilistic Hough transform line detection algorithm can be used to extract lines from a grayscale image. In a scenario where the first stacked object and the second stacked object have simple structures and little environmental interference, Hough transform line detection can also be used.

[0286] However, in order to be applicable to the first stacked object and the second stacked object with a more complex structure and improve the detection accuracy, the probabilistic Hough transform straight line detection algorithm is improved in the embodiment of the present application. Specifically, the controller may extract a straight line from the grayscale image (i.e., the first grayscale image or the second grayscale image) according to the improved probabilistic Hough transform straight line detection algorithm, including the following steps:

[0287] The controller sorts the pixel points in the grayscale image according to the grayscale value to obtain a pixel sequence set, wherein the pixel sequence set includes multiple pixel sequences, and each pixel sequence contains pixel points corresponding to the same grayscale value; the pixel sequences are taken out in sequence according to the sorting as the current pixel sequence, and pixel points are randomly selected in the current pixel sequence to find the straight line with the highest probability under the angle range, and the number of pixels on the straight line meets the straight line length requirement and the sum of the grayscale values ​​is the largest; if the distance between the found straight line and the extracted straight line is greater than or equal to the maximum straight line spacing threshold, the straight line is extracted, and the current pixel sequence is deleted from the pixel sequence set, and the next pixel sequence is continued to be used as the current pixel sequence until the pixel sequence set is empty.

[0288] It should be noted that the above-mentioned angle range, maximum straight line spacing threshold and number of straight lines can all be straight line parameters selected by the probabilistic Hough transform straight line detection algorithm in the parameter space.

[0289] Among them, for the pixel points in the current pixel sequence: calculate the corresponding series of straight line parameters (r and θ in the polar coordinate system) and vote in the parameter space. The angle range is used to limit the search direction of the straight line. The setting of the angle range needs to be adjusted according to the specific application scenario.

[0290] In one example, the parameters of the straight line include at least one of: pixel distance resolution, angle resolution, angle range, minimum straight line length threshold, maximum straight line spacing threshold, and the number of straight lines.

[0291] Specifically, (1) all pixels in the grayscale image are sorted from large to small according to grayscale values ​​to form a pixel sequence set; (2) pixel sequences are taken out from the pixel sequence set in order from large to small grayscale values, and pixel points are randomly taken out from the pixel sequence; (3) a straight line with the highest probability under the angle range is found among the taken out pixel points, the number of pixels on the straight line meets the straight line length threshold requirement (such as the above-mentioned minimum straight line length threshold) and the sum of grayscale values ​​is the largest; (4) all pixel points on the straight line are found, and it is determined whether the interval between the found straight line and the extracted straight line meets the maximum straight line spacing threshold. If not, return to (2), and if it meets, execute (5); (5) the straight line is taken out, and the pixel points on the straight line are removed from the pixels to be taken out; (6) it is determined whether all pixel points in the pixel sequence set have been traversed. If so, the taken out straight line set is returned; if not, it is determined whether the number of taken out straight lines meets the requirement. If not, continue (2), and if so, return the taken out straight line set.

[0292] It should be noted that the method of generating the boundary line of the second grayscale image is the same as the method of determining the boundary line of the first grayscale image, which will not be described in detail here.

[0293] The above step 407, ie, "the controller compares the relative posture data with the threshold value to confirm the alignment state of the first stacking object and the second stacking object" is described in detail below in conjunction with the embodiments.

[0294] In the embodiment of the present application, the controller can confirm the alignment state of the first stacking object and the second stacking object based on the relative posture data and the threshold. When the relative posture data is greater than or equal to the threshold, the alignment state is confirmed to be misaligned; when the relative posture data is less than the threshold, the alignment state is confirmed to be aligned.

[0295] In one example, when the relative posture data is greater than or equal to a threshold, the method further includes:

[0296] The controller controls the unmanned forklift to adjust its posture according to the relative posture data;

[0297] The controller reacquires the target point cloud of the first stacked object and the second stacked object through the sensor;

[0298] The controller re-determines the relative pose data according to the target point clouds of the first stacked object and the second stacked object;

[0299] The controller reconfirms the alignment status based on the relative pose data until the relative pose data is less than the threshold.

[0300] In one example, when the relative posture data is less than a threshold, the method further includes:

[0301] The controller controls the unmanned forklift to place the first stacking object on the second stacking object to complete the stacking.

[0302] Here, the threshold is used to measure whether to control the unmanned forklift to perform stacking of the first stacking object and the second stacking object, and the threshold can be set according to the alignment accuracy of the first stacking object and the second stacking object.

[0303] When the first pseudo image is a first grayscale image and the second pseudo image is a second grayscale image, an implementation method of the method proposed in the embodiment of the present application is introduced below in combination with an actual application scenario. Fig.13 As shown, the execution subject is an unmanned forklift.

[0304] exist Fig.13 In the example, the controller determines the relative position data between the first stacking object and the second stacking object according to the first grayscale image and the second grayscale image. The method includes the following steps:

[0305] Step 1301: The controller obtains original point clouds of the first stacking object and the second stacking object through a sensor.

[0306] Step 1302: The controller converts the original point cloud of the first stacking object from the coordinate system where the sensor is located to the coordinate system where the unmanned forklift is located, and according to the odometer information collected by the unmanned forklift, performs de-distortion processing on the original point cloud of the first stacking object and the original point cloud of the second stacking object, respectively, to obtain the target point cloud of the first stacking object; and the controller converts the original point cloud of the second stacking object from the coordinate system where the sensor is located to the coordinate system where the unmanned forklift is located, and according to the odometer information collected by the unmanned forklift, performs de-distortion processing on the original point cloud of the second stacking object and the original point cloud of the second stacking object, respectively, to obtain the target point cloud corresponding to the second stacking object.

[0307] Step 1303: The controller projects the target point cloud of the first stacking object and the target point cloud of the second stacking object onto the XOY plane respectively, and generates a first grayscale image corresponding to the first stacking object and a second grayscale image corresponding to the second stacking object.

[0308] Step 1304: The controller determines relative posture data between the first stacking object and the second stacking object according to the first grayscale image and the second grayscale image.

[0309] Step 1305: The controller determines whether the relative posture data is less than a threshold.

[0310] Step 1306 : When the relative posture data is greater than or equal to the threshold, the controller confirms that the alignment relationship between the first stacking object and the second stacking object is misaligned, and executes step 1307 .

[0311] Step 1307: The controller controls the unmanned forklift to adjust its posture according to the relative posture data, and continues to execute steps 1301 to 1305 until the relative posture difference is less than the threshold.

[0312] Step 1308: When the relative posture data is less than a threshold, the controller confirms that the alignment state of the first stacking object and the second stacking object is aligned.

[0313] Step 1309: The controller controls the unmanned forklift to place the first stacking object on the second stacking object to complete the stacking.

[0314] When the first pseudo image is a first grayscale image and the second pseudo image is a second grayscale image, an implementation method of the method proposed in the embodiment of the present application is introduced below in combination with an actual application scenario. Fig.14 As shown, the execution subject is an unmanned forklift, and the method includes the following steps:

[0315] Step 1401: The controller obtains original point clouds of the first stacking object and the second stacking object through a sensor.

[0316] Step 1402: The controller converts the original point cloud of the first stacking object from the coordinate system where the sensor is located to the coordinate system where the unmanned forklift is located, and according to the odometer information collected by the unmanned forklift, performs de-distortion processing on the original point cloud of the first stacking object and the original point cloud of the second stacking object, respectively, to obtain the target point cloud of the first stacking object; and the controller converts the original point cloud of the second stacking object from the coordinate system where the sensor is located to the coordinate system where the unmanned forklift is located, and according to the odometer information collected by the unmanned forklift, performs de-distortion processing on the original point cloud of the second stacking object and the original point cloud of the second stacking object, respectively, to obtain the target point cloud corresponding to the second stacking object.

[0317] Step 1403: Project the target point cloud of the first stacking object and the target point cloud of the second stacking object onto the XOY plane to generate a first grayscale image corresponding to the first stacking object and a second grayscale image corresponding to the second stacking object.

[0318] Step 1404: The controller determines the boundary line of the first grayscale image; the controller determines the boundary point cloud of the first stacking object corresponding to the boundary line of the first grayscale image from the target point cloud of the first stacking object; the controller determines the first target point of the first stacking object based on the boundary point cloud of the first stacking object; the controller determines the boundary line of the second grayscale image; the controller determines the boundary point cloud of the second stacking object corresponding to the boundary line of the second grayscale image from the target point cloud of the second stacking object; the controller determines the second target point of the second stacking object based on the boundary point cloud of the second stacking object.

[0319] Step 1405: The controller determines the first center line of the first stacking object based on the third edge line corresponding to the third boundary point cloud and the fifth edge line corresponding to the fifth boundary point cloud; the controller determines the second center line of the second stacking object based on the fourth edge line corresponding to the fourth boundary point cloud and the sixth edge line corresponding to the sixth boundary point cloud; the controller determines the relative angle difference based on the first center line and the second center line, and the first edge line corresponding to the first boundary point cloud and the second edge line corresponding to the second boundary point cloud.

[0320] Step 1406: The controller determines relative position data between the first stacking object and the second stacking object according to the relative position difference and the relative angle difference between the first target point and the second target point.

[0321] Step 1407: The controller determines whether the relative posture data is less than a threshold.

[0322] Step 1408 : When the relative posture data is greater than or equal to the threshold, the controller confirms that the alignment relationship between the first stacking object and the second stacking object is misaligned, and executes step 1409 .

[0323] Step 1409: The controller controls the unmanned forklift to adjust its posture according to the relative posture data, and continues to execute steps 1401 to 1407 until the relative posture difference is less than the threshold.

[0324] Step 1410: When the relative posture data is less than a threshold, the controller confirms that the alignment state of the first stacking object and the second stacking object is aligned.

[0325] Step 1411: The controller controls the unmanned forklift to place the first stacking object on the second stacking object to complete the stacking.

[0326] In the embodiment of the present application, controlling the unmanned forklift to adjust its posture based on relative posture data may include: controlling the unmanned forklift to adjust the posture of the first stacking object on the fork and / or controlling the unmanned forklift to adjust the posture of the chassis (i.e., adjusting the posture of the unmanned forklift relative to the second stacking object).

[0327] The embodiment of the present application can also control the unmanned forklift to adjust its posture based on the time when the sensors and odometers carried by the unmanned forklift collect data and the control system processes the data, specifically:

[0328] Determine the posture adjustment information of the unmanned forklift based on the relative posture data; obtain the odometer information collected by the unmanned forklift at the target time (i.e., the time of obtaining the target point cloud); determine the posture adjustment information corresponding to the current time according to the odometer information and posture adjustment information corresponding to the target time and the odometer information corresponding to the current time; control the unmanned forklift to adjust its posture according to the posture adjustment information corresponding to the current time.

[0329] Here, the posture adjustment information may be information for adjusting the position and posture of the unmanned forklift; the current moment and the target moment are different moments, and generally the target moment is a historical moment compared to the current moment.

[0330] For example, the posture adjustment information corresponding to the current moment can be determined based on the following formula:

[0331] E n ^=H n+1 -1 ·H n ·E n

[0332] Among them, H n is the odometer information corresponding to the target time, E n is the pose adjustment information, H n+1 -1 is the odometer information corresponding to the current moment, E n ^ is the posture adjustment information corresponding to the current moment.

[0333] The present application takes into account that the time when the data collected by the sensors and odometers carried by the unmanned forklift is not synchronized with the time when the data is processed by the control system. In order to ensure that the data processed by the control system is the posture adjustment information corresponding to the current moment, the posture adjustment information corresponding to the current moment is determined based on the odometer information corresponding to the collected time, the posture adjustment information corresponding to the historical moment (that is, the posture adjustment information determined based on the relative posture data) and the odometer information corresponding to the current moment. This can ensure that the unmanned forklift accurately adjusts its posture according to the posture adjustment information corresponding to the current moment, thereby ensuring the alignment accuracy of the first stacking object and the second stacking object at the current moment.

[0334] In some embodiments, before confirming the alignment state, the unmanned forklift moves the first stacking object to the preparation position to complete a pre-alignment action relative to the second stacking object.

[0335] Pre-alignment means: adjusting the position of the handling equipment so that the first stacking object and the second stacking object are basically aligned in the Y-axis direction. That is, the coordinate difference △Y of the first stacking object and the rotation angle difference △Ψ of the Z-axis between the first stacking object and the second stacking object are within the preset threshold. The threshold can be flexibly adjusted according to different handling equipment and stacking objects, for example, it can be -5cm<△Y<5cm, -2°

[0337] In the pre-alignment stage, the unmanned forklift carries the first stacking object and moves to the second stacking object. It first raises the fork to the optimal detection height, then calculates the posture of the second stacking object by detecting the pillars and other features of the second stacking object, and adjusts the chassis or fork to complete the pre-alignment. In this stage, in order to ensure that the unmanned forklift is basically aligned with the second stacking object in the direction of travel and rotation, the forward and backward path planning strategies are used to achieve large error correction to ensure that the first stacking object is basically aligned with the second stacking object.

[0338] The above method provided in the embodiment of the present application can be applied to a variety of application scenarios, including but not limited to: unmanned warehouse scenarios, unmanned loading and unloading scenarios.

[0339] Among them, unmanned warehouses may include unmanned forklifts, storage shelves, picking platforms, RCS control systems and warehouse management systems (Warehouse Management System, WMS), etc.

[0340] Among them, the unmanned loading and unloading scenarios include unmanned forklifts, RCS control systems, and trucks.

[0341] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0342] According to an embodiment of another aspect, a control system is provided for executing to implement any one of the alignment status confirmation methods disclosed in the embodiments of the present application.

[0343] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0344] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a controller, the method described in any one of the aforementioned method embodiments is implemented.

[0345] And a handling device, comprising:

[0346] Handling equipment body;

[0347] A sensor mounted on the handling equipment body;

[0348] one or more controllers; and

[0349] A memory coupled to one or more controllers, the memory is used to store program instructions, and when the program instructions are read and executed by one or more controllers, all or part of the steps in any one of the alignment status confirmation methods described in the above embodiments are executed.

[0350] In some embodiments, the handling equipment body includes an unmanned forklift body.

[0351] The present application also provides a computer program product, including a computer program, which implements any one of the methods described in the aforementioned method embodiments when executed by a controller.

[0352] It can be seen from the above description of the implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can essentially be embodied in the form of a computer program product, which can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc.

[0353] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for confirming an alignment state, characterized in that: include: The controller acquires target point clouds of the first stacking object and the second stacking object through a sensor; The controller determines a first pseudo image corresponding to the first stacking object according to the target point cloud of the first stacking object; The controller determines a second pseudo image corresponding to the second stacking object according to the target point cloud of the second stacking object; The controller determines relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image; The controller compares the relative posture data with a threshold value to confirm an alignment state of the first stacked object and the second stacked object.

2. The method according to claim 1, characterized in that: The first pseudo image is a first grayscale image, and the second pseudo image is a second grayscale image.

3. The method according to claim 1, characterized in that The method further comprises: Before confirming the alignment state, the controller controls the transport device to transport the first stacked object to a preparation position to complete a pre-alignment action relative to the second stacked object.

4. The method according to claim 1, characterized in that: The controller determines relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image, including: The controller determines a boundary line of the first pseudo image; The controller determines a target point of the first pseudo image according to a boundary line of the first pseudo image; The controller determines a boundary line of the second pseudo image; The controller determines a target point of the second pseudo image according to a boundary line of the second pseudo image; The controller determines relative position data between the first stacking object and the second stacking object according to a boundary line of the first pseudo image and a target point of the first pseudo image, and a boundary line of the second pseudo image and a target point of the second pseudo image.

5. The method according to claim 4, characterized in that The controller determines a target point of the first pseudo image according to a boundary line of the first pseudo image, including: The controller determines a center line of the first pseudo image according to a third boundary line and a fifth boundary line of the first pseudo image; and uses an intersection point of the center line of the first pseudo image and a first boundary line of the first pseudo image as a target point of the first pseudo image; The controller determines the target point of the second pseudo image according to the boundary line of the second pseudo image, including: The controller determines the center line of the second pseudo image according to the fourth boundary line and the sixth boundary line of the second pseudo image; and uses the intersection of the center line of the second pseudo image and the second boundary line of the second pseudo image as the target point of the second pseudo image.

6. The method according to claim 5, characterized in that The third boundary line and the fifth boundary line in the first pseudo image are parallel to each other; The second boundary line and the fourth boundary line in the second pseudo image are parallel to each other; The third boundary line in the first pseudo image is perpendicular to the first boundary line of the first pseudo image, and the fifth boundary line of the first pseudo image is perpendicular to the first boundary line of the first pseudo image; The fourth boundary line in the second pseudo image is perpendicular to the second boundary line of the second pseudo image, and the sixth boundary line of the second pseudo image is perpendicular to the second boundary line of the second pseudo image.

7. The method according to any one of claims 4 to 6, characterized in that: The target point of the first pseudo image is the midpoint of a first boundary line of the first pseudo image; The target point of the second pseudo image is a midpoint of a second boundary line of the second pseudo image.

8. The method according to claim 1, characterized in that The controller determines relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image, including: The controller determines a boundary line of the first pseudo image; The controller determines a boundary point cloud of the first stacking object corresponding to a boundary line of the first pseudo image from the target point cloud of the first stacking object; The controller determines a first target point of the first stacked object according to a boundary point cloud of the first stacked object; The controller determines a boundary line of the second pseudo image; The controller determines a boundary point cloud of the second stacking object corresponding to a boundary line of the second pseudo image from the target point cloud of the second stacking object; The controller determines a second target point of the second stacked object according to the boundary point cloud of the second stacked object; The controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object; The controller determines relative position data between the first stacking object and the second stacking object according to a relative position difference between the first target point and the second target point and the relative angle difference.

9. The method according to claim 8, characterized in that The boundary point cloud of the first stacked object includes a first boundary point cloud, a third boundary point cloud and a fifth boundary point cloud; The boundary point cloud of the second stacked object includes a second boundary point cloud, a fourth boundary point cloud and a sixth boundary point cloud.

10. The method according to claim 9, characterized in that The controller determines a first target point of the first stacking object according to a boundary point cloud of the first stacking object, including: The controller determines a first edge line of the first stacking object according to the first boundary point cloud; The controller determines a third edge line of the first stacking object according to the third boundary point cloud; The controller determines a fifth edge line of the first stacked object according to the fifth boundary point cloud; The controller determines a first center line according to the third edge line and the fifth edge line; The controller determines the first target point according to the first center line and the first edge line; The controller determines a second target point of the second stacking object according to the boundary point cloud of the second stacking object, including: The controller determines a second edge line of the second stacked object according to the second boundary point cloud; The controller determines a fourth edge line of the second stacking object according to the fourth boundary point cloud; The controller determines a sixth edge line of the second stacking object according to the sixth boundary point cloud; The controller determines a second center line according to the second edge line and the sixth edge line; The controller determines the second target point according to the second center line and the second edge line.

11. The method according to claim 10, characterized in that The first target point is the intersection of the first center line and the first edge line; the second target point is the intersection of the second center line and the second edge line.

12. The method according to claim 10, characterized in that The controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, comprising: The controller determines a relative angle difference between the first stacking object and the second stacking object according to a first angle between the first edge line and the second edge line and according to the first angle.

13. The method according to claim 10, characterized in that The controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, comprising: The controller calculates a second angle between the first center line and the second center line, and determines a relative angle difference between the first stacking object and the second stacking object according to the second angle.

14. The method according to claim 10, characterized in that The controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, comprising: The controller calculates a second angle between the first center line and the second center line; The controller calculates a first angle between the first edge line and the second edge line; The controller obtains a relative angle difference between the first stacking object and the second stacking object by weighting the first angle and the second angle.

15. The method according to claim 10, characterized in that The controller determines a relative angle difference between the first stacking object and the second stacking object according to a boundary point cloud of the first stacking object and a boundary point cloud of the second stacking object, comprising: The controller determines a first angle value of the first stacked object; The controller determines a second angle value of the second stacked object; The controller determines a relative angle difference between the first stacking object and the second stacking object according to a difference between the first angle value and the second angle value.

16. The method according to claim 15, characterized in that The controller determines a first angle value of the first stacked object, including: The controller determines a first angle value of the first stacking object according to an angle of the first edge line; or The controller determines a first angle value of the first stacking object according to an angle of the first center line; or The controller obtains a first angle value of the first stacked object by weighting the angle of the first center line and the angle of the first edge line.

17. The method according to claim 15, characterized in that The controller determines a second angle value of the second stacked object, including: The controller determines a second angle value of the second stacking object according to the angle of the second edge line; or The controller determines a second angle value of the second stacking object according to the angle of the second center line; or The controller obtains a second angle value of the second stacked object by weighting the angle of the second center line and the angle of the second edge line.

18. The method according to claim 1, characterized in that The controller determines a first pseudo image corresponding to the first stacking object according to the target point cloud of the first stacking object, including: The controller projects the target point cloud of the first stacking object onto a horizontal plane to generate a first pseudo image; The controller determines, according to the target point cloud of the second stacking object, a second pseudo image corresponding to the second stacking object, including: The controller projects the target point cloud of the second stacking object onto a horizontal plane to generate a second pseudo image.

19. The method according to claim 17, characterized in that The controller projects the target point cloud of the first stacking object onto a horizontal plane to generate a first pseudo image, including: The controller projects the target point cloud of the first stacking object onto a horizontal plane using at least two projection resolutions to obtain images of at least two projection resolutions corresponding to the target point cloud of the first stacking object; scales the images of at least two projection resolutions corresponding to the target point cloud of the first stacking object to a uniform size and superimposes them to obtain the first pseudo image; The controller projects the target point cloud of the second stacking object onto a horizontal plane to generate a second pseudo image, including: The controller projects the target point cloud of the second stacking object onto a horizontal plane using at least two projection resolutions to obtain images of at least two projection resolutions corresponding to the target point cloud of the second stacking object; and scales the images of at least two projection resolutions corresponding to the target point cloud of the second stacking object to a uniform size and superimposes them to obtain the second pseudo image.

20. The method according to claim 2, characterized in that The controller determines a first pseudo image corresponding to the first stacking object according to the target point cloud of the first stacking object, including: The controller projects the target point cloud of the first stacking object onto a horizontal plane using at least two projection resolutions to obtain images of at least two projection resolutions corresponding to the target point cloud of the first stacking object; scales the images of at least two projection resolutions corresponding to the target point cloud of the first stacking object to a uniform size and superimposes them, and converts the superimposed images into a first grayscale image; The controller determines, according to the target point cloud of the second stacking object, a second pseudo image corresponding to the second stacking object, including: The controller projects the target point cloud of the second stacking object onto a horizontal plane using at least two projection resolutions to obtain images of at least two projection resolutions corresponding to the target point cloud of the first stacking object; scales the images of at least two projection resolutions corresponding to the target point cloud of the first stacking object to a uniform size and superimposes them, and converts the superimposed images into a second grayscale image.

21. The method according to claim 1, characterized in that The controller compares the relative posture data with a threshold value to confirm an alignment state of the first stacking object and the second stacking object, including: If the relative posture data is greater than or equal to a threshold, the controller confirms that the alignment state is misaligned; If the relative posture data is less than a threshold, the controller confirms that the alignment state is aligned.

22. The method according to claim 21, characterized in that The method further comprises: When the alignment state is misaligned, the controller controls the handling device to adjust the posture; The controller reacquires the target point clouds of the first stacking object and the second stacking object through the sensor; The controller re-determines the relative posture data; The controller reconfirms the alignment state according to the relative posture data until the relative posture data is less than a threshold.

23. The method according to claim 21, characterized in that The method further comprises: When the alignment state is aligned, the controller controls the transport device to place the first stacking object on the second stacking object to complete stacking.

24. A control system, characterized in that: The invention comprises a controller and a memory, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 23.

25. A handling device, characterized in that: The invention comprises a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 23.

Images