Alignment method, control system and carrying equipment

Abstract

The embodiment of the invention discloses an alignment method, a control system and carrying equipment. According to the main technical scheme, the method comprises the steps that a controller obtains target data of a first stacking object and a second stacking object through a first sensor; the controller extracts first target data of the first stacking object and second target data of the second stacking object from the target data; the controller calculates the relative pose of the first stacking object relative to the second stacking object according to the first target data and the second target data; and the controller controls carrying equipment to move according to the relative pose, so that the first stacked object is aligned with the second stacked object. According to the method and the device, the target data of the two stacked objects are acquired and processed at the same time to correct the relative pose between the two stacked objects, so that the accuracy and the efficiency of the stacking operation are remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of warehousing logistics, and in particular to an alignment method, a control system and a handling device. Background Art

[0002] Systems that use handling equipment such as AGV (automated guided vehicle) have the advantages of being highly unmanned, automated, and intelligent, which improves production efficiency and operational levels for industries such as warehousing, manufacturing, and logistics. As one of the more typical scenarios, handling equipment is often responsible for the handling of various goods. In the process of handling, it is inevitable to stack goods. Goods are usually packaged in cartons, etc., or stored in cages, wooden boxes, plastic boxes, etc.

[0003] Considering the space utilization, the handling equipment will involve stacking objects such as cartons, cages, wooden boxes, etc. Considering the stability, the handling equipment needs to accurately stack one stacking object on top of another stacking object. In this process, if the two stacking objects cannot be aligned, it may affect the safety of the operation. Summary of the invention

[0004] The present application provides an alignment method, a control system and a handling device, which can improve the accuracy and efficiency of stacking operations.

[0005] This application provides the following solutions:

[0006] According to a first aspect, there is provided an alignment method, comprising:

[0007] The controller acquires target data of the first stacking object and the second stacking object through the first sensor;

[0008] The controller extracts first target data of the first stacked object and second target data of the second stacked object from the target data;

[0009] The controller calculates a relative posture of the first stacking object with respect to the second stacking object according to the first target data and the second target data;

[0010] The controller controls the movement of the handling device according to the relative position and posture to align the first stacking object with the second stacking object.

[0011] Optionally, before the controller acquires the target data of the first stacking object and the second stacking object through the first sensor, the method further includes:

[0012] The controller controls the transport device to transport the first stacking object to a stacking preparation position;

[0013] The controller acquires fourth target data of the second stacking object through the first sensor;

[0014] The controller controls the movement of the transport device according to the fourth target data so that a posture error of the transport device relative to the second stacking object is within a first preset threshold range.

[0015] Optionally, the controller controls the movement of the transport device according to the fourth target data so that the posture of the transport device relative to the second stacking object is within a first preset threshold range, including:

[0016] The controller determines the posture of the second stacking object according to the fourth target data;

[0017] According to the posture of the second stacking object, the transport device is controlled to move so that a posture error of the transport device relative to the second stacking object is within a first preset threshold range.

[0018] Optionally, the controller acquires target data of the first stacking object and the second stacking object through the first sensor, including:

[0019] When the controller controls the transport device to transport the first stacking object to a stacking operation position, the controller acquires the target data through a first sensor.

[0020] Optionally, the controller controls the transport device to transport the first stacking object to a stacking operation position, including:

[0021] The transport device is controlled to move toward the second stacking object until a projection length of a fork of the transport device on the second stacking object reaches a first length threshold.

[0022] Optionally, the controller controls the transport device to transport the first stacking object to a stacking operation position, including:

[0023] The transport device is controlled to move toward the second stacking object until a longitudinal distance between a first sensor of the transport device and the second stacking object is within a first preset distance.

[0024] Optionally, controlling the movement of the handling device according to the relative position to align the first stacking object with the second stacking object comprises:

[0025] According to the relative position and posture, the transport device is controlled to move until the relative position and posture is within a second preset threshold range.

[0026] Optionally, the method further comprises:

[0027] When the relative posture is not within the second preset threshold range, the controller controls the handling device to adjust the posture;

[0028] The controller reacquires target data of the first stacking object and target data of the second stacking object through the first sensor;

[0029] The controller re-determines a relative posture between a posture of the first stacking object and a posture of the second stacking object according to the re-acquired target data.

[0030] Optionally, controlling the movement of the transport equipment includes:

[0031] When the longitudinal distance between the transport device and the stacking end point is greater than a preset distance threshold, only the body of the transport device is controlled to move;

[0032] When the longitudinal distance between the transport device and the stacking end point is not greater than the distance threshold, the body of the transport device is controlled to move while the fork is controlled to move in the lateral direction.

[0033] Optionally, controlling the body of the transporting equipment to move while controlling the fork to move laterally includes:

[0034] Calculating a lateral error of the vehicle body relative to the second stacking object, and calculating a lateral target value of the fork according to the lateral error;

[0035] The longitudinal error of the vehicle body relative to the second stacking object is calculated, and the movement of the vehicle body is controlled according to the longitudinal error until the longitudinal error is within a preset longitudinal threshold range; at the same time, according to the lateral target value, the fork is controlled to move in the lateral direction so that the lateral error of the fork relative to the second stacking object is within a preset lateral threshold range.

[0036] Optionally, the method further comprises:

[0037] The longitudinal distance between the handling device and the stacking end point is acquired in real time, and the movement speed of the vehicle body and / or the movement speed of the fork is controlled according to the longitudinal distance.

[0038] Optionally, after controlling the transport device to move according to the relative posture until the relative posture is within a second preset threshold range, the method further includes:

[0039] Obtaining a current position of the first stacked object;

[0040] determining coordinates of at least one corner point of the first stacked object according to a current position of the first stacked object and a size of the first stacked object;

[0041] Obtaining a current position of the second stack object;

[0042] determining coordinates of at least one corner point of the second stacked object according to the current position of the second stacked object and the size of the second stacked object;

[0043] wherein at least one corner point of the first stacked object corresponds to at least one corner point of the second stacked object;

[0044] calculating a coordinate difference between at least one corner point of the first stacked object and at least one corner point of the second stacked object;

[0045] When the coordinate difference is within a third preset threshold range, it is confirmed that the first stacking object and the second stacking object are aligned successfully; otherwise, it is confirmed that the first stacking object and the second stacking object are aligned unsuccessfully.

[0046] Optionally, the method further comprises:

[0047] When the first stacking object and the second stacking object fail to be aligned, the controller determines a target posture of the transport device according to a current relative posture of the first stacking object and the second stacking object;

[0048] The controller controls the movement of the transport device to adjust the posture of the transport device to the target posture;

[0049] The controller reacquires target data of the first stacking object and the second stacking object through a first sensor;

[0050] re-determining the relative position and posture of the first stacking object and the second stacking object according to the re-acquired target data;

[0051] According to the re-determined relative position and posture, it is determined whether the first stacking object and the second stacking object are aligned.

[0052] Optionally, the method further comprises:

[0053] When the first stacking object and the second stacking object fail to be aligned, the controller determines a target posture of the transport device according to a current relative posture of the first stacking object and the second stacking object;

[0054] The controller controls the movement of the transport device to adjust the posture of the transport device to the target posture;

[0055] The controller reacquires target data of the first stacking object through the first sensor;

[0056] The controller re-determines the posture of the first stacked object according to the re-acquired target data;

[0057] Re-determining the relative posture of the first stacked object and the second stacked object according to the re-determined posture of the first stacked object and the posture of the second stacked object when the alignment fails;

[0058] According to the re-determined relative position and posture, it is determined whether the first stacking object and the second stacking object are aligned.

[0059] Optionally, the controller controls the transport device to move and adjusts the posture of the transport device to the target posture, including:

[0060] The controller controls the transport device to move to a first preset position;

[0061] The controller controls the transport device to move from the first preset position to a position corresponding to the target posture.

[0062] Optionally, the method further comprises:

[0063] When the first stacking object and the second stacking object fail to be aligned, the controller reacquires target data of the first stacking object and the second stacking object through the first sensor;

[0064] The controller controls the transport device to adjust its posture so that the first stacking object and the second stacking object are realigned.

[0065] Optionally, before the controller reacquires the target data of the first stacking object and the second stacking object through the first sensor, the method further includes:

[0066] The controller controls the transport device to transport the first stacking object to a stacking preparation position;

[0067] The controller acquires fourth target data of the second stacking object through the first sensor;

[0068] The controller determines the posture of the second stacking object according to the fourth target data;

[0069] The controller controls the movement of the transport device according to the posture of the second stacking object, so that the posture error of the transport device relative to the second stacking object is within the first preset threshold range.

[0070] Optionally, the method further comprises:

[0071] When the first stacking object and the second stacking object fail to be aligned, the controller controls the transport device to move back to the stacking operation position, and the controller reacquires target data of the first stacking object and the second stacking object through the first sensor;

[0072] The controller controls the transport device to adjust its posture so that the first stacking object and the second stacking object are realigned.

[0073] Optionally, the method further comprises:

[0074] The number of alignment failures is counted, and when the number of alignment failures is greater than a first preset number, an alarm prompt is output.

[0075] Optionally, after the first stacked object is aligned with the second stacked object, the method further comprises:

[0076] The transport device is controlled to stack the first stacking object on the second stacking object.

[0077] Optionally, after controlling the handling device to stack the first stacking object on the second stacking object, the method further includes:

[0078] It is determined whether the first stacking object is successfully stacked on the second stacking object.

[0079] Optionally, the determining whether the first stacking object is successfully stacked on the second stacking object includes:

[0080] The sensed stacking state of the first stacking object relative to the second stacking object is confirmed by using a sensed detection method.

[0081] Optionally, the determining whether the first stacking object is successfully stacked on the second stacking object further includes:

[0082] The pressure sensing stacking state of the second stacking object relative to the first stacking object is confirmed by using a pressure sensing detection method.

[0083] Optionally, using a perception detection method to confirm the perceived stacking state of the first stacking object relative to the second stacking object includes:

[0084] The controller scans the first stacking object and the second stacking object through a first sensor to obtain structural feature data of a stacking area of ​​the first stacking object and the second stacking object;

[0085] Based on the structural characteristic data, a perceived stacking state of the first stacking object relative to the second stacking object is determined.

[0086] Optionally, confirming the pressure-sensitive stacking state of the second stacking object relative to the second stacking object by using a pressure-sensitive detection method includes:

[0087] The controller obtains comprehensive pressure data through the second sensor, and confirms the pressure-sensitive stacking state of the second stacking object relative to the second stacking object according to the comprehensive pressure data.

[0088] Optionally, the method further comprises:

[0089] When the first stacking object is not successfully stacked on the second stacking object, the controller controls the transport device to lift the first stacking object;

[0090] The controller controls the transport device to re-stack the first stacked object on the second stacked object.

[0091] Optionally, the method further comprises:

[0092] When the first stacking object is not successfully stacked on the second stacking object, the controller controls the transport device to lift the first stacking object;

[0093] The controller reacquires target data of the first stacking object and the second stacking object through a first sensor;

[0094] The controller controls the transport device to adjust its posture so that the first stacking object and the second stacking object are realigned;

[0095] The controller controls the transport device to re-stack the first stacked object on the second stacked object.

[0096] Optionally, before the controller reacquires the target data of the first stacking object and the second stacking object through the first sensor, the method further includes:

[0097] The controller controls the transport device to transport the first stacking object to a stacking preparation position;

[0098] The controller acquires fourth target data of the second stacking object through the first sensor;

[0099] The controller controls the movement of the transport device according to the acquired fourth target data so that a posture error of the transport device relative to the second stacking object is within a first preset threshold range.

[0100] Optionally, before the controller reacquires the target data of the first stacking object and the second stacking object through the first sensor, the method further includes:

[0101] The controller controls the transport device to move to the stacking operation position again.

[0102] Optionally, the method further comprises:

[0103] The controller counts the number of unsuccessful stacking times, and outputs an alarm prompt when the number of unsuccessful stacking times is greater than a second preset number.

[0104] Optionally, the method further comprises:

[0105] The controller counts the number of alignment failures and the number of unsuccessful stacking;

[0106] When the sum of the alignment failure times and the unsuccessful stacking times is greater than a third preset number, an alarm prompt is output.

[0107] Optionally, before the controller controls the transport device to transport the first stacking object to the stacking preparation position, the method further includes:

[0108] The controller controls the transport device to move to a pickup position;

[0109] The controller acquires third target data of the first stacking object through the first sensor;

[0110] According to the third target data of the first stacked object, controlling the movement of the transport device to obtain the first stacked object;

[0111] The controller controls the transport device to transport the first stacking object to a stacking preparation area.

[0112] Optionally, the controller controls the transport device to transport the first stacking object to a stacking preparation position, including:

[0113] The controller controls the movement of the vehicle body and the lifting of the first stacking object simultaneously during the transport process.

[0114] According to a second aspect, a control system is provided, comprising a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the steps of any one of the methods described in the first aspect.

[0115] According to a third aspect, a handling device is provided, comprising a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the steps of any one of the methods described in the first aspect.

[0116] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0117] The present application obtains target data of two stacking objects through a first sensor, and a controller extracts first target data of the first stacking object and second target data of the second stacking object from the target data, and then calculates the relative posture of the first stacking object relative to the second stacking object, and controls the movement of the handling device according to the relative posture to align the first stacking object with the second stacking object. Compared with the existing solution, the relative posture between the two stacking objects can be corrected by simultaneously obtaining and processing the target data of the two stacking objects, thereby significantly improving the accuracy and efficiency of the stacking operation.

[0118] Of course, any invention of the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0120] Figure 1 A schematic diagram of the structure of the handling equipment provided in the embodiment of the present application;

[0121] Figure 2 is a schematic diagram of a first target area and a second target area;

[0122] Figure 3 is a schematic diagram of a third target area and a fourth target area;

[0123] Figure 4 is a schematic diagram of a fifth target area, a sixth target area, a seventh target area and an eighth target area;

[0124] Figure 5 A flow chart of a method for aligning a first stacking object and a second stacking object provided in an embodiment of the present application;

[0125] Figure 6 A schematic diagram of a handling device carrying a first stacking object to a stacking operation position;

[0126] Figure 7 A flowchart for calculating the relative posture of a first stacking object relative to a second stacking object provided in an embodiment of the present application;

[0127] Figure 8 A schematic diagram of a segmented image provided in an embodiment of the present application;

[0128] Fig. 9 A schematic diagram of a foot cup and a column provided in an embodiment of the present application;

[0129] Fig.10 Schematic diagram of the first target area and the second target area provided in the embodiment of the present application Figure 1 ;

[0130] Fig.11 Schematic diagram of the first target area and the second target area provided in the embodiment of the present application Figure 2 ;

[0131] Fig.12 A schematic diagram of the border lines of a first stacking object and a second stacking object provided in an embodiment of the present application;

[0132] Fig.13 A flowchart for calculating the relative posture of a first stacking object relative to a second stacking object provided in an embodiment of the present application;

[0133] Fig.14 A schematic diagram of generating a first grayscale image provided in an embodiment of the present application;

[0134] Fig.15 A schematic diagram of generating a second grayscale image provided in an embodiment of the present application;

[0135] Fig.16 A flowchart for determining relative position data between a first stacking object and a second stacking object provided in an embodiment of the present application;

[0136] Fig.17 A schematic diagram of a first grayscale image provided in an embodiment of the present application;

[0137] Fig.18 A schematic diagram of a second grayscale image provided in an embodiment of the present application;

[0138] Fig.19 A flowchart for determining relative position data between a first stacking object and a second stacking object provided in an embodiment of the present application;

[0139] Fig. 20 A schematic diagram of calculating a first target point and a second target point provided in an embodiment of the present application;

[0140] Fig.21 A schematic diagram of the direction indicated by an arrow at a first target point and the direction indicated by an arrow at a second target point at different viewing angles provided in an embodiment of the present application;

[0141] Fig. 22 A flowchart for implementing the second stage of controlling the movement of the transport equipment provided in an embodiment of the present application;

[0142] Fig.23 A schematic diagram of corner points of a first stacking object and a second stacking object provided in an embodiment of the present application;

[0143] Fig.24 A schematic diagram of the entire stacking process provided in an embodiment of the present application;

[0144] Fig.25 A flowchart of a stacking status confirmation method provided in an embodiment of the present application;

[0145] Fig.26 A schematic diagram of successful stacking provided in an embodiment of the present application;

[0146] Fig. 27 A schematic diagram of a stacking failure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0147] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0148] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0149] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, the term "according to" used in this article is not limited to being based on a certain object. For example, determining B based on A can mean: directly determining B based on A, or determining B partially based on A.

[0150] In the related art, the stacking of two stacked objects is mainly achieved by detecting the posture of the stacked object below. However, this method may be affected by various factors such as inaccurate posture of the stacked object above, uneven ground, cumulative errors in the odometer, and errors in the handling equipment itself, resulting in the two stacked objects not being able to be completely aligned when stacked, affecting operational safety.

[0151] In view of this, the present application provides a new idea, an alignment method, a control system and a handling device.

[0152] The method provided in the embodiment of the present application can be applied to Figure 1 In the handling equipment shown, Figure 1 As shown, it is a schematic diagram of the structure of the handling equipment provided in the embodiment of the present application, the handling equipment 1 stacks the first stacking object A onto the second stacking object B, wherein the handling equipment 1 includes a handling equipment body 10, a sensor 30, a stacking execution component 20, a controller 40 and a memory 50. The controller 40 can be a control mainboard, a control box, a control unit, a vehicle-mounted computer, a computing platform, a tablet computer, a computer, etc. on the handling equipment body 10, or a system or device that plays a computing or controlling role in a local server or a cloud server, or can also be a handheld controller, a remote controller, etc. Other forms. This is not limited in the embodiments of the present application.

[0153] The handling equipment 1 involved in the embodiment of the present application can be an unmanned forklift, a pallet truck, a crane, an AGV (Automated Guided Vehicle), an AMR (Autonomous Mobile Robot) or a humanoid robot, etc., and the corresponding stacking execution component 20 can be a fork, a robotic arm, etc.

[0154] Normally, the handling device 1 receives a handling task from the RCS (Robot Control System). After receiving the handling task, the handling device 1 can be controlled by its own controller 40 to execute the handling task. For example, the handling device 1 is controlled to first move to the first stacking object A, and the first stacking object A is forked by the stacking execution component 20, and then the first stacking object A is moved to the vicinity of the second stacking object B, and the first stacking object A is stacked on the second stacking object B.

[0155] The RCS may be integrated in the transport device 1 and form a whole with the transport device 1, or may be a separate device different from the transport device 1, such as a tablet computer, a laptop computer, a computer (Personal Computer, PC), a local server, a cloud server, etc. This is not limited in the embodiments of the present application.

[0156] The sensor 30 may be in the form of a sensor module, and may include a radar for collecting point cloud data, such as a laser radar, which may be installed on the stacking execution component 20 to facilitate the collection of point cloud data. Furthermore, the laser radar may be a three-dimensional laser radar. It may also include a camera for collecting image data, which may be installed on the stacking execution component 20 to facilitate the collection of image data. It may also include a pressure sensor for collecting pressure data, which may be installed on the stacking execution component 20.

[0157] The memory 50 is mainly used to store data collected by the sensor 30, such as point cloud data, image data or pressure data. For example, the memory 50 can be a non-volatile computer storage medium.

[0158] The following is an introduction to the noun concepts involved in the embodiments of the present application.

[0159] Stacking refers to arranging and stacking several objects up and down according to certain rules.

[0160] Stacking objects refer to the objects involved in stacking, which can be the goods themselves, or goods with simple packaging such as wrapping, or containers that can hold and carry goods, such as cages, wooden boxes, plastic boxes, pallets, etc.

[0161] Stacking process: refers to the handling device lifting the first stacking object, so that the first stacking object moves and approaches the second stacking object, and then by adjusting the posture of the handling device, the first stacking object is aligned with the second stacking object, and then the first stacking object is placed on the second stacking object to complete the stacking.

[0162] The first stacking object refers to the stacking object located at the top, and the second stacking object refers to the stacking object located at the bottom.

[0163] Stacking state: refers to the relative position state of two stacking objects during the stacking process.

[0164] Alignment: refers to the state where two or more stacked objects are arranged along a straight line in the vertical direction, and at least part of the border lines between the stacked objects are parallel or overlapped. The vertical direction refers to the Z-axis direction in the coordinate system of the handling equipment (i.e. the height direction of the handling equipment).

[0165] The first stacking object has a first target area, a third target area, a fifth target area and a seventh target area, which refer to specific areas on the first stacking object used for detection and analysis, such as the frame structure, foot cup, column, corner structure and other key structural areas of the first stacking object.

[0166] The second stacking object has a second target area, a fourth target area, a sixth target area and an eighth target area, which refer to specific areas on the second stacking object used for detection and analysis, such as the frame structure, foot cup, column, corner structure and other key structural areas of the second stacking object.

[0167] It should also be noted that, in the embodiment of the present application, the second stacking object may not only be goods or a container for accommodating goods, but may also be a stationary base serving as a stacking foundation.

[0168] like Figure 2 The figure shows a schematic diagram of the first target area and the second target area, wherein the stacking object A located at the top is the first stacking object, the stacking object B located at the bottom is the second stacking object, the bottom structural area of ​​the first stacking object is the first target area A1, and the top structural area of ​​the second stacking object is the second target area B2.

[0169] like Figure 3 The figure shows a schematic diagram of the third target area and the fourth target area. Wherein, stacking object A is the first stacking object, and stacking object B is the second stacking object. When the controller controls the transport device to move to the pickup position, the structural areas on both sides of stacking object A (such as columns and sockets, etc.) are the third target area A3 when observed along the length direction of the transport device (which is also the driving direction of the transport device); when the controller controls the transport device to transport the first stacking object to the stacking preparation position, the structural areas on both sides of stacking object B (such as columns and sockets, etc.) are the fourth target area B4 when observed along the length direction of the transport device.

[0170] like Figure 4 As shown, it is a schematic diagram of the fifth target area, the sixth target area, the seventh target area and the eighth target area. Among them, the stacking object A located at the top is the first stacking object, and the stacking object B located at the bottom is the second stacking object. The structural area on one side of the bottom of the first stacking object is the fifth target area A5, the structural area on the other side of the bottom is the seventh target area A7, and the structural area on one side of the top of the second stacking object is the sixth target area B6, and the structural area on the other side of the top is the eighth target area B8. Among them, the fifth target area A5 and the seventh target area A7 are respectively located on opposite sides of the first stacking object. The sixth target area B6 and the eighth target area B8 are respectively located on opposite sides of the second stacking object. In the stacking process, the first target area A5 and the second target area B6 are located on the same side, and the third target area A7 and the fourth target area B8 are located on the other same side. Figure 4 (a) and (b) show two different target area configurations.

[0171] In addition, the fifth target area and the seventh target area may also be located on two adjacent sides of the first stacked object, and the sixth target area and the eighth target area may also be located on two adjacent sides of the second stacked object.

[0172] It should be noted that if Figure 1 In the embodiment of the present application, in a coordinate system with the geometric center of the transport device 1 as the origin O, the front and rear travel direction of the transport device (i.e., the longitudinal direction of the transport device body 10) is the X-axis, wherein the positive direction of the X-axis is the direction away from the attachment (such as a fork) of the transport device 1, the lateral direction of the transport device 1 is the Y-axis, the positive direction of the Y-axis is perpendicular to the paper and outward (not shown in the figure), and the height direction of the transport device 1 is the Z-axis.

[0173] Target data: refers to data of a target area of ​​a stacked object acquired by the first sensor, including point cloud data and / or image data, etc.

[0174] First target data: refers to data of the first target area of ​​the first stacking object acquired by the first sensor, including point cloud data and / or image data. The first target data can be used to calculate the position and posture, stacking state, etc. of the first stacking object.

[0175] The third target data refers to the data of the third target area of ​​the first stacked object acquired by the first sensor, including point cloud data and / or image data. The third target data can be used to calculate the position and posture of the first stacked object.

[0176] The fifth target data refers to the data of the fifth target area of ​​the first stacking object acquired by the first sensor, including point cloud data and image data. The fifth target data can be used to calculate the position and posture, stacking state, etc. of the first stacking object.

[0177] The seventh target data refers to the data of the seventh target area of ​​the first stacking object acquired by the first sensor, including point cloud data and / or image data. The seventh target data can be used to calculate the position and posture, stacking state, etc. of the first stacking object.

[0178] Second target data: refers to data of the second target area of ​​the second stacking object acquired by the first sensor, including point cloud data and / or image data. The second target data can be used to calculate the position and posture, stacking state, etc. of the second stacking object.

[0179] Fourth target data: refers to data of a fourth target area of ​​the second stacked object acquired by the first sensor, including point cloud data and / or image data. The fourth target data can be used to calculate the position and posture of the second stacked object.

[0180] The sixth target data refers to the data of the sixth target area of ​​the second stacking object acquired by the first sensor, including point cloud data and / or image data. The sixth target data can be used to calculate the position and posture, stacking state, etc. of the second stacking object.

[0181] The eighth target data refers to the data of the eighth target area of ​​the second stacking object acquired by the first sensor, including point cloud data and / or image data. The eighth target data can be used to calculate the position and posture, stacking state, etc. of the second stacking object.

[0182] In addition, during the stacking process, the positions involved in the embodiment of the present application are as follows:

[0183] Pick-up position: refers to the position where the handling equipment forks the first stacked object;

[0184] Stacking preparation area: refers to an area having a second stacking object on which a first stacking object can be stacked.

[0185] Stacking preparation position: refers to the position before reaching the stacking operation position, at which the handling equipment can obtain the position and posture of the second stacking object.

[0186] Stacking operation position: refers to the position where the first sensor on the handling device can simultaneously obtain the target data of the first stacking object and the second stacking object.

[0187] Figure 5 A flowchart of an alignment method provided in an embodiment of the present application. Figure 5 As shown in , the method may include the following steps:

[0188] Step 501, the controller obtains target data of a first stacking object and a second stacking object through a first sensor;

[0189] Step 502, the controller extracts first target data of the first stacking object and second target data of the second stacking object from the target data;

[0190] Step 503: The controller calculates a relative position of the first stacking object with respect to the second stacking object according to the first target data and the second target data;

[0191] Step 504: The controller controls the movement of the transport device according to the relative position and posture to align the first stacking object with the second stacking object.

[0192] As can be seen from the above process, the present application obtains the target data of the two stacking objects through the first sensor, and the controller extracts the first target data of the first stacking object and the second target data of the second stacking object from the target data, and then calculates the relative posture of the first stacking object relative to the second stacking object, and controls the movement of the handling equipment according to the relative posture to align the first stacking object with the second stacking object. Compared with the existing solution, the relative posture between the two stacking objects can be corrected by simultaneously obtaining and processing the target data of the two stacking objects, thereby significantly improving the accuracy and efficiency of the stacking operation.

[0193] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the "first" and "second" and other limitations involved in the present disclosure do not have limitations in terms of size, order and quantity, and are only used to distinguish them in name, for example, "first stacking object" and "second stacking object" are used to distinguish two stacking objects, "first target data", "second target data" and "fourth target data" are used to distinguish different target data, and so on.

[0194] First, the above step 501, ie, "the first sensor acquires target data of the first stacking object and the second stacking object", is described in detail in conjunction with the embodiment.

[0195] In the embodiment of the present application, the first sensor is used to obtain the target data of the first stacking object and the second stacking object. Specifically, when the controller controls the handling device to carry the first stacking object to the stacking operation position, the first sensor collects the first stacking object and the second stacking object to obtain the target data of the first stacking object and the second stacking object.

[0196] As an achievable manner, the controller controls the transport device to transport the first stacking object to the stacking operation position, specifically including:

[0197] The handling device is controlled to move toward the second stacking object until a projection length of a fork of the handling device on the second stacking object reaches a first length threshold.

[0198] like Figure 6As shown, it is a schematic diagram of a handling device carrying a first stacking object to a stacking operation position. Among them, the projection length d1 of the fork of the handling device on the second stacking object B reaches the first length threshold, and the position of the handling device at this time is the stacking operation position. In the embodiment of the present application, the first length threshold can be set to 10 cm-30 cm. It should be noted that in the embodiment of the present application, in order to ensure that the first sensor can be at the optimal detection distance, so as to ensure that the handling device can be quickly adjusted after entering the alignment stage, thereby improving the alignment efficiency and accuracy, the embodiment of the present application sets the first length threshold within 10 cm-30 cm. In actual applications, the first length threshold can be set according to actual needs, and the embodiment of the present application does not limit this.

[0199] As another achievable manner, the controller controls the transport device to transport the first stacking object to the stacking operation position, specifically including:

[0200] The transport device is controlled to move toward the second stacking object until the distance between the first sensor of the transport device and the second stacking object in the longitudinal direction is within a first preset distance.

[0201] Still Figure 6 For example, the longitudinal distance d2 between the first sensor of the handling device and the second stacking object reaches the first preset distance, and the position of the handling device is the stacking operation position.

[0202] When the handling device reaches the stacking operation position, it enters the alignment phase, and the first sensor starts to collect target data of the first stacking object and the second stacking object, and aligns the first stacking object with the second stacking object according to the collected target data.

[0203] The first sensor in the embodiment of the present application may include a radar module and / or a camera module, wherein the radar module may include one or more radars, wherein the radar may be a laser radar, such as a 3D laser radar, and the camera module may include one or more cameras. The specific form of the first sensor in the embodiment of the present application is not limited.

[0204] When the first sensor is a 3D laser radar, the target data may be point cloud data, which includes three-dimensional geometric information of the first stacked object and the second stacked object; when the first sensor is a camera, the target data may include image data; when the first sensor is a combination of a 3D laser radar and a camera, the target data includes point cloud data and image data.

[0205] Furthermore, before the handling equipment enters the alignment phase, it may also include a picking phase and a pre-alignment phase.

[0206] In the picking stage, the controller controls the transport device to move to the picking position, the first sensor obtains the third target data of the first stacked object, and then controls the transport device to move according to the third target data to obtain the first stacked object, and then controls the transport device to move the first stacked object to the stacking preparation area. Specifically, the first stacked object can be moved in front of the second stacked object, such as within 5 meters of the second stacked object.

[0207] The controller can simultaneously control the movement of the vehicle body and the lifting of the first stacking object during the transport process.

[0208] In the pre-alignment stage, the controller controls the transporting equipment to transport the first stacking object to the stacking preparation position, the first sensor obtains the fourth target data of the second stacking object, and then determines the posture of the second stacking object according to the fourth target data, and controls the movement of the transporting equipment based on the posture of the second stacking object so that the posture error of the transporting equipment relative to the second stacking object is within a first preset threshold range.

[0209] The position error of the handling device relative to the second stacking object refers to the positioning and orientation of the handling device relative to the second stacking object in three-dimensional space. Specifically, it includes the coordinate difference (Δy1) in the Y direction and the rotation angle difference (Δθ1) in the Z direction between the handling device and the second stacking object.

[0210] Optionally, the first preset threshold range can be set within the range of ±10 cm for the Y-axis coordinate difference, ie, -10 cm<Δy1<10 cm, and within the range of ±5° for the Z-axis rotation angle difference, ie, -5°<Δθ1<5°.

[0211] It should be noted that the threshold range is only an example and can be adjusted according to different stacking objects and handling equipment in practice.

[0212] The above step 502, ie, "the controller extracts the first target data of the first stacking object and the second target data of the second stacking object from the target data" is described in detail below in conjunction with an embodiment.

[0213] After the controller acquires the target data, it is necessary to extract the first target data of the first stacking object and the second target data of the second stacking object from the target data.

[0214] Optionally, the first target data may be data in a first target area of ​​a first stacked object, and the second target data may be data in a second target area of ​​a second stacked object.

[0215] The first target area refers to the bottom stacking structure of the first stacking object, and the second target area refers to the top stacking structure of the second stacking object.

[0216] Taking the example that both the first stacking object and the second stacking object are cages, since the cage is usually a frame structure, the bottom stacking structure of the first stacking object refers to the bottom frame structure, and the top stacking structure of the second stacking object refers to the top frame structure.

[0217] In addition, when the second stacking object is a static base, the second target area refers to a boundary line of the top surface of the static base.

[0218] The above step 503, namely "the controller calculates the relative posture of the first stacking object with respect to the second stacking object according to the first target data and the second target data", is described in detail below in conjunction with the embodiments.

[0219] After the controller extracts the first target data and the second target data from the target data, it can determine the posture of the first stacking object according to the first target data, and determine the posture of the second stacking object according to the second target data, and then calculate the relative posture between them by comparing the first posture and the second posture.

[0220] As an implementable manner, when the first sensor is a combination of a 3D laser radar and a camera, the acquired target data of the first stacked object and the second stacked object includes an image and a point cloud of the first stacked object, and an image and a point cloud of the second stacked object.

[0221] like Figure 7 As shown, calculating the relative position of the first stacking object with respect to the second stacking object can be implemented as follows:

[0222] Step 701: The controller obtains an image and a point cloud of a first stacking object, and an image and a point cloud of a second stacking object.

[0223] When the controller controls the transporting equipment to transport the first stacking object to the stacking operation position, and controls the transporting equipment to lift the first stacking object, the radar scans the first stacking object and the second stacking object to obtain the point clouds of the first stacking object and the second stacking object, and at the same time, the camera photographs the first stacking object and the second stacking object to obtain images of the first stacking object and the second stacking object.

[0224] Taking into account the distortion in the images and point clouds collected when the first stacked object and the second stacked object are photographed and scanned by the camera and the radar at the same time, in the embodiment of the present application, when obtaining the image and point cloud of the first stacked object, as well as the image and point cloud of the second stacked object, it can specifically include: obtaining the original image of the first stacked object and the original image of the second stacked object by the camera, and obtaining the original point cloud of the first stacked object and the original point cloud of the second stacked object by the radar; de-distorting the original image according to the pre-calibrated camera intrinsic parameters; converting the original point cloud from the coordinate system where the radar is located to the coordinate system where the handling equipment is located, and de-distorting the original point cloud according to the odometer information and the timestamp corresponding to the original image; and time-synchronizing the de-distorted image and the de-distorted point cloud to obtain the time-synchronized image and point cloud.

[0225] Here, the timestamp corresponding to the original image may be the timestamp corresponding to when the camera captured the original image (i.e., the original image of the first stacked object or the original image of the second stacked object). Camera intrinsic parameters, also known as camera internal parameters or intrinsic parameters, are parameters that describe the internal properties of the camera. These parameters include focal length, principal point (optical center) coordinates, and distortion coefficients (such as k1, k2, k3 for radial distortion, and p1, p2 for tangential distortion). The intrinsic parameters are usually determined during camera calibration.

[0226] Image distortion is mainly caused by the optical properties of the camera lens. When light passes through the lens, radial distortion and tangential distortion will occur due to refraction and manufacturing process limitations. Radial distortion will cause straight lines in the image to become curved, and the farther away from the center of the image, the more severe the distortion. Tangential distortion is caused by the lens and the photosensitive element not being completely parallel. Therefore, the original image is dedistorted by using the pre-calibrated camera internal parameters to obtain the dedistorted image.

[0227] The distortion in the original point cloud refers to the shape distortion problem caused by radar movement or external factors. The odometer information provides the motion information of the radar when collecting the original point cloud, which usually includes position, speed, acceleration, etc. Next, the compensation transformation matrix is ​​determined according to the timestamp of the image and the odometer information; the compensation transformation matrix is ​​applied to the original point cloud to obtain the de-distorted point cloud.

[0228] It should be noted that the odometer measures the movement of the transport equipment to estimate the distance it has moved, and usually combines sensor data to calculate physical quantities such as the position, speed and posture of the transport equipment. The odometer can be set near the wheels of the transport equipment to record the number of rotations of the wheels to estimate the distance moved; the odometer can also be set at the center of the chassis of the transport equipment.

[0229] Step 702: The controller extracts first object image data of a first object area from the image of the first stacked object, and extracts second object image data of a second object area from the image of the second stacked object.

[0230] First target image data: refers to image data of a first target area extracted by the first sensor from the image of the first stacked object. The first target image data can be used to extract first target point cloud data corresponding to the first target area from the point cloud of the first stacked object.

[0231] Second target image data: refers to image data of the second target area extracted by the first sensor from the image of the second stacked object. The second target image data can be used to extract second target point cloud data corresponding to the second target area from the point cloud of the second stacked object.

[0232] The first target region of the first stacked object usually has a boundary in the image, which separates the first target region from other parts in the image, and the first target region describes the partial shape, size, position and possible texture or color features of the first stacked object. The second target region of the second stacked object usually has a boundary in the image, which separates the second target region from other parts in the image, and the second target region describes the partial shape, size, position and possible texture or color features of the second stacked object.

[0233] In one example, the controller extracts first target image data of a first target area from an image of a first stacked object and extracts second target image data of a second target area from an image of a second stacked object according to an image segmentation model; wherein image segmentation is a process of dividing an image into a plurality of non-intersecting regions, each region corresponding to an object in the image. The image segmentation method may include one or more methods such as threshold segmentation, edge detection, region growing, clustering, and deep learning methods.

[0234] In another example, the controller extracts first target image data of a first target area from an image of a first stacked object, and extracts second target image data of a second target area from an image of a second stacked object according to a target detection model. Target detection is the process of identifying and locating a specific target or object in an image. The target detection method is mainly implemented based on a deep learning model.

[0235] The embodiment of the present application may use an image segmentation or target detection method to accurately determine a first target region of a first stacked object and a second target region of a second stacked object in an image.

[0236] It should be noted that, in the process of object detection or image segmentation, a binary mask (or mask) of the same size as the image is usually generated. Each pixel value in the mask indicates whether the pixel belongs to the first stacked object or the second stacked object (usually 1 means it belongs, and 0 means it does not belong). This mask can be used to extract the first target area of ​​the first stacked object and the second target area of ​​the second stacked object.

[0237] For example, a camera captures an image containing a first stacked object and a second stacked object. After image segmentation, the segmentation result can be as follows: Figure 8 As shown in the figure, the first part (blue part) is at least a portion of the fifth foot cup Aj5 and at least a portion of the third foot cup Aj3 of the first stacking object, and the second part (red part) is at least a portion of the sixth column Bj6 and at least a portion of the fourth column Bj4 of the second stacking object.

[0238] Here, at least a partial area of ​​the foot cup of the first stacked object and at least a partial area of ​​the pillar of the second stacked object presented in the captured image can be changed by adjusting the number of cameras and the orientation of the cameras.

[0239] Among them, the foot cup refers to the supporting component installed at the bottom of the cage, which is usually used to stabilize the cage, bear weight, and protect the cage from direct contact with the ground. In industrial, warehousing and logistics scenarios, cages (also called turnover cages or storage cages) are often equipped with foot cups to achieve better mobility, stacking and durability.

[0240] The column refers to the vertical support structure around the cage, which is usually used to bear the weight of the cage, fix the cage frame structure and provide stacking function. The column is one of the core components of the cage, and its design has a direct impact on the strength, stability and use function of the cage.

[0241] The first target area in the embodiment of the present application at least includes: the foot cup, frame line and other key structural areas of the first stacking object. The second target area at least includes: the pillar, frame line and other key structural areas of the second stacking object.

[0242] In one example, the first target area includes at least a portion of at least one foot cup, which may be the area corresponding to all or part of the foot cup. The second target area includes at least a portion of at least one column, which may be the area corresponding to all or part of the column.

[0243] For example, the first stacking object is the first material cage, and the second stacking object is the second material cage; the first target area includes at least a portion of the first foot cup of the first material cage and at least a portion of the third foot cup; the second target area includes at least a portion of the second column of the second material cage and at least a portion of the fourth column.

[0244] For example, the first target area also includes at least part of the fifth foot cup and at least part of the seventh foot cup of the first material basket; the second target area also includes at least part of the sixth column and at least part of the eighth column of the second material basket. Fig. 9 , Fig.10 , Fig.11 right Figure 2 The first target area and the second target area in the embodiment are described in detail respectively.

[0245] exist Fig. 9 In the embodiment, the first target area includes at least a portion of the first foot cup Aj1, at least a portion of the third foot cup Aj3, at least a portion of the fifth foot cup Aj5, and at least a portion of the seventh foot cup Aj7; the second target area includes at least a portion of the second column Bj2, at least a portion of the fourth column Bj4, at least a portion of the sixth column Bj6, and at least a portion of the eighth column Bj8.

[0246] exist Fig.10 In the embodiment, the first target area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3; the second target area B2 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4.

[0247] exist Fig.11 In the embodiment, the first target area A1 includes at least a portion of the fifth foot cup Aj5 and at least a portion of the seventh foot cup Aj7; the second target area B2 includes at least a portion of the sixth column Bj6 and at least a portion of the eighth column Bj8.

[0248] In one example, the first target area includes at least one border line of the first stacked object; and the second target image area includes at least one border line of the second stacked object.

[0249] For example, the first stacking object is a first material cage, and the second stacking object is a second material cage; the first target area includes a first border line and a third border line of the first material cage, and the first border line intersects with the third border line; the second target area includes a second border line and a fourth border line of the second material cage, and the second border line intersects with the fourth border line.

[0250] For example, the first target area also includes the fifth and seventh border lines of the first cage, and the fifth and seventh border lines intersect; the second target area also includes the sixth and eighth border lines of the second cage, and the sixth and eighth border lines intersect. Fig.12 right Figure 2 The first target area and the second target area in the embodiment are described in detail respectively.

[0251] exist Fig.12 In the figure, the first target area A1 includes a first border line a11, a third border line a13, a fifth border line a15 and a seventh border line a17; wherein the first border line a11 intersects with the third border line a13, the third border line a13 intersects with the fifth border line a15, the fifth border line a15 intersects with the seventh border line a17, and the seventh border line a17 intersects with the first border line a11.

[0252] The second target area B2 includes a second border line b22, a fourth border line b24, a sixth border line b26 and an eighth border line b28; wherein the second border line b22 intersects with the fourth border line b24, the fourth border line b24 intersects with the sixth border line b26, the sixth border line b26 intersects with the eighth border line b28, and the eighth border line b28 intersects with the second border line b22.

[0253] It should be noted that in Fig.12 In the embodiment of the present application, the first target area A1 is located at the bottom of the first stacking object A, and the second target area B2 is located at the top of the second stacking object B.

[0254] Step 703: The controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacked object based on the first target image data; the controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacked object based on the second target image data.

[0255] The first target point cloud data refers to the point cloud data of the first target area of ​​the first stacking object acquired by the sensor. The first target point cloud data can be used to calculate the position and posture, stacking state, etc. of the first stacking object.

[0256] Second target point cloud data: refers to the point cloud data of the second target area of ​​the second stacking object acquired by the sensor. The second target point cloud data can be used to calculate the position and posture, stacking state, etc. of the second stacking object.

[0257] In the embodiment of the present application, the controller pre-establishes a correspondence between the point cloud and the pixels of the image, and based on the correspondence and the first target area of ​​the first stacked object and the second target area of ​​the second stacked object, extracts first target point cloud data corresponding to the first target area of ​​the first stacked object from the point cloud of the first stacked object, and extracts second target point cloud data corresponding to the second target area of ​​the second stacked object from the point cloud of the second stacked object.

[0258] In the embodiment of the present application, before the controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacked object based on the first target image data; and before the controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacked object based on the second target image data, the method further includes:

[0259] After the controller calibrates the camera and the radar, first target point cloud data corresponding to the first target area is extracted from the target point cloud of the first stacked object based on the first target image data, including:

[0260] extracting first target point cloud data corresponding to the first target area from the target point cloud of the first stacked object according to the joint calibration parameter and the first target image data;

[0261] Extracting second target point cloud data corresponding to the second target area from the target point cloud of the second stacked object based on the second target image data includes:

[0262] According to the joint calibration parameters and the second target image data, second target point cloud data corresponding to the second target area is extracted from the target point cloud of the second stacked object.

[0263] In one example, the controller pre-calibrates a camera for collecting images and a radar for collecting point clouds to obtain joint calibration parameters.

[0264] In one example, the joint calibration parameters include at least one of the following: intrinsic parameters and extrinsic parameters of the camera, extrinsic parameters from the radar to the camera, and extrinsic parameters from the radar to the handling equipment.

[0265] The intrinsic parameters of a camera refer to the parameters that describe the internal properties of the camera. These parameters are usually determined during camera calibration and usually remain unchanged during the use of the camera. The intrinsic parameters mainly include the following aspects: focal length, optical center, distortion coefficients, and intrinsic matrix.

[0266] The camera's external parameters are the parameters that describe the camera's position and posture in the world coordinate system. The external parameters mainly include the following aspects: rotation matrix and translation vector.

[0267] Unlike camera intrinsic parameters, camera extrinsic parameters change as the camera's position in the world coordinate system or the moment of capture changes. For example, in stereo vision, if there are two cameras, when the cameras move, their relative position and orientation change, which will cause changes in extrinsic parameters.

[0268] The external parameters from radar to camera refer to determining the rotation and translation relationship between radar and camera so that their coordinate systems can be aligned.

[0269] The external parameters of the radar to the handling equipment refer to the determination of the relative position and direction relationship between the radar and the handling equipment (its body is used as the reference coordinate system, such as the three-dimensional coordinate system mentioned above). This calibration is a key step to ensure that the radar data can be accurately converted to the three-dimensional coordinate system, thereby achieving accurate environmental perception, positioning navigation and obstacle avoidance functions.

[0270] When the joint calibration parameters include the intrinsic and extrinsic parameters of the camera, the extrinsic parameters of the radar to the camera, and the extrinsic parameters of the radar to the handling equipment, the correspondence between the point cloud and the pixel can be expressed by the following formula:

[0271] P img =T img ·T camera-img ·T lidar-camera ·T 搬运设备-lidar ·P n

[0272] Among them, the coordinates of the midpoint of the point cloud in the coordinate system where the handling equipment is located are P n , the coordinates of the pixel in the image (the image format can be .img) are P img , the camera internal parameter is T img , the camera external parameter is T camera-img The external parameter from radar to camera is T lidar-camera , the external parameter from the transport equipment to the laser radar is T 搬运设备-lidar .

[0273] Step 704: The controller determines the position and posture of the first stacking object according to the first target point cloud data, and determines the position and posture of the second stacking object according to the second target point cloud data.

[0274] The controller of the embodiment of the present application extracts a first border point cloud from first target point cloud data corresponding to a first target area of ​​a first stacked object; and extracts a second border point cloud from second target point cloud data corresponding to a second target area of ​​a second stacked object; fits the first border point cloud according to the least squares method to obtain a border line equation corresponding to the first stacked object, and fits the second border point cloud according to the least squares method to obtain a border line equation corresponding to the second stacked object; determines the posture of the first stacked object according to the border line equation corresponding to the first stacked object, and determines the posture of the second stacked object according to the border line equation corresponding to the second stacked object.

[0275] In one example, the controller extracts a first border point cloud from first target point cloud data corresponding to a first target area of ​​a first stacked object according to RANSAC (Random Sample Consensus) or PROSAC (Progressive Sample Consensus); and extracts a second border point cloud from second target point cloud data corresponding to a second target area of ​​a second stacked object according to RANSAC; fits the first border point cloud according to the least squares method to obtain a border line equation corresponding to the first stacked object, and fits the second border point cloud according to the least squares method to obtain a border line equation corresponding to the second stacked object; determines the pose of the first stacked object according to the border line equation corresponding to the first stacked object, and determines the pose of the second stacked object according to the border line equation corresponding to the second stacked object.

[0276] Here, extracting a first frame point cloud from first target point cloud data corresponding to a first target region of the first stacked object according to RANSAC may include:

[0277] 1) Randomly select a group of points from the first target point cloud data corresponding to the first target area of ​​the first stacking object as initial samples; 2) Use the initial samples to estimate the parameters of the model (i.e., the model for extracting the border point cloud), which are the vertices, side lengths, angles and other parameters of the border of the first stacking object; 3) According to the model parameters, classify the other points in the first target point cloud data corresponding to the first target area of ​​the first stacking object into internal points (points that meet the model parameters) and external points (points that do not meet the model parameters); 4) Repeat the above steps of random sampling, model estimation and internal and external point classification, and record the number of internal points of the current model in each iteration; 5) In all iterations, select the model with the largest number of internal points as the final estimation result (i.e., the border point cloud of the first stacking object).

[0278] It should be noted that the process of extracting the second border point cloud from the second target point cloud data corresponding to the second target area of ​​the second stacking object according to RANSAC or PROSAC and the process of extracting the first border point cloud from the first target point cloud data corresponding to the first target area of ​​the first stacking object according to RANSAC or PROSAC will not be repeated here.

[0279] Next, fitting the first frame point cloud according to the least squares method to obtain a frame line equation corresponding to the first stacked object, and fitting the second frame point cloud according to the least squares method to obtain a frame line equation corresponding to the second stacked object may include:

[0280] 1) A suitable fitting model can be selected for the appearance of the first stacked object and the second stacked object, and a straight line fitting model can be used for the frame line equations of the first stacked object and the second stacked object; 2) The parameters of the first fitting model of the first stacked object and the parameters of the second fitting model corresponding to the second stacked object are respectively determined, such as the slope and the intercept and other parameters; 3) A first objective function (i.e., an error function) is constructed according to the first fitting model and the first frame point cloud; and a second objective function is constructed according to the parameters of the second fitting model and the second frame point cloud; wherein the objective function corresponding to the first stacked object represents the deviation or distance between the corresponding fitting model and the first frame point cloud, and the second stacked object represents the deviation or distance between the corresponding fitting model and the first frame point cloud, and the second stacked object represents the deviation or distance between the corresponding fitting model and the first frame point cloud. The objective function corresponding to the stacked object represents the deviation or distance between the corresponding fitting model and the second border point cloud; 4) by using the least squares method, respectively solve the model parameters when the first objective function and the second objective function reach the minimum value; wherein the least squares method finds the best fit by minimizing the sum of the squares of the distances from the point cloud of all first border lines to the first fitting model, or finds the best fit by minimizing the sum of the squares of the distances from the point cloud of all second border lines to the second fitting model; 5) solve the minimum value of the first objective function to obtain the border line equation corresponding to the first stacked object; and solve the minimum value of the second objective function to obtain the border line equation corresponding to the second stacked object.

[0281] For example, the distribution of each border point cloud in the embodiment of the present application directly reflects the shape of the stacked object. For example, a stacked object of a cube will generate a point cloud with six rectangular faces.

[0282] According to the shape of the first stacking object (such as a rectangle), the first frame point cloud, the third frame point cloud, the fifth frame point cloud and the seventh frame point cloud are distributed on the rectangular surface corresponding to the first stacking object; according to the shape of the second stacking object, the second frame point cloud, the fourth frame point cloud, the sixth frame point cloud and the eighth frame point cloud are distributed on the rectangular surface corresponding to the second stacking object.

[0283] The first border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the first border line equation; the third border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the third border line equation; the fifth border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the fifth border line equation; the seventh border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the seventh border line equation.

[0284] The second frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the second frame line equation; the fourth frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the fourth frame line equation; the sixth frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the sixth frame line equation; the eighth frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the eighth frame line equation.

[0285] The following takes the third border point cloud as an example, and uses the least squares method to fit the third border point cloud distributed on the rectangular surface to obtain the third border line equation.

[0286] For example, find the border line equation y=mx+b on the rectangular surface, where m is the slope and b is the intercept, so that the border line equation is as close as possible to the third border point cloud distributed on the rectangular surface. The goal of the least squares method is to find the values ​​of m and b so that the sum of the squares of the vertical distances (i.e., errors) from all points (i.e., all points in the third border point cloud distributed on the rectangular surface) to the border line equation is minimized; the sum of the squares of the vertical distances (i.e., errors) from all points to the border line equation is minimized to obtain the third border line equation.

[0287] It should be noted that, for other border point clouds, their edge lines can be determined in the same manner as the third border point cloud, which will not be described in detail here.

[0288] In some embodiments, the following steps may be used to determine the posture of the first stacked object according to the frame line equation corresponding to the first stacked object, and to determine the posture of the second stacked object according to the frame line equation corresponding to the second stacked object:

[0289] Take the first stack object as an example,

[0290] The border line equation corresponding to the first stacked object obtained by the straight line fitting algorithm may include four border line equations, namely, a first border line equation corresponding to the first border line a11, a third border line equation corresponding to the third border line a13, a fifth border line equation corresponding to the fifth border line a15, and a seventh border line equation corresponding to the seventh border line a17.

[0291] Determine the position information of the first stacked object according to the coordinates of any intersection point between the straight lines corresponding to the four frame line equations, for example: determine the position information of the first stacked object according to the coordinates of the intersection point between the first frame line a11 corresponding to the first frame line equation and the third frame line a13 corresponding to the third frame line equation;

[0292] Alternatively, a centerline equation is obtained based on two parallel straight lines, and the position information of the first stacked object is determined based on the coordinates of the intersection of the straight line corresponding to the centerline equation and the straight lines corresponding to other border line equations. For example: the first centerline is obtained based on the first border line a11 corresponding to the first border line equation and the fifth border line a15 corresponding to the fifth border line equation, and the position information of the first stacked object is determined based on the coordinates of the intersection of the first centerline and the third border line a13 corresponding to the third border line equation.

[0293] Then, the angle information of the first stacked object is determined according to the angle of any straight line of the first frame line a11 corresponding to the first frame line equation, the third frame line a13 corresponding to the third frame line equation, the fifth frame line a15 corresponding to the fifth frame line equation, and the seventh frame line a17 corresponding to the seventh frame line equation;

[0294] Then, the position and posture of the first stacking object is determined according to the position information and the angle information.

[0295] Similar steps may be adopted to determine the position and posture of the second stacked object using the frame line equation corresponding to the second stacked object.

[0296] Step 705: Calculate the relative posture of the first stacking object with respect to the second stacking object according to the posture of the first stacking object and the posture of the second stacking object.

[0297] The relative posture of the first stacked object and the second stacked object refers to the posture difference between the first stacked object and the second stacked object. Specifically, taking the coordinate system of this embodiment as an example, it includes the coordinate difference (Δx, Δy) of the two stacked objects in the X direction and the Y direction and the rotation angle difference (Δθ) on the Z axis.

[0298] As another achievable manner, when the first sensor is a 3D laser radar, the acquired target data of the first stacked object and the second stacked object include a point cloud of the first stacked object and a point cloud of the second stacked object.

[0299] like Fig.13 As shown, calculating the relative position of the first stacking object with respect to the second stacking object can be implemented as follows:

[0300] Step 131: The controller obtains target point clouds of the first stacking object and the second stacking object.

[0301] The transport device transports the first stacking object to the stacking operation position, lifts the first stacking object, and scans the first stacking object and the second stacking object through the first sensor carried by the transport device, and the first sensor obtains the target point cloud of the first stacking object and the second stacking object.

[0302] Considering that there is distortion in the target point cloud collected when the first stacked object and the second stacked object are scanned by the first sensor at the same time, in the embodiment of the present application, the first sensor obtaining the target point cloud of the first stacked object and the second stacked object may also include: the first sensor carried by the handling device collects the original point cloud of the first stacked object; converting the original point cloud of the first stacked object from the coordinate system where the first sensor is located to the coordinate system where the handling device is located, and de-distorting the original point cloud of the first stacked object according to the odometer information collected by the handling device to obtain the target point cloud; and

[0303] The first sensor carried by the handling equipment collects the original point cloud of the second stacked object; the original point cloud of the second stacked object is converted from the coordinate system where the first sensor is located to the coordinate system where the handling equipment is located, and the original point cloud of the second stacked object is dedistorted according to the odometer information collected by the handling equipment to obtain the target point cloud.

[0304] The distortion in the original point clouds of the first stacked object and the second stacked object refers to the shape distortion problem caused by the movement of the first sensor or external factors. The odometer information provides the movement information of the sensor when collecting the original point cloud, which usually includes position, speed, acceleration, etc.; the compensation transformation matrix can be determined according to the odometer information; and then the compensation transformation matrix is ​​applied to each point in the original point cloud to obtain the de-distorted point cloud (that is, the target point cloud of the first stacked object and the target point cloud of the second stacked object).

[0305] The present application converts the original point cloud collected by the first sensor from the coordinate system where the sensor is located to the coordinate system where the handling equipment is located, and dedistorts the original point cloud according to the odometer information, so as to obtain a distortion-free target point cloud.

[0306] Step 132: The controller determines a first pseudo image corresponding to the first stacking object according to the target point cloud of the first stacking object; the controller determines a second pseudo image corresponding to the second stacking object according to the target point cloud of the second stacking object.

[0307] Among them, pseudo-images refer to images generated by algorithms or processed by certain special techniques in the field of computer image processing or computer vision. They may not represent actual image data, or have a virtualized or approximate relationship with real images. In some cases, pseudo-images can also refer to simulated images or images used to represent some non-real world scenes.

[0308] The controller determines the projection points of each point in the target point cloud of the first stacking object and the target point cloud of the second stacking object on the same horizontal plane (such as the XOY plane), which usually involves converting the point cloud coordinates from the original coordinate system to the coordinate system where the XOY plane is located, and retaining the two-dimensional position information (such as the X and Y coordinates). Optionally, the controller projects the target point cloud of the first stacking object from above on the XOY plane to generate a first projection map (such as Fig.14 The first projection image is used as the first pseudo image of the first stacking object, and the target point cloud of the second stacking object is projected onto the XOY plane to generate a second projection image (as shown in the upper or lower figure of (a)). Fig.15 As shown in the upper or lower figure of (a), the second projection image is used as the second pseudo image of the second stacking object. Wherein, "X" in the XOY plane corresponds to the X-axis in the above three-dimensional coordinate system; "O" corresponds to the origin O in the above three-dimensional coordinate system; and "Y" corresponds to the Y-axis in the above three-dimensional coordinate system.

[0309] like Fig.14 In some embodiments, the controller projects the target point cloud of the first stacked object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (such as Fig.14 In (a), at least two projection images of the projection resolution corresponding to the target point cloud of the first stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Fig.14 In (b), the superimposed projection image is used as the first pseudo image of the first stacking object; in the same way, the controller projects the target point cloud of the second stacking object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (such as Fig.15 In (a), the projection images of at least two projection resolutions corresponding to the target point cloud of the second stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Fig.15 In (b), the superimposed projection image is used as the second pseudo image of the second stacking object.

[0310] In some embodiments, the controller projects the target point cloud of the first stacked object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (eg, Fig.14 In (a), at least two projection images of the projection resolution corresponding to the target point cloud of the first stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Fig.14 (b)), and then convert the superimposed projection image into the first grayscale image (such as Fig.14 In (c), the first grayscale image is used as the first pseudo image of the first stacking object; in the same way, the controller projects the target point cloud of the second stacking object onto the XOY plane using at least two projection resolutions to obtain at least two projection images (such as Fig.15 In (a), the projection images of at least two projection resolutions corresponding to the target point cloud of the second stacking object are scaled to a uniform size and superimposed to obtain a superimposed projection image (such as Fig.15 (b)), and then convert the superimposed projection image into a second grayscale image (such as Fig.15 In (c), the second grayscale image is used as the second pseudo image of the second stacking object. The purpose of scaling to a uniform size is to ensure that at least two projection images of the projection resolution are correctly aligned before superposition to avoid misalignment or ghosting after superposition.

[0311] The unification of the sizes of the pseudo images of at least two projection resolutions may include scaling the sizes of the pseudo images of the respective projection resolutions to a specified size.

[0312] Optionally, unifying the sizes of pseudo images of at least two projection resolutions may also include first determining the largest pseudo image among the pseudo images of at least two projection resolutions, and unifying the sizes of pseudo images of other projection resolutions to the size of the largest pseudo image.

[0313] It should be noted that the pixel values ​​in the above grayscale images (i.e., the first grayscale image and the second grayscale image) are usually used to represent the brightness of the image, i.e., the grayscale value. The larger the grayscale value, the brighter the pixel; conversely, the smaller the grayscale value, the darker the pixel. When we project the point cloud into the pixel, we can adjust the grayscale value of the pixel according to the attributes of each point in the point cloud.

[0314] During the process in which the controller projects the target point cloud of the first stacking object and the target point cloud of the second stacking object onto the XOY plane respectively and converts them into grayscale images, the grayscale value in the corresponding grayscale image can be determined based on the attributes of each point in the point cloud projected into the pixel (such as the number of points, the intensity of the points, the distance between the points, the angle of the points, and at least one of the elevation difference between the points in the point cloud).

[0315] After the controller determines the grayscale value, the corresponding pixel position is found on the grayscale image according to the two-dimensional position information (X and Y coordinates) of each projection point, and the determined grayscale value is used as the pixel value at the pixel position. The above process is repeated until all projection points are processed, thereby generating a complete grayscale image.

[0316] Optionally, the projection resolution used by the target point cloud of the first stacked object may be the same as the projection resolution used by the target point cloud of the second stacked object, such as the target point cloud of the first stacked object and the target point cloud of the second stacked object both use projection resolutions of 5mm (millimeter) / pixel and 1cm (centimeter) / pixel.

[0317] For example, projecting the target point cloud of the first stacking object onto the XOY plane using 5 mm / pixel and 1 cm / pixel, and generating a first grayscale image corresponding to the first stacking object may include the following steps:

[0318] Step 1: The controller projects the target point cloud of the first stacked object onto the XOY plane at 5 mm / pixel and 1 cm / pixel, and generates a pseudo image corresponding to 5 mm / pixel and a pseudo image corresponding to 1 cm / pixel (corresponding to Fig.14 The higher the projection resolution, the clearer the image. Fig.14 The lines in the upper and lower figures of (a) generally reflect the contour lines or characteristic lines of the first stacked object at a certain viewing angle (ie, the viewing angle corresponding to the top view).

[0319] Step 2: The controller scales the projection image corresponding to 5 mm / pixel and the projection image corresponding to 1 cm / pixel to a uniform size and superimposes them to obtain a superimposed projection image (such as Fig.14 The superimposed projection image is then converted into a grayscale image to obtain a first grayscale image corresponding to the first stacked object (as shown in (b)). Fig.14 (as shown in (c)).

[0320] The second grayscale image is generated in a similar manner to the first grayscale image.

[0321] It should be noted that Fig.14 The "O" in corresponds to the origin O in the above three-dimensional coordinate system, the "X" corresponds to the X-axis in the above three-dimensional coordinate system, and the "Y" corresponds to the Y-axis in the above three-dimensional coordinate system.

[0322] It should be noted that, in some embodiments, the projection image of each projection resolution can be divided into three channels, wherein the first channel uses the number of points in the point cloud projected into the pixel, and the more the number of point clouds, the higher the pixel value; the second channel uses the relative posture data of the distance between the point cloud and the target (i.e., the theoretical position of the stacked object) to generate, and the larger the relative posture data, the lower the pixel value; the third channel uses the elevation difference between the points in the point cloud, and within a certain height range, the larger the elevation difference, the higher the pixel value.

[0323] The present application determines the pixel value in the grayscale image based on at least one of the number of points in the point cloud projected into the pixel, the intensity of the points, the distance between points, the angle of the points, and the elevation difference between the points in the point cloud, so that the pixel value in the grayscale image reflects the multi-dimensional properties of the point cloud, so as to improve the accuracy of determining the boundary line from the grayscale image.

[0324] Step 133: The controller determines relative position data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image.

[0325] The position and posture of the first stacked object are determined by extracting key structures such as boundaries and target points in a first pseudo-image generated by projecting the target point cloud of the first stacked object on a horizontal plane; and the position and posture of the second stacked object are determined by extracting key features such as boundaries and target points in a second pseudo-image generated by projecting the target point cloud of the second stacked object on a horizontal plane; relative position and posture data are determined based on the position and posture of the first stacked object and the position and posture of the second stacked object.

[0326] In the embodiment of the present application, the controller determines the relative posture data between the first stacking object and the second stacking object according to the first pseudo image and the second pseudo image, including:

[0327] The controller determines a boundary line of the first pseudo image; the controller determines a target point of the first pseudo image based on the boundary line of the first pseudo image; the controller determines a boundary line of the second pseudo image; the controller determines a target point of the second pseudo image based on the boundary line of the second pseudo image; the controller determines relative posture data between the first stacking object and the second stacking object based on the boundary line of the first pseudo image and the target point of the first pseudo image, as well as the boundary line of the second pseudo image and the target point of the second pseudo image.

[0328] The boundary line of the first pseudo image in the embodiment of the present application may be: the boundary line of the first projection image generated by projecting the target point cloud of the first stacking object on the XOY plane from top view, such as Fig.14 The boundary lines in the upper or lower figure of (a) (corresponding to the edges of the rectangular box); or, the boundary lines of the superimposed projection images obtained by scaling the projection images of at least two projection resolutions corresponding to the target point cloud of the first stacking object to a uniform size and superimposing them, such as Fig.14 The boundary lines in the figure shown in (b) (corresponding to the edges of the rectangular frame); or, the boundary lines of the first grayscale image, Fig.14 The boundary lines in the figure shown in (c) (corresponding to the edges of the rectangular box).

[0329] The boundary line of the second pseudo image in the embodiment of the present application may be: the boundary line of the second projection image generated by projecting the target point cloud of the second stacking object on the XOY plane in a top-down manner, Fig.15 The boundary lines in the upper or lower figure of (a) (corresponding to the edges of the rectangular frame); or, the boundary lines of the superimposed projection images obtained by scaling the projection images of at least two projection resolutions corresponding to the target point cloud of the second stacking object to a uniform size and superimposing them, such as Fig.15 The boundary lines in the figure shown in (b) (corresponding to the edges of the rectangular frame); or, the boundary lines of the second grayscale image, such as Fig.15 The boundary lines in the figure shown in (c) (corresponding to the edges of the rectangular box).

[0330] In the embodiment of the present application, a probabilistic Hough transform straight line detection algorithm is used to extract a straight line from the first pseudo image and the second pseudo image, and then the boundary line between the first pseudo image and the second pseudo image is determined. This can improve the detection speed and reduce the consumption of computing resources while ensuring the detection accuracy.

[0331] Optionally, the boundary line of the first pseudo image includes: a first boundary line of the first pseudo image, a third boundary line of the first pseudo image, a fifth boundary line of the first pseudo image, and a seventh boundary line of the first pseudo image;

[0332] The boundary lines of the second dummy image include: a second boundary line of the second dummy image, a fourth boundary line of the second dummy image, a sixth boundary line of the second dummy image, and an eighth boundary line of the second dummy image.

[0333] In the embodiment of the present application, the controller determines the target point of the first pseudo image according to the boundary line of the first pseudo image, including:

[0334] The controller determines the center line of the first pseudo image according to the third boundary line of the first pseudo image and the fifth boundary line of the first pseudo image; and uses the intersection of the center line of the first pseudo image and the first boundary line of the first pseudo image as the target point of the first pseudo image;

[0335] The controller determines the target point of the second pseudo image according to the boundary line of the second pseudo image, including: the controller determines the center line of the second pseudo image according to the fourth boundary line of the second pseudo image and the sixth boundary line of the second pseudo image; and uses the intersection of the center line of the second pseudo image and the second boundary line of the second pseudo image as the target point of the second pseudo image.

[0336] In the embodiment of the present application, stacking objects with regular shapes or irregular shapes can be used to complete the stacking of stacking objects. The following takes stacking objects with regular shapes as an example.

[0337] Continuing from the above, the boundary lines of the regularly shaped stacked objects are parallel or perpendicular to each other in the pseudo-image; based on this, the third boundary line of the first pseudo-image and the fifth boundary line of the first pseudo-image in the embodiment of the present application are parallel to each other; the second boundary line of the second pseudo-image and the fourth boundary line of the second pseudo-image are parallel to each other; the third boundary line of the first pseudo-image is perpendicular to the first boundary line of the first pseudo-image, and the fifth boundary line of the first pseudo-image is perpendicular to the first boundary line of the first pseudo-image; the fourth boundary line of the second pseudo-image is perpendicular to the second boundary line of the second pseudo-image, and the sixth boundary line of the second pseudo-image is perpendicular to the second boundary line of the second pseudo-image.

[0338] It should be noted that the target point of the first pseudo image is the midpoint of the first boundary line of the first pseudo image; and the target point of the second pseudo image is the midpoint of the second boundary line of the second pseudo image.

[0339] The following is a detailed description using an example in which the first pseudo image is a first grayscale image and the second pseudo image is a second grayscale image.

[0340] exist Fig.16 In the method, determining the relative position data between the first stacking object and the second stacking object according to the first grayscale image and the second grayscale image may include the following steps:

[0341] Step 161: The controller determines a boundary line of the first grayscale image;

[0342] Step 162: The controller determines a target point of the first grayscale image according to a boundary line of the first grayscale image;

[0343] Step 163: The controller determines a boundary line of the second grayscale image;

[0344] Step 164: The controller determines a target point of the second grayscale image according to a boundary line of the second grayscale image;

[0345] Step 165: The controller determines the relative posture data between the first stacking object and the second stacking object according to the boundary line of the first grayscale image and the target point of the first grayscale image, and the boundary line of the second grayscale image and the target point of the second grayscale image.

[0346] In one example, the controller extracts boundary lines of a first grayscale image through a probabilistic Hough transform straight line detection algorithm, wherein the boundary lines of the first grayscale image include at least a first boundary line of the first grayscale image, a third boundary line of the first grayscale image, and a fifth boundary line of the first grayscale image; the boundary lines of the second grayscale image include at least a second boundary line of the second grayscale image, a fourth boundary line of the second grayscale image, and a sixth boundary line of the second grayscale image.

[0347] like Fig.17 In some embodiments, the boundary lines of the first grayscale image include a first boundary line TC1 of the first grayscale image, a third boundary line TC3 of the first grayscale image, a fifth boundary line TC5 of the first grayscale image, and a seventh boundary line TC7 of the first grayscale image.

[0348] In some embodiments, the third boundary line TC3 of the first grayscale image is parallel to the fifth boundary line TC5 of the first grayscale image, and the first boundary line TC1 of the first grayscale image is parallel to the seventh boundary line TC7 of the first grayscale image.

[0349] like Fig.18 In some embodiments, the boundary lines of the second grayscale image include a second boundary line TB2 of the second grayscale image, a fourth boundary line TB4 of the second grayscale image, a sixth boundary line TB6 of the second grayscale image, and an eighth boundary line TB8 of the second grayscale image.

[0350] In some embodiments, the fourth boundary line TB4 of the second grayscale image is parallel to the sixth boundary line TB6 of the second grayscale image, and the second boundary line TB2 of the second grayscale image is parallel to the eighth boundary line TB8 of the second grayscale image.

[0351] In one example, the controller determines a target point of the first grayscale image according to a boundary line of the first grayscale image, including:

[0352] exist Fig.17 In the figure, the controller determines the center line L1 (yellow dotted line) of the first grayscale image according to the third boundary line TC3 of the first grayscale image and the fifth boundary line TC5 of the first grayscale image; and takes the intersection of the center line L1 of the first grayscale image and the first boundary line TC1 of the first grayscale image as the target point Q1 of the first grayscale image.

[0353] The controller determines the target point of the second grayscale image according to the boundary line of the second grayscale image, and the target point of the second grayscale image includes:

[0354] exist Fig.18 In the figure, the controller determines the center line L2 (yellow dotted line) of the second grayscale image according to the fourth boundary line TB4 of the second grayscale image and the sixth boundary line TB6 of the second grayscale image; and takes the intersection of the center line L2 of the second grayscale image and the second boundary line TB2 of the second grayscale image as the target point Q2 of the second grayscale image.

[0355] In the embodiment of the present application, stacking objects with regular shapes or irregular shapes can be used to complete the stacking of stacking objects. The following takes stacking objects with regular shapes as an example.

[0356] Continuing from the above, the boundary lines of the regularly shaped stacked objects are parallel or perpendicular in the grayscale image; based on this, the third boundary line TC3 of the first grayscale image and the fifth boundary line TC5 of the first grayscale image in the embodiment of the present application are parallel to each other; the second boundary line TB2 of the second grayscale image and the fourth boundary line TB4 of the second grayscale image are parallel to each other; the third boundary line TC3 of the first grayscale image is perpendicular to the first boundary line TC1 of the first grayscale image, and the fifth boundary line TC5 of the first grayscale image is perpendicular to the first boundary line TC1 of the first grayscale image; the fourth boundary line TB4 of the second grayscale image is perpendicular to the second boundary line TB2 of the second grayscale image, and the sixth boundary line TB6 of the second grayscale image is perpendicular to the second boundary line TB2 of the second grayscale image.

[0357] In one example, the target point of the first grayscale image may be any point on the first grayscale image, such as the midpoint on the first boundary line TC1 of the first grayscale image (eg Fig.17 The target point of the second grayscale image can be any point on the second grayscale image, such as the midpoint of the second boundary line TB2 of the second grayscale image (such as Fig.18 The corresponding figures of this application take the midpoint as an example.

[0358] It should be noted that steps 161 to 162 are executed in parallel with steps 163 to 164; or steps 163 to 164 are executed first, and then steps 161 to 162; or, steps 161 to 162 are executed first, and then steps 163 to 164.

[0359] In some embodiments, the controller determines the posture data of the first stacked object based on the position information of the target point Q1 of the first grayscale image and the angle information of the midline L1 of the first grayscale image, and determines the posture data of the second stacked object based on the position information of the target point Q2 of the second grayscale image and the angle information of the midline L2 of the second grayscale image, and then performs a difference calculation on the posture data of the first stacked object and the posture data of the second stacked object to obtain the relative posture data of the first stacked object and the second stacked object.

[0360] In some embodiments, a method for determining relative pose data between a first stacking object and a second stacking object is disclosed, the specific steps being:

[0361] The controller determines the boundary line of the first pseudo image; the controller determines the boundary point cloud of the first stacking object corresponding to the boundary line of the first pseudo image from the target point cloud of the first stacking object; the controller determines the first target point of the first stacking object based on the boundary point cloud of the first stacking object; the controller determines the boundary line of the second pseudo image; the controller determines the boundary point cloud of the second stacking object corresponding to the boundary line of the second pseudo image from the target point cloud of the second stacking object; the controller determines the second target point of the second stacking object based on the boundary point cloud of the second stacking object; the controller determines the relative angle difference between the first stacking object and the second stacking object based on the boundary point cloud of the first stacking object and the boundary point cloud of the second stacking object; the controller determines the relative posture data between the first stacking object and the second stacking object based on the relative position difference and the relative angle difference between the first target point and the second target point.

[0362] The following is a specific description using the first pseudo image being the first grayscale image and the second pseudo image being the second grayscale image as an example. Fig.19 , determining the relative position data between the first stacking object and the second stacking object may include the following steps:

[0363] Step 191: The controller determines a boundary line of the first grayscale image;

[0364] Step 192: The controller determines a boundary point cloud of the first stacking object corresponding to a boundary line of the first grayscale image from the target point cloud of the first stacking object;

[0365] Step 193: The controller determines a first target point of the first stacking object according to the boundary point cloud of the first stacking object;

[0366] Step 194: The controller determines a boundary line of the second grayscale image;

[0367] Step 195: The controller determines a boundary point cloud of the second stacking object corresponding to a boundary line of the second grayscale image from the target point cloud of the second stacking object;

[0368] Step 196: The controller determines a second target point of the second stacking object according to the boundary point cloud of the second stacking object;

[0369] Step 197: The controller determines a relative angle difference between the first stacking object and the second stacking object according to the boundary point cloud of the first stacking object and the boundary point cloud of the second stacking object;

[0370] Step 198: The controller determines the relative posture data between the first stacking object and the second stacking object according to the relative position difference and the relative angle difference between the first target point and the second target point.

[0371] It should be noted that steps 191 to 193 are executed in parallel with steps 194 to 196; or steps 194 to 196 are executed first, and then steps 191 to 193; or steps 191 to 193 are executed first, and then steps 194 to 196.

[0372] Regarding the above step 192: the controller may determine, according to the first projection relationship, from the target point cloud of the first stacking object, a boundary point cloud of the first stacking object corresponding to the boundary line of the first grayscale image.

[0373] Here, the first projection relationship refers to the relationship between the target point cloud of the first stacking object and the first grayscale image.

[0374] Regarding the above step 195: the controller may determine, according to the second projection relationship, from the target point cloud of the second stacking object, a boundary point cloud of the second stacking object corresponding to the boundary line of the second grayscale image.

[0375] Here, the second projection relationship refers to the relationship between the target point cloud of the second stacking object and the second grayscale image.

[0376] The following takes the first boundary line TC1 of the first grayscale image, the third boundary line TC3 of the first grayscale image, and the fifth boundary line TC5 of the first grayscale image as examples, and describes the corresponding first boundary point cloud S1, third boundary point cloud S3, and fifth boundary point cloud S5 based on the first projection relationship.

[0377] The first boundary line TC1 of the first grayscale image (eg Fig.17 TC1 in the first grayscale image), the third boundary line TC3 of the first grayscale image (such as Fig.17 TC3 in the first grayscale image), the fifth boundary line TC5 in the first grayscale image (such as Fig.17 TC5 in the first grayscale image), respectively, based on the first projection relationship, a first boundary point cloud S1 (such as Fig. 20 S1 in the first grayscale image), and a third boundary point cloud S3 corresponding to the third boundary line TC3 of the first grayscale image (such as Fig. 20 S3 in ), and a fifth boundary point cloud S5 corresponding to the fifth boundary line TC5 of the first grayscale image (such as Fig. 20 in S5).

[0378] The following takes the second boundary line TB2 of the second grayscale image, the fourth boundary line TB4 of the second grayscale image, and the sixth boundary line TB6 of the second grayscale image as examples, and describes the corresponding second boundary point cloud T2, fourth boundary point cloud T4, and sixth boundary point cloud T6 based on the second projection relationship.

[0379] The second boundary line TB2 of the second grayscale image (eg Fig.18 TB2 in the second grayscale image), the fourth boundary line TB4 of the second grayscale image (such as Fig.18 TB4 in the second grayscale image), the sixth boundary line TB6 in the second grayscale image (such as Fig.18 TB6 in the figure), based on the above second projection relationship, the second boundary point cloud T2 (such as Fig. 20 T2 in the second grayscale image), and a fourth boundary point cloud T4 (such as Fig. 20 T4 in ), and the sixth boundary point cloud T6 corresponding to the sixth boundary line TB6 of the second grayscale image (such as Fig. 20 T6 in ).

[0380] The distribution of each boundary point cloud in the embodiment of the present application directly reflects the geometric shape of the stacked object. For example, a stacked object of a cube will generate a point cloud with six rectangular faces.

[0381] exist Fig. 20 In the figure, according to the geometric shape of the first stacking object (such as a rectangle), the first boundary point cloud S1, the third boundary point cloud S3 and the fifth boundary point cloud S5 are distributed on the rectangular surface corresponding to the first stacking object; according to the geometric shape of the second stacking object, the second boundary point cloud T2, the fourth boundary point cloud T4 and the sixth boundary point cloud T6 are distributed on the rectangular surface corresponding to the second stacking object.

[0382] exist Fig. 20 In the figure, the least squares method is used to fit the first boundary point cloud S1 distributed on the rectangular surface to obtain the first edge line W1; the least squares method is used to fit the third boundary point cloud S3 distributed on the rectangular surface to obtain the third edge line W3; the least squares method is used to fit the fifth boundary point cloud S5 distributed on the rectangular surface to obtain the fifth edge line W5.

[0383] exist Fig. 20 In the figure, the least squares method is used to fit the second boundary point cloud T2 distributed on the rectangular surface to obtain the second edge line V2; the least squares method is used to fit the fourth boundary point cloud T4 distributed on the rectangular surface to obtain the fourth edge line V4; the least squares method is used to fit the sixth boundary point cloud T6 distributed on the rectangular surface to obtain the sixth edge line V6.

[0384] Taking the third boundary point cloud S3 as an example, the least square method is used to fit the third boundary point cloud S3 distributed on the rectangular surface to obtain the third edge line W3 (at Fig. 20 The red line in the middle).

[0385] For example, find a straight line y=mx+b on the rectangular surface, where m is the slope and b is the intercept, so that the straight line is as close as possible to the third boundary point cloud S3 distributed on the rectangular surface. The goal of the least squares method is to find the values ​​of m and b so that the sum of the squares of the perpendicular distances (i.e., errors) from all points (i.e., all points in the third boundary point cloud S3 distributed on the rectangular surface) to the straight line is minimized; the sum of the squares of the perpendicular distances (i.e., errors) from all points to the straight line is minimized, and the determined straight line is taken as the third edge line W3.

[0386] It should be noted that, for other boundary point clouds, the edge lines thereof may be determined in the same manner as the third boundary point cloud, which will not be described in detail herein.

[0387] The first target point in the embodiment of the present application may be any point on the first stacking object, such as the midpoint on the first edge line W1 of the first stacking object (eg Fig. 20 The second target point may be any point on the second stacked object, such as the midpoint of the second edge line V2 of the second stacked object (e.g. Fig. 20 The corresponding figures of this application take the midpoint as an example.

[0388] In one example, the embodiment of the present application may determine the first target point (eg Fig. 20 The first target point P1 in , and the second target point (such as Fig. 20 The second target point P2 in ( ), specifically:

[0389] exist Fig. 20 In the process, the controller determines the first edge line W1 of the first stacking object according to the first boundary point cloud S1; the controller determines the third edge line W3 of the first stacking object according to the third boundary point cloud S3; the controller determines the fifth edge line W5 of the first stacking object according to the fifth boundary point cloud S5; the controller determines the first center line M1 according to the third edge line W3 and the fifth edge line W5; the controller determines the first target point P1 according to the first center line M1 and the first edge line W1.

[0390] exist Fig. 20 In the process, the controller determines the second edge line V2 of the second stacking object according to the second boundary point cloud T2; the controller determines the fourth edge line V4 of the second stacking object according to the fourth boundary point cloud T4; the controller determines the sixth edge line V6 of the second stacking object according to the sixth boundary point cloud T6; the controller determines the second center line M2 according to the fourth edge line V4 and the sixth edge line V6; the controller determines the second target point according to the second center line M2 and the second edge line V2.

[0391] The relative position data involved in this embodiment includes a relative position difference and a relative angle difference. The relative position difference can be determined according to the position difference between the first target point P1 and the second target point P2.

[0392] The following describes in detail several ways to determine the relative angle difference:

[0393] First way:

[0394] The controller determines a relative angle difference between the first stacking object and the second stacking object according to a first angle between the first edge line W1 and the second edge line V2 and the first angle.

[0395] exist Fig. 20 In the figure, the first angle is an angle on the XOY plane in the three-dimensional coordinate system, and the first angle is an angle formed by projecting the first edge line W1 and the second edge line V2 on the XOY plane.

[0396] Optionally, the first angle may also be: the angle between the third edge line W3 and the fourth edge line V4; or, the angle between the fifth edge line W5 and the sixth edge line V6; or, the angle between the seventh edge line and the eighth edge line.

[0397] Second way:

[0398] The controller calculates a second angle between the first center line M1 and the second center line M2, and determines a relative angle difference between the first stacking object and the second stacking object according to the second angle.

[0399] exist Fig. 20 In the figure, the second angle is an angle on the XOY plane in the three-dimensional coordinate system, and the second angle is an angle formed by projecting the first midline M1 and the second midline M2 onto the XOY plane.

[0400] The third way:

[0401] The controller calculates a second angle between the first center line M1 and the second center line M2;

[0402] The controller calculates a first angle between the first edge line W1 and the second edge line V2;

[0403] The controller obtains a relative angle difference between the first stacking object and the second stacking object by weighting the first angle and the second angle.

[0404] exist Fig. 20 In , the relative angle difference is obtained by weighting the second angle between the first center line M1 and the second center line M2, and the first angle between the first edge line W1 and the second edge line V2; wherein the weighted proportion distribution can be adjusted according to the actual situation, for example, when the boundary point cloud corresponding to the first edge line W1 and the second edge line V2 has a higher clarity, the weighted proportion of the first angle can be increased. Fig. 20 In the example, the angle difference can be obtained by the difference between the direction indicated by the arrow at the upper arrow root, i.e., the first target point P1, and the direction indicated by the arrow at the lower arrow root, i.e., the second target point P2. Fig.21 (a) to (c) adopt the Fig. 20 The direction indicated by the arrow of the first target point and the direction indicated by the arrow of the second target point are represented by different perspectives.

[0405] Fourth way:

[0406] The controller determines a first angle value of a first stacked object;

[0407] The controller determines a second angle value of a second stacked object;

[0408] The controller determines a relative angle difference between the first stacking object and the second stacking object according to a difference between the first angle value and the second angle value.

[0409] The controller determines the first angle value of the first stacking object, which may be:

[0410] The controller determines a first angle value of the first stacking object according to the angle of the first edge line W1; or

[0411] The controller determines a first angle value of the first stacking object according to the angle of the first midline M1; or

[0412] The controller obtains the first angle value of the first stacking object according to the weighted angle of the first center line M1 and the angle of the first edge line W1, where the weighted ratio can be adjusted according to actual conditions. For example, when the boundary point cloud data corresponding to the first edge line W1 is relatively clear, the angle weighted ratio of the first edge line W1 can be appropriately increased.

[0413] Similarly, the controller determines a second angle value of the second stacked object, which may be:

[0414] The controller determines a second angle value of the second stacking object according to the angle of the second edge line V2; or

[0415] The controller determines a second angle value of the second stacking object according to the angle of the second midline M2; or

[0416] The controller obtains the second angle value of the second stacking object by weighting the angle of the second center line M2 and the angle of the second edge line V2. The weighting ratio can be adjusted according to the actual situation. For example, when the boundary point cloud data corresponding to the second edge line V2 is relatively clear, the weighting ratio of the angle of the second edge line V2 can be appropriately increased.

[0417] It should also be noted that the first angle value of the first stacking object and the second angle value of the second stacking object can be flexibly selected according to actual conditions. For example, when the first angle value of the first stacking object is determined according to the angle of the first edge line W1, the second angle value of the second stacking object can be determined according to the angle of the second center line M2.

[0418] In the embodiment of the present application, the boundary line of the first grayscale image is obtained in the following manner: the controller extracts a straight line from the first grayscale image according to the probabilistic Hough transform straight line detection algorithm, and determines the boundary line of the first grayscale image from the extracted straight lines according to the size of the first stacked object; the boundary line of the second grayscale image is obtained in the following manner: the controller extracts a straight line from the second grayscale image according to the probabilistic Hough transform straight line detection algorithm, and determines the boundary line of the second grayscale image from the extracted straight lines according to the size of the second stacked object.

[0419] Here, the Probabilistic Hough Transform line detection algorithm reduces the amount of calculation by randomly selecting parameters of edge points in the parameter space.

[0420] In the rectangular coordinate system, the straight line of the first grayscale image can be expressed in polar coordinates as r=xcosθ+ysinθ, where r is the distance from the straight line to the origin, and θ is the direction angle of the straight line. Each edge point of the first grayscale image corresponds to a sine curve in the parameter space, and the intersection of these curves represents the straight line passing through these points. The probabilistic Hough transform determines the straight line parameters by randomly selecting edge points and calculating their corresponding parameter space curves, and then finding the intersection of these curves.

[0421] Here, the size of the first stacked object may include length, width and height, and these sizes are used as a basis for screening and matching straight lines from the extracted straight lines to determine the boundary line of the first grayscale image.

[0422] In the embodiment of the present application, a traditional probabilistic Hough transform line detection algorithm can be used to extract lines from a grayscale image. In a scenario where the first stacked object and the second stacked object have simple structures and little environmental interference, Hough transform line detection can also be used.

[0423] However, in order to be applicable to the first stacked object and the second stacked object with a more complex structure and improve the detection accuracy, the probabilistic Hough transform straight line detection algorithm is improved in the embodiment of the present application. Specifically, the controller may extract a straight line from the grayscale image (i.e., the first grayscale image or the second grayscale image) according to the improved probabilistic Hough transform straight line detection algorithm, including the following steps:

[0424] The controller sorts the pixel points in the grayscale image according to the grayscale value to obtain a pixel sequence set, wherein the pixel sequence set includes multiple pixel sequences, and each pixel sequence contains pixel points corresponding to the same grayscale value; the pixel sequences are taken out in sequence according to the sorting as the current pixel sequence, and pixel points are randomly selected in the current pixel sequence to find the straight line with the highest probability under the angle range, and the number of pixels on the straight line meets the straight line length requirement and the sum of the grayscale values ​​is the largest; if the distance between the found straight line and the extracted straight line is greater than or equal to the maximum straight line spacing threshold, the straight line is extracted, and the current pixel sequence is deleted from the pixel sequence set, and the next pixel sequence is continued to be used as the current pixel sequence until the pixel sequence set is empty.

[0425] It should be noted that the above-mentioned angle range, maximum straight line spacing threshold and number of straight lines can all be straight line parameters selected by the probabilistic Hough transform straight line detection algorithm in the parameter space.

[0426] Among them, for the pixel points in the current pixel sequence: calculate the corresponding series of straight line parameters (r and θ in the polar coordinate system) and vote in the parameter space. The angle range is used to limit the search direction of the straight line. The setting of the angle range needs to be adjusted according to the specific application scenario.

[0427] In one example, the parameters of the straight line include at least one of: pixel distance resolution, angle resolution, angle range, minimum straight line length threshold, maximum straight line spacing threshold, and the number of straight lines.

[0428] Specifically, 1) sort all the pixels in the grayscale image from large to small according to the grayscale value to form a pixel sequence set; 2) extract the pixel sequence from the pixel sequence set in order from large to small grayscale value, and randomly extract pixel points from the pixel sequence; 3) find the straight line with the highest probability under the angle range among the extracted pixel points, the number of pixels on the straight line meets the straight line length threshold requirement (such as the above-mentioned minimum straight line length threshold) and the sum of the grayscale values ​​is the largest; 4) find all the pixel points on the straight line, and determine whether the interval between the found straight line and the extracted straight line meets the maximum straight line spacing threshold. If not, return to 2), and if it meets, execute 5); 5) extract the straight line and remove the pixel points on the straight line from the pixels to be extracted; 6) determine whether all the pixel points in the pixel sequence set have been traversed. If so, return to the extracted straight line set; if not, determine whether the number of extracted straight lines meets the requirement. If not, continue to 2), and if it meets, return to the extracted straight line set.

[0429] It should be noted that the method of generating the boundary line of the second grayscale image is the same as the method of determining the boundary line of the first grayscale image, which will not be described in detail here.

[0430] The above step 504, namely "the controller controls the movement of the transport device according to the relative posture to align the first stacking object with the second stacking object" is described in detail below in conjunction with the embodiments.

[0431] In the embodiment of the present application, once the relative posture between the first stacking object and the second stacking object is calculated, the movement of the handling device can be controlled according to the relative posture until the relative posture is within a second preset threshold range.

[0432] Optionally, the second preset threshold range may be set to: -5mm<Δx2<5mm, -5mm<Δy2<5mm, -0.5°<Δθ2<0.5°. It should be noted that the threshold range is only an example and may be adjusted according to different stacking objects and handling equipment in practice.

[0433] Further, when the relative posture is not within a second preset threshold range, the controller controls the handling device to adjust the posture;

[0434] The controller reacquires target data of the first stacked object and target data of the second stacked object through the first sensor;

[0435] The controller re-determines the relative posture between the posture of the first stacking object and the posture of the second stacking object according to the re-acquired target data.

[0436] Furthermore, the process of controlling the movement of the handling equipment can be divided into two stages:

[0437] Phase 1: When the longitudinal distance between the handling device and the stacking end point is greater than a preset distance threshold, only the body of the handling device is controlled to move.

[0438] The stacking end point refers to the position of the handling device when the first stacking object is aligned with the second stacking object.

[0439] At this time, only the movement of the chassis is used to correct the errors of the X-axis and Y-axis.

[0440] The second stage: when the longitudinal distance between the handling device and the stacking end point is not greater than a preset distance threshold, the body of the handling device is controlled to move while the fork is controlled to move horizontally.

[0441] At this time, the errors of the X-axis and Y-axis are corrected by the lateral movement of the vehicle body and the fork. The lateral movement of the fork refers to the movement of the fork in a direction perpendicular to the forward direction of the vehicle body, that is, left and right in the Y-axis direction.

[0442] Optionally, the distance threshold may be set to, but is not limited to, 15 cm.

[0443] like Fig. 22 As shown, it is a flowchart of the second stage of controlling the movement of the handling equipment provided by the embodiment of the present application. It includes the following steps:

[0444] Step 221: Calculate the lateral error of the vehicle body relative to the second stacking object, and calculate the lateral target value of the fork according to the lateral error;

[0445] In the embodiment of the present application, when the longitudinal distance between the handling equipment and the stacking end point is less than 15 cm, the second stage of controlling the movement of the handling equipment is entered. At this time, the first sensor can be used to obtain the posture of the vehicle body and the Y-axis position of the fork, and the Y-axis position of the fork is recorded as `forkY_current`.

[0446] Based on the posture of the vehicle body and the second posture of the second stacking object, the lateral error and longitudinal error of the vehicle body relative to the second stacking object can be obtained. The lateral error of the vehicle body relative to the second stacking object is recorded as `dy`, and then the Y-axis target value of the fork is calculated as `forkY_target=forkY_current+dy`.

[0447] The horizontal direction here refers to the horizontal direction of the handling equipment body, that is, the Y-axis direction. The vertical direction refers to the front and rear driving direction of the vehicle body, that is, the X-axis direction.

[0448] Step 222: Calculate the longitudinal error of the vehicle body relative to the second stacking object, and control the movement of the vehicle body according to the longitudinal error until the longitudinal error is within a preset longitudinal threshold range; at the same time, control the fork to move in the lateral direction according to the lateral target value, so that the lateral error of the fork relative to the second stacking object is within a preset lateral threshold range.

[0449] According to the longitudinal error of the vehicle body relative to the second stacking object, the vehicle body is controlled to move. Meanwhile, according to the calculated lateral target value, the fork is controlled to move in the lateral direction.

[0450] Repeat the above vehicle body movement and fork lateral movement operations until the longitudinal error of the vehicle relative to the second stacking object is less than the preset longitudinal threshold, and the lateral error of the fork relative to the second stacking object is less than the preset lateral threshold. Here, the lateral error of the fork relative to the second stacking object is the error between the Y-axis position `forkY_current` of the fork and the Y-axis target value `forkY_current`.

[0451] Optionally, the longitudinal threshold range may be, but is not limited to, set at ±3 mm, and the lateral threshold range may be, but is not limited to, set at ±3 mm.

[0452] In addition, since a high perception and control accuracy is required in the alignment stage, the embodiment of the present application can obtain the longitudinal distance between the handling device and the stacking end point in real time, and control the movement speed of the vehicle body and / or fork according to the longitudinal distance. For example, when the longitudinal distance between the handling device and the stacking end point is less than 30cm, the movement speed of the vehicle body is less than 5cm / s, the heading change speed of the vehicle body is less than 0.2rad / s, and the lateral movement speed of the fork is less than 5cm / s.

[0453] Furthermore, when the relative position of the first stacking object with respect to the second stacking object is within the second preset threshold range, in order to ensure the accuracy of the alignment, the embodiment of the present application can also make a final judgment and confirmation on the alignment state of the first stacking object and the second stacking object.

[0454] Specifically, firstly, a current position and posture of the first stacked object is obtained, and then the coordinates of at least one corner point of the first stacked object are determined according to the current position and posture of the first stacked object and the size of the first stacked object;

[0455] Acquire a current position and posture of the second stacked object, and determine the coordinates of at least one corner point of the second stacked object according to the current position and posture of the second stacked object and the size of the second stacked object;

[0456] Calculate the coordinate difference between at least one corner point of the first stacked object and at least one corner point of the second stacked object;

[0457] When the coordinate difference is within the third preset threshold range, it is confirmed that the first stacking object and the second stacking object are aligned successfully; otherwise, it is confirmed that the first stacking object and the second stacking object are aligned unsuccessfully.

[0458] It should be noted that at least one corner point of the first stacking object corresponds to at least one corner point of the second stacking object. The so-called corresponding means that when the first stacking object and the second stacking object are aligned, they are adjacent to each other. Fig.23 As shown, the first stacking object A includes a first corner point D1, a third corner point D3, a fifth corner point D5 and a seventh corner point D7, and the second stacking object B includes a second corner point D2, a fourth corner point D4, a sixth corner point D6 and an eighth corner point D8. When the relative position of the first stacking object A relative to the second stacking object B is within the second preset threshold range, the first corner point D1 and the second corner point D2 are adjacent to each other, the third corner point D3 and the fourth corner point D4 are adjacent to each other, the fifth corner point D5 and the sixth corner point D6 are adjacent to each other, and the seventh corner point D7 and the eighth corner point D8 are adjacent to each other.

[0459] Taking the determination of the coordinates of the four corner points of the first stacking object and the coordinates of the four corner points of the second stacking object as an example, when confirming the alignment status, first determine the coordinates of the first corner point D1, the third corner point D3, the fifth corner point D5 and the seventh corner point D7 of the first stacking object; and the coordinates of the second corner point D2, the fourth corner point D4, the sixth corner point D6 and the eighth corner point D8 of the second stacking object; respectively calculate the first coordinate difference of the first corner point D1 relative to the second corner point D2, the second coordinate difference of the third corner point D3 relative to the fourth corner point D4, the third coordinate difference of the fifth corner point D5 relative to the sixth corner point D6, and the fourth coordinate difference of the seventh corner point D7 relative to the eighth corner point D8; when the first coordinate difference, the second coordinate difference, the third coordinate difference and the fourth coordinate difference are all within the third preset threshold range, confirm that the first stacking object and the second stacking object are aligned successfully; otherwise, confirm that the first stacking object and the second stacking object are aligned unsuccessfully.

[0460] Optionally, the third preset threshold may be, but is not limited to, set to: the coordinate differences in the X and Y directions are both within a range of -7 mm to 7 mm.

[0461] Furthermore, considering that the failure to align the first stacked object and the second stacked object may be caused by inaccurate odometer information, the embodiment of the present application further proposes to re-determine whether the first stacked object and the second stacked object are aligned by re-adjusting the posture of the handling equipment.

[0462] As an implementable manner, when the first stacked object and the second stacked object fail to be aligned, the controller determines a target posture of the handling device according to a current relative posture of the first stacked object and the second stacked object;

[0463] The controller controls the movement of the transport device and adjusts the posture of the transport device to a target posture;

[0464] The controller reacquires target data of the first stacking object and the second stacking object through the first sensor;

[0465] re-determining the relative position and posture of the first stacked object and the second stacked object according to the re-acquired target data;

[0466] According to the re-determined relative position and posture, it is determined whether the first stacking object and the second stacking object are aligned.

[0467] As another achievable manner, when the first stacked object and the second stacked object fail to be aligned, the controller determines a target posture of the handling device according to a current relative posture of the first stacked object and the second stacked object;

[0468] The controller controls the movement of the transport device and adjusts the posture of the transport device to a target posture;

[0469] The controller reacquires target data of the first stacking object through the first sensor;

[0470] The controller re-determines the position and posture of the first stacking object according to the re-acquired target data;

[0471] Re-determining the relative posture of the first stacked object and the second stacked object according to the re-determined posture of the first stacked object and the posture of the second stacked object when the alignment fails;

[0472] According to the re-determined relative position and posture, it is determined whether the first stacking object and the second stacking object are aligned.

[0473] In the above two implementations, the controller controls the movement of the transport device to adjust the posture of the transport device to the target posture, specifically including:

[0474] The controller controls the transport device to move from a first preset position to a position corresponding to a target posture.

[0475] The first preset position here refers to a position at a preset distance from the second stacking object. The preset distance can be flexibly adjusted, for example, set to 50 cm.

[0476] Furthermore, when the alignment of the first stacking object and the second stacking object fails, the embodiment of the present application also proposes an alignment failure retry mechanism.

[0477] As an achievable manner, when the first stacking object and the second stacking object fail to be aligned, the controller controls the transport device to transport the first stacking object and move it back to the stacking preparation position;

[0478] The controller reacquires fourth target data of the second stacked object through the first sensor;

[0479] The controller determines the posture of the second stacking object according to the reacquired fourth target data; and controls the movement of the handling device according to the posture of the second stacking object, so that the posture error of the handling device relative to the second stacking object is within a first preset threshold range;

[0480] The controller reacquires target data of the first stacking object and the second stacking object through the first sensor;

[0481] The controller controls the transport device to adjust its posture so that the first stacked object and the second stacked object are realigned.

[0482] In this way, when the alignment of the first stacking object and the second stacking object fails, the handling device can be controlled to return to the starting position of the pre-alignment stage, that is, the stacking preparation position, and re-execute the operations of the pre-alignment stage and the alignment stage to achieve realignment.

[0483] As another achievable manner, when the first stack object and the second stack object fail to be aligned, the controller controls the transport device to transport the first stack object to move back to the stacking operation position, and reacquires target data of the first stack object and the second stack object through the first sensor;

[0484] The controller controls the transport device to adjust its posture so that the first stacked object and the second stacked object are realigned.

[0485] In this way, when the alignment of the first stacking object and the second stacking object fails, the handling device can be controlled to return to the starting position of the alignment phase, that is, the stacking operation position, and the operation of the alignment phase is re-executed to achieve realignment.

[0486] In the above alignment failure retry mechanism, the re-pre-alignment and realignment processes are implemented in the same principle as the aforementioned pre-alignment stage and alignment stage, and will not be described in detail here.

[0487] Furthermore, the embodiment of the present application may also include:

[0488] The number of alignment failures is counted, and when the number of alignment failures is greater than a first preset number, an alarm prompt is output.

[0489] Through the above mechanism, when the number of alignment failures is too many, an alarm prompt can be used to prompt the user to intervene, thereby avoiding invalid alignment retries.

[0490] After the picking stage, the pre-alignment stage and the alignment stage, the embodiment of the present application may further include a stacking stage.

[0491] After the first stacking object is aligned with the second stacking object, the transport device is controlled to stack the first stacking object on the second stacking object.

[0492] Specifically, the first stacking object can be stacked on the second stacking object by lowering the height of the fork of the handling device. When the first stacking object is successfully stacked on the second stacking object, the controller controls the handling device to perform an end stacking operation, such as controlling the handling device to perform a fork extraction operation.

[0493] In order to avoid the situation where the first stacking object is stuck or partially stacked, the embodiment of the present application can further confirm the stacking state of the first stacking object relative to the second stacking object to determine whether the first stacking object is successfully stacked on the second stacking object.

[0494] As an achievable method, a perception detection method may be used to confirm the perceived stacking state of the first stacking object relative to the second stacking object.

[0495] Specifically, the first sensor scans the first stacking object and the second stacking object to obtain structural feature data of the stacking area of ​​the first stacking object and the second stacking object; and confirms the perceived stacking state of the first stacking object relative to the second stacking object based on the structural feature data.

[0496] As another achievable manner, a pressure-sensitive detection manner may also be used to confirm the pressure-sensitive stacking state of the second stacking object relative to the second stacking object.

[0497] Specifically, the second sensor acquires comprehensive pressure data, and confirms the pressure-sensitive stacking state of the second stacking object relative to the second stacking object according to the comprehensive pressure data. The second sensor here may be a pressure-sensitive switch.

[0498] As a preferred method, the perceived stacking state of the first stacking object relative to the second stacking object can be confirmed by using a perception detection method. When the perceived stacking state is a perceived stacking success, the pressure-sensing stacking state of the first stacking object relative to the second stacking object can be confirmed by using a pressure-sensing detection method. When the pressure-sensing stacking state is a pressure-sensing stacking success, it is confirmed that the first stacking object is successfully stacked on the second stacking object; when the perceived stacking state is a perceived stacking failure, or the pressure-sensing stacking state is a pressure-sensing stacking failure, it is confirmed that the first stacking object is not successfully stacked on the second stacking object.

[0499] In addition, when confirming the perceived stacking status and the pressure-sensitive stacking status, it is also possible to monitor whether the stacking status confirmation time exceeds the preset time. If the perceived stacking status or the pressure-sensitive stacking status is not confirmed within the preset time, the handling equipment can also be controlled to send an alarm prompt. This can avoid falling into an infinite waiting state when encountering special situations.

[0500] Furthermore, considering that the first stacking object is not successfully stacked on the second stacking object, it may be caused by the presence of an obvious obstacle on the second stacking object. Therefore, when the first stacking object is not successfully stacked on the second stacking object, the transport device can be controlled to lift the first stacking object. After lifting, the obstacle may be removed. At this time, the controller controls the transport device to restack the first stacking object on the second stacking object to further reconfirm whether the stacking is successful.

[0501] Furthermore, when the first stacking object is not successfully stacked on the second stacking object, the embodiment of the present application also proposes a stacking failure retry mechanism when stacking fails.

[0502] As an implementable manner, when the first stacking object is not successfully stacked on the second stacking object, the controller controls the handling device to lift the first stacking object;

[0503] The controller reacquires target data of the first stacking object and the second stacking object through the first sensor;

[0504] The controller controls the handling device to adjust the posture so that the first stacked object and the second stacked object are realigned;

[0505] The controller controls the transport device to re-stack the first stacked objects on the second stacked objects.

[0506] In this manner, the body of the handling device does not need to move, and only needs to lift the first stacking object, and re-acquire target data through the first sensor to realign and restack.

[0507] As another achievable manner, when the first stacking object is not successfully stacked on the second stacking object, the controller controls the handling device to lift the first stacking object;

[0508] The controller controls the transport device to transport the first stacking object to move back to the stacking preparation position;

[0509] The controller reacquires fourth target data of the second stacked object through the first sensor;

[0510] The controller controls the movement of the transport device according to the reacquired fourth target data, so that the position error of the transport device relative to the second stacking object is within a first preset threshold range;

[0511] The controller reacquires target data of the first stacking object and the second stacking object through the first sensor;

[0512] The controller controls the handling device to adjust the posture so that the first stacked object and the second stacked object are realigned;

[0513] The controller controls the transport device to re-stack the first stacked objects on the second stacked objects.

[0514] In this way, after controlling the handling device to lift the first stacking object, the handling device is controlled to return to the starting position of the pre-alignment stage, i.e., the stacking preparation position, and re-execute the operations of the pre-alignment stage, the alignment stage and the stacking stage to achieve realignment and restacking.

[0515] As another achievable manner, when the first stacking object is not successfully stacked on the second stacking object, the controller controls the handling device to lift the first stacking object;

[0516] The controller controls the transport device to transport the first stacking object to move back to the stacking operation position;

[0517] The controller reacquires target data of the first stacking object and the second stacking object through the first sensor;

[0518] The controller controls the handling device to adjust the posture so that the first stacked object and the second stacked object are realigned;

[0519] The controller controls the transport device to re-stack the first stacked objects on the second stacked objects.

[0520] In this way, after controlling the handling device to lift the first stacking object, the handling device is controlled to return to the starting position of the alignment phase, that is, the stacking operation position, and the operations of the alignment phase and the stacking phase are re-executed to achieve realignment and restacking.

[0521] In the stacking failure retry mechanism described above, the processes of re-pre-alignment, realignment and restacking are implemented in the same principle as the aforementioned pre-alignment stage, alignment stage and stacking stage, and will not be described in detail here.

[0522] Furthermore, the embodiment of the present application may also include:

[0523] The number of unsuccessful stacking is counted, and when the number of unsuccessful stacking is greater than a second preset number, an alarm prompt is output.

[0524] Through the above mechanism, when the number of unsuccessful stacking attempts is too many, an alarm prompt can be used to prompt the user to intervene, thereby avoiding invalid stacking retries.

[0525] Furthermore, the embodiment of the present application may also include:

[0526] The controller counts the number of alignment failures and unsuccessful stacking;

[0527] When the sum of the number of alignment failures and the number of unsuccessful stacking is greater than a third preset number, an alarm prompt is output.

[0528] Through the above mechanism, when the sum of the number of alignment failures and the number of unsuccessful stacking is too large, an alarm prompt can be used to prompt the user to intervene, thereby avoiding invalid retries.

[0529] The embodiment of the present application provides a full-process operation of the handling equipment in the picking stage for the first stacking object, and the pre-alignment stage, alignment stage and stacking stage for the first stacking object and the second stacking object, ensuring the accuracy and efficiency of the stacking operation.

[0530] The method proposed in the embodiment of the present application is described in detail below by taking the first stacking object being cage A and the second stacking object being cage B as an example.

[0531] refer to Fig.24 , is a schematic diagram of the entire stacking process provided by the embodiment of the present application. The method proposed in the embodiment of the present application can be divided into four parts: a picking stage, a pre-alignment stage, an alignment stage, and a stacking stage.

[0532] Pickup stage

[0533] Step 2401, the controller controls the transport device to move to the pickup position and fork the A material cage;

[0534] Step 2402, the controller controls the transporting equipment to move cage A to the stacking preparation area.

[0535] First, the controller controls the handling equipment to move to the front of cage A. During the movement, the first sensor is called to detect and calculate the position of cage A, and the path is planned in real time and the handling equipment is controlled to complete the forking of cage A.

[0536] Optionally, the first sensor can be a dual laser radar and dual camera architecture, which can perform high-density detection on the core feature area of ​​​​Cage A. The detection density is twice that of the original single laser radar, thereby shortening the integration time by half, thereby significantly improving the detection efficiency, and allowing the entire picking process to be completed in servo mode. That is, the handling equipment continuously detects key features such as the columns or sockets of Cage A during movement without stopping, thereby achieving efficient picking operations while detecting and moving to the stacking preparation area.

[0537] Pre-alignment stage

[0538] Step 2403, the controller controls the transporting equipment to move cage A to the stacking preparation position.

[0539] Step 2404: The controller obtains fourth target data of the B material cage through the first sensor, and determines the position and posture of the B material cage according to the fourth target data.

[0540] Step 2405: The controller controls the movement of the transport device according to the posture of the B material cage, so that the posture error of the transport device relative to the B material cage is within a first preset threshold range.

[0541] The purpose of the pre-alignment phase is to ensure that the handling equipment is basically aligned with the B cage before entering the alignment phase, reducing the need for large error correction when entering the alignment phase and improving overall efficiency and accuracy.

[0542] In the pre-alignment stage, the handling equipment moves with cage A to the front of cage B, firstly lifts the fork to the optimal detection height, i.e. the stacking preparation position, then detects the features of cage B such as the column, calculates the posture of cage B, and adjusts the vehicle body to complete the pre-alignment. In this stage, the controller can use a variety of path planning methods to ensure that the vehicle body is basically aligned with cage B in the direction of travel and rotation, and adopts forward and backward path planning strategies to achieve large error correction.

[0543] It should be noted that in the embodiment of the present application, the first sensor remains activated from the servo pre-alignment to the stacking stage and the entire retry process. The controller adopts different control strategies according to different stages to ensure accuracy and stability. In the traditional stacking operation scheme, the handling equipment will turn off the first sensor after running a certain distance after forking the A cage, resulting in a lack of perception support (blind walking) in the last distance of the driving process, increasing the risk of cumulative errors.

[0544] Alignment Phase

[0545] Step 2406, the controller controls the transporting device to transport cage A to the stacking operation position;

[0546] Specifically, the transport device is controlled to move toward the B material cage until the longitudinal distance between the transport device and the stacking end point reaches 10-30 cm.

[0547] Step 2407, the controller obtains target data of cage A and cage B through the first sensor, and extracts first target data of cage A and second target data of cage B from the target data;

[0548] Step 2408, the controller calculates the relative position of the A cage relative to the B cage according to the first target data and the second target data;

[0549] Step 2409: The controller controls the movement of the transport device according to the relative posture until the relative posture is within a second preset threshold range.

[0550] Specifically, when the longitudinal distance between the handling device and the stacking end point is greater than 15 cm, only the body of the handling device is controlled to move;

[0551] When the longitudinal distance between the handling device and the stacking end point is not greater than 15 cm, the body of the handling device is controlled to move while the fork is controlled to move laterally.

[0552] Step 2410, determine whether material cage A and material cage B are aligned.

[0553] When it is determined that the A cage and the B cage are aligned, the stacking operation can be performed at this time, and step 2411 is executed. When it is determined that the A cage and the B cage are not aligned, the stacking operation cannot be performed at this time, and the retry process is entered, and step 2417 is executed. The retry process can return to the pre-alignment stage or the alignment stage.

[0554] Specifically, whether cage A and cage B are aligned can be determined as follows:

[0555] According to the relative posture, determine the current posture of cage A;

[0556] According to the current position and size of cage A, determine the coordinates of the first corner point, the third corner point, the fifth corner point and the seventh corner point of cage A; and according to the current position and size of cage B, determine the coordinates of the second corner point, the fourth corner point, the sixth corner point and the eighth corner point of cage B;

[0557] Calculate a first coordinate difference between the first corner point and the second corner point, a second coordinate difference between the third corner point and the fourth corner point, a third coordinate difference between the fifth corner point and the sixth corner point, and a fourth coordinate difference between the seventh corner point and the eighth corner point;

[0558] When the first coordinate difference, the second coordinate difference, the third coordinate difference and the fourth coordinate difference are all within the third preset threshold range, it is confirmed that the alignment of material cage A and material cage B is successful, and the stacking operation can be performed; otherwise, it is confirmed that the alignment of material cage A and material cage B fails, and the stacking operation cannot be performed.

[0559] After the pre-alignment phase is completed, the handling equipment will immediately enter the alignment phase. At this time, the detection object of the first sensor is switched from the B cage in the pre-alignment phase to the stacking alignment surface of the A and B cages. Through this dual-cage synchronous observation, the relative posture between the upper and lower cages can be calculated in real time and transmitted to the controller for correction. The whole process simulates a human-like operation method and continuously corrects the position deviation in the stack.

[0560] During the alignment phase, the handling equipment continuously adjusts the body and forks based on the feedback information from the first sensor until the relative position of cage A relative to cage B is corrected to within the threshold range (i.e., the first preset threshold) set by the control (e.g., the coordinate difference in the x and y directions is within the range of ±5mm, and the rotation angle difference around the Z axis is within ±0.5°). When the relative position meets the set requirements, the first sensor will make a final judgment and confirmation to ensure that the coordinate difference of the four corner points of the upper and lower cages meets the preset threshold range (i.e., the third preset threshold) (e.g., the coordinate difference in the x and y directions is within ±10mm). At this time, the forks can be safely lowered to complete the stacking operation.

[0561] If the set threshold is not reached during the correction process, the vehicle and fork will continue to be adjusted until the accuracy requirement is met. If the coordinate difference of the four corner points of the upper and lower cages is still found to not meet the preset threshold range during the final judgment, the alignment failure retry mechanism will be activated and the retry process will be entered to ensure that cage A and cage B can be aligned successfully. This retry mechanism is designed to cope with large deviations and improve the alignment success rate through multiple adjustments and corrections.

[0562] The servo closed-loop alignment technology used in the embodiment of the present application realizes the synchronous observation and control correction of the upper and lower cages through the configured first sensor. During the entire process, the first sensor remains activated at all times, monitors the cage status in real time, and forms a closed-loop feedback between perception and control. This can effectively reduce the accumulation of various errors during the stacking process, such as posture deviations caused by uneven ground, deformation of the gantry, perception and control errors during pickup, odometer errors, and the sliding of the cage on the fork during the stacking process.

[0563] Stacking stage

[0564] Step 2411, control the handling equipment to stack cage A on cage B.

[0565] Step 2412, determine whether cage A is successfully stacked on cage B. If it is determined that cage A is successfully stacked on cage B, execute step 2413, otherwise enter the retry process and execute step 2414. The retry process can return to the pre-alignment stage or the alignment stage.

[0566] Step 2413: the controller controls the transport device to perform a fork-out operation. The process ends.

[0567] Step 2414, counting the number of unsuccessful stacking attempts.

[0568] Step 2415, determining whether the number of unsuccessful stacking is greater than a second preset number, when the number of unsuccessful stacking is greater than the second preset number, executing step 2419, otherwise executing step 2416.

[0569] Step 2416, the controller controls the handling device to lift cage A. If the pre-alignment stage is returned, the execution returns to step 2403, and if the alignment stage is returned, the execution returns to step 2406 (such as Fig.24 (shown by the dashed line).

[0570] Step 2417, counting the number of alignment failures.

[0571] Step 2418, determine whether the number of alignment failures is greater than a first preset number. If the number of alignment failures is greater than the first preset number, execute step 2419. Otherwise, if the pre-alignment stage is returned, then return to step 2403. If the alignment stage is returned, then return to step 2406 (such as Fig.24 (shown by the dashed line).

[0572] Step 2419: Output an alarm prompt. The process ends.

[0573] Through closed-loop control of the entire stacking process, the embodiment of the present application simplifies the stacking error into a combination of a single control error and a single sensing error, greatly improving the accuracy, not only increasing the stacking success rate, but also reducing the tolerance requirements for the cage, making the entire stacking process more efficient and safer.

[0574] Due to the use of closed-loop detection technology, in the embodiment of the present application, even if the upper cage slides slightly during the retry process, there is no need to put the cage back on the ground for forking, which greatly simplifies the complexity of the retry process and improves efficiency. The introduction of the retry mechanism significantly improves the success rate of stacking, making it close to 100%, thereby effectively ensuring the stability and reliability of the entire stacking process.

[0575] like Fig.25 As shown, it is a specific implementation flowchart for determining whether material cage A is successfully stacked on material cage B provided in an embodiment of the present application. Fig.25 The process shown is Fig.24 The specific implementation of step 2412 in the embodiment includes:

[0576] Step 2501: The controller collects point cloud data through the first sensor, and the point cloud data is the point cloud data of the A material cage and the B material cage.

[0577] Step 2502, pre-processing the point cloud data, including filtering the point cloud data, removing noise points, and retaining valid point clouds.

[0578] Step 25031, extracting the fifth target data of the fifth target area of ​​cage A and the sixth target data of the sixth target area of ​​cage B from the preprocessed point cloud data.

[0579] In the embodiment of the present application, the fifth target area includes at least the bottom beam on one side of the first stacking object, and the sixth target area includes at least the top beam on one side of the second stacking object.

[0580] During the stacking process, when viewed along the X-axis, as a possible approach, Figure 4 As shown in (a), the fifth target area includes the bottom beam on the left side (or right side) of the first stacked object, and the sixth target area includes the top beam on the left side (or right side) of the second stacked object; as another achievable method, Figure 4 As shown in (b), the fifth target area includes the bottom beam of the front side (or rear side) of the first stacked object, and the sixth target area includes the top beam of the front side (or rear side) of the second stacked object.

[0581] It should be noted that the fifth target area and the sixth target area here only need to be located on the same side of the first stacking object and the second stacking object respectively, and at least include their respective bottom beams and top beams. The embodiment of the present application does not limit their specific locations.

[0582] Specifically, the controller can obtain pre-stored specification parameters (such as size, shape and other information) of the first stacking object and the second stacking object from the memory, and then determine the fifth target data of the fifth target area and the sixth target data of the sixth target area from the target data based on the specification parameters.

[0583] Step 25032, extract the seventh target data of the seventh target area of ​​cage A and the eighth target data of the eighth target area of ​​cage B from the preprocessed point cloud data.

[0584] In the embodiment of the present application, the seventh target area includes at least the bottom beam on one side of the first stacking object, and the eighth target area includes at least the top beam on one side of the second stacking object.

[0585] The method of obtaining the seventh target data of the seventh target area is the same as the method of obtaining the fifth target data of the fifth target area, which will not be described in detail herein.

[0586] The method of acquiring the eighth target data of the eighth target area is the same as the method of acquiring the sixth target data of the sixth target area, which will not be repeated here.

[0587] It should be noted that the fifth target area and the seventh target area can be located on opposite sides of the first stacked object, or on adjacent sides of the first stacked object; similarly, the sixth target area and the eighth target area can be located on opposite sides of the second stacked object, or on adjacent sides of the second stacked object. In the embodiment of the present application, it is sufficient to ensure that the fifth target area and the sixth target area are located on the same side, and the seventh target area and the eighth target area are located on the same side.

[0588] Step 25041, extract the first boundary of cage A according to the fifth target data, and extract the second boundary of cage B according to the sixth target data.

[0589] As a feasible method, in the embodiment of the present application, the fifth target data and the sixth target data are projected onto a two-dimensional plane to obtain a first projection image, and then the contour of the first projection image is extracted to obtain a first boundary of the first stacked object and a second boundary of the second stacked object.

[0590] The two-dimensional plane here may include but is not limited to a plane used for projecting and analyzing target data in the coordinate system of the handling equipment.

[0591] When the fifth target area includes the bottom beam on the left side (or right side) of the first stacked object and the sixth target area includes the top beam on the left side (or right side) of the second stacked object, this two-dimensional plane is the XOZ plane in the coordinate axis centered on the transport equipment, wherein the X-axis represents the longitudinal direction (also the direction of travel) of the transport equipment, the Y-axis represents the lateral direction of the transport equipment, and the Z-axis represents the height direction of the transport equipment.

[0592] When the fifth target area includes the bottom beam on the front side (or rear side) of the first stacked object and the sixth target area includes the top beam on the front side (or rear side) of the second stacked object, this two-dimensional plane is the YOZ plane in the coordinate axis centered on the handling equipment.

[0593] It should be noted that the projection direction is not limited to being perpendicular to the XOZ direction and the YOZ direction, and any other projection direction perpendicular to the Z axis can be used as the projection direction of the embodiment.

[0594] Step 25042, extract the third boundary of material cage A according to the seventh target data, and extract the fourth boundary of material cage B according to the eighth target data.

[0595] The implementation method of this step is the same as the implementation principle of step 25041, which will not be repeated here.

[0596] Step 25051, calculate the width of the first gap between cage A and cage B based on the first boundary and the second boundary, and compare the width of the first gap with the first threshold to confirm the first stacking state of cage A stacking relative to cage B.

[0597] In the embodiment of the present application, the first stacking state includes a first safe state and a first unsafe state, wherein the first safe state means that the width of the first gap is less than or equal to the first threshold.

[0598] by Figure 4 Taking the target area configuration method in (a) as an example, the first stacking state is described.

[0599] like Fig.26 As shown, Fig.26 (a) is a side view when the first stacking state is successful stacking, and the gap width formed by stacking the first stacking object A and the second stacking object B calculated according to the first boundary corresponding to the first edge J1 of the first stacking object A and the second boundary corresponding to the second edge J2 of the second stacking object B is less than the first threshold value; Fig.26 (b) is a projection diagram of the YOZ plane obtained by projecting the first edge J1 and the second edge J2.

[0600] like Fig. 27 As shown, Fig. 27 (a) is a side view when the first stacking state is stacking failure, and the gap width formed by stacking the first stacking object A and the second stacking object B calculated according to the first boundary corresponding to the first edge J1 of the first stacking object A and the second boundary corresponding to the second edge J2 of the second stacking object B is greater than or equal to the first threshold; Fig. 27 (b) is a projection diagram of the YOZ plane obtained by projecting the first edge J1 and the second edge J2.

[0601] Specifically, when calculating the width of the first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, if the width of the gap formed between the first boundary and the second boundary is inconsistent, the width of the first gap is determined based on the maximum width of the gap formed between the first boundary and the second boundary.

[0602] It should be noted that the first threshold in the embodiment of the present application can be automatically learned based on historical data, or it can be set according to actual needs, and this is not limited in the embodiment of the present application.

[0603] Step 25052, calculate the width of the second gap between cage A and cage B based on the third boundary and the fourth boundary, and compare the width of the second gap with the first threshold to confirm the second stacking state of cage A relative to cage B.

[0604] In the embodiment of the present application, the second stacking state includes a second safe state and a second unsafe state, wherein the second safe state means that the width of the second gap is less than or equal to the first threshold.

[0605] The implementation method of this step is the same as the implementation principle of step 25051, which will not be repeated here.

[0606] It should also be noted that the above-mentioned steps 25031 and 25032, steps 25041 and 25042, and steps 25051 and 25052 may be steps executed in parallel, or only steps 25031, 25041 and 25051 may be executed, or only steps 25032, 25042 and 25052 may be executed.

[0607] Step 2506: confirm the perceived stacking state of cage A relative to cage B according to the first stacking state and / or the second stacking state. When the perceived stacking state is perceived stacking success, execute step 2507, otherwise confirm that cage A is not successfully stacked on cage B.

[0608] In this step, when only step 25031, step 25041 and step 25051 are executed in the above process, the perceived stacking state of cage A relative to cage B is confirmed based on the first stacking state; when only step 25032, step 25042 and step 25052 are executed, the perceived stacking state of cage A relative to cage B is confirmed based on the second stacking state; when step 25031 and step 25032, step 25041 and step 25042, and step 25051 and step 25052 are executed in parallel, the perceived stacking state of cage A relative to cage B is confirmed based on the first stacking state and the second stacking state.

[0609] Wherein, confirming the perceived stacking state of cage A relative to cage B according to the first stacking state and the second stacking state specifically includes:

[0610] When the first stacking state is the first safe state and the second stacking state is the second safe state, the perceived stacking state is confirmed to be a perceived stacking success; when the first stacking state is the first unsafe state and / or the second stacking state is the second unsafe state, the perceived stacking state is confirmed to be a perceived stacking failure.

[0611] That is to say, when the width of the first gap is less than or equal to the first threshold, and the width of the second gap is less than or equal to the first threshold, the perceived stacking state is confirmed to be perceived stacking success; when the width of the first gap is greater than the first threshold, or the width of the second gap is greater than the first threshold, the perceived stacking state is confirmed to be perceived stacking failure.

[0612] Step 2507, the second sensor obtains comprehensive pressure data, and confirms the pressure-sensitive stacking state of cage A relative to cage B according to the comprehensive pressure data. When the pressure-sensitive stacking state is pressure-sensitive stacking success, it is confirmed that cage A is successfully stacked on cage B; when the pressure-sensitive stacking state is pressure-sensitive stacking failure, it is confirmed that cage A is not successfully stacked on cage B.

[0613] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0614] According to another aspect of the embodiment, a control system is provided for executing to implement any method disclosed in the embodiments of the present application.

[0615] According to another aspect of the embodiment, a transport device is also provided to implement any method disclosed in the embodiments of the present application.

[0616] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiment of the stacking device of the stacking object, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The embodiment of the stacking device of the stacking object described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0617] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0618] The present application also provides a computer program product, including a computer program, which implements any one of the methods described in the aforementioned method embodiments when executed by a processor.

[0619] It can be seen from the above description of the implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can essentially be embodied in the form of a computer program product, which can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc.

[0620] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An alignment method, characterized in that: include: The controller acquires target data of the first stacking object and the second stacking object through the first sensor; The controller extracts first target data of the first stacking object and second target data of the second stacking object from the target data; The controller calculates a relative posture of the first stacking object with respect to the second stacking object according to the first target data and the second target data; The controller controls the movement of the handling device according to the relative position and posture to align the first stacking object with the second stacking object.

2. The method according to claim 1, characterized in that Before the controller acquires target data of the first stacking object and the second stacking object through the first sensor, the method further includes: The controller controls the transport device to transport the first stacking object to a stacking preparation position; The controller acquires fourth target data of the second stacking object through the first sensor; The controller controls the movement of the transport device according to the fourth target data so that a posture error of the transport device relative to the second stacking object is within a first preset threshold range.

3. The method according to claim 2, characterized in that The controller controls the movement of the transport device according to the fourth target data so that the posture of the transport device relative to the second stacking object is within a first preset threshold range, including: The controller determines the posture of the second stacking object according to the fourth target data; According to the posture of the second stacking object, the transport device is controlled to move so that a posture error of the transport device relative to the second stacking object is within a first preset threshold range.

4. The method according to claim 1, characterized in that The controller acquires target data of the first stacking object and the second stacking object through the first sensor, including: When the controller controls the transport device to transport the first stacking object to a stacking operation position, the controller acquires the target data through the first sensor.

5. The method according to claim 4, characterized in that The controller controls the transport device to transport the first stacking object to a stacking operation position, including: The transport device is controlled to move toward the second stacking object until a projection length of a fork of the transport device on the second stacking object reaches a first length threshold.

6. The method according to claim 4, characterized in that The controller controls the transport device to transport the first stacking object to a stacking operation position, including: The transport device is controlled to move toward the second stacking object until a longitudinal distance between a first sensor of the transport device and the second stacking object is within a first preset distance.

7. The method according to claim 4, characterized in that The controlling the movement of the handling device according to the relative position to align the first stacking object with the second stacking object comprises: According to the relative position and posture, the transport device is controlled to move until the relative position and posture is within a second preset threshold range.

8. The method according to claim 7, characterized in that The method further comprises: When the relative posture is not within the second preset threshold range, the controller controls the handling device to adjust the posture; The controller reacquires target data of the first stacking object and target data of the second stacking object through the first sensor; The controller re-determines a relative posture between a posture of the first stacking object and a posture of the second stacking object according to the re-acquired target data.

9. The method according to claim 1, characterized in that: The controlling the movement of the handling equipment comprises: When the longitudinal distance between the transport device and the stacking end point is greater than a preset distance threshold, only the body of the transport device is controlled to move; When the longitudinal distance between the transport device and the stacking end point is not greater than the distance threshold, the body of the transport device is controlled to move while the fork is controlled to move in the lateral direction.

10. The method according to claim 9, characterized in that The method of controlling the body of the transporting equipment to move while controlling the fork to move laterally includes: Calculating a lateral error of the vehicle body relative to the second stacking object, and calculating a lateral target value of the fork according to the lateral error; The longitudinal error of the vehicle body relative to the second stacking object is calculated, and the movement of the vehicle body is controlled according to the longitudinal error until the longitudinal error is within a preset longitudinal threshold range; at the same time, according to the lateral target value, the fork is controlled to move in the lateral direction so that the lateral error of the fork relative to the second stacking object is within a preset lateral threshold range.

11. The method according to claim 9, characterized in that The method further comprises: The longitudinal distance between the handling device and the stacking end point is acquired in real time, and the movement speed of the vehicle body and / or the movement speed of the fork is controlled according to the longitudinal distance.

12. The method according to claim 8, characterized in that After controlling the movement of the transport device according to the relative posture until the relative posture is within a second preset threshold range, the method further includes: Obtaining a current position of the first stacked object; determining coordinates of at least one corner point of the first stacked object according to a current position of the first stacked object and a size of the first stacked object; Obtaining a current position of the second stack object; determining coordinates of at least one corner point of the second stacked object according to the current position of the second stacked object and the size of the second stacked object; wherein at least one corner point of the first stacked object corresponds to at least one corner point of the second stacked object; calculating a coordinate difference between at least one corner point of the first stacked object and at least one corner point of the second stacked object; When the coordinate difference is within a third preset threshold range, it is confirmed that the first stacking object and the second stacking object are aligned successfully; otherwise, it is confirmed that the first stacking object and the second stacking object are aligned unsuccessfully.

13. The method according to claim 12, characterized in that The method further comprises: When the first stacking object and the second stacking object fail to be aligned, the controller determines a target posture of the transport device according to a current relative posture of the first stacking object and the second stacking object; The controller controls the movement of the transport device to adjust the posture of the transport device to the target posture; The controller reacquires target data of the first stacking object and the second stacking object through the first sensor; re-determining the relative position and posture of the first stacking object and the second stacking object according to the re-acquired target data; According to the re-determined relative position and posture, it is determined whether the first stacking object and the second stacking object are aligned.

14. The method according to claim 12, characterized in that The method further comprises: When the first stacking object and the second stacking object fail to be aligned, the controller determines a target posture of the transport device according to a current relative posture of the first stacking object and the second stacking object; The controller controls the movement of the transport device to adjust the posture of the transport device to the target posture; The controller reacquires target data of the first stacking object through the first sensor; The controller re-determines the position and posture of the first stacked object according to the re-acquired target data; Re-determining the relative posture of the first stacked object and the second stacked object according to the re-determined posture of the first stacked object and the posture of the second stacked object when the alignment fails; According to the re-determined relative position and posture, it is determined whether the first stacking object and the second stacking object are aligned.

15. The method according to claim 13 or 14, characterized in that The controller controls the movement of the transport device to adjust the posture of the transport device to the target posture, including: The controller controls the transport device to move to a first preset position; The controller controls the transport device to move from the first preset position to a position corresponding to the target posture.

16. The method according to claim 12, characterized in that The method further comprises: When the first stacking object and the second stacking object fail to be aligned, the controller reacquires target data of the first stacking object and the second stacking object through the first sensor; The controller controls the transport device to adjust its posture so that the first stacking object and the second stacking object are realigned.

17. The method according to claim 16, characterized in that Before the first sensor reacquires the target data of the first stacked object and the second stacked object, the method further includes: The controller controls the transport device to transport the first stacking object to a stacking preparation position; The controller acquires fourth target data of the second stacking object through the first sensor; The controller determines the posture of the second stacking object according to the fourth target data; The controller controls the movement of the transport device according to the posture of the second stacking object, so that the posture error of the transport device relative to the second stacking object is within the first preset threshold range.

18. The method according to claim 12, characterized in that The method further comprises: When the first stacking object and the second stacking object fail to be aligned, the controller controls the transport device to move back to the stacking operation position, and the controller reacquires target data of the first stacking object and the second stacking object through the first sensor; The controller controls the transport device to adjust its posture so that the first stacking object and the second stacking object are realigned.

19. The method according to claim 12, characterized in that The method further comprises: The number of alignment failures is counted, and when the number of alignment failures is greater than a first preset number, an alarm prompt is output.

20. The method according to claim 12, characterized in that After the first stacked object is aligned with the second stacked object, the method further includes: The transport device is controlled to stack the first stacking object on the second stacking object.

21. The method according to claim 20, characterized in that After controlling the handling device to stack the first stacking object on the second stacking object, the method further includes: It is determined whether the first stacking object is successfully stacked on the second stacking object.

22. The method according to claim 21, characterized in that The determining whether the first stacking object is successfully stacked on the second stacking object includes: The sensed stacking state of the first stacking object relative to the second stacking object is confirmed by using a sensed detection method.

23. The method according to claim 22, characterized in that The determining whether the first stacking object is successfully stacked on the second stacking object further includes: The pressure sensing stacking state of the second stacking object relative to the first stacking object is confirmed by using a pressure sensing detection method.

24. The method according to claim 22, characterized in that Confirming the sensed stacking state of the first stacking object relative to the second stacking object by using a sensed detection method includes: The controller scans the first stacking object and the second stacking object through the first sensor to obtain structural feature data of the stacking area of ​​the first stacking object and the second stacking object; Based on the structural characteristic data, a perceived stacking state of the first stacking object relative to the second stacking object is determined.

25. The method according to claim 23, characterized in that Confirming the pressure-sensitive stacking state of the second stacking object relative to the second stacking object by using a pressure-sensitive detection method includes: The controller acquires comprehensive pressure data through the second sensor, and confirms the pressure-sensitive stacking state of the second stacking object relative to the second stacking object according to the comprehensive pressure data.

26. The method according to claim 21, characterized in that The method further comprises: When the first stacking object is not successfully stacked on the second stacking object, the controller controls the transport device to lift the first stacking object; The controller controls the transport device to re-stack the first stacked object on the second stacked object.

27. The method according to claim 21, characterized in that The method further comprises: When the first stacking object is not successfully stacked on the second stacking object, the controller controls the transport device to lift the first stacking object; The controller reacquires target data of the first stacking object and the second stacking object through the first sensor; The controller controls the transport device to adjust its posture so that the first stacking object and the second stacking object are realigned; The controller controls the transport device to re-stack the first stacked object on the second stacked object.

28. The method according to claim 27, characterized in that Before the controller reacquires the target data of the first stacking object and the second stacking object through the first sensor, the method further includes: The controller controls the transport device to transport the first stacking object to a stacking preparation position; The controller acquires fourth target data of the second stacking object through the first sensor; The controller controls the movement of the transport device according to the acquired fourth target data so that a posture error of the transport device relative to the second stacking object is within a first preset threshold range.

29. The method according to claim 27, characterized in that Before the controller reacquires the target data of the first stacking object and the second stacking object through the first sensor, the method further includes: The controller controls the transport device to move to the stacking operation position again.

30. The method according to claim 27, characterized in that The method further comprises: The controller counts the number of unsuccessful stacking times, and outputs an alarm prompt when the number of unsuccessful stacking times is greater than a second preset number.

31. The method according to claim 27, characterized in that The method further comprises: The controller counts the number of alignment failures and the number of unsuccessful stacking; When the sum of the alignment failure times and the unsuccessful stacking times is greater than a third preset number, an alarm prompt is output.

32. The method according to claim 2, characterized in that Before the controller controls the transport device to transport the first stacking object to a stacking preparation position, the method further includes: The controller controls the transport device to move to a pickup position; The controller acquires third target data of the first stacking object through the first sensor; According to the third target data of the first stacked object, controlling the movement of the transport device to obtain the first stacked object; The controller controls the transport device to transport the first stacking object to a stacking preparation area.

33. The method according to claim 32, characterized in that The controller controls the transport device to transport the first stacking object to a stacking preparation position, including: The controller controls the movement of the vehicle body and the lifting of the first stacking object simultaneously during the transport process.

34. A control system, characterized in that: The invention comprises a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 33.

35. A handling device, characterized in that: It comprises a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 33.

Images