Alignment state confirmation method, control system and carrying equipment

Abstract

The embodiment of the invention discloses an alignment state confirmation method, a control system and carrying equipment. According to the main technical scheme, a controller obtains an image and a point cloud through a sensor; extracting first target image data of a first target area from the image of the first stacking object, and extracting second target image data of a second target area from the image of the second stacking object; based on the first target image data, extracting first target point cloud data corresponding to the first target area from the point cloud of the first stacking object; based on the second target image data, extracting second target point cloud data corresponding to the second target area from the point cloud of the second stacking object; determining the pose of the first stacking object according to the first target point cloud data, and determining the pose of the second stacking object according to the second target point cloud data; and determining a difference value between the pose of the first stacked object and the pose of the second stacked object, and comparing the difference value with a threshold value to confirm the alignment state of the first stacked object and the second stacked object.

Description

Technical Field

[0001] The present application relates to the field of warehousing logistics technology or machine vision, and in particular to an alignment status confirmation method, a control system and a handling device. Background Art

[0002] Systems that use handling equipment such as AGV (automated guided vehicle) have the advantages of being highly unmanned, automated, and intelligent, which improves production efficiency and operational levels for industries such as warehousing, manufacturing, and logistics. As one of the more typical scenarios, handling equipment is often responsible for the handling of various goods. In the process of handling, it is inevitable to stack goods. Goods are usually packaged in cartons, etc., or stored in cages, wooden boxes, plastic boxes, etc.

[0003] Considering the space utilization, the handling equipment will involve stacking objects such as cartons, cages, wooden boxes, etc. in the process of handling goods. Considering the stability, the handling equipment needs to accurately stack one stacking object on top of another stacking object. In this process, if the two stacking objects cannot be aligned, it may affect the safety of the operation. Summary of the invention

[0004] The present application provides an alignment state confirmation method, a control system and a handling device, which are used to control the alignment of a first stacking object and a second stacking object during stacking.

[0005] This application provides the following solutions:

[0006] In a first aspect, a method for confirming an alignment state is provided, the method comprising: a controller acquiring an image and a point cloud of a first stacking object, and an image and a point cloud of a second stacking object through a sensor; the controller extracts first target image data of a first target area from the image of the first stacking object, and extracts second target image data of a second target area from the image of the second stacking object; the controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacking object based on the first target image data; the controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacking object based on the second target image data; the controller determines a posture of the first stacking object according to the first target point cloud data, and determines a posture of the second stacking object according to the second target point cloud data; the controller determines a difference between a posture of the first stacking object and a posture of the second stacking object, and compares the difference with a threshold value to confirm the alignment state of the first stacking object and the second stacking object.

[0007] Optionally, the method further includes: before confirming the alignment state, the controller controls the transport device to transport the first stacking object to a stacking preparation position to complete a pre-alignment action relative to the second stacking object.

[0008] Optionally, the controller extracts first target image data of the first target area from the image of the first stacked object, and extracts second target image data of the second target area from the image of the second stacked object, including: the controller extracts the first target image data of the first target area from the image of the first stacked object according to the image segmentation model, and extracts the second target image data of the second target area from the image of the second stacked object.

[0009] Optionally, the controller extracts first target image data of the first target area from the image of the first stacked object, and extracts second target image data of the second target area from the image of the second stacked object, including: the controller extracts the first target image data of the first target area from the image of the first stacked object according to the target detection model, and extracts the second target image data of the second target area from the image of the second stacked object.

[0010] Optionally, the controller extracts first target point cloud data corresponding to the first target area from the target point cloud of the first stacking object based on the first target image data, including: extracting the first target point cloud data corresponding to the first target area from the target point cloud of the first stacking object according to the joint calibration parameters and the first target image data; the controller extracts second target point cloud data corresponding to the second target area from the target point cloud of the second stacking object based on the second target image data, including: extracting the second target point cloud data corresponding to the second target area from the target point cloud of the second stacking object according to the joint calibration parameters and the second target image data.

[0011] Optionally, the method further includes: the controller pre-calibrates the camera for acquiring the image and the radar for acquiring the point cloud to obtain joint calibration parameters.

[0012] Optionally, the joint calibration parameters include at least one of the following: an intrinsic parameter and an extrinsic parameter of the camera, an extrinsic parameter from the radar to the camera, and an extrinsic parameter from the radar to the handling equipment.

[0013] Optionally, the difference is compared with a threshold to confirm the alignment state of the first stacked object and the second stacked object, including: if the difference is greater than or equal to the threshold, confirming the alignment state is misaligned; if the difference is less than the threshold, confirming the alignment state is aligned.

[0014] Optionally, the method also includes: when the alignment state is misaligned, the controller controls the handling equipment to adjust the posture; the controller re-acquires the image and point cloud of the first stacking object, and the image and point cloud of the second stacking object through the sensor; the controller redetermines the difference between the posture of the first stacking object and the posture of the second stacking object; the controller re-confirms the alignment state until the difference is less than a threshold.

[0015] Optionally, the method further includes: when the alignment state is aligned, the controller controls the handling device to place the first stacking object on the second stacking object to complete the stacking.

[0016] Optionally, controlling the transport equipment to adjust the posture includes: controlling the transport equipment to adjust the posture of the chassis or the posture of the fork.

[0017] Optionally, controlling the transport equipment to adjust its posture includes: the controller determines posture adjustment information of the transport equipment based on the alignment state; the controller obtains odometer information corresponding to a target time, the target time being the time corresponding to when the sensor obtains the image and point cloud of the first stacked object, and the image and point cloud of the second stacked object; the controller determines the posture adjustment information corresponding to the current time based on the odometer information corresponding to the target time, the posture adjustment information, and the odometer information corresponding to the current time; the controller controls the transport equipment to adjust its posture based on the posture adjustment information corresponding to the current time.

[0018] Optionally, the controller obtains an image and a point cloud of the first stacked object, and an image and a point cloud of the second stacked object through a sensor, including: a camera obtains an image of the first stacked object and an original image of the second stacked object, and a radar obtains an original point cloud of the first stacked object and an original point cloud of the second stacked object; the controller dedistorts the original image of the first stacked object according to a pre-calibrated camera intrinsic parameter, and dedistorts the original image of the second stacked object according to a pre-calibrated camera intrinsic parameter; the controller converts the original point cloud from a coordinate system where the radar is located to a coordinate system where the handling equipment is located, and dedistorts the original point cloud of the first stacked object according to the odometer information and the timestamp corresponding to the original image, and dedistorts the original point cloud of the second stacked object; the controller synchronizes the dedistorted image and the dedistorted point cloud to obtain the image and the point cloud of the first stacked object, and the image and the point cloud of the second stacked object.

[0019] Optionally, determining the posture of the first stacking object according to the first target point cloud data includes: the controller extracting a first border point cloud from the first target point cloud data; the controller fitting the first border point cloud according to the least squares method to obtain a border line equation corresponding to the first stacking object; the controller determines the posture of the first stacking object according to the border line equation corresponding to the first stacking object.

[0020] Optionally, determining the posture of the second stacking object according to the second target point cloud data includes: extracting a second border point cloud from the second target point cloud data; fitting the second border point cloud according to the least squares method to obtain a border line equation corresponding to the second stacking object; and determining the posture of the second stacking object according to the border line equation corresponding to the second stacking object.

[0021] Optionally, the first target area is located on a first side of the first stacked object, and the second target area is located on a second side of the second stacked object, and the first side and the second side are on the same side when the first stacked object and the second stacked object are aligned.

[0022] Optionally, the first stacking object is a first cage, and the second stacking object is a second cage; the first target area includes at least a portion of the first foot cup and at least a portion of the third foot cup of the first cage; the second target area includes at least a portion of the second column and at least a portion of the fourth column of the second cage.

[0023] Optionally, the first target area also includes at least a portion of the fifth cup and at least a portion of the seventh cup of the first material basket; the second target area also includes at least a portion of the sixth column and at least a portion of the eighth column of the second material basket.

[0024] Optionally, the first stacking object is a first material cage, and the second stacking object is a second material cage; the first target area includes a first border line and a third border line of the first material cage, and the first border line intersects with the third border line; the second target area includes a second border line and a fourth border line of the second material cage, and the second border line intersects with the fourth border line.

[0025] Optionally, the first target area also includes the fifth border line and the seventh border line of the first material cage, and the fifth border line intersects with the seventh border line; the second target area also includes the sixth border line and the eighth border line of the second material cage, and the sixth border line intersects with the eighth border line.

[0026] In a second aspect, a control system is provided, including a controller and a memory, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement any one of the methods described in the first aspect.

[0027] According to a third aspect, a handling device is provided, comprising a controller and a memory, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement any one of the methods described in the first aspect.

[0028] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0029] 1) The controller of the present application extracts the first target image data of the first target area from the image of the first stacking object acquired by the sensor, and extracts the second target image data of the second target area from the image of the second stacking object acquired by the sensor, and then the controller extracts the first target point cloud data corresponding to the first target area from the point cloud of the first stacking object acquired by the sensor based on the first target image data; the controller extracts the second target point cloud data corresponding to the second target area from the point cloud of the second stacking object acquired by the sensor based on the second target image data, and then the controller determines the posture of the first stacking object according to the first target point cloud data, and determines the posture of the second stacking object according to the second target point cloud data; finally, the controller determines the difference between the posture of the first stacking object and the posture of the second stacking object, and compares the difference with the threshold value to confirm the alignment state of the first stacking object and the second stacking object. In this way, the difference between the posture of the first stacking object and the posture of the second stacking object can be detected during the stacking process, and the alignment state between the first stacking object and the second stacking object can be confirmed based on the difference and the threshold value, which can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the posture data of the first stacking object and the second stacking object, and then determine the alignment state of the two.

[0030] 2) The present application can accurately determine the first target area of ​​the first stacked object and the second target area of ​​the second stacked object in the image according to the image segmentation model and the target detection model, thereby improving the detection efficiency.

[0031] 3) The present application can establish a spatial relationship between the camera and the radar, and this spatial relationship is determined by the intrinsic and extrinsic parameters of the camera, the extrinsic parameters from the radar to the camera, and the extrinsic parameters from the radar to the handling equipment; then, based on the above spatial relationship, the first target point cloud data corresponding to the first target area of ​​the first stacked object and the second target point cloud data corresponding to the second target area of ​​the second stacked object can be accurately determined from the target point cloud.

[0032] 4) The present application takes into account the time asynchrony of the data collected by the sensors and odometers carried by the handling equipment. In order to ensure that the data processed by the control system is the posture adjustment information data corresponding to the current moment, the posture adjustment information corresponding to the current moment will be determined based on the odometer information corresponding to the target moment, the posture adjustment information corresponding to the target moment, and the odometer information corresponding to the current moment, to ensure that the handling equipment accurately adjusts its posture according to the posture adjustment information corresponding to the current moment, thereby ensuring the alignment accuracy of the first stacking object and the second stacking object at the current moment.

[0033] 5) The controller of the present application will use the pre-calibrated camera internal parameters to dedistort the original image captured by the camera; and convert the original point cloud data collected by the sensor from the coordinate system where the radar is located to the coordinate system where the handling equipment is located, and dedistort the original point cloud according to the odometer information and the timestamp corresponding to the original image; finally, the dedistorted image and the dedistorted point cloud are time-synchronized to obtain a time-synchronized and distortion-free point cloud and image.

[0034] 6) In the present application, after the controller extracts the first border point cloud and the second border point cloud from the first target point cloud data corresponding to the first target area of ​​the first stacked object and the second target point cloud data corresponding to the second target area of ​​the second stacked object, the controller uses the least squares method to fit the first border point cloud and the second border point cloud respectively, to obtain the border line equation corresponding to the first stacked object and the border line equation corresponding to the second stacked object; then, the border line equation corresponding to the first stacked object and the border line equation corresponding to the second stacked object are used for calculation to accurately determine the difference between the posture of the first stacked object and the posture of the second stacked object.

[0035] 7) The present application dynamically controls the operation of the handling equipment based on the comparison result between the difference and the threshold; when the difference is greater than the threshold, the handling equipment is controlled to adjust its posture, and the data collected by the sensor carried by the handling equipment is re-acquired to redetermine the difference based on the re-collected data; until the difference is less than the threshold, the controller controls the handling equipment to place the first stacking object on the second stacking object to complete the stacking, forming a closed-loop servo detection control process. During the servo detection process, there is no need to stop the operation of the handling equipment, thereby improving the stacking efficiency.

[0036] Of course, any invention of the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A system schematic diagram of a handling device applicable to an embodiment of the present application;

[0039] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present application;

[0040] Figure 3 A flowchart of an alignment status confirmation method provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a foot cup and a column provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a segmented image provided in an embodiment of the present application;

[0043] Figure 6 Schematic diagram of the first target area and the second target area provided in the embodiment of the present application Figure 1 ;

[0044] Figure 7 Schematic diagram of the first target area and the second target area provided in the embodiment of the present application Figure 2 ;

[0045] Figure 8 A schematic diagram of a border line provided in an embodiment of the present application;

[0046] Fig. 9 A flowchart of an alignment status confirmation method provided in an embodiment of the present application.

[0047] Among them, 100-handling equipment; 101-handling equipment body; 102-sensor; 103-stacking execution component; 104-controller; 105-memory; A-first stacking object; B-second stacking object; A1-first target image area; B1-second target image area; Aj1-first foot cup; Aj3-third foot cup; Aj5-fifth foot cup; Aj7-seventh foot cup; Bj2-second column; Bj4-fourth column; Bj6-sixth column; Bj8-eighth column; a11-first border line; a13-third border line; a15-fifth border line; a17-seventh border line; b22-second border line; b24-fourth border line; b26-sixth border line; b28-eighth border line. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0049] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0050] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, the term "according to" used in this article is not limited to being based on a certain object. For example, determining B based on A can mean: determining B based only on A, or determining B based partially on A.

[0051] In the related art, the stacking of upper stacking objects is completed based on the posture of the lower stacking object relative to the handling equipment, without paying attention to the influence of various factors such as inaccurate posture of the upper stacking object when picking up goods, uneven ground, cumulative errors in the odometer, and errors in the handling equipment itself, which causes the upper stacking objects to be misaligned during stacking, affecting work safety.

[0052] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system schematic diagram of the handling equipment on which the present application is based is first described. Figure 1 A schematic diagram of an exemplary transport device to which the present invention can be applied is shown in FIG. Figure 1 As shown in , the transport device 100 includes a transport device body 101 , a sensor 102 , a stacking execution component 103 , a controller 104 , and a memory 105 .

[0053] The handling equipment 100 involved in the embodiment of the present application may be an unmanned forklift, a pallet truck, a crane, an AGV (Automated Guided Vehicle), an AMR (Automatic Mobile Robot), a humanoid robot, etc. The corresponding stacking execution component 103 may be a fork, a robotic arm, etc.

[0054] The controller 104 is used to control the stacking execution component 103 of the transport device 100 to execute the stacking of the first stacking object and the second stacking object. The controller 104 can be a control mainboard, a control box, a control unit, a vehicle-mounted computer, a computing platform, a tablet computer, a computer, or other systems or devices that perform computing or control functions on the transport device body 101, or a system or device that performs computing or control functions in a local server or a cloud server, or can be other forms such as a handheld controller or a remote controller. This is not limited in the embodiments of the present application.

[0055] The sensor 102 may be in the form of a sensor module, including at least a radar for collecting point cloud data, such as a laser radar, and / or a visual sensor for collecting image data, such as a camera.

[0056] The memory 105 is mainly used to store data collected by the sensor 102, such as point cloud data and image data.

[0057] First, the concepts of terms involved in the embodiments of the present application are introduced.

[0058] Stacking refers to arranging and stacking several objects up and down according to certain rules.

[0059] Stacking objects refer to the objects involved in stacking, which can be the goods themselves, or goods with simple packaging such as wrapping, or containers that can hold and carry goods, such as stacking objects, wooden boxes, plastic boxes, pallets, etc.

[0060] Stacking process: refers to the handling device lifting the first stacking object, so that the first stacking object moves and approaches the second stacking object, and then aligns the first stacking object with the second stacking object by adjusting the posture of the handling device, and then places the first stacking object on the second stacking object to complete the stacking.

[0061] The first stacking object refers to the stacking object located at the top, and the second stacking object refers to the stacking object located at the bottom.

[0062] Alignment state: refers to the state in which two or more stacked objects are arranged in a vertical direction, and at least part of the border lines between the stacked objects are parallel or overlapped. The vertical direction refers to the Z-axis direction in the coordinate system of the handling equipment (such as Figure 1 and Figure 2 The Z axis in the image).

[0063] like Figure 2 As shown, the geometric center of the transport device can be taken as the O point of the coordinate system, the front and rear travel direction of the transport device (i.e., the longitudinal direction of the transport device body 101) can be taken as the X-axis, wherein the positive direction of the X-axis is the direction away from the transport device attachment (such as a fork), the height direction of the transport device can be taken as the Z-axis, and the lateral direction of the transport device body 101 can be taken as the Y-axis, wherein the positive direction of the Y-axis is perpendicular to the paper and outward ( Figure 2 not shown).

[0064] The first stacking object has a first target area, which refers to a key structural area on the first stacking object that contacts the second stacking object after stacking. For example, the first target area at least includes key structural areas such as a foot cup and a frame line.

[0065] The second stacking object has a second target area, which refers to the key structural area on the second stacking object that contacts the first stacking object after stacking. For example, the second target area at least includes key structural areas such as columns and frame lines.

[0066] First target image data: refers to image data of a first target area extracted by a sensor from an image of a first stacked object. The first target image data can be used to extract first target point cloud data corresponding to the first target area from a point cloud of the first stacked object.

[0067] Second target image data: refers to image data of the second target area extracted by the sensor from the image of the second stacked object. The second target image data can be used to extract second target point cloud data corresponding to the second target area from the point cloud of the second stacked object.

[0068] First target point cloud data: refers to the point cloud data of the first target area of ​​the first stacked object acquired by the sensor. The first target point cloud data can be used to calculate the position and alignment status of the first stacked object.

[0069] Second target point cloud data: refers to the point cloud data of the second target area of ​​the second stacked object acquired by the sensor. The second target point cloud data can be used to calculate the position and alignment status of the second stacked object.

[0070] Figure 3 A flowchart of a method for confirming an alignment state provided in an embodiment of the present application, the method can be performed by Figure 1 The handling equipment in the system architecture shown is executed. Figure 3 As shown in , the method may include the following steps:

[0071] Step 301: The controller obtains an image and a point cloud of a first stacking object, and an image and a point cloud of a second stacking object.

[0072] Step 303: The controller extracts first object image data of a first object area from the image of the first stacked object, and extracts second object image data of a second object area from the image of the second stacked object.

[0073] Step 305: The controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacked object based on the first target image data; the controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacked object based on the second target image data.

[0074] Step 307: The controller determines the position and posture of the first stacking object according to the first target point cloud data, and determines the position and posture of the second stacking object according to the second target point cloud data.

[0075] Step 309: The controller determines a difference between the posture of the first stacked object and the posture of the second stacked object, and compares the difference with a threshold value to confirm an alignment state of the first stacked object and the second stacked object.

[0076] It can be seen from the above process that the present application can detect the difference between the posture of the first stacking object and the posture of the second stacking object during the stacking process, and confirm the alignment state between the first stacking object and the second stacking object based on the difference and the threshold, which can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the posture data of the first stacking object and the second stacking object, and then determine the alignment state of the two.

[0077] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the "first" and "second" and other limitations involved in the present disclosure do not have limitations in terms of size, order and quantity, and are only used to distinguish in name, for example, "first stacking object" and "second stacking object" are used to distinguish two stacking objects.

[0078] First, the above step 301, namely "the controller obtains the image and point cloud of the first stacking object and the image and point cloud of the second stacking object through the sensor" is described in detail in conjunction with the embodiment.

[0079] First, a stacking scenario involved in the embodiment of the present application is briefly introduced. Figure 2 As shown, when the handling device receives a handling task of stacking the first stacking object A onto the second stacking object B, the controller (such as Figure 1 The controller 104 shown controls the fork of the transport device to pick up the first stacking object A, and then controls the transport device to move to the vicinity of the second stacking object B, and then aligns the first target area A1 of the first stacking object A and the second target area B1 of the second stacking object B up and down by controlling the posture of the transport device, and then stacks the first stacking object A on the second stacking object B.

[0080] In the embodiment of the present application, multiple cameras and radars can be installed according to different types of sensors to simultaneously photograph and scan the first stacked object and the second stacked object.

[0081] The odometer estimates the distance traveled by the transport equipment by measuring its movement, and is usually combined with sensor data to calculate physical quantities such as the position, speed, and posture of the transport equipment.

[0082] The sensor involved in the embodiment of the present application may include a radar module and a camera module, wherein the radar module may include one or more radars, and the camera module may include one or more cameras. The sensor can be installed at a preset distance below the midpoint between the roots of the stacking execution components (such as forks) of the handling equipment, so that the field of view of the radar module and the camera module can cover the first target area of ​​the first pair of stacked objects and the second target area of ​​the second stacked object. Therefore, the sensor can obtain an image including the first stacked object and the second stacked object, and obtain a point cloud including the first stacked object and the second stacked object. Optionally, the radar is a laser radar (Lidar). The laser radar may include a three-dimensional laser radar.

[0083] The odometer can be arranged near the wheels of the transporting equipment to record the number of revolutions of the wheels and estimate the distance moved; the odometer can also be arranged at the center of the chassis of the transporting equipment.

[0084] For example, the camera, radar and odometer can be controlled by a controller built into the transport equipment ( Figure 1 The controller 104 shown in the figure is controlled, for example, based on a SoC (System-on-a-Chip), which is not specifically limited in the embodiments of the present application.

[0085] Before determining the posture adjustment information of the handling equipment, it is also necessary to synchronize the odometer and the sensor in time. The synchronization method can adopt common synchronization methods (such as hardware synchronization method, software synchronization method (such as timestamp alignment, interpolation synchronization, etc.)). The radar can use PPS (Pulse Per Second) + GPRMC (Global Positioning System Recommended Minimum Navigation Information) synchronization, and the camera can use PTP (Precision Time Protocol) or hardware trigger synchronization. In addition, the radar and camera need to be jointly calibrated in advance to obtain the internal parameters of the camera, the external parameters of the camera and radar, and the external parameters of the radar and the handling equipment.

[0086] In the embodiment of the present application, when the transport device lifts the first stacked object and the sensor simultaneously photographs and scans the first stacked object and the second stacked object, the controller obtains the image and point cloud of the first stacked object, as well as the image and point cloud of the second stacked object through the sensor. Specifically: the controller controls the transport device to transport the first stacked object to the stacking operation position, and controls the transport device to lift the first stacked object, scans the first stacked object and the second stacked object by radar to obtain the point clouds of the first stacked object and the second stacked object, and simultaneously photographs the first stacked object and the second stacked object by the camera to obtain the image of the first stacked object and the second stacked object.

[0087] Here, the stacking operation position may be a position where the sensor on the handling device can simultaneously obtain the target image and the target point cloud of the first stacking object and the second stacking object, and in this area, the fork of the handling device can stack the first stacking object and the second stacking object within the variable posture range of the fork. For example, according to the position of the second stacking object (such as the lower stacking object), a position within a preset distance range in front of the second stacking object is used as the stacking operation position.

[0088] Taking into account the distortion in the images and point clouds collected when the first stacked object and the second stacked object are photographed and scanned by the camera and the radar at the same time, the data collected by the sensor carried by the handling equipment in the embodiment of the present application may also include: acquiring the original image of the first stacked object and the original image of the second stacked object by the camera, and acquiring the original point cloud of the first stacked object and the original point cloud of the second stacked object by the radar; de-distorting the original image according to the pre-calibrated camera intrinsic parameters; converting the original point cloud from the coordinate system where the radar is located to the coordinate system where the handling equipment is located, and de-distorting the original point cloud according to the odometer information and the timestamp corresponding to the original image; and time-synchronizing the de-distorted image and the de-distorted point cloud to obtain the time-synchronized image and point cloud.

[0089] Here, the timestamp corresponding to the original image may be the timestamp corresponding to when the camera captured the original image (i.e., the original image of the first stacked object or the original image of the second stacked object). Camera intrinsic parameters, also known as camera internal parameters or intrinsic parameters, are parameters that describe the internal properties of the camera. These parameters include focal length, principal point (optical center) coordinates, and distortion coefficients (such as k1, k2, k3 for radial distortion, and p1, p2 for tangential distortion). The intrinsic parameters are usually determined during camera calibration.

[0090] Image distortion is mainly caused by the optical properties of the camera lens. When light passes through the lens, radial distortion and tangential distortion will occur due to refraction and manufacturing process limitations. Radial distortion will cause straight lines in the image to become curved, and the farther away from the center of the image, the more severe the distortion. Tangential distortion is caused by the lens and the photosensitive element not being completely parallel. Therefore, the original image is dedistorted by using the pre-calibrated camera internal parameters to obtain the dedistorted image.

[0091] The distortion in the original point cloud refers to the shape distortion problem caused by radar movement or external factors. The odometer information provides the motion information of the radar when collecting the original point cloud, which usually includes position, speed, acceleration, etc. Next, the compensation transformation matrix is ​​determined according to the timestamp of the image and the odometer information; the compensation transformation matrix is ​​applied to the original point cloud to obtain the de-distorted point cloud.

[0092] This application dedistorts the original image captured by the camera through pre-calibrated camera internal parameters; and converts the original point cloud data collected by the sensor from the coordinate system where the radar is located to the coordinate system where the handling equipment is located, and dedistorts the original point cloud according to the odometer information and the timestamp corresponding to the original image; finally, the dedistorted image and the dedistorted point cloud are time-synchronized to obtain a time-synchronized and distortion-free image and point cloud.

[0093] It should be noted that the embodiments of the present application can be applied to scenarios with multiple stacked objects.

[0094] The above step 303, namely "the controller extracts first target image data of the first target area from the image of the first stacked object, and extracts second target image data of the second target area from the image of the second stacked object" is described in detail below in conjunction with an embodiment.

[0095] The controller of the embodiment of the present application extracts first target image data of the first target area from the image of the first stacked object, and extracts second target image data of the second target area from the image of the second stacked object, which generally involves technologies such as image segmentation, target detection or object recognition. The first target area of ​​the first stacked object generally has a boundary in the image, which separates the first stacked object from other parts in the image, and the first target area describes the shape, size, position and possible texture or color features of the first stacked object. The second target area of ​​the second stacked object generally has a boundary in the image, which separates the second stacked object from other parts in the image, and the second target area describes the shape, size, position and possible texture or color features of the second stacked object.

[0096] In one example, the controller extracts first target image data of a first target area from an image of a first stacked object and extracts second target image data of a second target area from an image of a second stacked object according to an image segmentation model; wherein image segmentation is a process of dividing an image into a plurality of non-intersecting regions, each region corresponding to an object in the image. Image segmentation methods may include threshold segmentation, edge detection, region growing, clustering, deep learning methods, etc.

[0097] In another example, the controller extracts first target image data of a first target area from an image of a first stacked object, and extracts second target image data of a second target area from an image of a second stacked object according to a target detection model. Target detection is the process of identifying and locating a specific target or object in an image. The target detection method is mainly implemented based on a deep learning model.

[0098] The controller of the present application may use an image segmentation or target detection method to accurately determine a first target region of a first stacked object and a second target region of a second stacked object in an image.

[0099] It should be noted that, in the process of object detection or image segmentation, a binary mask (or mask) of the same size as the image is usually generated. Each pixel value in the mask indicates whether the pixel belongs to the first stacked object or the second stacked object (usually 1 means it belongs, and 0 means it does not belong). This mask can be used to extract the first target area of ​​the first stacked object and the second target area of ​​the second stacked object.

[0100] For example, the camera module captures an image containing a first stacked object and a second stacked object. After image segmentation, the segmentation result can be as follows: Figure 5 As shown in the figure, the first part (blue part) is at least a portion of the fifth foot cup Aj5 and at least a portion of the third foot cup Aj3 of the first stacking object, and the first part (red part) is at least a portion of the sixth column Bj6 and at least a portion of the fourth column Bj4 of the second stacking object.

[0101] Here, at least a portion of the foot cup of the first stacked object and at least a portion of the column of the second stacked object presented in the captured image can be changed by adjusting the number of cameras in the camera module and the orientation of the cameras.

[0102] Among them, the foot cup refers to the supporting component installed at the bottom of the cage, which is usually used to stabilize the cage, bear weight, and protect the cage from direct contact with the ground. In industrial, warehousing and logistics scenarios, cages (also called turnover cages or storage cages) are often equipped with foot cups to achieve better mobility, stacking and durability.

[0103] The column refers to the vertical support structure around the cage, which is usually used to bear the weight of the cage, fix the cage frame structure and provide stacking function. The column is one of the core components of the cage, and its design has a direct impact on the strength, stability and use function of the cage.

[0104] The first target area in the embodiment of the present application at least includes: the foot cup, frame line and other key structural areas of the first stacking object. The second target area at least includes: the pillar, frame line and other key structural areas of the second stacking object.

[0105] In one example, the first target area includes at least a portion of at least one foot cup, which may be the area corresponding to all or part of the foot cup. The second target area includes at least a portion of at least one column, which may be the area corresponding to all or part of the column.

[0106] For example, the first stacking object is the first material cage, and the second stacking object is the second material cage; the first target area includes at least a portion of the first foot cup of the first material cage and at least a portion of the third foot cup; the second target area includes at least a portion of the second column of the second material cage and at least a portion of the fourth column.

[0107] For example, the first target area also includes at least part of the fifth foot cup and at least part of the seventh foot cup of the first material basket; the second target area also includes at least part of the sixth column and at least part of the eighth column of the second material basket. Figure 4 , Figure 6 , Figure 7 Describe them separately.

[0108] exist Figure 4 In the embodiment, the first target area includes at least a portion of the first foot cup Aj1, at least a portion of the third foot cup Aj3, at least a portion of the fifth foot cup Aj5, and at least a portion of the seventh foot cup Aj7; the second target area includes at least a portion of the second column Bj2, at least a portion of the fourth column Bj4, at least a portion of the sixth column Bj6, and at least a portion of the eighth column Bj8.

[0109] exist Figure 6 In the embodiment, the first target area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3; the second target area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4.

[0110] exist Figure 7In the embodiment, the first target area A1 includes at least a portion of the fifth foot cup Aj5 and at least a portion of the seventh foot cup Aj7; the second target area B1 includes at least a portion of the sixth column Bj6 and at least a portion of the eighth column Bj8.

[0111] In one example, the first target area includes at least one border line of the first stacked object; and the second target image area includes at least one border line of the second stacked object.

[0112] For example, the first stacking object is a first material cage, and the second stacking object is a second material cage; the first target area includes a first border line and a third border line of the first material cage, and the first border line intersects with the third border line; the second target area includes a second border line and a fourth border line of the second material cage, and the second border line intersects with the fourth border line.

[0113] For example, the first target area also includes the fifth and seventh border lines of the first cage, and the fifth and seventh border lines intersect; the second target area also includes the sixth and eighth border lines of the second cage, and the sixth and eighth border lines intersect. Figure 8 Give a description.

[0114] exist Figure 8 In the figure, the first target area A1 includes a first border line a11, a third border line a13, a fifth border line a15 and a seventh border line a17; wherein the first border line a11 intersects with the third border line a13, the third border line a13 intersects with the fifth border line a15, the fifth border line a15 intersects with the seventh border line a17, and the seventh border line a17 intersects with the first border line a11.

[0115] The second target area B1 includes a second border line b22, a fourth border line b24, a sixth border line b26 and an eighth border line b28; wherein the second border line b22 intersects with the fourth border line b24, the fourth border line b24 intersects with the sixth border line b26, the sixth border line b26 intersects with the eighth border line b28, and the eighth border line b28 intersects with the second border line b22.

[0116] It should be noted that in Figure 8 In the embodiment of the present application, the first target area A1 is located at the bottom of the first stacking object A, and the second target area B1 is located at the top of the second stacking object B.

[0117] The above step 305, i.e., "the controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacked object based on the first target image data; the controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacked object based on the second target image data" is described in detail below in conjunction with the embodiment.

[0118] In the embodiment of the present application, the controller can pre-establish a correspondence between the point cloud and the pixels of the image, and based on the correspondence and the first target area of ​​the first stacked object and the second target area of ​​the second stacked object, extract the first target point cloud data corresponding to the first target area of ​​the first stacked object from the point cloud of the first stacked object, and extract the second target point cloud data corresponding to the second target area of ​​the second stacked object from the point cloud of the second stacked object.

[0119] In the embodiment of the present application, before the controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacked object based on the first target image data; and before the controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacked object based on the second target image data, the method further includes: calibrating the camera and the radar. After the controller calibrates the camera and the radar, the first target point cloud data corresponding to the first target area is extracted from the target point cloud of the first stacked object based on the first target image data, including: extracting the first target point cloud data corresponding to the first target area from the target point cloud of the first stacked object based on the joint calibration parameters and the first target image data; extracting the second target point cloud data corresponding to the second target area from the target point cloud of the second stacked object based on the second target image data, including: extracting the second target point cloud data corresponding to the second target area from the target point cloud of the second stacked object based on the joint calibration parameters and the second target image data.

[0120] In one example, the controller pre-calibrates a camera for collecting images and a radar for collecting point clouds to obtain joint calibration parameters.

[0121] In one example, the joint calibration parameters include at least one of the following: intrinsic parameters and extrinsic parameters of the camera, extrinsic parameters from the radar to the camera, and extrinsic parameters from the radar to the handling equipment.

[0122] The intrinsic parameters of a camera refer to the parameters that describe the internal properties of the camera. These parameters are usually determined during camera calibration and usually remain unchanged during the use of the camera. The intrinsic parameters mainly include the following aspects: focal length, optical center, distortion coefficients, and intrinsic matrix.

[0123] The camera's external parameters are the parameters that describe the camera's position and posture in the world coordinate system. The external parameters mainly include the following aspects: rotation matrix and translation vector.

[0124] Unlike camera intrinsic parameters, camera extrinsic parameters change as the camera's position in the world coordinate system or the moment of capture changes. For example, in stereo vision, if there are two cameras, when the cameras move, their relative position and orientation change, which will cause changes in extrinsic parameters.

[0125] The external parameters from radar to camera refer to determining the rotation and translation relationship between radar and camera so that their coordinate systems can be aligned.

[0126] The external parameters of the radar to the handling equipment refer to the determination of the relative position and direction relationship between the radar and the handling equipment (its body is used as the reference coordinate system, such as the three-dimensional coordinate system mentioned above). This calibration is a key step to ensure that the radar data can be accurately converted to the three-dimensional coordinate system, thereby achieving accurate environmental perception, positioning navigation and obstacle avoidance functions.

[0127] When the joint calibration parameters include the intrinsic and extrinsic parameters of the camera, the extrinsic parameters of the radar to the camera, and the extrinsic parameters of the radar to the handling equipment, the correspondence between the point cloud and the pixel can be expressed by the following formula:

[0128] P img =T img ·T camera-img ·T lidar-camera ·T 搬运设备-lidar ·P n

[0129] Among them, the coordinates of the midpoint of the point cloud in the coordinate system where the handling equipment is located are P n , the coordinates of the pixel in the image (the image format can be .img) are P img , the camera internal parameter is T img , the camera external parameter is T camera-img , the external parameter from radar to camera is T lidar-camera , the external parameter from the transport equipment to the laser radar is T 搬运设备-lidar .

[0130] The present application pre-calibrates the camera for collecting images and the radar for collecting point clouds jointly, and can establish a spatial relationship between the camera and the radar. This spatial relationship is determined by the intrinsic and extrinsic parameters of the camera, the extrinsic parameters from the radar to the camera, and the extrinsic parameters from the radar to the handling equipment. Then, according to the above spatial relationship, first target point cloud data corresponding to the first target area can be accurately extracted from the point cloud of the first stacked object based on the first target image data; and second target point cloud data corresponding to the second target area can be extracted from the point cloud of the second stacked object based on the second target image data.

[0131] The above step 307, namely "the controller determines the position and posture of the first stacking object according to the first target point cloud data, and determines the position and posture of the second stacking object according to the second target point cloud data" is described in detail below in conjunction with the embodiments.

[0132] The controller of the embodiment of the present application extracts a first border point cloud from first target point cloud data corresponding to a first target area of ​​a first stacked object; and extracts a second border point cloud from second target point cloud data corresponding to a second target area of ​​a second stacked object; fits the first border point cloud according to the least squares method to obtain a border line equation corresponding to the first stacked object, and fits the second border point cloud according to the least squares method to obtain a border line equation corresponding to the second stacked object; determines the posture of the first stacked object according to the border line equation corresponding to the first stacked object, and determines the posture of the second stacked object according to the border line equation corresponding to the second stacked object.

[0133] In one example, the controller extracts a first border point cloud from first target point cloud data corresponding to a first target area of ​​a first stacked object according to RANSAC (Random Sample Consensus) or PROSAC (Progressive Sample Consensus); and extracts a second border point cloud from second target point cloud data corresponding to a second target area of ​​a second stacked object according to RANSAC; fits the first border point cloud according to the least squares method to obtain a border line equation corresponding to the first stacked object, and fits the second border point cloud according to the least squares method to obtain a border line equation corresponding to the second stacked object; determines the pose of the first stacked object according to the border line equation corresponding to the first stacked object, and determines the pose of the second stacked object according to the border line equation corresponding to the second stacked object.

[0134] Here, extracting the first border point cloud from the first target point cloud data corresponding to the first target area of ​​the first stacking object according to RANSAC may include: (1) randomly selecting a group of points from the first target point cloud data corresponding to the first target area of ​​the first stacking object as initial samples; (2) using the initial samples to estimate the parameters of the model (i.e., the model for extracting the border point cloud), which are the vertices, side lengths, angles and other parameters of the border of the first stacking object; (3) according to the model parameters, classifying other points in the first target point cloud data corresponding to the first target area of ​​the first stacking object into internal points (points that meet the model parameters) and external points (points that do not meet the model parameters); (4) repeating the above steps of random sampling, model estimation and classification of internal points and external points, and recording the number of internal points of the current model in each iteration; (5) in all iterations, selecting the model with the largest number of internal points as the final estimation result (i.e., the border point cloud of the first stacking object).

[0135] It should be noted that the process of extracting the second border point cloud from the second target point cloud data corresponding to the second target area of ​​the second stacking object according to RANSAC or PROSAC and the process of extracting the first border point cloud from the first target point cloud data corresponding to the first target area of ​​the first stacking object according to RANSAC or PROSAC will not be repeated here.

[0136] Next, fitting the first frame point cloud according to the least squares method to obtain a frame line equation corresponding to the first stacked object, and fitting the second frame point cloud according to the least squares method to obtain a frame line equation corresponding to the second stacked object, may include: (1) selecting a suitable fitting model for the shapes of the first stacked object and the second stacked object, and a straight line fitting model may be used for the frame line equations of the first stacked object and the second stacked object; (2) respectively determining the parameters of the first fitting model of the first stacked object and the parameters of the second fitting model corresponding to the second stacked object, such as the slope and intercept parameters; (3) constructing a first objective function (i.e., an error function) according to the first fitting model and the first frame point cloud; and constructing a second objective function according to the parameters of the second fitting model and the second frame point cloud; wherein , the objective function corresponding to the first stacking object represents the deviation or distance between the corresponding fitting model and the first border point cloud, and the objective function corresponding to the second stacking object represents the deviation or distance between the corresponding fitting model and the second border point cloud; (4) using the least squares method, respectively solving the model parameters that make the first objective function and the second objective function reach the minimum value; wherein the least squares method finds the best fit by minimizing the sum of the squares of the distances from the point cloud of all first border lines to the first fitting model, or finds the best fit by minimizing the sum of the squares of the distances from the point cloud of all second border lines to the second fitting model; (5) solving the minimum value of the first objective function to obtain the border line equation corresponding to the first stacking object; and solving the minimum value of the second objective function to obtain the border line equation corresponding to the second stacking object.

[0137] For example, the distribution of each border point cloud in the embodiment of the present application directly reflects the shape of the stacked object. For example, a stacked object of a cube will generate a point cloud with six rectangular faces.

[0138] According to the shape of the first stacking object (such as a rectangle), the first frame point cloud, the third frame point cloud, the fifth frame point cloud and the seventh frame point cloud are distributed on the rectangular surface corresponding to the first stacking object; according to the shape of the second stacking object, the second frame point cloud, the fourth frame point cloud, the sixth frame point cloud and the eighth frame point cloud are distributed on the rectangular surface corresponding to the second stacking object.

[0139] The first border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the first border line equation; the third border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the third border line equation; the fifth border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the fifth border line equation; the seventh border point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the seventh border line equation.

[0140] The second frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the second frame line equation; the fourth frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the fourth frame line equation; the sixth frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the sixth frame line equation; the eighth frame point cloud distributed on the rectangular surface is fitted by the least squares method to obtain the eighth frame line equation.

[0141] The following takes the third border point cloud as an example, and uses the least squares method to fit the third border point cloud distributed on the rectangular surface to obtain the third border line equation.

[0142] For example, find the border line equation y=mx+b on the rectangular surface, where m is the slope and b is the intercept, so that the border line equation is as close as possible to the third border point cloud distributed on the rectangular surface. The goal of the least squares method is to find the values ​​of m and b so that the sum of the squares of the vertical distances (i.e., errors) from all points (i.e., all points in the third border point cloud distributed on the rectangular surface) to the border line equation is minimized; the sum of the squares of the vertical distances (i.e., errors) from all points to the border line equation is minimized to obtain the third border line equation.

[0143] It should be noted that, for other border point clouds, their edge lines can be determined in the same manner as the third border point cloud, which will not be described in detail here.

[0144] In some embodiments, for determining the posture of the first stacked object according to the border line equation corresponding to the first stacked object, and determining the posture of the second stacked object according to the border line equation corresponding to the second stacked object, the following steps can be adopted: taking the first stacked object as an example, the border line equation corresponding to the first stacked object obtained by the straight line fitting algorithm may include four border line equations, namely, a first border line equation corresponding to the first border line a11, a third border line equation corresponding to the third border line a13, a fifth border line equation corresponding to the fifth border line a15, and a seventh border line equation corresponding to the seventh border line a17.

[0145] Determine the position information of the first stacked object according to the coordinates of any intersection point between the straight lines corresponding to the four frame line equations, for example: determine the position information of the first stacked object according to the coordinates of the intersection point between the first frame line a11 corresponding to the first frame line equation and the third frame line a13 corresponding to the third frame line equation;

[0146] Alternatively, a centerline equation is obtained based on two parallel straight lines, and the position information of the first stacked object is determined based on the coordinates of the intersection of the straight line corresponding to the centerline equation and the straight lines corresponding to other border line equations. For example: the first centerline is obtained based on the first border line a11 corresponding to the first border line equation and the fifth border line a15 corresponding to the fifth border line equation, and the position information of the first stacked object is determined based on the coordinates of the intersection of the first centerline and the third border line a13 corresponding to the third border line equation.

[0147] Then, the angle information of the first stacked object is determined based on the angle of any one of the first border line a11 corresponding to the first border line equation, the third border line a13 corresponding to the third border line equation, the fifth border line a15 corresponding to the fifth border line equation, and the seventh border line a17 corresponding to the seventh border line equation; and then the posture of the first stacked object is determined based on the position information and the angle information.

[0148] Similar steps may be adopted to determine the position and posture of the second stacked object using the frame line equation corresponding to the second stacked object.

[0149] The above step 309, i.e., "the controller determines the difference between the posture of the first stacking object and the posture of the second stacking object, and compares the difference with a threshold value to confirm the alignment state of the first stacking object and the second stacking object" is described in detail below in conjunction with the embodiments.

[0150] The embodiment of the present application can confirm the alignment status of the first stacking object and the second stacking object in the following manner, including: if the difference is greater than or equal to a threshold, confirming that the alignment status is misaligned; if the difference is less than the threshold, confirming that the alignment status is aligned.

[0151] In the embodiment of the present application, when the difference is greater than or equal to a threshold value, the alignment state of the first stacked object and the second stacked object is confirmed to be misaligned, and the larger the difference between the difference and the threshold value, the lower the degree of alignment between the first stacked object and the second stacked object; when the difference is less than the threshold value, the alignment state of the first stacked object and the second stacked object is confirmed to be aligned, and the smaller the difference between the difference and the threshold value, the higher the degree of alignment between the first stacked object and the second stacked object.

[0152] Among them, the determination of the threshold can be flexibly adjusted according to different handling devices and stacking objects. For example, it can be: -3 cm < X-axis coordinate difference < 3 cm, -3 cm < Y-axis coordinate difference < 3 cm, -2° < Z-axis rotation angle difference < 2°.

[0153] In one example, when the difference is greater than or equal to the threshold in the embodiments of the present application, the method further includes: the controller controls the handling device to adjust its pose; the controller re-obtains the image and point cloud of the first stacking object, and the image and point cloud of the second stacking object through the sensor; the controller re-determines the difference between the pose of the first stacking object and the pose of the second stacking object, and the controller re-confirms the alignment state until the difference is less than the threshold.

[0154] In one example, when the difference is less than the threshold in the embodiments of the present application, the method further includes: the controller controls the handling device to place the first stacking object on the second stacking object to complete the stacking.

[0155] In some embodiments, the first target area in the embodiments of the present application is located at the bottom (i.e., the first side) of the first stacking object, and the second target area is located at the top (i.e., the second side) of the second stacking object. The first side and the second side are on the same side when the first stacking object and the second stacking object are aligned. The following will be described in detail with reference to Figure 4 、 Figure 6 、 Figure 7 and Figure 8 for a detailed description.

[0156] For example, Figure 4 , in some embodiments, the first target area is located at the bottom of the first stacking object A; the first target area includes at least partial areas of the first foot cup Aj1, the third foot cup Aj3, the fifth foot cup Aj5, and the seventh foot cup Aj7. The second target area is located at the top of the second stacking object B; the second target area includes at least partial areas of the second column Bj2, the fourth column Bj4, the sixth column Bj6, and the eighth column Bj8. When the alignment state is confirmed to be aligned, the first target area and the second target area are arranged in the vertical direction, the first foot cup Aj1 is aligned with the second column Bj2, the third foot cup Aj3 is aligned with the fourth column Bj4, the fifth foot cup Aj5 is aligned with the sixth column Bj6, and the seventh foot cup Aj7 is aligned with the eighth column Bj8.

[0157] For example, Figure 6In some embodiments, the first target area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3. The second target area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4. In the Z-axis direction, when the alignment state is confirmed to be aligned, the first foot cup Aj1 is aligned with the second column Bj2, and the third foot cup Aj3 is aligned with the fourth column Bj4.

[0158] like Figure 7 In some embodiments, the first target area A1 includes at least a portion of the fifth foot cup Aj5 and at least a portion of the seventh foot cup Aj7. The second target area B1 includes at least a portion of the sixth column Bj6 and at least a portion of the eighth column Bj8. In the Z-axis direction, when the alignment state is confirmed to be aligned, the fifth foot cup Aj5 is aligned with the sixth column Bj6, and the seventh foot cup Aj7 is aligned with the eighth column Bj8.

[0159] like Figure 8 In some embodiments, the first target area A1 is located at the bottom (i.e., the first side) of the first stacking object A; the first target area A1 includes a first frame line a11, a third frame line a13, a fifth frame line a15, and a seventh frame line a17. The second target area B1 is located at the top (i.e., the second side) of the second stacking object B; the second target area B1 includes a second frame line b22, a fourth frame line b24, a sixth frame line b26, and an eighth frame line b28. When the alignment state is confirmed to be aligned, the first target area A1 and the second target area B1 are arranged in a vertical direction, the first frame line a11 and the second frame line b22 are parallel to each other, the third frame line a13 and the fourth frame line b24 are parallel to each other, the fifth frame line a15 and the sixth frame line b26 are parallel to each other, and the seventh frame line a17 and the eighth frame line b28 are parallel to each other.

[0160] In the embodiment of the present application, controlling the transport device to adjust the posture may include controlling the transport device to adjust the posture of the first stacking object on the fork and / or controlling the posture of the chassis of the transport device (ie, adjusting the posture of the transport device relative to the second stacking object).

[0161] In one example, controlling a transport device to adjust its posture includes: determining posture adjustment information of the transport device based on an alignment state; obtaining odometer information corresponding to a target time, the target time being a time corresponding to when a sensor obtains an image and a point cloud of a first stacked object, and an image and a point cloud of a second stacked object; determining posture adjustment information corresponding to a current time according to the odometer information corresponding to the target time, the posture adjustment information, and the odometer information corresponding to the current time; and controlling the transport device to adjust its posture according to the posture adjustment information corresponding to the current time.

[0162] Here, the posture adjustment information may be information for adjusting the position and posture (ie, angle) of the transport equipment; the current moment and the target moment are different moments, and generally the target moment is a historical moment compared to the current moment.

[0163] For example, the posture adjustment information corresponding to the current moment can be determined based on the following formula:

[0164] E n ^=H n+1 -1 ·H n ·E n

[0165] Among them, H n is the odometer information corresponding to the target time, E n is the pose adjustment information, H n+1 -1 is the odometer information corresponding to the current moment, E n ^ is the posture adjustment information corresponding to the current moment.

[0166] The present application takes into account the problem that the time when data collected by the sensors and odometers carried by the handling equipment are not synchronized with the time when the data is processed by the control system. In order to ensure that the data processed by the control system is the posture adjustment information corresponding to the current moment, the posture adjustment information corresponding to the current moment is determined based on the odometer information corresponding to the target moment, the posture adjustment information corresponding to the target moment, and the odometer information corresponding to the current moment. This can ensure that the handling equipment accurately adjusts its posture according to the posture adjustment information corresponding to the current moment, thereby ensuring the alignment accuracy of the first stacking object and the second stacking object at the current moment.

[0167] The following describes an implementation of the method proposed in the embodiment of the present application in combination with an actual application scenario. Fig. 9 As shown, the execution subject is a controller in the handling equipment, and the method includes the following steps:

[0168] Step 801: The controller obtains an original image and an original point cloud of a first stacking object, and an original image and an original point cloud of a second stacking object through a sensor.

[0169] Step 802: The controller performs dedistortion processing on the original image and original point cloud of the first stacked object, and the original image and original point cloud of the second stacked object, respectively, to obtain a dedistorted point cloud and a dedistorted point cloud; and performs time synchronization on the dedistorted point cloud and the dedistorted image to obtain an image and point cloud of the first stacked object, and an image and point cloud of the second stacked object.

[0170] The original image and original point cloud of the first stacking object, and the original image and original point cloud of the second stacking object are respectively dedistorted, including: dedistorting the original image of the first stacking object and the original point cloud of the second stacking object according to the pre-calibrated camera internal parameters; converting the original point cloud of the first stacking object from the coordinate system where the radar is located to the coordinate system where the handling equipment is located, and dedistorting the original point cloud of the first stacking object according to the odometer information and the timestamp corresponding to the original image of the first stacking object; and converting the original point cloud of the second stacking object from the coordinate system where the radar is located to the coordinate system where the handling equipment is located, and dedistorting the original point cloud of the second stacking object according to the odometer information and the timestamp corresponding to the original image of the second stacking object; and time synchronizing the dedistorted image and the dedistorted point cloud to obtain the image and point cloud of the first stacking object, and the image and point cloud of the second stacking object.

[0171] Step 803: The controller extracts first object image data of a first object area from the image of the first stacked object, and extracts second object image data of a second object area from the image of the second stacked object.

[0172] Step 804: The controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacked object based on the first target image data; the controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacked object based on the second target image data.

[0173] Step 805: The controller determines the position and posture of the first stacking object according to the first target point cloud data, and determines the position and posture of the second stacking object according to the second target point cloud data.

[0174] Step 806: The controller determines a difference between the pose of the first stacked object and the pose of the second stacked object.

[0175] Step 807: The controller determines whether the difference is less than a threshold.

[0176] Step 808 : When the difference is greater than or equal to the threshold, the controller determines that the alignment relationship between the first stacking object and the second stacking object is misaligned, and executes step 809 .

[0177] Step 809: The controller controls the transport device to adjust its posture according to the difference, and continues to execute steps 801 to 807 until the difference is less than the threshold, and then executes steps 810 and 811.

[0178] Step 810: When the difference is less than a threshold, the controller confirms that the alignment state of the first stacking object and the second stacking object is aligned.

[0179] Step 811: The controller controls the transport device to place the first stacking object on the second stacking object to complete the stacking.

[0180] In the embodiment of the present application, before confirming the alignment state, the method further includes: the controller controls the transport device to transport the first stacking object to the stacking preparation position to complete a pre-alignment action relative to the second stacking object.

[0181] The stacking preparation position refers to a position before reaching the stacking operation position, at which the transport device can obtain the position and posture of the second stacking object.

[0182] Pre-alignment means: adjusting the position of the handling equipment so that the first stacking object and the second stacking object are basically aligned in the Y-axis direction. That is, the coordinate difference △Y of the first stacking object and the second stacking object in the Y-axis and the rotation angle difference △Ψ of the Z-axis are within the preset threshold, which can be flexibly adjusted according to different handling equipment and stacking objects, for example, -5cm<△Y<5cm, -3°<△Ψ<3°.

[0183] In the pre-alignment stage, the handling equipment moves with the first stacking object to the second stacking object, firstly lifts the fork to the optimal detection height, then detects the features such as the column of the second stacking object, calculates the posture of the column, and adjusts the chassis or fork to complete the pre-alignment.

[0184] The above method provided in the embodiment of the present application can be applied to a variety of application scenarios, including but not limited to: unmanned warehouse scenarios, unmanned loading and unloading scenarios.

[0185] Among them, unmanned warehouses may include unmanned forklifts, storage shelves, picking platforms, RCS control systems and warehouse management systems (Warehouse Management System, WMS), etc.

[0186] Among them, the unmanned loading and unloading scenarios include unmanned forklifts, RCS control systems, and trucks.

[0187] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0188] According to an embodiment of another aspect, a control system is provided for executing steps of implementing any one of the method embodiments described above.

[0189] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0190] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a controller, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0191] An embodiment of the present application also provides a transport device, which includes: a transport device body; a sensor mounted on the transport device body; one or more controllers; and a memory coupled to the one or more controllers, the memory being used to store program instructions, and when the program instructions are read and executed by one or more controllers, the steps of the method implementing any one of the aforementioned method embodiments are executed.

[0192] In some examples, the handling device body includes a handling device body.

[0193] In some examples, the sensors include a camera and a radar that have been jointly calibrated.

[0194] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a controller.

[0195] It can be seen from the above description of the implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can essentially be embodied in the form of a computer program product, which can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc.

[0196] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for confirming an alignment state, characterized in that: include: The controller acquires the image and point cloud of the first stacked object and the image and point cloud of the second stacked object through the sensor; The controller extracts first object image data of a first object area from the image of the first stacked object, and extracts second object image data of a second object area from the image of the second stacked object; The controller extracts first target point cloud data corresponding to the first target area from the point cloud of the first stacked object based on the first target image data; The controller extracts second target point cloud data corresponding to the second target area from the point cloud of the second stacked object based on the second target image data; The controller determines the posture of the first stacking object according to the first target point cloud data, and determines the posture of the second stacking object according to the second target point cloud data; The controller determines a difference between a posture of the first stacked object and a posture of the second stacked object, and compares the difference with a threshold value to confirm an alignment state of the first stacked object and the second stacked object.

2. The method according to claim 1, characterized in that The method further comprises: Before confirming the alignment state, the controller controls the transport device to transport the first stacking object to a stacking preparation position to complete a pre-alignment action relative to the second stacking object.

3. The method according to claim 1, characterized in that The controller extracts first target image data of a first target area from the image of the first stacked object, and extracts second target image data of a second target area from the image of the second stacked object, including: The controller extracts first object image data of a first object area from the image of the first stacked object according to an image segmentation model, and extracts second object image data of a second object area from the image of the second stacked object.

4. The method according to claim 1, characterized in that: The controller extracts first target image data of a first target area from the image of the first stacked object, and extracts second target image data of a second target area from the image of the second stacked object, including: The controller extracts first object image data of a first object area from the image of the first stacked object according to the object detection model, and extracts second object image data of a second object area from the image of the second stacked object.

5. The method according to claim 1, characterized in that The controller extracts first target point cloud data corresponding to the first target area from the target point cloud of the first stacked object based on the first target image data, including: the controller extracts first target point cloud data corresponding to the first target area from the target point cloud of the first stacked object according to a joint calibration parameter and the first target image data; The controller extracts second target point cloud data corresponding to the second target area from the target point cloud of the second stacked object based on the second target image data, including: extracting second target point cloud data corresponding to the second target area from the target point cloud of the second stacked object according to the joint calibration parameters and the second target image data.

6. The method according to claim 5, characterized in that The method further comprises: The controller pre-calibrates the camera for acquiring images and the radar for acquiring point clouds to obtain joint calibration parameters.

7. The method according to claim 6, characterized in that The joint calibration parameters include at least one of the following: The internal and external parameters of the camera, the external parameters of the radar to the camera, and the external parameters of the radar to the handling equipment.

8. The method according to claim 1, characterized in that The step of comparing the difference value with a threshold value to confirm the alignment state of the first stacking object and the second stacking object comprises: If the difference is greater than or equal to a threshold, confirming that the alignment state is misaligned; If the difference is less than a threshold, the alignment state is confirmed to be aligned.

9. The method according to claim 8, characterized in that The method further comprises: When the alignment state is misaligned, the controller controls the handling device to adjust the posture; The controller reacquires the image and point cloud of the first stacked object and the image and point cloud of the second stacked object through the sensor; The controller re-determines a difference between the posture of the first stacked object and the posture of the second stacked object; The controller reconfirms the alignment state according to the difference until the difference is less than a threshold value.

10. The method according to claim 8, characterized in that The method further comprises: When the alignment state is aligned, the controller controls the transport device to place the first stacking object on the second stacking object to complete the stacking.

11. The method according to claim 9, characterized in that Controlling the handling equipment to adjust the posture includes: controlling the handling equipment to adjust the posture of the chassis or the posture of the fork.

12. The method according to claim 9, characterized in that Control the handling equipment to adjust its position, including: The controller determines the posture adjustment information of the handling device based on the alignment state; The controller obtains odometer information corresponding to a target time, where the target time is a time corresponding to when the sensor obtains an image and a point cloud of the first stacking object and an image and a point cloud of the second stacking object; The controller determines the posture adjustment information corresponding to the current moment according to the odometer information corresponding to the target moment, the posture adjustment information and the odometer information corresponding to the current moment; The controller adjusts the posture of the transport device according to the posture adjustment information corresponding to the current moment.

13. The method according to claim 1, characterized in that The controller acquires an image and a point cloud of a first stacking object and an image and a point cloud of a second stacking object through a sensor, including: The camera acquires an image of the first stacked object and an original image of the second stacked object, and the radar acquires an original point cloud of the first stacked object and an original point cloud of the second stacked object; The controller performs a dedistortion process on the original image of the first stacked object according to the pre-calibrated camera intrinsic parameters, and performs a dedistortion process on the original image of the second stacked object according to the pre-calibrated camera intrinsic parameters; The controller converts the original point cloud of the first stacked object from the coordinate system where the radar is located to the coordinate system where the handling device is located, and performs dedistortion processing on the original point cloud of the first stacked object according to the odometer information and the timestamp corresponding to the original image of the first stacked object; and converts the original point cloud of the second stacked object from the coordinate system where the radar is located to the coordinate system where the handling device is located, and performs dedistortion processing on the original point cloud of the second stacked object according to the odometer information and the timestamp corresponding to the original image of the second stacked object; The controller performs time synchronization on the dedistorted image and the dedistorted point cloud to obtain the image and point cloud of the first stacked object and the image and point cloud of the second stacked object.

14. The method according to claim 1, characterized in that The controller determines the position and posture of the first stacking object according to the first target point cloud data, including: The controller extracts a first border point cloud from the first target point cloud data; The controller fits the first frame point cloud according to the least square method to obtain a frame line equation corresponding to the first stacked object; The controller determines the position and posture of the first stacked object according to a frame line equation corresponding to the first stacked object.

15. The method according to claim 1, characterized in that The determining the position and posture of the second stacking object according to the second target point cloud data includes: Extracting a second frame point cloud from the second target point cloud data; Fitting the second frame point cloud according to the least square method to obtain a frame line equation corresponding to the second stacked object; The position and posture of the second stacked object is determined according to a frame line equation corresponding to the second stacked object.

16. The method according to claim 1, characterized in that The first target area is located on a first side of the first stacked object, and the second target area is located on a second side of the second stacked object, and the first side and the second side are on the same side when the first stacked object and the second stacked object are aligned.

17. The method according to claim 1, characterized in that The first stacking object is a first material cage, and the second stacking object is a second material cage; The first target area includes at least a portion of the first foot cup and at least a portion of the third foot cup of the first cage; The second target area includes at least a portion of the second column and at least a portion of the fourth column of the second material cage.

18. The method according to claim 17, characterized in that The first target area also includes at least a portion of the fifth foot cup and at least a portion of the seventh foot cup of the first material cage; The second target area also includes at least a partial area of ​​the sixth column and at least a partial area of ​​the eighth column of the second material cage.

19. The method according to claim 1, characterized in that The first stacking object is a first material cage, and the second stacking object is a second material cage; The first target area includes a first border line and a third border line of the first cage, and the first border line intersects with the third border line; The second target area includes a second border line and a fourth border line of the second cage, and the second border line intersects with the fourth border line.

20. The method according to claim 19, characterized in that The first target area further includes a fifth border line and a seventh border line of the first cage, wherein the fifth border line intersects with the seventh border line; The second target area also includes a sixth border line and an eighth border line of the second cage, and the sixth border line intersects with the eighth border line.

21. A control system, characterized in that: The invention comprises a controller and a memory, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 20.

22. A handling device, characterized in that: The invention comprises a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 20.

Images