Alignment state confirmation method, control system and carrying equipment

Abstract

The embodiment of the invention discloses an alignment state confirmation method, a control system and carrying equipment. According to the main technical scheme, the method comprises the steps that a controller obtains target images of a first stacking object and a second stacking object through a sensor; the controller obtains a first target image area of the first stacking object and a second target image area of the second stacking object from the target image; the controller determines a pixel difference value between the first target image area and the second target image area, and compares the pixel difference value with a threshold value to confirm an alignment state of the first stacked object and the second stacked object. The embodiment of the invention is used for realizing alignment of the first stacking object and the second stacking object during stacking.

Description

Technical Field

[0001] The present application relates to the technical field of warehousing and logistics, and in particular to an alignment status confirmation method, a control system and a handling device. Background Art

[0002] Systems that use handling equipment such as AGV (automated guided vehicle) have the advantages of being highly unmanned, automated, and intelligent, which improves production efficiency and operational levels for industries such as warehousing, manufacturing, and logistics. As one of the more typical scenarios, handling equipment is often responsible for the handling of various goods. In the process of handling, it is inevitable to stack goods. Goods are usually packaged in cartons, etc., or stored in cages, wooden boxes, plastic boxes, etc.

[0003] Considering the space utilization, the handling equipment will involve stacking objects such as cartons, cages, wooden boxes, etc. in the process of handling goods. Considering the stability, the handling equipment needs to accurately stack one stacking object on top of another stacking object. In this process, if the two stacking objects cannot be aligned, it may affect the safety of the operation. Summary of the invention

[0004] The present application provides an alignment state confirmation method, a control system and a handling device, which are used to control the alignment of a first stacking object and a second stacking object during stacking.

[0005] This application provides the following solutions:

[0006] In a first aspect, a method for confirming an alignment status is provided, the method comprising: a controller acquiring a target image of a first stacking object and a second stacking object through a sensor; the controller acquiring a first target image area of ​​the first stacking object and a second target image area of ​​the second stacking object from the target image; the controller determining a pixel difference between the first target image area and the second target image area, and comparing the pixel difference with a threshold value to confirm the alignment status of the first stacking object and the second stacking object.

[0007] Optionally, the method further comprises:

[0008] Before confirming the alignment state, the transport device transports the first stacking object to the stacking preparation position to complete a pre-alignment action relative to the second stacking object.

[0009] Optionally, the controller acquires target images of the first stacking object and the second stacking object through a sensor, including:

[0010] The controller acquires original images of the first stacked object and the second stacked object through the sensor;

[0011] The original image is dedistorted according to the pre-calibrated sensor internal parameters to obtain the target image.

[0012] Optionally, the controller determines a pixel difference between the first target image area and the second target image area, comprising:

[0013] A pixel relationship vector between a first center point in the first target image region and a second center line in the second target image region is used as a pixel difference value.

[0014] Optionally, the pixel relationship vector is a perpendicular vector from the first center point to a second center line in the second target image area.

[0015] Optionally, the first stacking object is a first material cage, and the second stacking object is a second material cage;

[0016] The first center point is the geometric center point or center vertex of the first foot cup of the first material cage, and the second center line is the center line of the second column of the second material cage.

[0017] Optionally, the controller determines a pixel difference between the first target image area and the second target image area, comprising:

[0018] A pixel relationship vector between a first center point in the first target image region and a second center point in the second target image region is used as a pixel difference value.

[0019] Optionally, the pixel relationship vector is a line vector connecting the first center point to the second center point.

[0020] Optionally, the first stacking object is a first material cage, and the second stacking object is a second material cage;

[0021] The first center point is the geometric center point or center vertex of the first foot cup of the first material cage, and the second center point is the geometric center point or center vertex of the second column of the second material cage.

[0022] Optionally, the controller determines a pixel difference between the first target image area and the second target image area, comprising:

[0023] The pixel relationship vector between the first center line in the first target image area and the fourth center point in the second target image area is used as the pixel difference value.

[0024] Optionally, the pixel relationship vector is a perpendicular vector from the fourth center point to the first center line.

[0025] Optionally, the first stacking object is a first material cage, and the second stacking object is a second material cage;

[0026] The first center line is the center line of the first foot cup of the first material cage, and the fourth center point is the geometric center point or center vertex of the second column of the second material cage.

[0027] Optionally, comparing the pixel difference with a threshold value to confirm the alignment state of the first stacked object and the second stacked object includes:

[0028] If the pixel difference is greater than or equal to the first threshold, it is determined that the alignment state is misaligned;

[0029] If the pixel difference is less than the first threshold, the alignment state is determined to be aligned.

[0030] Optionally, the method further comprises:

[0031] When the alignment state is misaligned, the handling equipment is controlled to adjust the posture according to the pixel difference;

[0032] The controller reacquires the target image of the first stacked object and the second stacked object through the sensor;

[0033] The controller re-determines the pixel difference between the first target image area and the second target image area;

[0034] The controller reconfirms the alignment state according to the pixel difference until the pixel difference is less than a first threshold.

[0035] Optionally, controlling the handling device to adjust the posture according to the pixel difference includes:

[0036] Determine the control amount using inverse kinematics based on pixel differences;

[0037] The handling equipment is controlled to adjust its posture based on the control amount.

[0038] Optionally, comparing the pixel difference with a threshold value to confirm the alignment state of the first stacked object and the second stacked object includes:

[0039] Determine the control amount using inverse kinematics based on pixel differences;

[0040] If the control amount is greater than or equal to the second threshold, it is determined that the alignment state is misaligned;

[0041] If the control amount is less than the second threshold, it is determined that the alignment state is aligned.

[0042] Optionally, the method further comprises:

[0043] When the alignment state is misaligned, the handling equipment is controlled to adjust the posture according to the control amount;

[0044] The controller reacquires the target image of the first stacked object and the second stacked object through the sensor;

[0045] The controller re-determines the pixel difference between the first target image area and the second target image area;

[0046] The controller redetermines the control amount based on the pixel difference;

[0047] The controller reconfirms the alignment state according to the control amount until the control amount is less than a second threshold.

[0048] Optionally, the method further comprises:

[0049] When the alignment state is aligned, the handling device is controlled to place the first stacking object on the second stacking object to complete the stacking.

[0050] Optionally, controlling the transport equipment to adjust the posture includes: controlling the transport equipment to adjust the posture of the chassis or the posture of the fork.

[0051] Optionally, the first stacking object is a first material cage, and the second stacking object is a second material cage;

[0052] The first target image area includes at least a portion of the first foot cup of the first material cage and at least a portion of the third foot cup;

[0053] The second target image area includes at least a portion of the second column and at least a portion of the fourth column of the second material basket.

[0054] Optionally, the first target image area further includes at least a portion of the fifth foot cup and at least a portion of the seventh foot cup of the first material cage;

[0055] The second target image area also includes at least a partial area of ​​the sixth column and at least a partial area of ​​the eighth column of the second material basket.

[0056] In a second aspect, a control system is provided, comprising a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement any of the methods described in the first aspect.

[0057] According to a third aspect, a handling device is provided, comprising a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement any one of the methods described in the first aspect.

[0058] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0059] 1) The present application confirms the alignment state between the first stacked object and the second stacked object based on the pixel difference between the first target image area of ​​the first stacked object and the target image area of ​​the second stacked object. This scheme can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the position and posture data of the first stacked object and the second stacked object, and then determine the alignment state of the two.

[0060] 2) The present application performs dedistortion processing on the original image collected by the sensor through the pre-calibrated sensor internal parameters, and can obtain a distortion-free target image.

[0061] 3) The pixel relationship vector in the present application includes a vertical vector from the center point of the foot cup to the center line of the column; and / or a connection vector from the center point of the foot cup to the center vertex of the column. The pixel difference is determined based on these multiple methods, and the position data of the first stacking object and the second stacking object can be accurately calculated to determine the alignment status of the two.

[0062] 4) When the alignment state between the first stacked object and the second stacked object is misaligned, the controller of the present application controls the handling equipment to adjust its posture according to the pixel difference, and controls the sensor to re-acquire the target images of the first stacked object and the second stacked object; the controller then redetermines the pixel difference between the first target image area and the second target image area; the controller reconfirms the alignment state between the first stacked object and the second stacked object according to the redetermined pixel difference until the pixel difference is less than the first threshold value, thereby forming a closed-loop servo detection process. In this servo detection process, there is no need to stop the operation of the handling equipment, thereby improving the handling efficiency.

[0063] 5) When the alignment state between the first stacking object and the second stacking object is misaligned, the controller of the present application controls the transport equipment to adjust its posture according to the control amount, and controls the sensor to re-acquire the target images of the first stacking object and the second stacking object; the controller then re-determines the control amount based on the first target image area and the second target image area; the controller reconfirms the alignment state between the first stacking object and the second stacking object according to the re-determined control amount until the control amount is less than the second threshold value, thereby forming a closed-loop servo detection process. In this servo detection process, there is no need to stop the operation of the transport equipment, thereby improving the transport efficiency.

[0064] Of course, any invention of the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A system schematic diagram of a handling device applicable to an embodiment of the present application;

[0067] Figure 2 is a schematic diagram of the region of the first stacking object and the second stacking object when they are aligned;

[0068] Figure 3 A flowchart of an alignment status confirmation method provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of the layout of a camera on a handling device;

[0070] Figure 5 A schematic diagram of the layout of two cameras on the handling equipment;

[0071] Figure 6 to Figure 18 A schematic diagram for determining pixel difference;

[0072] Fig.19 A schematic diagram of controlling the position adjustment of the handling device based on pixel difference as an iteration basis;

[0073] Fig. 20 Schematic diagram of controlling the position adjustment of the handling equipment using the control quantity as the iteration basis Figure 1 ;

[0074] Fig.21 Schematic diagram of controlling the position adjustment of the handling equipment using the control quantity as the iteration basis Figure 2 .

[0075] Wherein: 100-handling device; 101-handling device body; 102-sensor; 103-stack execution unit 103; 104-controller; 105-memory; A1-first target image area; B1-second target image area; Aj1-first foot cup; Aj3-third foot cup; Aj5-fifth foot cup; Aj7-seventh foot cup; Bj2-second column; Bj4-fourth column; Bj6-sixth column; Bj8-eighth column; a1-center vertex of the first foot cup; a3-center vertex of the third foot cup; a5-center vertex of the fifth foot cup; a7-center vertex of the seventh foot cup; a1'-geometric center point of the first foot cup; a3'-geometric center point of the third foot cup; a5'-the geometric center point of the fifth cup; a7'-the geometric center point of the seventh cup; b2-the center line of the second column; b4-the center line of the fourth column; b6-the center line of the sixth column; b8-the center line of the eighth column; c1-the center line of the first cup; c3-the center line of the third cup; c5-the center line of the fifth cup; c7-the center line of the seventh cup; d2-the center vertex of the second column; d4-the center vertex of the fourth column; d6-the center vertex of the sixth column; d8-the center vertex of the eighth column; d2'-the geometric center point of the second column; d4'-the geometric center point of the fourth column; d6'-the geometric center point of the sixth column; d8'-the geometric center point of the eighth column. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0077] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0078] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, the term "according to" used in this article is not limited to being based on a certain object. For example, determining B based on A can mean: determining B based only on A, or determining B based partially on A.

[0079] In the related art, the stacking of upper and lower stacking objects is completed based on the posture of the lower stacking object relative to the handling equipment, without paying attention to the influence of various factors such as inaccurate posture of the upper stacking object, uneven ground, cumulative error of the odometer, and error of the handling equipment itself, which causes the upper and lower stacking objects to be misaligned during stacking, affecting operation safety.

[0080] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system schematic diagram of the handling equipment on which the present application is based is first described. Figure 1 A schematic diagram of a system of a handling device to which the present application embodiment can be applied is shown in FIG. Figure 1 As shown in , the transport device 100 includes a transport device body 101 , a sensor 102 , a stacking execution component 103 , a controller 104 , and a memory 105 .

[0081] The handling equipment 100 involved in the embodiment of the present application may be an unmanned forklift, a pallet truck, a crane, an AGV (Automated Guided Vehicle), an AMR (Automatic Mobile Robot), a humanoid robot, etc. The corresponding stacking execution component 103 may be a fork, a robotic arm, etc.

[0082] The controller 104 is used to control the stacking execution component 103 of the transport device 100 to execute the stacking of the first stacking object and the second stacking object. The controller 104 can be a control mainboard, a control box, a control unit, a vehicle-mounted computer, a computing platform, a tablet computer, a computer, etc. on the transport device body 101, or a system or device that performs a computing or controlling function in a local server or a cloud server, or can be a handheld controller, a remote controller, etc., etc. This is not limited in the embodiments of the present application.

[0083] This is not limited in the embodiments of the present application.

[0084] The sensor 102 may be in the form of a sensor module, a visual sensor for collecting image data, such as a camera. The camera may be installed on the stacking execution component 103 to facilitate the collection of image data.

[0085] The memory 105 is mainly used to store data collected by the sensor 102, such as image data.

[0086] First, the concepts of terms involved in the embodiments of the present application are introduced.

[0087] Stacking refers to arranging and stacking several objects up and down according to certain rules.

[0088] Stacking objects refer to the objects involved in stacking, which can be the goods themselves, or goods with simple packaging such as wrapping, or containers that can hold and carry goods, such as cages, wooden boxes, plastic boxes, pallets, etc.

[0089] Stacking process: refers to the handling device lifting the first stacking object, so that the first stacking object moves and approaches the second stacking object, and then aligns the first stacking object with the second stacking object by adjusting the posture of the handling device, and then places the first stacking object on the second stacking object to complete the stacking.

[0090] The first stacking object refers to the stacking object located at the top, and the second stacking object refers to the stacking object located at the bottom.

[0091] Alignment state: refers to the state in which two or more stacked objects are arranged in a vertical direction, and at least part of the border lines between the stacked objects are parallel or overlapped. The vertical direction refers to the Z-axis direction in the coordinate system of the handling equipment (such as Figure 2 The Z axis in the figure). Where, Figure 2 The three-dimensional coordinate system in is composed of an origin O, an X-axis, a Y-axis, and a Z-axis. The coordinate axes intersect each other perpendicularly. Optionally, the geometric center point of the transport equipment is point O, the front and rear travel direction of the transport equipment (i.e., the longitudinal direction of the transport equipment body 101) is the X-axis, wherein the positive direction of the X-axis is the direction away from the transport equipment attachment (such as a fork), the lateral direction of the transport equipment is the Y-axis, and the height direction of the transport equipment is the Z-axis.

[0092] Figure 1 "X" corresponds to the X-axis in the above three-dimensional coordinate system, "O" corresponds to the origin O in the above three-dimensional coordinate system; "Z" corresponds to the Z-axis in the above three-dimensional coordinate system, and "Y" corresponds to the Y-axis in the above three-dimensional coordinate system. The positive direction of the Y-axis is perpendicular to the paper and outward (not shown in the figure).

[0093] The first stacking object has a first target area, which refers to a specific area on the first stacking object used for detection and analysis. For example, the first target area at least includes key structural areas such as the foot cup of the first stacking object.

[0094] Optionally, the first stacking object is a first material cage, and the first target area may include at least a portion of the first foot cup and at least a portion of the third foot cup of the first material cage; optionally, the first target area also includes at least a portion of the fifth foot cup and at least a portion of the seventh foot cup of the first material cage, etc.

[0095] The foot cup of the cage refers to the supporting component installed at the bottom of the cage, which is usually used to stabilize the cage, bear weight, and protect the cage from direct contact with the ground. In industrial, warehousing and logistics scenarios, cages (also called turnover cages or storage cages) are often equipped with foot cups to achieve better mobility, stacking and durability.

[0096] The second stacking object has a second target area, which refers to a specific area on the second stacking object used for detection and analysis. For example, the second target area at least includes key structural areas such as columns of the second stacking object.

[0097] Optionally, the second stacking object is a second material cage, and the first target area may include at least a portion of the second column and at least a portion of the fourth column of the second stacking object; optionally, the second target area also includes at least a portion of the sixth column and at least a portion of the eighth column of the second stacking object, etc.

[0098] The column of the cage refers to the vertical support structure around the cage, which is usually used to bear the weight of the cage, fix the cage frame structure and provide stacking function. The column is one of the core components of the cage, and its design has a direct impact on the strength, stability and use function of the cage.

[0099] Target image: refers to image data acquired by the sensor, including at least a first target image area and a second target image area, wherein the first target image area is image data corresponding to the first target area, and the second target image area is image data corresponding to the second target area.

[0100] Pixel: It is the basic unit of an image, representing a point in the image. Each pixel contains location information.

[0101] Pixel difference: The position difference between two pixels in an image.

[0102] Controlled quantity: refers to the moving distance of the body or parts of the handling equipment, rotational angular velocity, speed, rotation angle, displacement, etc.

[0103] Figure 3 A flowchart of a method for confirming an alignment state provided in an embodiment of the present application, the method can be performed by Figure 1 The controller in the handling equipment in the system shown is executed. Figure 3 As shown in , the method may include the following steps:

[0104] Step 301: The controller acquires target images of the first stacking object and the second stacking object through a sensor;

[0105] Step 303: the controller acquires a first target image region of the first stacking object and a second target image region of the second stacking object from the target image;

[0106] Step 305: The controller determines a pixel difference between the first target image region and the second target image region, and compares the pixel difference with a threshold value to confirm an alignment state of the first stacking object and the second stacking object.

[0107] It can be seen from the above process that the present application confirms the alignment state between the first stacked object and the second stacked object based on the pixel difference between the first target image area of ​​the first stacked object and the target image area of ​​the second stacked object. This scheme can avoid the influence of the external environment (such as uneven ground) and the error of the handling equipment itself, and can accurately calculate the position and posture data of the first stacked object and the second stacked object, and then determine the alignment state of the two.

[0108] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the "first" and "second" limitations involved in the present disclosure do not have limitations in terms of size, order, and quantity, and are only used to distinguish in name, for example, "first stacking object" and "second stacking object" are used to distinguish two objects.

[0109] First, the above step 301, ie, "the controller obtains the target images of the first stacking object and the second stacking object through the sensor" is described in detail in conjunction with the embodiment.

[0110] First, a brief introduction to the stacking scenarios involved in the embodiments of the present application is given. Figure 4 and Figure 5 As shown, when the handling device receives a handling task of stacking a first stacking object onto a second stacking object, the controller (such as Figure 1 The controller 104 shown controls the fork of the transport device to pick up the first stacked object, and then controls the transport device to move to the vicinity of the second stacked object, and aligns the first stacked object and the second stacked object up and down by controlling the posture of the transport device, and then stacks the first stacked object on top of the second stacked object.

[0111] The sensor can be arranged at a preset distance below the midpoint between the fork arms of the handling device (for example, on the vehicle body or on the fork arm structure), or at other locations according to actual conditions. The odometer can be arranged near the wheels of the handling device to record the number of rotations of the wheels and estimate the distance moved; the odometer can also be arranged at the center of the chassis of the handling device.

[0112] The odometer estimates the distance moved by measuring the movement of the transport equipment, and is usually combined with sensor data to calculate physical quantities such as the position, speed, and posture of the transport equipment.

[0113] For example, the above-mentioned sensors and odometers can be controlled by a controller built into the transport equipment (such as Figure 1 The controller 104 in the embodiment is controlled, for example, based on a SoC (System-on-a-Chip), which is not specifically limited in the embodiments of the present application.

[0114] Before determining the posture adjustment information of the handling equipment, it is also necessary to synchronize the odometer with the sensor in terms of time. For example, the sensor can use PTP (Precision Time Protocol) or hardware trigger synchronization.

[0115] exist Figure 4 In the embodiment, the first stacking object is placed on the fork of the handling device, and the number of the second stacking object is at least one. When the number of the second stacking objects is multiple, the first stacking object is stacked on the second stacking object located on the top layer among the multiple second stacking objects through the handling device.

[0116] In the embodiment of the present application, when the transport device transports the first stacking object to the stacking operation position, the transport device lifts the first stacking object beyond the stacking height, and the sensor is able to capture the target area of ​​the first stacking object and the second stacking object, the sensor captures the target images of the first stacking object and the second stacking object.

[0117] Here, the stacking operation position may be a position where the sensor on the handling device can simultaneously obtain the target area images of the first stacking object and the second stacking object, and at this position, the fork of the handling device can stack the first stacking object and the second stacking object within the variable posture range of the fork. For example, according to the position of the second stacking object, a position within a preset distance range in front of the second stacking object is used as the stacking operation position.

[0118] The sensor involved in the embodiment of the present application may include a camera module, wherein the camera module may include one or more cameras. The following is an example of a camera. The camera may be installed at a preset distance below the midpoint of the root of the stacking execution component (e.g., fork) of the handling equipment, so that the field of view of the camera can cover the target area of ​​the first stacking object and the second stacking object respectively.

[0119] The layout of the cameras in the embodiment of the present application is determined based on the size of the stacked object and the field of view parameters of the camera. The layout is used to characterize the number and / or orientation of the cameras so that the field of view of each camera can cover the target area of ​​the first stacked object and the second stacked object.

[0120] The embodiment of the present application adjusts the layout of the cameras so that the field of view of each camera can cover at least one foot cup of the first stacked object and at least one column of the second stacked object.

[0121] The embodiment of the present application can use a fisheye camera or an RGB camera to collect the target image. Compared with the traditional implementation method, the laser sensor or TOF (Time of Flight) is used to perform pose calculation after collecting the point cloud or depth map. Since the laser sensor and TOF camera are expensive, and the point cloud of the laser sensor is sparse and has no color information, the FOV (Field of View) of the TOF camera is small, the fisheye camera and RGB camera have a wider field of view and lower cost.

[0122] If a camera is provided, the camera can be provided at the root of the fork arm.

[0123] Taking a fisheye camera as an example, in one example, the controller obtains target images of the first stacked object and the second stacked object through a sensor, including:

[0124] The fisheye camera photographs the first stacked object and the second stacked object to obtain target images of the first stacked object and the second stacked object.

[0125] exist Figure 4 In the figure, the field of view of the fisheye camera is shown by two dotted lines, and the fisheye camera can simultaneously capture all the foot cups of the first stacked object and all the pillars of the second stacked object.

[0126] If two cameras are set, the two cameras can be respectively set at the two ends of the root of the fork arm (such as the first end and the second end), and the lenses of the two cameras are set opposite to each other. The camera at the first end faces the direction of the camera at the second end, and the camera at the second end faces the direction of the camera at the first end.

[0127] Taking an RGB camera as an example, in one example, the controller obtains target images of the first stacked object and the second stacked object through the sensor, including:

[0128] A first RGB camera in a first orientation is used to photograph the first stacked object and the second stacked object to obtain a first image, and a second RGB camera in a second orientation is controlled to photograph the first stacked object and the second stacked object to obtain a second image; wherein the first orientation and the second orientation are two different orientations, and the target image includes the first image and the second image.

[0129] like Figure 5 ,refer to Figure 2The layout of the foot cups and columns in the figure, the field of view of the first camera 510 (blue camera) is shown by the blue dotted lines 511 and 512, and the camera can simultaneously photograph the fifth foot cup and the third foot cup of the first stacked object, and the sixth column and the fourth column of the second stacked object, the fifth foot cup and the third foot cup are located on the first side of the first stacked object, and the sixth column and the fourth column are located on the first side of the second stacked object; the field of view of the second camera 520 (red camera) is shown by the red dotted lines 521 and 522, and the camera can simultaneously photograph the first foot cup and the seventh foot cup of the first stacked object, and the second column and the eighth column of the second stacked object, the first foot cup and the seventh foot cup are located on the second side of the first stacked object, and the second column and the eighth column are located on the second side of the second stacked object.

[0130] Taking into account the presence of distortion in the image captured when the first stacked object and the second stacked object are simultaneously photographed by the camera, in the embodiment of the present application, the sensor acquiring the target images of the first stacked object and the second stacked object may also include: the controller acquiring the original images of the first stacked object and the second stacked object by acquiring the sensor; and de-distorting the original image according to the pre-calibrated camera intrinsic parameters to obtain the target image.

[0131] Image distortion is mainly caused by the optical properties of the camera lens. When light passes through the lens, radial distortion and tangential distortion will occur due to refraction and manufacturing process limitations. Radial distortion will cause straight lines in the image to become curved, and the farther away from the center of the image, the more severe the distortion. Tangential distortion is caused by the fact that the lens and the photosensitive element are not completely parallel. Therefore, the original image is dedistorted by pre-calibrated camera internal parameters to obtain the dedistorted image (i.e., the target image).

[0132] It should be noted that the embodiments of the present application can be applied to scenarios with multiple stacked objects.

[0133] The above step 303, ie, "the controller obtains the first target image region of the first stacking object and the second target image region of the second stacking object from the target image" is described in detail below in conjunction with the embodiments.

[0134] The controller of the embodiment of the present application can identify and locate the first target image area of ​​the first stacking object in the target image through the target detection algorithm, and identify and locate the second target image area of ​​the second stacking object in the target image through the target detection algorithm.

[0135] The first stacking object in the embodiment of the present application is a first material cage, and the first target image area includes at least a partial area of ​​at least one foot cup, for example, it may be at least a partial area of ​​the first foot cup, at least a partial area of ​​the third foot cup, at least a partial area of ​​the fifth foot cup, and at least a partial area of ​​the seventh foot cup; the second target image area includes at least a partial area of ​​at least one column, for example, it may be at least a partial area of ​​the second column, at least a partial area of ​​the fourth column, at least a partial area of ​​the sixth column, and at least a partial area of ​​the eighth column.

[0136] The first target image area and the second target image area are described below by taking the example that the first material basket includes the first foot cup, the third foot cup, the fifth foot cup and the seventh foot cup, and the second material basket includes the second column, the fourth column, the sixth column and the eighth column.

[0137] In one example, the first target image area includes at least a portion of a first foot cup of a first cage; the second stacking object is a second cage; and the second target image area includes at least a portion of a second column of the second cage.

[0138] exist Figure 6 In the embodiment, the first target image area A1 includes at least a portion of the first foot cup Aj1; the second target image area B1 includes at least a portion of the second pillar Bj2.

[0139] In one example, the first target image area includes at least a portion of the first foot cup of the first material basket and at least a portion of the third foot cup; the second target image area includes at least a portion of the second column of the second material basket and at least a portion of the fourth column.

[0140] exist Figure 7 , Fig. 9 , Fig.11 , Fig.12 , Fig.14 , Fig.15 , Fig.16 , Fig.17 In the figure, the first target image area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3; the second target image area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4.

[0141] In one example, the first target image area includes at least a portion of the first foot cup of the first material basket, at least a portion of the third foot cup, and at least a portion of the fifth foot cup of the first material basket;

[0142] The second target image area includes at least a portion of the second column of the second material basket, at least a portion of the fourth column, and at least a portion of the sixth column of the second material basket.

[0143] In one example, the first target image area includes at least a portion of the first foot cup of the first material basket, at least a portion of the third foot cup, at least a portion of the fifth foot cup of the first material basket, and at least a portion of the seventh foot cup;

[0144] The second target image area includes at least a portion of the second column of the second material basket, at least a portion of the fourth column, at least a portion of the sixth column of the second material basket, and at least a portion of the eighth column.

[0145] exist Figure 8 , Fig.10 , Fig.13 , Fig.18 In the figure, the first target image area A1 includes at least a portion of the first foot cup Aj1, at least a portion of the third foot cup Aj3, at least a portion of the fifth foot cup Aj5, and at least a portion of the seventh foot cup Aj7; the second target image area B1 includes at least a portion of the second column Bj2, at least a portion of the fourth column Bj4, at least a portion of the sixth column Bj6, and at least a portion of the eighth column Bj8.

[0146] Among them, at least a part of the area may refer to at least including part or all of the area. Taking the first foot cup as an example, the first target image area at least includes: part or all of the first foot cup. Taking the second column as an example, the second target image area at least includes: part or all of the second column.

[0147] The above step 305, i.e. "the controller determines the pixel difference between the first target image area and the second target image area, and compares the pixel difference with a threshold value to confirm the alignment state of the first stacking object and the second stacking object" is described in detail below in conjunction with the embodiments.

[0148] In a first manner, the controller uses a first pixel relationship vector between a first center point in a first target image region and a second center line in a second target image region as a first pixel difference value.

[0149] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacking object is aligned with the second column of the second stacking object, and further confirms that the first stacking object is aligned with the second stacking object.

[0150] The first pixel relationship vector is a vertical vector from the first center point to the second center line in the second target image area. Here, the vertical vector is generated by drawing a vertical line in the direction of the second center line in the second target image area according to the coordinate point of the first center point.

[0151] In one example, the first stacking object is a first material cage; the first center point in the first target image area is the center vertex of a first foot cup of the first material cage; the second stacking object is a second material cage; and the second center line in the second target image area is the center line of a second column of the second material cage.

[0152] The central vertex of the first foot cup may refer to: the central point of the surface of the first foot cup (ie, the surface facing the top area of ​​the second cage).

[0153] The center line of the second pillar may refer to: a virtual line passing through the geometric center point of the columnar structure (ie, the second pillar), and the virtual line may represent the symmetry axis of the second pillar.

[0154] like Figure 6 , taking the center vertex of the first foot cup and the center line of the second column as examples for explanation.

[0155] exist Figure 6 In the figure, the first target image area A1 includes at least part of the first foot cup Aj1, and the first center point in the first target image area A1 includes the central vertex a1 of the first foot cup Aj1; the second target image area B1 includes at least part of the second column Bj2, and the second center line in the second target image area B1 includes the center line b2 of the second column Bj2. Draw a vertical line (i.e., Figure 6 The direction of the vertical vector is represented by the red arrow pointing from a1 to b2.

[0156] In one example, the first center point in the first target image area may also be the geometric center point of the first foot cup of the first material cage.

[0157] The geometric center point of the first cup may refer to: the intersection point of all hyperplanes that divide the first cup into two geometrically equal parts.

[0158] The second method is based on a first pixel relationship vector between a first center point in the first target image area and a second center line in the second target image area, and a second pixel relationship vector between a third center point in the first target image area and a fourth center line in the second target image area, the controller uses the first pixel relationship vector as the first pixel difference, and the controller uses the second pixel relationship vector as the second pixel difference.

[0159] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacking object is aligned with the second column of the second stacking object; when the second pixel difference is less than the threshold, the controller confirms that the third foot cup of the first stacking object is aligned with the fourth column of the second stacking object, and then confirms that the first stacking object is aligned with the second stacking object.

[0160] The second pixel relationship vector is generated in a similar manner to the first pixel relationship vector.

[0161] like Figure 7 , the first target image area A1 includes at least part of the first foot cup Aj1 and at least part of the third foot cup Aj3, and the first target image area A1 includes the central vertex a1 of the first foot cup Aj1 and the central vertex a3 of the third foot cup Aj3; the second target image area B1 includes at least part of the second column Bj2 and at least part of the fourth column Bj4, and the second target image area B1 includes the center line b2 of the second column Bj2 and the center line b4 of the fourth column Bj4. According to the coordinates (x1, y1, z1) of the central vertex a1 of the first foot cup Aj1, a vertical line is drawn to the center line b2 of the second column Bj2 (i.e., Figure 7 The direction of the vertical vector is represented by the red arrow pointing from a1 to b2; and a vertical line is drawn from the coordinates (x3, y3, z3) of the central vertex a3 of the third foot cup Aj3 to the center line b4 of the fourth column Bj4 (i.e., Figure 7 The direction of the vertical vector is represented by the red arrow pointing from a3 to b4.

[0162] In one example, the first center point in the first target image area may be the geometric center point of the first foot cup of the first material basket, and the third center point may be the geometric center point of the third foot cup of the first material basket.

[0163] like Fig. 9 The first target image area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3, and the first target image area A1 includes the geometric center point a1' of the first foot cup Aj1 and the geometric center point a3' of the third foot cup Aj3; the second target image area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4, and the second target image area B1 includes the center line b2 of the second column Bj2 and the center line b4 of the fourth column Bj4.

[0164] According to the coordinates (x1', y1', z1') of the geometric center point a1' of the first foot cup Aj1, a vertical line is drawn to the center line b2 of the second column Bj2 (ie, Fig. 9The direction of the vertical vector is represented by the red arrow pointing from a1' to b2; and a vertical line is drawn from the coordinates (x3', y3', z3') of the geometric center point a3' of the third foot cup Aj3 to the center line b4 of the fourth column Bj4 (i.e., Fig. 9 The direction of the vertical vector is represented by the red arrow pointing from a3' to b4.

[0165] The third method is based on the first pixel relationship vector between the first center point in the first target image area and the second center line in the second target image area, the second pixel relationship vector between the third center point in the first target image area and the fourth center line in the second target image area, the third pixel relationship vector between the fifth center point in the first target image area and the sixth center line in the second target image area, and the fourth pixel relationship vector between the seventh center point in the first target image area and the eighth center line in the second target image area, the controller uses the first pixel relationship vector as the first pixel difference, the second pixel relationship vector as the second pixel difference, the third pixel relationship vector as the third pixel difference, and the fourth pixel relationship vector as the fourth pixel difference.

[0166] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacked object is aligned with the second column of the second stacked object. When the second pixel difference is less than the threshold, the controller confirms that the third foot cup of the first stacked object is aligned with the fourth column of the second stacked object. When the third pixel difference is less than the threshold, the controller confirms that the fifth foot cup of the first stacked object is aligned with the sixth column of the second stacked object. When the fourth pixel difference is less than the threshold, the controller confirms that the seventh foot cup of the first stacked object is aligned with the eighth column of the second stacked object. Then the first stacked object is confirmed to be aligned with the second stacked object.

[0167] The generation method of the second, third and fourth pixel relationship vectors is similar to the generation method of the first pixel relationship vector.

[0168] like Figure 8The first target image area A1 includes at least a portion of the first foot cup Aj1, at least a portion of the third foot cup Aj3, at least a portion of the fifth foot cup Aj5, and at least a portion of the seventh foot cup Aj7. The first center point in the first target image area A1 includes the center vertex a1 of the first foot cup Aj1, the center vertex a3 of the third foot cup Aj3, the center vertex a5 of the fifth foot cup Aj5, and the center vertex a7 of the seventh foot cup Aj7. The second target image area B1 includes at least a portion of the second column Bj2, at least a portion of the fourth column Bj4, at least a portion of the sixth column Bj6, and at least a portion of the eighth column Bj8. The second center line in the second target image area B1 includes the center line b2 of the second column Bj2, the center line b4 of the fourth column Bj4, the center line b6 of the sixth column Bj6, and the center line b8 of the eighth column Bj8.

[0169] According to the coordinates (x1, y1, z1) of the center vertex a1 of the first foot cup Aj1, a vertical line is drawn to the center line b2 of the second column Bj2 (i.e., Figure 8 The direction of the vertical vector is represented by the red arrow pointing from a1 to b2; and a vertical line is drawn from the coordinates (x3, y3, z3) of the central vertex a3 of the third foot cup Aj3 to the center line b4 of the fourth column Bj4 (i.e., Figure 8 The direction of the vertical vector is represented by the red arrow pointing from a3 to b4; and a vertical line is drawn from the coordinates (x5, y5, z5) of the central vertex a5 of the fifth foot cup Aj5 to the center line b6 of the sixth column Bj6 (i.e., Figure 8 The direction of the vertical vector is represented by the red arrow from a5 to b6; and a vertical line is drawn from the coordinates (x7, y7, z7) of the central vertex a7 of the seventh foot cup Aj7 to the center line b8 of the eighth column Bj8 (i.e., Figure 8 The direction of the vertical vector is represented by the red arrow from a7 to b8.

[0170] Among them, the coordinates of the center vertex a1 of the first cup Aj1, the coordinates of the center vertex a3 of the third cup Aj3, the coordinates of the center vertex a5 of the fifth cup Aj5, and the coordinates of the center vertex a7 of the seventh cup Aj7 are all Figure 2 Coordinates in the three-dimensional coordinate system.

[0171] In one example, the first center point in the first target image area can be the geometric center point of the first foot cup of the first cage, the third center point can be the geometric center point of the third foot cup of the first cage, the fifth center point can be the geometric center point of the fifth foot cup of the first cage, and the seventh center point can be the geometric center point of the seventh foot cup of the first cage.

[0172] like Fig.10 The first target image area A1 includes at least a portion of the first foot cup Aj1, at least a portion of the third foot cup Aj3, at least a portion of the fifth foot cup Aj5 and at least a portion of the seventh foot cup Aj7, and the first target image area A1 includes a geometric center point a1' of the first foot cup Aj1, a geometric center point a3' of the third foot cup Aj3, a geometric center point a5' of the fifth foot cup Aj5 and a geometric center point a7' of the seventh foot cup Aj7; the second target image area B1 includes at least a portion of the second column Bj2, at least a portion of the fourth column Bj4, at least a portion of the sixth column Bj6 and at least a portion of the eighth column Bj8, and the second target image area B1 includes a center line b2 of the second column Bj2, a center line b4 of the fourth column Bj4, a center line b6 of the sixth column Bj6 and a center line b8 of the eighth column Bj8.

[0173] According to the coordinates (x1', y1', z1') of the geometric center point a1' of the first foot cup Aj1, a vertical line is drawn to the center line b2 of the second column Bj2 (ie, Fig.10 The direction of the vertical vector is represented by the red arrow pointing from a1' to b2; and a vertical line is drawn from the coordinates (x3', y3', z3') of the geometric center point a3' of the third foot cup Aj3 to the center line b4 of the fourth column Bj4 (i.e., Fig.10 The direction of the vertical vector is represented by the red arrow from a3' to b4; and a vertical line is drawn from the coordinates (x5', y5', z5') of the geometric center point a5' of the fifth foot cup Aj5 to the center line b6 of the sixth column Bj6 (i.e., Fig.10 The direction of the vertical vector is represented by the red arrow from a5' to b6; and a vertical line is drawn from the coordinates (x7', y7', z7') of the geometric center point a7' of the seventh foot cup Aj7 to the center line b8 of the eighth column Bj8 (i.e., Fig.10 A vertical vector is generated by the direction of the red line between a7' and b8 in the figure. The direction of the vertical vector is indicated by the red arrow pointing from a7' to b8.

[0174] Among them, the coordinates of the geometric center point a1' of the first cup Aj1, the coordinates of the geometric center point a3' of the third cup Aj3, the coordinates of the geometric center point a5' of the fifth cup Aj5, and the coordinates of the geometric center point a7' of the seventh cup Aj7 are all Figure 2 Coordinates in the three-dimensional coordinate system.

[0175] In a fourth manner, the controller of the embodiment of the present application uses a first pixel relationship vector between a first center point in the first target image area and a second center point in the second target image area as a first pixel difference.

[0176] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacking object is aligned with the second column of the second stacking object, and further confirms that the first stacking object is aligned with the second stacking object.

[0177] The first pixel relationship vector is a line vector connecting the first center point to the second center point.

[0178] Here, a connection vector is generated according to the coordinate point of the first center point and the coordinate point of the second center point and based on the direction from the first center point to the second center point.

[0179] In one example, the first stacking object is a first material cage; the first center point in the first target image area is the center vertex of a first foot cup of the first material cage; the second stacking object is a second material cage; and the second center point in the second target image area is the center vertex of a second column of the second material cage.

[0180] The central vertex of the second column may refer to the central point of the surface of the second column (ie, the surface facing the bottom area of ​​the first cage).

[0181] The central vertex of the first foot cup may refer to: the central point of the surface of the first foot cup (ie, the surface facing the top area of ​​the second cage).

[0182] In one example, when the first center point in the first target image area is the center vertex of the first foot cup of the first basket, the second center point in the second target image area may be the geometric center point of the second column of the second basket.

[0183] In one example, the first center point in the first target area may be the geometric center point of the first foot cup of the first cage, and the second center point in the second target image area may be the geometric center point of the second column of the second cage.

[0184] In one example, the first center point in the first target area may be the geometric center point of the first foot cup of the first cage, and the second center point in the second target image area may be the central vertex of the second column of the second cage.

[0185] In a fifth method, based on a first pixel relationship vector between a first center point in the first target image area and a second center point in the second target image area, and a second pixel relationship vector between a third center point in the first target image area and a fourth center point in the second target image area, the controller uses the first pixel relationship vector as a first pixel difference and the controller uses the second pixel relationship vector as a second pixel difference.

[0186] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacking object is aligned with the second column of the second stacking object; when the second pixel difference is less than the threshold, the controller confirms that the third foot cup of the first stacking object is aligned with the fourth column of the second stacking object, and then confirms that the first stacking object is aligned with the second stacking object.

[0187] The second pixel relationship vector is generated in a similar manner to the first pixel relationship vector.

[0188] like Fig.11 The first target image area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3, and the first target image area A1 includes the central vertex a1 of the first foot cup Aj1 and the central vertex a3 of the third foot cup Aj3; the second target image area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4, and the second target image area B1 includes the central vertex a2 of the second column Bj2 and the central vertex d4 of the fourth column Bj4.

[0189] According to the coordinates (x1, y1, z1) of the center vertex a1 of the first foot cup Aj1 and the coordinates (x2, y2, z2) of the center vertex d2 of the second column Bj2, the connection vector (i.e. Fig.11 The direction of the connection vector is indicated by the red arrow pointing from a1 to d2.

[0190] According to the coordinates (x3, y3, z3) of the central vertex a3 of the third foot cup Aj3 and the coordinates (x4, y4, z4) of the central vertex d4 of the fourth pillar Bj4, based on the direction from the central vertex a3 of the third foot cup Aj3 to the central vertex d4 of the fourth pillar Bj4, a connection vector (i.e. Fig.11 The direction of the connection vector is indicated by the red arrow from a3 to d4.

[0191] Among them, the coordinates of the central vertex a1 of the first foot cup Aj1, the coordinates of the central vertex a3 of the third foot cup Aj3, the coordinates of the central vertex d2 of the second column Bj2, and the coordinates of the central vertex d4 of the fourth column Bj4 are all Figure 2 Coordinates in the three-dimensional coordinate system.

[0192] In one example, when the first center point in the first target image area is the center vertex of the first foot cup of the first cage and the third center point is the center vertex of the third foot cup of the first cage, the second center point in the second target image area can be the geometric center point of the second column of the second cage and the fourth center point can be the geometric center point of the fourth column of the second cage.

[0193] The geometric center point of the second column may be the intersection point of the diagonal lines of the columnar structure in the second target image area, and the geometric center point of the fourth column may be the intersection point of the diagonal lines of the columnar structure in the second target image area.

[0194] like Fig.12 The first target image area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3, and the first center point in the first target image area A1 includes the center vertex a1 of the first foot cup Aj1 and the center vertex a3 of the third foot cup Aj3; the second target image area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4, and the second center point in the second target image area B1 includes the geometric center point d2' of the second column Bj2 and the geometric center point d4' of the fourth column Bj4.

[0195] According to the coordinates (x1, y1, z1) of the center vertex a1 of the first foot cup Aj1 and the coordinates (x2', y2', z2') of the geometric center point d2' of the second column Bj2, the connection vector (i.e. Fig.12 The direction of the connection vector is indicated by the red arrow pointing from a1 to d2'.

[0196] According to the coordinates (x3, y3, z3) of the central vertex a3 of the third foot cup Aj3 and the coordinates (x4', y4', z4') of the geometric center point d4' of the fourth pillar Bj4, the connection vector (i.e. Fig.12 The direction of the connection vector is indicated by the red arrow from a3 to d4'.

[0197] In one example, when the first center point in the first target image area is the geometric center point of the first foot cup of the first cage and the third center point is the geometric center point of the third foot cup of the first cage, the second center point in the second target image area can be the geometric center point of the second column of the second cage and the fourth center point can be the geometric center point of the fourth column of the second cage.

[0198] like Fig.14 The first target image area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3, and the first target image area A1 includes the geometric center point a1' of the first foot cup Aj1 and the geometric center point a3' of the third foot cup Aj3; the second target image area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4, and the second target image area B1 includes the geometric center point d2' of the second column Bj2 and the geometric center point d4' of the fourth column Bj4.

[0199] According to the coordinates (x1', y1', z1') of the geometric center point a1' of the first foot cup Aj1 and the coordinates (x2', y2', z2') of the geometric center point d2' of the second column Bj2, the connection vector (i.e. Fig.14 The direction of the connection vector is indicated by the red arrow pointing from a1' to d2'.

[0200] According to the coordinates (x3', y3', z3') of the geometric center point a3' of the third foot cup Aj3 and the coordinates (x4', y4', z4') of the geometric center point d4' of the fourth pillar Bj4, the connection vector (i.e. Fig.14 The direction of the connection vector is indicated by the red arrow pointing from a3' to d4'.

[0201] The coordinates of the geometric center point d2' of the second column Bj2 and the coordinates of the geometric center point d4' of the fourth column Bj4 are Figure 2 Coordinates in the three-dimensional coordinate system.

[0202] In one example, when the first center point in the first target image area is the geometric center point of the first foot cup of the first cage and the third center point is the geometric center point of the third foot cup of the first cage, the second center point in the second target image area can be the center vertex of the second column of the second cage and the fourth center point can be the center vertex of the fourth column of the second cage.

[0203] like Fig.15 The first target image area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3, and the first target image area A1 includes the geometric center point a1' of the first foot cup Aj1 and the geometric center point a3' of the third foot cup Aj3; the second target image area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4, and the second target image area B1 includes the central vertex d2 of the second column Bj2 and the central vertex d4 of the fourth column Bj4.

[0204] According to the coordinates (x1', y1', z1') of the geometric center point a1' of the first foot cup Aj1 and the coordinates (x2, y2, z2) of the central vertex d2 of the second column Bj2, the connection vector (i.e. Fig.15 The direction of the connection vector is indicated by the red arrow pointing from a1' to d2.

[0205] According to the coordinates (x3', y3', z3') of the geometric center point a3' of the third foot cup Aj3 and the coordinates (x4, y4, z4) of the central vertex d4 of the fourth column Bj4, the connection vector (i.e. Fig.15 The direction of the connection vector is indicated by the red arrow from a3' to d4.

[0206] The coordinates of the center vertex d2 of the second column Bj2 and the coordinates of the center vertex d4 of the fourth column Bj4 are Figure 2 Coordinates in the three-dimensional coordinate system.

[0207] A sixth method is to use a first pixel relationship vector between a first center point in a first target image area and a second center point in a second target image area, a second pixel relationship vector between a third center point in the first target image area and a fourth center point in the second target image area, a third pixel relationship vector between a fifth center point in the first target image area and a sixth center point in the second target image area, and a fourth pixel relationship vector between a seventh center point in the first target image area and an eighth center point in the second target image area, the controller uses the first pixel relationship vector as a first pixel difference, the second pixel relationship vector as a second pixel difference, the third pixel relationship vector as a third pixel difference, and the fourth pixel relationship vector as a fourth pixel difference.

[0208] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacked object is aligned with the second column of the second stacked object. When the second pixel difference is less than the threshold, the controller confirms that the third foot cup of the first stacked object is aligned with the fourth column of the second stacked object. When the third pixel difference is less than the threshold, the controller confirms that the fifth foot cup of the first stacked object is aligned with the sixth column of the second stacked object. When the fourth pixel difference is less than the threshold, the controller confirms that the seventh foot cup of the first stacked object is aligned with the eighth column of the second stacked object. Then the first stacked object is confirmed to be aligned with the second stacked object.

[0209] The second, third and fourth pixel relationship vectors are generated in a similar manner to the first pixel relationship vector.

[0210] like Fig.13 , the first target image area A1 includes at least a portion of the first foot cup Aj1, at least a portion of the third foot cup Aj3, at least a portion of the fifth foot cup Aj5 and at least a portion of the seventh foot cup Aj7, and the first target image area A1 includes the central vertex a1 of the first foot cup Aj1, the central vertex a3 of the third foot cup Aj3, the central vertex a5 of the fifth foot cup Aj5 and the central vertex a7 of the seventh foot cup Aj7; the second target image area B1 includes at least a portion of the second column Bj2, at least a portion of the fourth column Bj4, at least a portion of the sixth column Bj6 and at least a portion of the eighth column Bj8, and the second target image area B1 includes the geometric center point d2' of the second column Bj2, the geometric center point d4' of the fourth column Bj4, the geometric center point d6' of the sixth column Bj6 and the geometric center point d8' of the eighth column Bj8.

[0211] According to the coordinates (x1, y1, z1) of the center vertex a1 of the first foot cup Aj1 and the coordinates (x2', y2', z2') of the geometric center point d2' of the second column Bj2, the connection vector (i.e. Fig.13 The direction of the connection vector is indicated by the red arrow pointing from a1 to d2'.

[0212] According to the coordinates (x3, y3, z3) of the central vertex a3 of the third foot cup Aj3 and the coordinates (x4', y4', z4') of the geometric center point d4' of the fourth pillar Bj4, the connection vector (i.e. Fig.13 The direction of the connection vector is indicated by the red arrow from a3 to d4'.

[0213] According to the coordinates (x5, y5, z5) of the central vertex a5 of the fifth foot cup Aj5 and the coordinates (x6', y6', z6') of the geometric center point d6' of the sixth column Bj6, the connection vector (i.e. Fig.13 The direction of the connection vector is indicated by the red arrow from a5 to d6'.

[0214] According to the coordinates (x7, y7, z7) of the central vertex a7 of the seventh cup Aj7 and the coordinates (x8', y8', z8') of the geometric center point d8' of the eighth column Bj8, the connection vector (i.e. Fig.13 The direction of the connection vector is indicated by the red arrow from a7 to d8'.

[0215] The coordinates of the geometric center point d2' of the second column Bj2 and the coordinates of the geometric center point d4' of the fourth column Bj4 are Figure 2 Coordinates in the three-dimensional coordinate system.

[0216] In an example, when the first center point in the first target image area is the center vertex of the first foot cup of the first cage, the third center point is the center vertex of the third foot cup of the first cage, the fifth center point is the center vertex of the fifth foot cup of the first cage, and the seventh center point is the center vertex of the seventh foot cup of the first cage, the second center point in the second target image area can be the center vertex of the second column of the second cage, the fourth center point can be the center vertex of the fourth column of the second cage, the sixth center point can be the center vertex of the sixth column of the second cage, and the eighth center point can be the center vertex of the eighth column of the second cage.

[0217] In an example, when the first center point in the first target image area is the geometric center point of the first foot cup of the first cage, the third center point is the geometric center point of the third foot cup of the first cage, the fifth center point is the geometric center point of the fifth foot cup of the first cage, and the seventh center point is the geometric center point of the seventh foot cup of the first cage, the second center point in the second target image area may be the geometric center point of the second column of the second cage, the fourth center point may be the geometric center point of the fourth column of the second cage, the sixth center point may be the geometric center point of the sixth column of the second cage, and the eighth center point may be the geometric center point of the eighth column of the second cage.

[0218] In an example, when the first center point in the first target image area is the geometric center point of the first foot cup of the first cage, the third center point is the geometric center point of the third foot cup of the first cage, the fifth center point is the geometric center point of the fifth foot cup of the first cage, and the seventh center point is the geometric center point of the seventh foot cup of the first cage, the second center point in the second target image area can be the center vertex of the second column of the second cage, the fourth center point can be the center vertex of the fourth column of the second cage, the sixth center point can be the center vertex of the sixth column of the second cage, and the eighth center point can be the center vertex of the eighth column of the second cage.

[0219] In a seventh manner, the controller of the embodiment of the present application uses the first pixel relationship vector between the first center line in the first target image area and the second center point in the second target image area as the first pixel difference. When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacked object is aligned with the second column of the second stacked object, and further confirms that the first stacked object is aligned with the second stacked object.

[0220] The pixel relationship vector is a perpendicular vector from the second center point to the first center line.

[0221] Here, based on the coordinate point of the second center point, a vertical vector is generated in the direction of a perpendicular line to the first center line.

[0222] In one example, the first stacking object is a first cage; the first center line is the center line of a first foot cup of the first cage; the second stacking object is a second cage; and the second center point is the center vertex of a second column of the second cage.

[0223] It should be noted that when the first target image area includes at least a portion of the first column, the center line of the first foot cup may be: a virtual line passing through the geometric center point of the columnar structure (ie, the first column), and the virtual line may represent the symmetry axis of the first column.

[0224] In one example, the second center point in the second target image area may also be the geometric center point of the second column of the second cage.

[0225] An eighth method, based on a first pixel relationship vector between a first center line in a first target image area and a second center point in a second target image area, and a second pixel relationship vector between a third center line in the first target image area and a fourth center point in the second target image area, the controller uses the first pixel relationship vector as a first pixel difference, and the controller uses the second pixel relationship vector as a second pixel difference.

[0226] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacking object is aligned with the second column of the second stacking object; when the second pixel difference is less than the threshold, the controller confirms that the third foot cup of the first stacking object is aligned with the fourth column of the second stacking object, and then confirms that the first stacking object is aligned with the second stacking object.

[0227] like Fig.16 In the figure, the first target image area A1 includes at least part of the first foot cup Aj1 and at least part of the third foot cup Aj3, and the first target image area A1 includes the center line c1 of the first foot cup Aj1 and the center line c3 of the third foot cup Aj3; the second target image area B1 includes at least part of the second column Bj2 and at least part of the fourth column Bj4, and the second target image area B1 includes the central vertex d2 of the second column Bj2 and the central vertex d4 of the fourth column Bj4.

[0228] According to the center line c1 of the first foot cup Aj1 and the coordinates (x2, y2, z2) of the central vertex d2 of the second column Bj2, the vertical vector (i.e. Fig.16 The direction of the perpendicular vector is indicated by the red arrow from d2 to c1.

[0229] According to the center line c3 of the third foot cup Aj3 and the coordinates (x4, y4, z4) of the central vertex d4 of the fourth column Bj4, the vertical vector (i.e. Fig.16 The direction of the perpendicular vector is indicated by the red arrow from d4 to c3.

[0230] The coordinates of the center vertex d2 of the second column Bj2 and the coordinates of the center vertex d4 of the fourth column Bj4 are Figure 2 Coordinates in the three-dimensional coordinate system.

[0231] In one example, the second center point in the second target image area may be the geometric center point of the second column of the second basket, and the fourth center point may be the geometric center point of the fourth column of the second basket.

[0232] like Fig.17The first target image area A1 includes at least a portion of the first foot cup Aj1 and at least a portion of the third foot cup Aj3, and the first target image area A1 includes the center line c1 of the first foot cup Aj1 and the center line c3 of the third foot cup Aj3; the second target image area B1 includes at least a portion of the second column Bj2 and at least a portion of the fourth column Bj4, and the second target image area B1 includes the geometric center point d2' of the second column Bj2 and the geometric center point d4' of the fourth column Bj4.

[0233] According to the center line c1 of the first foot cup Aj1 and the coordinates (x2', y2', z2') of the geometric center point d2' of the second column Bj2, a vertical vector (i.e. Fig.17 The direction of the perpendicular vector is indicated by the red arrow pointing from d2' to c1.

[0234] According to the center line c3 of the third foot cup Aj3 and the coordinates (x4', y4', z4') of the geometric center point d4' of the fourth column Bj4, the vertical vector (i.e. Fig.17 The direction of the perpendicular vector is indicated by the red arrow pointing from d4' to c3.

[0235] A ninth method, based on a first pixel relationship vector between a first center line in a first target image area and a second center point in a second target image area, a second pixel relationship vector between a third center line in a first target image area and a fourth center point in a second target image area, a third pixel relationship vector between a fifth center line in a first target image area and a sixth center point in a second target image area, and a fourth pixel relationship vector between a seventh center line in a first target image area and an eighth center point in a second target image area, a controller uses the first pixel relationship vector as a first pixel difference, the second pixel relationship vector as a second pixel difference, the third pixel relationship vector as a third pixel difference, and the fourth pixel relationship vector as a fourth pixel difference.

[0236] When the first pixel difference is less than the threshold, the controller confirms that the first foot cup of the first stacked object is aligned with the second column of the second stacked object. When the second pixel difference is less than the threshold, the controller confirms that the third foot cup of the first stacked object is aligned with the fourth column of the second stacked object. When the third pixel difference is less than the threshold, the controller confirms that the fifth foot cup of the first stacked object is aligned with the sixth column of the second stacked object. When the fourth pixel difference is less than the threshold, the controller confirms that the seventh foot cup of the first stacked object is aligned with the eighth column of the second stacked object. Then the first stacked object is confirmed to be aligned with the second stacked object.

[0237] The generation method of the second, third and fourth pixel relationship vectors is similar to the generation method of the first pixel relationship vector.

[0238] like Fig.18 The first target image area A1 includes at least a portion of the first foot cup Aj1, at least a portion of the third foot cup Aj3, at least a portion of the fifth foot cup Aj5, and at least a portion of the seventh foot cup Aj7. The first target image area A1 includes the center line c1 of the first foot cup Aj1, the center line c3 of the third foot cup Aj3, the center line c5 of the fifth foot cup Aj5, and the center line c7 of the seventh foot cup Aj7; the second target image area B1 includes at least a portion of the second column Bj2, at least a portion of the fourth column Bj4, at least a portion of the sixth column Bj6, and at least a portion of the eighth column Bj8. The second target image area B1 includes the geometric center point d2' of the second column Bj2, the geometric center point d4' of the fourth column Bj4, the geometric center point d6' of the sixth column Bj6, and the geometric center point d8' of the eighth column Bj8.

[0239] According to the center line c1 of the first foot cup Aj1 and the coordinates (x2', y2', z2') of the geometric center point d2' of the second column Bj2, a vertical vector (i.e. Fig.18 The direction of the perpendicular vector is indicated by the red arrow pointing from d2' to c1.

[0240] According to the center line c3 of the third foot cup Aj3 and the coordinates (x4', y4', z4') of the geometric center point d4' of the fourth column Bj4, the vertical vector (i.e. Fig.18 The direction of the perpendicular vector is indicated by the red arrow pointing from d4' to c3.

[0241] According to the center line c5 of the fifth foot cup Aj5 and the coordinates (x6', y6', z6') of the geometric center point d6' of the sixth column Bj6, the vertical vector (i.e. Fig.18 The direction of the perpendicular vector is indicated by the red arrow pointing from d6' to c5.

[0242] According to the center line c7 of the seventh foot cup Aj7 and the coordinates (x8', y8', z8') of the geometric center point d8' of the eighth column Bj8, the vertical vector (i.e. Fig.18 The direction of the perpendicular vector is indicated by the red arrow pointing from d8' to c7.

[0243] The coordinates of the geometric center point d2' of the second column Bj2 and the coordinates of the geometric center point d4' of the fourth column Bj4 are Figure 2 Coordinates in the three-dimensional coordinate system.

[0244] In one example, the second center point in the above-mentioned second target image area can be the center vertex of the second column of the second material basket, the fourth center point can be the center vertex of the fourth column of the second material basket, the sixth center point can be the center vertex of the sixth foot cup of the second material basket, and the eighth center point can be the center vertex of the eighth column of the second material basket.

[0245] In the embodiment of the present application, the alignment state of the first stacking object and the second stacking object may be confirmed in the following manner, including:

[0246] In the first way, the embodiment of the present application compares the pixel difference with a threshold value to confirm the alignment state of the first stacking object and the second stacking object, including:

[0247] If the pixel difference is greater than or equal to the first threshold, it is determined that the alignment state is misaligned;

[0248] If the pixel difference is less than the first threshold, the alignment state is determined to be aligned.

[0249] In the embodiment of the present application, when the pixel difference is greater than or equal to the first threshold, it is confirmed that the alignment state of the first stacked object and the second stacked object is misaligned, and the larger the difference between the pixel difference and the first threshold, the lower the degree of alignment between the first stacked object and the second stacked object; when the pixel difference is less than the first threshold, it is confirmed that the alignment state of the first stacked object and the second stacked object is aligned, and the smaller the difference between the pixel difference and the first threshold, the higher the degree of alignment between the first stacked object and the second stacked object.

[0250] In one example, in the embodiment of the present application, when the pixel difference is greater than or equal to the first threshold, the method further includes:

[0251] The controller controls the handling equipment to adjust its posture according to the pixel difference;

[0252] The controller reacquires the target image of the first stacked object and the second stacked object through the sensor;

[0253] The controller re-determines the pixel difference between the first target image area and the second target image area;

[0254] The controller reconfirms the alignment state according to the re-determined pixel difference value until the pixel difference value is less than the first threshold value.

[0255] In one example, when the pixel difference is less than the first threshold value, the method further includes:

[0256] When the alignment state is aligned, the controller controls the transport device to place the first stacking object on the second stacking object to complete the stacking.

[0257] Here, the pixel difference and the first threshold are used to measure the alignment state of the first stacking object and the second stacking object, and the first threshold can be set according to the alignment accuracy of the first stacking object and the second stacking object.

[0258] exist Fig.19 In the example, the controller calculates the pixel difference in a point-to-line manner, and uses the pixel difference as the basis for iteration to control the handling equipment to adjust its posture, as follows:

[0259] Step 1001: The controller obtains an original image through a sensor;

[0260] Step 1002: The controller performs dedistortion processing on the original image collected by the sensor in real time according to the pre-calibrated camera internal parameters to obtain a target image.

[0261] Step 1003: The controller acquires a first target image region of the first stacking object and a second target image region of the second stacking object from the target image.

[0262] exist Fig.19 In the example, the first target image area includes at least a portion of the first foot cup and at least a portion of the third foot cup of the first stacked object, and the second target image area includes at least a portion of the second column and at least a portion of the fourth column of the second stacked object.

[0263] Step 1004: The controller uses the vertical vector from the geometric center point or center vertex of the first foot cup of the first stacked object to the center line of the second column of the second stacked object, and the vertical vector from the geometric center point or center vertex of the third foot cup of the first stacked object to the center line of the fourth column of the second stacked object as the pixel difference.

[0264] Step 1005: The controller determines whether the pixel difference is less than a first threshold.

[0265] Step 1006 : When the pixel difference is greater than or equal to the first threshold, the controller determines that the alignment state between the first stacking object and the second stacking object is misaligned, and executes step 1007 .

[0266] Step 1007: The controller controls the transport device to adjust its posture according to the pixel difference, and continues to execute steps 1001 to 1005.

[0267] Step 1008: When the pixel difference is less than a first threshold, the controller confirms that the alignment state of the first stacking object and the second stacking object is aligned.

[0268] Step 1009: The controller controls the transport device to place the first stacking object on the second stacking object to complete the stacking.

[0269] The embodiment of the present application can control the handling equipment to adjust its posture according to the pixel difference and / or the control amount.

[0270] Continuing from the above, after determining the pixel difference in step 305, the method further includes:

[0271] Determine the control amount using inverse kinematics based on pixel differences;

[0272] Based on the control amount, the handling equipment is controlled to adjust its posture.

[0273] The controller of the embodiment of the present application uses inverse kinematics (IKP) to calculate the control amount from the current posture to the target posture according to the pixel difference, so as to control the handling equipment to adjust its posture.

[0274] Among them, the current posture refers to the posture of the handling equipment at the current moment (such as the posture of the fork at the current moment or the posture of the chassis of the handling equipment); the target posture usually refers to the posture that the handling equipment is expected to reach, which is determined based on the stacking task requirements and the working environment.

[0275] exist Fig. 20 In the process, the controller calculates the pixel difference in a point-to-line manner, and uses inverse kinematics to determine the control amount based on the pixel difference. The controller then controls the handling equipment to adjust its posture based on the control amount, as follows:

[0276] Step 1101: The controller obtains an original image through a sensor;

[0277] Step 1102: The controller performs dedistortion processing on the original image collected by the sensor in real time according to the pre-calibrated camera internal parameters to obtain a target image.

[0278] Step 1103: The controller acquires a first target image region of the first stacking object and a second target image region of the second stacking object from the target image.

[0279] exist Fig. 20 In the example, the first target image area includes at least a portion of the first foot cup and at least a portion of the third foot cup of the first stacked object, and the second target image area includes at least a portion of the second column and at least a portion of the fourth column of the second stacked object.

[0280] Step 1104: The controller uses the vertical vector from the geometric center point or center vertex of the first foot cup of the first stacked object to the center line of the second column of the second stacked object, and the vertical vector from the geometric center point or center vertex of the third foot cup of the first stacked object to the center line of the fourth column of the second stacked object as the pixel difference.

[0281] Step 1105: The controller determines whether the pixel difference is less than a first threshold.

[0282] Step 1106 : When the pixel difference is greater than or equal to the first threshold, the controller determines that the alignment state of the first stacking object and the second stacking object is misaligned, and executes step 1107 .

[0283] Step 1107: The controller uses inverse kinematics to determine the control amount according to the pixel difference.

[0284] In some embodiments, taking the control variable as speed as an example, the control variable may be determined by the following calculation method:

[0285]

[0286] in, is the speed control quantity, which can include speeds in multiple directions, such as v x , v y , w z ; K is a constant, is the inverse operation of the speed-based Jacobian matrix, is the pixel difference.

[0287] Step 1108: The controller controls the transport device to adjust its posture according to the control amount, and continues to execute steps 1101 to 1105.

[0288] Step 1109: When the pixel difference is less than the first threshold, the controller confirms that the alignment state of the first stacking object and the second stacking object is aligned.

[0289] Step 1110: The controller controls the transport device to place the first stacking object on the second stacking object to complete the stacking.

[0290] The second method: using inverse kinematics to determine the control amount based on pixel difference;

[0291] If the control amount is greater than or equal to the second threshold, it is determined that the alignment state is misaligned;

[0292] If the control amount is less than the second threshold, it is determined that the alignment state is aligned.

[0293] In one example, when the control amount is greater than or equal to the second threshold, the method further includes:

[0294] When the alignment state is misaligned, the handling equipment is controlled to adjust the posture according to the control amount;

[0295] The controller reacquires the target image of the first stacked object and the second stacked object through the sensor;

[0296] The controller re-determines the pixel difference between the first target image area and the second target image area;

[0297] The controller redetermines the control amount based on the pixel difference;

[0298] The controller reconfirms the alignment state according to the control amount until the control amount is less than a second threshold.

[0299] In one example, when the control amount is less than the second threshold value, the method further includes:

[0300] When the alignment state is aligned, the handling device is controlled to place the first stacking object on the second stacking object to complete the stacking.

[0301] exist Fig.21 In the process, the controller calculates the pixel difference and uses inverse kinematics to determine the control amount based on the pixel difference. The controller then controls the handling equipment to adjust the posture based on the control amount, as follows:

[0302] Step 1201: The controller obtains an original image through a sensor;

[0303] Step 1202: The controller performs dedistortion processing on the original image collected by the sensor in real time according to the pre-calibrated camera internal parameters to obtain a target image.

[0304] Step 1203: The controller obtains a first target image region of the first stacking object and a second target image region of the second stacking object from the target image.

[0305] The first target image area includes at least a portion of the first foot cup and at least a portion of the third foot cup of the first stacked object, and the second target image area includes at least a portion of the second column and at least a portion of the fourth column of the second stacked object.

[0306] Step 1204: The controller uses the vertical vector from the geometric center point or center vertex of the first foot cup of the first stacked object to the center line of the second column of the second stacked object, and the vertical vector from the geometric center point or center vertex of the third foot cup of the first stacked object to the center line of the fourth column of the second stacked object as the pixel difference.

[0307] Step 1205: The controller uses inverse kinematics to determine the control amount according to the pixel difference.

[0308] Step 1206: The controller determines whether the control amount is less than a second threshold.

[0309] Step 1207 : When the control amount is greater than or equal to the second threshold, the controller confirms that the alignment state between the first stacking object and the second stacking object is misaligned, and executes step 1208 .

[0310] Step 1208: The controller controls the transport device to adjust its posture according to the control amount, and continues to execute steps 1201 to 1208 until the control amount is less than the second threshold, and then executes steps 1209 and 1210.

[0311] Step 1209: When the control amount is less than the second threshold, the controller confirms that the alignment state of the first stacking object and the second stacking object is aligned.

[0312] Step 1210: The controller controls the transport device to place the first stacking object on the second stacking object to complete the stacking.

[0313] Here, the control amount and the second threshold are used to measure whether the handling device is controlled to align the first stacked object with the second stacked object, and the second threshold can be set according to the alignment accuracy of the first stacked object and the second stacked object.

[0314] Controlling the transport device to adjust the posture based on the pixel difference may include controlling the transport device to adjust the posture of the fork and / or controlling the transport device to adjust the posture of the chassis.

[0315] In some embodiments, before confirming the alignment state, the handling device carries the first stacking object to the stacking preparation position to complete the pre-alignment action relative to the second stacking object. The stacking preparation position refers to the position before reaching the stacking operation position, at which the handling device can obtain the posture of the second stacking object.

[0316] Stacking operation position: refers to a position where a sensor on the handling device can simultaneously obtain target data (such as a target image) of the first stacking object and the second stacking object.

[0317] Pre-alignment means: adjusting the position of the handling equipment so that the first stacking object and the second stacking object are basically aligned in the Y-axis direction. That is, the coordinate difference △Y of the first stacking object and the rotation angle difference △Ψ of the Z-axis between the first stacking object and the second stacking object are within a preset threshold value, which can be flexibly adjusted according to different handling equipment and stacking objects, for example, -5cm<△Y<5cm, -3°<△Ψ<3°.

[0318] In the pre-alignment stage, the handling equipment moves with the first stacking object to the second stacking object, first lifts the fork to the optimal detection height, then detects the column and other features of the second stacking object, calculates the position of the column, and adjusts the chassis or fork to complete the pre-alignment. In this stage, in order to ensure that the unmanned forklift is basically aligned with the second stacking object in the direction of travel and rotation, the forward and backward path planning strategies are used to achieve large error correction to ensure that the first stacking object is basically aligned with the second stacking object.

[0319] The above method provided in the embodiment of the present application can be applied to a variety of application scenarios, including but not limited to: unmanned warehouse scenarios, unmanned loading and unloading scenarios.

[0320] Among them, unmanned warehouses may include handling equipment, storage shelves, picking platforms, RCS control systems and warehouse management systems (Warehouse Management System, WMS), etc.

[0321] Among them, the unmanned loading and unloading scenarios include handling equipment, RCS control systems, and trucks.

[0322] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0323] According to an embodiment of another aspect, a control system is provided for executing to implement any one of the alignment status confirmation methods disclosed in the embodiments of the present application.

[0324] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0325] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a controller, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0326] The present application also provides a transport device, comprising:

[0327] Handling equipment body;

[0328] A sensor mounted on the handling equipment body;

[0329] one or more controllers; and

[0330] A memory coupled to one or more controllers, the memory is used to store program instructions, and when the program instructions are read and executed by one or more controllers, all or part of the steps in any one of the alignment status confirmation methods described in the above embodiments are executed.

[0331] In some embodiments, the handling device body includes a handling device body.

[0332] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a controller.

[0333] It can be seen from the above description of the implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can essentially be embodied in the form of a computer program product, which can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc.

[0334] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for confirming an alignment state, characterized in that: include: The controller acquires target images of the first stacking object and the second stacking object through the sensor; The controller acquires a first target image region of the first stacking object and a second target image region of the second stacking object from the target image; The controller determines a pixel difference value between the first object image area and the second object image area, and compares the pixel difference value with a threshold value to confirm an alignment state of the first stacked object and the second stacked object.

2. The method according to claim 1, characterized in that The method further comprises: Before confirming the alignment state, the controller controls the transport device to transport the first stacking object to a stacking preparation position to complete a pre-alignment action relative to the second stacking object.

3. The method according to claim 1, characterized in that The controller acquires target images of the first stacking object and the second stacking object through a sensor, including: The controller acquires original images of the first stacked object and the second stacked object through the sensor; The original image is dedistorted according to the pre-calibrated sensor internal parameters to obtain the target image.

4. The method according to claim 1, characterized in that: The controller determines a pixel difference between the first target image area and the second target image area, comprising: A pixel relationship vector between a first center point in the first target image region and a second center line in the second target image region is used as a pixel difference value.

5. The method according to claim 4, characterized in that The pixel relationship vector is a perpendicular vector from the first center point to the second center line.

6. The method according to claim 5, characterized in that The first stacking object is a first material cage, and the second stacking object is a second material cage; The first center point is the geometric center point or center vertex of the first foot cup of the first material cage, and the second center line is the center line of the second column of the second material cage.

7. The method according to claim 1, characterized in that The controller determines a pixel difference between the first target image area and the second target image area, comprising: A pixel relationship vector between a first center point in the first target image region and a second center point in the second target image region is used as a pixel difference value.

8. The method according to claim 7, characterized in that The pixel relationship vector is a line vector connecting the first center point to the second center point.

9. The method according to claim 8, characterized in that The first stacking object is a first material cage, and the second stacking object is a second material cage; The first center point is the geometric center point or center vertex of the first foot cup of the first material cage, and the second center point is the geometric center point or center vertex of the second column of the second material cage.

10. The method according to claim 1, characterized in that The controller determines a pixel difference between the first target image area and the second target image area, comprising: The pixel relationship vector between the first center line in the first target image area and the fourth center point in the second target image area is used as the pixel difference.

11. The method according to claim 10, characterized in that The pixel relationship vector is a perpendicular vector from the fourth center point to the first center line.

12. The method according to claim 11, characterized in that The first stacking object is a first material cage, and the second stacking object is a second material cage; The first center line is the center line of the first foot cup of the first cage, and the fourth center point is the geometric center point or center vertex of the second column of the second cage.

13. The method according to claim 1, characterized in that The comparing the pixel difference with a threshold value to confirm the alignment state of the first stacking object and the second stacking object includes: If the pixel difference is greater than or equal to a first threshold, determining that the alignment state is misaligned; If the pixel difference is less than a first threshold, it is determined that the alignment state is aligned.

14. The method according to claim 13, characterized in that The method further comprises: When the alignment state is misaligned, controlling the handling device to adjust the posture according to the pixel difference; The controller reacquires the target images of the first stacked object and the second stacked object through the sensor; The controller re-determines the pixel difference between the first target image area and the second target image area; The controller reconfirms the alignment state according to the pixel difference value until the pixel difference value is less than a first threshold value.

15. The method according to claim 14, characterized in that The step of controlling the handling device to adjust the posture according to the pixel difference comprises: Determining a control amount using inverse kinematics according to the pixel difference; The transport device is controlled to adjust its posture based on the control amount.

16. The method according to claim 13, characterized in that The method further comprises: When the alignment state is aligned, the transport device is controlled to place the first stacking object on the second stacking object to complete the stacking.

17. The method according to claim 14, characterized in that Controlling the handling equipment to adjust the posture includes: controlling the handling equipment to adjust the posture of the chassis or the posture of the fork.

18. The method according to claim 1, characterized in that The first stacking object is a first material cage, and the second stacking object is a second material cage; The first target image area includes at least a portion of the first foot cup and at least a portion of the third foot cup of the first material cage; The second target image area includes at least a portion of the second column and at least a portion of the fourth column of the second material cage.

19. The method according to claim 18, characterized in that The first target image area also includes at least a portion of the fifth foot cup and at least a portion of the seventh foot cup of the first material cage; The second target image area also includes at least a partial area of ​​the sixth column and at least a partial area of ​​the eighth column of the second material basket.

20. A control system, characterized in that: The invention comprises a controller and a memory, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 19.

21. A handling device, characterized in that: The invention comprises a memory and a controller, wherein the memory is used to store program instructions, and the controller is used to execute the program instructions to implement the method according to any one of claims 1 to 19.

Images