Object stacking control method and apparatus, electronic device, storage medium, and computer program product

Abstract

The application provides an object stacking control method and device, electronic equipment, storage medium and computer program product, which relates to the technical field of logistics technology. The method comprises: acquiring stacking feature data of objects in a predetermined area; determining a stacking strategy based on the stacking feature data of each object, and stacking subsequent objects. In this way, the position space available for stacking packages can be determined according to the features of the packages (objects) stacked in the predetermined area each time, and a stacking strategy matching the position space is formulated. The stacking strategy is used for stacking, and since each package is stacked in a matching position space, each package is more stable, thereby reducing the occurrence of unstacking and package falling when facing different packages of mixed material categories.

Description

Technical Field

[0001] This application relates to the field of logistics technology, and in particular to an object stacking control method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] In related technologies, during the process of handling and palletizing packages by robotic arms, they can usually only handle packages of the same standard category and cannot flexibly handle packages of different types of mixed materials. When faced with packages of different types of mixed materials, it is easy to cause situations such as palletizing overturning, packages falling off, and other displacement of the package position inside the container. Summary of the Invention

[0003] The embodiments of this application provide an object stacking control method, apparatus, electronic device, storage medium, and computer program product that can reduce the occurrence of stacking and package falling when dealing with different packages of mixed materials.

[0004] The technical solution of this application is implemented as follows:

[0005] This application provides an object stacking control method, including:

[0006] Obtain stacking feature data of objects within a predetermined area;

[0007] Based on the stacking feature data of each object, a stacking strategy is determined, and subsequent objects are stacked.

[0008] In the above scheme, obtaining the spatial features of the location of objects within the predetermined area includes:

[0009] In response to a stacking event within the predetermined region, a first spatial feature of the location of an object within a first region is obtained; wherein the first region includes the predetermined region;

[0010] Based on the first location spatial feature filtering, the location spatial features of objects within the predetermined area are determined.

[0011] In the above scheme, the first location spatial feature includes: first point cloud data; the location spatial feature includes: point cloud blocks;

[0012] The step of filtering based on the first location spatial feature to determine the location spatial features of objects within the predetermined area includes:

[0013] The first point cloud data is transformed into the coordinate system corresponding to the predetermined area to determine the second point cloud data;

[0014] Extract the third point cloud data belonging to the predetermined area from the second point cloud data;

[0015] The third point cloud data is clustered to determine multiple point cloud blocks; wherein each point cloud block is used to characterize the spatial and positional features of the top corresponding object within the predetermined area.

[0016] In the above scheme, determining the stacking strategy based on the spatial characteristics of each object includes:

[0017] The stacking strategy is determined based on the space between each cloud block.

[0018] In the above scheme, determining the stacking strategy based on the space between each cloud block includes any one of the following:

[0019] Based on the space between each point cloud block, a first stacking strategy is determined for the next object stacked within the predetermined area; wherein the first stacking strategy is used to characterize the attributes of the next object;

[0020] Based on the space between each point cloud block and the required size of the subsequent objects, a second stacking strategy for the next object is adjusted; wherein, the second stacking strategy is used to represent adjusting the stacking position of the next object to leave the required space for the subsequent objects.

[0021] The method in the above scheme further includes:

[0022] The object state within the predetermined area is determined by comparing each point cloud block with a historical point cloud block sequence.

[0023] In the above scheme, determining the object state within the predetermined area based on the comparison between each point cloud block and the historical point cloud block sequence includes any one of the following:

[0024] If the average height of each point cloud block is greater than the average height of the top point cloud block sequence in the historical point cloud block sequence, then a new object is determined for the predetermined area, and the historical point cloud block sequence is updated based on the point cloud block corresponding to the new object.

[0025] If there is no historical point cloud block sequence, then the historical point cloud block sequence is formed based on each of the point cloud blocks;

[0026] If the average height of each point cloud block is lower than the average height of the top point cloud block sequence in the historical point cloud block sequence, then it is determined that objects will be reduced for the predetermined area, and the historical point cloud block sequence is updated based on the point cloud blocks corresponding to the reduced objects.

[0027] The method in the above scheme further includes:

[0028] Add the point cloud block corresponding to the newly added object to the historical point cloud block sequence to determine the predicted point cloud block sequence;

[0029] Based on the comparison between the predicted point cloud block sequence and the point cloud block sequence formed by each of the point cloud blocks, a detection result is determined for whether the state of the object within the predetermined area is abnormal.

[0030] In the above scheme, determining the detection result of whether the object state within the predetermined area is abnormal based on the comparison between the predicted point cloud block sequence and the point cloud block sequence formed by each point cloud block includes:

[0031] Each point cloud column in the predicted point cloud block sequence is compared with the corresponding point cloud column in the point cloud block sequence. If the height of the corresponding column changes or the number of objects changes, it is determined that the object state in the corresponding column of the point cloud block sequence within the predetermined area is abnormal.

[0032] This application embodiment also provides an object stacking control device, including:

[0033] The data acquisition unit is used to acquire stacking characteristic data of objects within a predetermined area;

[0034] A determining unit is configured to determine a stacking strategy based on the stacking feature data of each of the objects, and to stack the objects subsequently.

[0035] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps in the above-described method.

[0036] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method.

[0037] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method.

[0038] In this embodiment, stacking feature data of objects within a predetermined area is obtained; based on the stacking feature data of each object, a stacking strategy is determined, and subsequent objects are stacked. In this way, the available space for stacking packages can be determined based on the stacking features of packages (objects) within the predetermined area each time, and a stacking strategy matching that space can be formulated. Using this stacking strategy for palletizing, since each package is stacked in the matching space, the packages are more stable, thus reducing the occurrence of stacking collapse and package falling when dealing with packages of mixed materials and types. Attached Figure Description

[0039] Figure 1 Illustration of the effects of the related technologies provided in the embodiments of this application Figure 1 ;

[0040] Figure 2 Illustration of the effects of the related technologies provided in the embodiments of this application Figure 2 ;

[0041] Figure 3 A flowchart illustrating the object stacking control method provided in this application embodiment. Figure 1 ;

[0042] Figure 4 This application provides an illustration of the effect of the object stacking control method in its embodiments. Figure 1 ;

[0043] Figure 5 A flowchart illustrating the object stacking control method provided in this application embodiment. Figure 2 ;

[0044] Figure 6 A flowchart illustrating the object stacking control method provided in this application embodiment. Figure 3 ;

[0045] Figure 7 A flowchart illustrating the object stacking control method provided in this application embodiment. Figure 4 ;

[0046] Figure 8 A flowchart illustrating the object stacking control method provided in this application embodiment. Figure 5 ;

[0047] Figure 9 This application provides an example of the effect of the object stacking control method. Figure 2 ;

[0048] Figure 10 A flowchart illustrating the object stacking control method provided in this application embodiment. Figure 6 ;

[0049] Figure 11 A flowchart illustrating the object stacking control method provided in this application embodiment. Figure 7 ;

[0050] Figure 12 This is a schematic diagram of the object stacking control device provided in the embodiments of this application;

[0051] Figure 13 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0056] Combination Figure 1 In the relevant technologies, a sorting process exists in frontline logistics operations. Within this process, couriers categorize and stack items according to type and order of arrival / departure within the logistics sorting center. This process mainly consists of six steps: unloading from the truck, feeding small packages, handling bundled bags, handling NC (Non-Standard Parts) items, stacking in cages, and loading into the truck. Combined with... Figure 2 The existing solutions for robotic arms to automatically place packages into cage carts in logistics scenarios mainly consist of the following hardware components: six-sided scanner, fully automated sorting conveyor belt equipment, grid slots, palletizing robotic arms, and cage cart containers.

[0057] In related technologies, robotic arms can only stack standard packages of the same shape, size, and material into a designated pallet trolley. When stacking standard items, the packages inside the container typically do not shift position; the stacking status of the packages within the container is stored according to pre-planned and calculated original placement positions, enabling continuous stacking by the robotic arm. Current container status recognition technologies, while ensuring stacking efficiency, can only handle packages of the same standard category and cannot flexibly handle packages of mixed materials.

[0058] The reason for this situation is that the packages that need to be sorted in the current logistics scenario vary greatly in terms of material, shape, size, and weight. Non-standard packages include soft packages, irregularly shaped packages, irregularly shaped packages, packages of various sizes, and white insulated boxes. When a large number of non-standard packages accumulate in the container, if the packages are of mixed sizes and materials, each stacking action of the robotic arm can easily cause the packages to tip over, fall, or shift in position within the container. If the existing container status recognition technology is still used when the packages are shifted, phenomena such as high-throwing and compression during stacking due to package displacement can easily occur.

[0059] Furthermore, if robotic arm palletizing technology can only handle standard product categories, it will lead to low operational productivity in current applications. In current robotic arm standard product palletizing solutions, because multiple sales departments handle similar standard product packages in logistics sorting scenarios, the arrival of packages at the sorting bins is not a continuous, high-frequency action. This prevents the robotic arm from reaching its full palletizing capacity, as it spends most of its time waiting for available packages. In the current technology product system, if the robotic arm's palletizing capacity (the number of packages it can pick up and stack) is limited, it will be difficult to meet the business demands of sorting centers during peak periods.

[0060] To address the aforementioned technical problems, this application provides an object stacking control method. Please refer to [link to relevant documentation]. Figure 3 This is a flowchart illustrating the object stacking control method provided in this application embodiment. Figure 1 , will combine Figure 3 The steps shown are explained below:

[0061] S101. Obtain the spatial characteristics of the location of objects within the predetermined area.

[0062] In this embodiment of the application, the object stacking control device can acquire the positional spatial features of the objects (packages) within a predetermined area, which can characterize the position and space of each object.

[0063] In this embodiment of the application, the object stacking control device can acquire three-dimensional (3D) point cloud data blocks or other image data that can characterize the position and space of objects within a predetermined area through an image acquisition component.

[0064] The predetermined area may include containers, flatbed carts, pallets, or a defined area for placing packages. The objects may include packages or other items used for stacking. The object stacking control device may include a terminal or server for controlling the robotic arm. The object stacking control device is communicatively connected to the image acquisition components.

[0065] Point clouds are data sets consisting of a large number of discrete points in three-dimensional space. Each point contains three-dimensional coordinates (X,Y,Z) and can be supplemented with information such as color, reflection intensity, and normal vector.

[0066] The image acquisition component may include a camera (exemplary: an RGB-D camera). A camera stand can be erected above the container to mount the camera and capture real-time changes inside the container. Therefore, from the camera's perspective, a camera coordinate system exists to identify the positions of various objects captured by the camera, with the camera location as the origin of this three-dimensional coordinate system. RGB-D Depth Camera: A 3D Time-of-Flight (TOF) camera module with RGB functionality, featuring a 320×240 resolution and an Inertial Measurement Unit (IMU). This device integrates a color camera and a depth camera module, enabling simultaneous acquisition of color images and depth information.

[0067] In this embodiment, the control device can also detect the spatial position features of objects within the predetermined area by installing an infrared rangefinder and a monocular camera on top of the predetermined area. However, for a robotic arm, it needs to understand the entire size, shape, and posture of each object within the predetermined area. Using an infrared detector to obtain the height and a monocular camera to obtain the object's length and width information, and then fusing the results of both, may not achieve the required three-dimensional precision. A monocular camera consists of a camera and a lens. It captures images through a single lens, primarily providing two-dimensional image information, such as the size, color, and shape of objects. While it cannot directly determine the distance or depth of an object, depth can be estimated using complex algorithms and machine learning techniques; however, the accuracy of this method is limited by the algorithm and scene complexity. An infrared rangefinder is a device that converts incident infrared radiation signals into electrical signals. It is an instrument for precise distance measurement using modulated infrared light, with a typical range of 1-5 kilometers.

[0068] In this embodiment, the control device can also utilize the numerous two-dimensional (2D) surveillance cameras already deployed in the sorting center to identify changes in the state of objects within a predetermined area. However, for the robotic arm, understanding the fine spatial characteristics of objects within the predetermined area is crucial, resulting in a low tolerance for error. Using RGB images acquired from the side of a 2D camera for object size and position identification may lead to insufficient accuracy.

[0069] S102. Based on the spatial characteristics of each object, determine a stacking strategy and stack the subsequent objects.

[0070] In this embodiment, the object stacking control device can analyze the positional and spatial characteristics of each object to determine a stacking strategy, including the size and material of the next object or the stacking position of the next object. Then, based on the stacking strategy, the robotic arm is controlled to stack subsequent objects.

[0071] In this embodiment of the application, the object stacking control device can also compare and analyze the positional spatial characteristics of each object's representative position and spatial characteristics with the positional spatial characteristics in historical time periods to determine whether the stacking status of objects within a predetermined area is abnormal.

[0072] The problem of recognizing the stacking state of packages inside a container, as addressed in this application embodiment, is to enable a robotic arm system to detect and acquire the stacking state of packages inside a container in real time when the robotic arm continuously stacks mixed-category packages. Figure 4 Based on the real-time spatial characteristics of the location of the package 200 inside the container 100 (point cloud data 300), the stacking strategy is adaptively adjusted to maximize the loading rate of the container 100 and reduce abnormal phenomena such as tipping, falling, and squeezing caused by the movement of the package 200.

[0073] In this embodiment, the spatial characteristics of objects within a predetermined area are obtained; based on the spatial characteristics of each object, a stacking strategy is determined, and subsequent objects are stacked. In this way, the available space for stacking packages can be determined based on the spatial characteristics of each package (object) within the predetermined area, and a stacking strategy matching that space is formulated. Using this stacking strategy for palletizing, since each package is stacked in its matching space, the packages are more stable, thus reducing the occurrence of stacking collapse and package falling when dealing with packages of mixed materials and types.

[0074] Please see Figure 5 This is a flowchart illustrating the object stacking control method provided in this application embodiment. Figure 2 , Figure 3 The shown S101 can also be implemented through S201 to S202, which will be explained in conjunction with the steps:

[0075] S201. In response to a stacking event within the predetermined region, obtain a first spatial feature of the location of an object within the first region; wherein the first region includes the predetermined region.

[0076] In this embodiment of the application, the object stacking control device can be communicatively connected to the image acquisition component. After each stacking of objects within a predetermined area, the image acquisition component sends the first positional spatial features of the first area acquired to characterize the space and position of the objects to the control device.

[0077] In this embodiment of the application, the robotic arm used for stacking objects can be communicatively connected to the image acquisition component, and after the robotic arm stacks the objects once, it controls the image acquisition component to perform a first positional spatial feature acquisition.

[0078] The first region includes the predetermined region. The first region can be an area larger than the predetermined region.

[0079] For example, after each action of placing packages into the container is completed, the camera is triggered to take a picture and obtain 3D point cloud data. The system adds an interface to transmit the 3D point cloud data captured by each camera at the moment the picture is triggered to the interface of the container status recognition algorithm, so as to support the function of real-time container status recognition.

[0080] S202. Filter based on the first location spatial features to determine the location spatial features of objects within the predetermined area.

[0081] In this embodiment of the application, the control device can extract the spatial features of objects belonging only to the predetermined area from the first spatial features of the object corresponding to the first area, based on the range of the predetermined area.

[0082] In this embodiment, a corresponding spatial region range parameter can be configured for a predetermined region. The spatial region range parameter is used to characterize the range vertex coordinates of the predetermined region in the coordinate system corresponding to the predetermined region. The control device can extract the positional spatial features within the predetermined region based on the spatial region range parameter and the first positional spatial features.

[0083] For example, panoramic 3D point cloud data captured by a camera can be cropped and filtered to process only the point cloud data within the container space.

[0084] In this embodiment, in response to a stacking event within a predetermined area, a first spatial feature of an object within a first area is obtained; wherein the first area includes the predetermined area. The spatial feature of the object within the predetermined area is determined by filtering based on the first spatial feature. This filtering process reduces the amount of data processed by the control device, improving processing efficiency and consequently increasing the efficiency of determining the stacking strategy.

[0085] Please see Figure 6 This is a flowchart illustrating the object stacking control method provided in this application embodiment. Figure 3 , Figure 5 The shown S202 can also be implemented by S301 to S302, which will be explained in conjunction with the steps:

[0086] S301. Convert the first point cloud data into the coordinate system corresponding to the predetermined area to determine the second point cloud data.

[0087] In this embodiment, the first location spatial feature includes first point cloud data. Each point cloud in the first point cloud data is formed based on the camera coordinate system of the image acquisition component. Since the origin and coordinate axis distribution of the camera coordinate system and the coordinate system corresponding to the predetermined area are different, a transfer matrix is ​​needed to transform and align the point cloud coordinates. The control device can determine the second point cloud data by transforming each point cloud in the first point cloud data into the coordinate system corresponding to the predetermined area based on the transfer matrix.

[0088] Among them, the coordinate system of each region is predetermined: the coordinate system is constructed by taking one of the four corners of the upper plane of the predetermined region as the zero point of the container coordinate system.

[0089] S302. Extract the third point cloud data belonging to the predetermined area from the second point cloud data.

[0090] In this embodiment of the application, the control device can extract third point cloud data belonging to a predetermined area from the second point cloud data based on the spatial area range parameter.

[0091] In this embodiment, the control device updates the container space range parameters of the predetermined area to be identified in advance based on the mapping relationship between each predetermined area and different camera positions. Furthermore, the control device also configures default algorithm parameters for the object state recognition algorithm within the predetermined area, including point cloud downsampling threshold, point cloud clustering algorithm parameter thresholds (number of clusters, etc.), package drop / tipping judgment threshold, and anomaly handling recovery parameters.

[0092] S303. Cluster the third point cloud data to determine multiple point cloud blocks; wherein each point cloud block is used to characterize the spatial and positional features of the top corresponding object within the predetermined area.

[0093] In this embodiment, the spatial features include point cloud blocks, and the third point cloud data includes multiple point clouds. The control device can cluster the multiple point clouds to determine multiple point cloud blocks. Each point cloud block is used to characterize the spatial and positional features of the corresponding top object within the predetermined area.

[0094] In this embodiment, after filtering the second point cloud data, the top point cloud of objects within a predetermined area is obtained. The control device can use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to cluster the planar point clouds of packages scattered in the graph into clusters. Then, the K-Nearest Neighbors (KNN) algorithm is used to classify them, splitting all top point clouds into multiple point cloud blocks. Each point cloud block identifies the location and spatial features of a package within the predetermined area.

[0095] In this embodiment, the first point cloud data is transformed into the coordinate system corresponding to the predetermined area to determine the second point cloud data. Third point cloud data belonging to the predetermined area is then extracted from the second point cloud data. Thus, by transforming the first point cloud data into the coordinate system corresponding to the predetermined area, the third point cloud data within the predetermined area can be determined more conveniently and quickly, reducing the amount of data processed by the control device and improving the efficiency of determining the stacking strategy.

[0096] Please see Figure 7 This is a flowchart illustrating the object stacking control method provided in this application embodiment. Figure 4 , Figure 3 The shown S102 can also be implemented via S401, which will be explained in conjunction with the steps:

[0097] S401. Based on the space between each cloud block, determine the stacking strategy and stack the subsequent objects.

[0098] In this embodiment of the application, the control device can analyze the space within a predetermined area that can be used for stacking objects based on the location and space corresponding to each cloud block, determine the stacking strategy based on the determined space, and stack the objects subsequently.

[0099] The control device can determine a first stacking strategy for the next object stacked in the predetermined area based on the space between each point cloud block; wherein the first stacking strategy is used to characterize the attributes of the next object.

[0100] For example, the control device can control the attributes (size, material, etc.) of the packages being placed in the compartments in real time based on the real-time detected spatial characteristics within the container. For instance, when the container is nearly full, the control system will schedule the collection bags to fall into the compartments to provide human-machine collaboration and improve the container's loading rate. When the container's stacking space is limited by mixed-category packages, and ordinary medium-sized packages cannot be placed, and the system calculates that not adjusting the package size would result in a significant waste of container space, the system will trigger a compartment scheduling mechanism to reduce the package size.

[0101] The control device adjusts the second stacking strategy of the next object based on the space between each point cloud block and the required size of the subsequent objects; wherein the second stacking strategy is used to adjust the stacking position of the next object to leave the required space for the subsequent objects.

[0102] For example, the control device can dynamically adjust the stacking strategy based on the real-time detection of the package stacking status within the container. For instance, considering the current strategy for scheduling packages to fall into the slots and the sequence of packages about to fall into the slots (including information such as package size, material, and weight), the control strategy can reserve sufficient stacking space for large packages. When the stacking space becomes cramped due to mixed product categories, and ordinary medium-sized packages cannot be placed, the device continuously monitors the updates to the slot scheduling strategy, reserving space for small packages in the cramped areas. This increases the effective number of packages that can be stacked within the container, significantly improving the loading rate of the robotic arm's package stacking.

[0103] In this embodiment, the stacking strategy is determined based on the space between each point cloud block. This allows for the determination of available stacking locations based on the spatial characteristics represented by each point cloud block, enabling the development of a stacking strategy that matches these locations. Using this stacking strategy for palletizing ensures greater stability for each package, especially when dealing with packages of mixed materials and types, thus reducing the occurrence of overstacking and package falling.

[0104] Please see Figure 8 This is a flowchart illustrating the object stacking control method provided in this application embodiment. Figure 5 The following steps will be explained:

[0105] S501. Based on the comparison between each point cloud block and the historical point cloud block sequence, determine the object state within the predetermined area.

[0106] In this embodiment, the control device determines the point cloud blocks after each stacking of objects within a historical time period, forming a historical point cloud block sequence. For example, in conjunction with... Figure 9 The historical point cloud block sequence includes multiple historical point cloud blocks 400. These multiple historical point cloud blocks 400 are stacked in different columns to form the historical point cloud block sequence. After determining each current point cloud block, the location-space features represented by each point cloud block are compared with the location-space features represented by the historical point cloud block sequence to determine the object state within a predetermined area.

[0107] In this embodiment, the control device can determine the feature value of the corresponding point cloud block based on the spatial characteristics of each point cloud block, or it can determine the feature value corresponding to each historical point cloud block in the historical point cloud block sequence. The feature value of each point cloud block is compared with the feature value of the topmost historical point cloud block in the historical point cloud sequence to determine the state of the object within the predetermined area.

[0108] If the average height of each point cloud block is greater than the average height of the top point cloud block sequence in the historical point cloud block sequence, then a new object is determined for the predetermined area, and the historical point cloud block sequence is updated based on the point cloud block corresponding to the new object.

[0109] For example, if the average height (z-axis) of each point cloud block is higher than the average height of the top historical point cloud block in a sequence of historical point cloud blocks with approximately the same xy-axis plane position within the coordinate system of the predetermined region, a new object is added. The minimum and maximum x-axis and y-axis values ​​of this new object are calculated using the boundary points of the point cloud blocks. The height of the new object is calculated from the top height of the historical point cloud block sequence in the same xy-plane, with the highest point of that top point cloud block serving as the highest point of the new object's z-axis. The height difference between these two points is the height of the new object. The detected new object is added to the cached historical point cloud block sequence and updated as the latest object stacking state sequence within the predetermined region.

[0110] Wherein, if there is no historical point cloud block sequence, the historical point cloud block sequence is formed based on each of the point cloud blocks;

[0111] For example, if the historical point cloud block sequence is empty, the minimum and maximum values ​​of the x-axis and y-axis of the new object are calculated using the boundary points of the point cloud blocks. The height of the new object is calculated from the height of the plane at the bottom of the predetermined area, and the highest point of the top point cloud block is taken as the highest point of the z-axis of the new object. The height difference is the height of the new object.

[0112] If the average height of each point cloud block is lower than the average height of the top point cloud block sequence in the historical point cloud block sequence, then it is determined that objects will be reduced for the predetermined area, and the historical point cloud block sequence is updated based on the point cloud blocks corresponding to the reduced objects.

[0113] For example, if the average height (z-axis) of each point cloud block is lower than the height of a historical point cloud block sequence that is approximately the same in position on the xy-axis plane, then the historical package column on the xy-axis plane of the historical point cloud block sequence is traversed, and the first historical package with a height higher than the current average height is removed from the historical point cloud block sequence. Then, it is determined whether the average height (z-axis) of each point cloud block is higher than the height of the historical stacked package column that is approximately the same in position on the xy-axis plane after the removal of the first historical package. If it is higher, the second historical package is removed, and so on, until the average height (z-axis) of each point cloud block after the Nth package is extracted is the same as the height after the removal of the Nth historical package. This indicates that n packages have been reduced within the predetermined area.

[0114] In this embodiment, the object state within the predetermined area is determined based on a comparison between each point cloud block and a historical point cloud block sequence. Thus, when objects within the predetermined area increase or decrease, the state of objects within the predetermined area can be accurately determined by comparing each point cloud block with the historical point cloud sequence, thereby providing the user with timely information on the object state within the predetermined area.

[0115] Please see Figure 10 This is a flowchart illustrating the object stacking control method provided in this application embodiment. Figure 6 The following steps will be explained:

[0116] S601. Add the point cloud block corresponding to the newly added object to the historical point cloud block sequence to determine the predicted point cloud block sequence.

[0117] In this embodiment of the application, after determining the newly added object within the predetermined area, the control device can add the point cloud block corresponding to the newly added object to the historical point cloud block sequence according to the column to which the position of the newly added object belongs, and determine the predicted point cloud block sequence.

[0118] The predicted point cloud block sequence is a stack of multiple historical point cloud blocks and point cloud blocks of newly added objects arranged in columns, thus the predicted point cloud block sequence includes multiple columns.

[0119] S602. Based on the comparison between the predicted point cloud block sequence and the point cloud block sequence formed by each point cloud block, determine the detection result of whether the state of the object in the predetermined area is abnormal.

[0120] In this embodiment, the control device can compare the predicted point cloud block sequence with the point cloud block sequence formed by each point cloud block in terms of height or number of objects. If there is a height difference or a difference in the number of objects between the predicted point cloud block sequence and the point cloud block sequence formed by each point cloud block, then the detection result of an abnormal object state within the predetermined area is determined. Otherwise, the detection result of a normal object state within the predetermined area is determined.

[0121] In this embodiment, the control device compares each point cloud column in the predicted point cloud block sequence with the corresponding point cloud column in the point cloud block sequence. If the height of the corresponding column changes or the number of objects changes, it is determined that the object state in the corresponding column of the point cloud block sequence within the predetermined area is abnormal. Otherwise, it is determined that the object state within the predetermined area is normal.

[0122] In this embodiment, the current point cloud block sequence formed by each point cloud block is also stacked in columns. Each column corresponds to a column in the predicted point cloud block sequence. The control device can compare each column corresponding to the predicted point cloud block sequence with the corresponding column in the point cloud block sequence. If the height changes or the number of objects changes, it is determined that the object status corresponding to that column in the predetermined area has become abnormal. That is, the packages in that column in the predetermined area have been overstacked or fallen off.

[0123] The height of each column can be determined according to the Z-axis coordinate of that column in the coordinate system of the predetermined area.

[0124] For example, after the control device identifies a new object, it can add the point cloud block of the new object to the corresponding column in the historical point cloud block sequence to form a predicted point cloud block sequence. If there is a height difference or a difference in the number of objects between the current point cloud block sequence and the predicted point cloud block sequence, it is determined that a stacking phenomenon has occurred within the predetermined area during the time interval between the two algorithm calculations. If the logical object position output by the current object stacking strategy is different from the new object position determined by the current object stacking state sequence, an object placement falling phenomenon has occurred within the predetermined area.

[0125] In this embodiment, a point cloud block corresponding to the newly added object is added to the historical point cloud block sequence to determine a predicted point cloud block sequence. Based on the comparison between the predicted point cloud block sequence and the point cloud block sequence formed by each of the point cloud blocks, a detection result for whether the object state within the predetermined area is abnormal is determined. Thus, when an object within the predetermined area is overstacked, the comparison between the predicted point cloud block sequence and the point cloud sequence can accurately determine whether the object within the predetermined area is abnormal, thereby providing the user with timely detection results regarding whether the object within the predetermined area is abnormal.

[0126] Please see Figure 11 This is a flowchart illustrating the object stacking control method provided in this application embodiment. Figure 7 The following steps will be explained:

[0127] S11, Coordinate System Transformation: Fusion calculation of multiple coordinate systems to calibrate the 3D position of the container relative to the camera.

[0128] In this embodiment, the camera coordinate system is as follows: Within the logistics sorting center, a camera bracket is erected above the container by the robotic arm palletizing system to mount the camera and capture real-time changes inside the container. Therefore, from the camera's perspective, a camera coordinate system exists to identify the positions of various objects captured by the camera. In this three-dimensional coordinate system, the camera location is the origin of the camera coordinate system. The container coordinate system is constructed using one of the four corners of the container's (cage cart / bottom pallet) upper plane as the zero point. Coordinate system transformation: Since the origins and coordinate axes of the camera coordinate system and the container coordinate system are different, a transformation matrix is ​​needed to align and transform multiple coordinate systems, unifying the calibration method of the container's position across multiple coordinate systems.

[0129] S12. Feature parameter settings for recognizing the size and position of packages inside containers, recognizing packages tipping over, and recognizing packages falling.

[0130] In this embodiment, the container state recognition algorithm feature parameters are configured as follows: For each new container and its mapping relationship with different camera positions, the container spatial region range parameters of the multiple containers to be identified are updated in advance. Furthermore, default algorithm parameters for the container state recognition algorithm are configured, including point cloud downsampling threshold, point cloud clustering algorithm parameter thresholds (number of clusters, etc.), package drop / tipping judgment threshold, and anomaly handling recovery parameters, etc.

[0131] S13, Camera photo trigger: After the packages are stacked, the camera is triggered to take a photo, and the point cloud data is transmitted through the interface.

[0132] In this embodiment of the application, after each action of placing packages into the container is completed, the camera is triggered to take a picture and obtain 3D point cloud data. The system adds an interface to transmit the 3D point cloud data captured by each camera at the moment of triggering the picture to the interface of the container state recognition algorithm, so as to support the function of real-time container state recognition.

[0133] S14. Camera point cloud preprocessing: Extract and filter the point cloud of the container and downsample the data.

[0134] In this embodiment, camera point cloud preprocessing involves cropping and filtering the panoramic 3D point cloud data captured by the camera based on the container space location calibrated in the camera coordinate system, processing only the point cloud data within the container space. Furthermore, to increase the algorithm's computation speed, a downsampling method is used to batch filter the source 3D point cloud data in advance.

[0135] S15. Cluster the packages based on the point cloud of the top surface of the packages and extract the top plane position of the packages.

[0136] In this embodiment, after filtering the overall point cloud data in step S14, the top point cloud of the packages stacked inside the container is obtained. The DBSCAN clustering algorithm is used to cluster the planar point clouds of the packages scattered in the image into clusters. Then, the KNN algorithm is used to classify them, splitting all the top point clouds into multiple point cloud blocks, each point cloud block identifying the location of a package inside the container.

[0137] S16. Temporal point cloud fusion processing: Combine the historical identified package status sequence, compare the height of the current package's top plane with the height of historical packages with similar plane positions, and use the height difference as the height of the new package.

[0138] In this embodiment, the multiple top point cloud blocks extracted in step S15 are compared with the historical package stacking state sequence cached by the system. The height difference between the current package's top plane height and the height of historical packages with similar plane positions is used as the height of the new package.

[0139] S17. Identify the position displacement of packages inside the container and detect package falling or stacking phenomena: Compare the cached historical package stacking state sequence with the height of the top plane of the currently extracted top package. If the height decreases, there is a package falling or stacking phenomenon.

[0140] S18. Dynamic scheduling control of parcel types: Based on the real-time calculated parcel stacking status inside the container, control the size and material of future parcels.

[0141] S19. Dynamic adjustment of parcel stacking strategy: Based on the real-time calculated parcel stacking status inside the container, the stacking strategy is dynamically adjusted (height priority, flatness priority, gap waiting).

[0142] In this embodiment, low-cost RGB-D cameras deployed in the sorting center are used to trigger images taken at the container locations. The 3D point clouds transmitted back by the cameras are analyzed, taking into account the temporal relationships between the captured point clouds, to capture the real-time state of package stacking within the containers. The displacement of packages after stacking is detected and updated in real-time. The detected package stacking state is synchronized in real-time to the upstream package scheduling module and the downstream palletizing module, enabling dynamic control of the type of packages placed in the containers and adaptive adjustment and switching of the package stacking strategy based on the current package state within the containers.

[0143] Please see Figure 12 This is a schematic diagram of the object stacking control device provided in the embodiments of this application.

[0144] This application embodiment also provides an object stacking control device 800, including: a data acquisition unit 801 and a determination unit 802.

[0145] Data acquisition unit 801 is used to acquire the spatial characteristics of the location of objects within a predetermined area;

[0146] The determining unit 802 is used to determine a stacking strategy based on the spatial characteristics of each of the objects, and to stack the objects for subsequent objects.

[0147] In this embodiment of the application, the data acquisition unit 801 in the object stacking control device 800 is used to acquire the first positional spatial features of objects in the first region in response to a stacking event in the predetermined region; wherein, the first region includes the predetermined region;

[0148] Based on the first location spatial feature filtering, the location spatial features of objects within the predetermined area are determined.

[0149] In this embodiment, the first point cloud data; the spatial features include: the third point cloud data; the data acquisition unit 801 in the object stacking control device 800 is used to convert the first point cloud data into the coordinate system corresponding to the predetermined area to determine the second point cloud data; and extract the third point cloud data belonging to the predetermined area from the second point cloud data. The third point cloud data is clustered to determine multiple point cloud blocks; wherein each point cloud block is used to characterize the spatial and positional features of the top corresponding object within the predetermined area;

[0150] In this embodiment of the application, the determining unit 802 in the object stacking control device 800 is used to determine the stacking strategy based on the space between each point cloud block.

[0151] In this embodiment of the application, the determining unit 802 in the object stacking control device 800 is used to determine a first stacking strategy for the next object stacked in the predetermined area based on the space between each point cloud block; wherein, the first stacking strategy is used to characterize the attributes of the next object.

[0152] Based on each point cloud block and the required size for future stacking of the objects, a second stacking strategy for the next object is adjusted; wherein, the second stacking strategy is used to represent adjusting the stacking position of the next object to leave the required space for future stacking of the objects.

[0153] In this embodiment of the application, the determining unit 802 in the object stacking control device 800 is used to determine the object state within the predetermined area based on the comparison between each point cloud block and the historical point cloud block sequence.

[0154] In this embodiment of the application, the determining unit 802 in the object stacking control device 800 is used to determine that a new object is added to the predetermined area if the average height of each point cloud block is greater than the average height of the top point cloud block sequence in the historical point cloud block sequence, and to update the historical point cloud block sequence based on the point cloud block corresponding to the new object.

[0155] If there is no historical point cloud block sequence, then the historical point cloud block sequence is formed based on each of the point cloud blocks;

[0156] If the average height of each point cloud block is lower than the average height of the top point cloud block sequence in the historical point cloud block sequence, then it is determined that objects will be reduced for the predetermined area, and the historical point cloud block sequence is updated based on the point cloud blocks corresponding to the reduced objects.

[0157] In this embodiment of the application, the determining unit 802 in the object stacking control device 800 is used to remove the point cloud block corresponding to the newly added object from each point cloud block and determine the predicted historical point cloud block sequence.

[0158] Based on the comparison between the predicted historical point cloud block sequence and the historical point cloud block sequence, a detection result is determined as to whether the state of an object within the predetermined area is abnormal.

[0159] In this embodiment of the application, the determining unit 802 in the object stacking control device 800 is used to compare each point cloud column in the predicted point cloud block sequence with the corresponding point cloud column in the point cloud block sequence. If the height of the corresponding column changes or the number of objects changes, it is determined that the object state in the corresponding column of the point cloud block sequence in the predetermined area has become abnormal.

[0160] It should be noted that, in the embodiments of this application, if the above-described object stacking control method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an object stacking control device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0161] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of a method in the object stacking control method.

[0162] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0163] It should be noted that, Figure 13 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 13 As shown, this application embodiment provides an electronic device 900, including a memory 902 and a processor 901. The memory 902 stores a computer program that can run on the processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein;

[0164] Processor 901 typically controls the overall operation of electronic device 900.

[0165] The memory 902 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 901 and various modules in the electronic device 900. It can be implemented by flash memory or random access memory (RAM).

[0166] Correspondingly, this application embodiment also provides a computer program product, including a computer program that can be executed by a processor 901 of an electronic device 900 to complete the steps in the method on one side of the object stacking control device 800.

[0167] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or units can be electrical, mechanical, or other forms.

[0170] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0172] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0173] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0174] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for controlling object stacking, characterized in that, include: Obtain the spatial characteristics of objects within a predetermined area; Based on the spatial characteristics of each object, a stacking strategy is determined, and subsequent objects are stacked.

2. The object stacking control method according to claim 1, characterized in that, The step of obtaining the spatial location features of objects within a predetermined area includes: In response to a stacking event within the predetermined region, a first spatial feature of the location of an object within a first region is obtained; wherein the first region includes the predetermined region; Based on the first location spatial feature filtering, the location spatial features of objects within the predetermined area are determined.

3. The object stacking control method according to claim 2, characterized in that, The first location spatial feature includes: first point cloud data; the location spatial feature includes: point cloud blocks; The step of filtering based on the first location spatial feature to determine the location spatial features of objects within the predetermined area includes: The first point cloud data is transformed into the coordinate system corresponding to the predetermined area to determine the second point cloud data; Extract the third point cloud data belonging to the predetermined area from the second point cloud data; The third point cloud data is clustered to determine multiple point cloud blocks; wherein each point cloud block is used to characterize the spatial and positional features of the top corresponding object within the predetermined area.

4. The object stacking control method according to any one of claims 1 to 3, characterized in that, Determining the stacking strategy based on the spatial characteristics of each object includes: The stacking strategy is determined based on the space between each cloud block.

5. The object stacking method according to claim 4, characterized in that, The stacking strategy determined based on the space between each cloud block includes any one of the following: Based on the space between each point cloud block, a first stacking strategy is determined for the next object stacked within the predetermined area; wherein the first stacking strategy is used to characterize the attributes of the next object; Based on the space between each point cloud block and the required size of the subsequent objects, a second stacking strategy for the next object is adjusted; wherein, the second stacking strategy is used to represent adjusting the stacking position of the next object to leave the required space for the subsequent objects.

6. The object stacking control method according to claim 4, characterized in that, The method further includes: The object state within the predetermined area is determined by comparing each point cloud block with a historical point cloud block sequence.

7. The object stacking control method according to claim 6, characterized in that, The determination of the object state within the predetermined area based on the comparison between each point cloud block and the historical point cloud block sequence includes any one of the following: If the average height of each point cloud block is greater than the average height of the top point cloud block sequence in the historical point cloud block sequence, then a new object is determined for the predetermined area, and the historical point cloud block sequence is updated based on the point cloud block corresponding to the new object. If there is no historical point cloud block sequence, then the historical point cloud block sequence is formed based on each of the point cloud blocks; If the average height of each point cloud block is lower than the average height of the top point cloud block sequence in the historical point cloud block sequence, then it is determined that objects will be reduced for the predetermined area, and the historical point cloud block sequence is updated based on the point cloud blocks corresponding to the reduced objects.

8. The object stacking control method according to claim 4, characterized in that, The method further includes: Add the point cloud block corresponding to the new object to the historical point cloud block sequence to determine the predicted point cloud block sequence; Based on the comparison between the predicted point cloud block sequence and the point cloud block sequence formed by each of the point cloud blocks, a detection result is determined for whether the state of the object within the predetermined area is abnormal.

9. The object stacking control method according to claim 8, characterized in that, The determination of the detection result regarding whether the state of an object within the predetermined region is abnormal, based on the comparison between the predicted point cloud block sequence and the point cloud block sequence formed by each of the point cloud blocks, includes: Each point cloud column in the predicted point cloud block sequence is compared with the corresponding point cloud column in the point cloud block sequence. If the height of the corresponding column changes or the number of objects changes, it is determined that the object state in the corresponding column of the point cloud block sequence within the predetermined area is abnormal.

10. An object stacking control device, characterized in that, include: The data acquisition unit is used to acquire the spatial characteristics of objects within a predetermined area; The determining unit is configured to determine a stacking strategy based on the spatial characteristics of each of the objects, and to stack the objects in subsequent orders.

11. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images