A pallet and container stack counting method and system based on visual algorithm

Abstract

The present application relates to a method and system for counting and counting pallet container stacks based on a visual algorithm. The method comprises: obtaining a top surface image, a top surface depth image, and four side images of a pallet container stack; performing target detection or instance segmentation on the top surface image and the side surface image of the pallet container stack to obtain a container detection frame and a pallet detection frame contained in the image; calculating the number of containers on the top layer of the container stack based on the container detection frame and the top surface depth image of the top surface image; calculating the number of containers on the non-top layer of the container stack based on the container detection frame of the side surface image; and calculating the sum of the number of containers on the top layer and the non-top layer of the container stack as the total number of containers in the container stack. In this technical solution, a visual algorithm is used to obtain container detection frames in the top surface image and the side surface image of the pallet container stack, and then the total number of containers in the container stack is determined based on the container detection frames. The container counting process is automated, saving labor costs while improving detection efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision algorithms, and in particular to a method and system for counting pallet and container stacks based on vision algorithms. Background Art

[0002] Inventory counting is a necessary process in warehousing and logistics. Intermediary warehouses in the production process should conduct inventory counts at both inbound and outbound locations to ensure consistency between physical inventory and recorded inventory. For warehouses with high inbound and outbound volumes and frequent traffic, improving inventory counting efficiency while maintaining accuracy is crucial.

[0003] Traditional warehouse inventory counting requires manual counting. For warehouses with a large number of incoming and outgoing goods, manual counting is inefficient, costly, and prone to errors.

[0004] Another automated counting solution, using weighing, requires manual verification of the item type and calculation of the total based on the weight of each item. This method requires prior knowledge of the weight of each item in inventory, and can introduce errors in counting items with large fluctuations in the weight of individual boxes. Summary of the Invention

[0005] In order to at least to some extent overcome the problem that the inventory counting method in warehousing logistics in the related art is not only inefficient but also prone to errors, the present application provides a pallet and cargo box stack inventory counting method and system based on a visual algorithm.

[0006] The scheme of this application is as follows:

[0007] According to a first aspect of an embodiment of the present application, a method for counting pallet and container stacks based on a visual algorithm is provided, comprising:

[0008] Obtain the top image, top depth image and four side images of the pallet container stack;

[0009] Perform object detection or instance segmentation on the top and side images of the pallet cargo box stack to obtain the cargo box detection frame and pallet detection frame contained in the image;

[0010] Calculate the number of containers on the top layer of the container stack based on the container detection frame and top depth image of the top surface image.

[0011] Count the number of containers on the non-top layer of the stack based on the container detection frame in the side image.

[0012] Calculate the total number of cartons in the stack by adding the number of cartons on the top and bottom layers of the stack.

[0013] Preferably, calculating the number of containers on the top layer of the container stack based on the container detection frame of the top surface image and the top surface depth image includes:

[0014] Calculate the depth range of each container detection frame in the top surface image based on the top surface depth image;

[0015] Calculate the average depth value of the cargo box detection box in the top surface image;

[0016] According to the average depth value, outliers generated by the container detection frames of non-top containers are removed, and the number of remaining container detection frames is used as the number of containers on the top layer of the container stack.

[0017] Preferably, the method further comprises:

[0018] Based on the number of cargo box detection frames on the top layer and the top surface depth image, calculate the area occupied by all cargo boxes on the top layer and the area occupied by a single cargo box;

[0019] Get the pallet area;

[0020] Based on the area occupied by all cargo boxes on the top floor, the area occupied by a single cargo box, and the area of the pallet, determine whether there is space on the top floor to accommodate a single cargo box;

[0021] If yes, make sure the top layer of the cargo stack is not full;

[0022] If not, make sure the top layer of the cargo box stack is full;

[0023] Record the status of the top layer of the stack.

[0024] Preferably, calculating the number of containers on non-top layers of a container stack based on the container detection frame in the side image includes:

[0025] The detection frames of the four side images are layered on the y-axis to determine whether the layering results are consistent;

[0026] If the stratification results are consistent, determine whether the number of layers is greater than 1;

[0027] If the number of layers is greater than 1, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of non-top layers.

[0028] If the number of layers is not greater than 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

[0029] Preferably, the method of calculating the number of containers on non-top layers of a container stack based on the container detection frame in the side image further includes:

[0030] If the layering results are inconsistent, determine whether the difference between the left and right boundaries of the bottom layer detection frame of the four side images and the left and right boundaries of the pallet detection frame is within the preset difference threshold;

[0031] If the difference is within the preset threshold, the bottom layer of the stack is determined to be full, and the top layer of the stack is determined to be full.

[0032] If the top layer of the container stack is not full, determine whether the layering results of the four side images are consistent after removing the top layer of the container stack;

[0033] If the stratification results of the four side images are consistent, the number of single-layer cargo boxes is calculated based on the single-layer detection frames of the four side images, and the number of cargo boxes on the non-top layer of the cargo stack is calculated based on the number of single-layer cargo boxes and the number of non-top layers.

[0034] Preferably, the method further comprises:

[0035] If the difference between the left and right boundaries of the bottom layer detection frame of the container stack in the four side images and the left and right boundaries of the pallet detection frame is not within the preset difference threshold, it is determined that the bottom layer of the container stack is not full, and whether there is a surface with a layer number of 1 in the four side images;

[0036] If the layer number of one of the four side images is 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

[0037] Preferably, the number of single-layer cargo boxes is calculated based on the single-layer detection frames of the four side images, including:

[0038] Compare the number of current single-layer detection frames of the two sets of relative side images to see if they are consistent;

[0039] If they are consistent, calculate the product of the current single-layer detection frame numbers of the two adjacent side images as the current single-layer cargo box number;

[0040] If only one set of side images has the same number of current single-layer detection frames, any one surface in the opposite side images with the same number of current single-layer detection frames is determined as the reference surface;

[0041] Classify the current single-layer detection frame of the container stack in the reference plane according to shape or area;

[0042] The sum of the number of first-category detection frames on the left side of the reference plane and the number of second-category detection frames on the right side of the reference plane is calculated as the current number of single-layer cargo boxes.

[0043] Preferably, the method further comprises:

[0044] If the layering results of the detection frames of the four side images are inconsistent, and the bottom layer of the stack is full, and the top layer of the stack is also full, an error message indicating that the quantity cannot be calculated is displayed;

[0045] If the layering results of the detection frames of the four side images are inconsistent, the bottom layer of the stack is full, but the top layer is not, and after removing the top layer, the layering results of the four side images are inconsistent, an error message "Unable to calculate quantity" is displayed;

[0046] The layering results of the detection frames in the four side images are inconsistent, the bottom layer of the cargo box stack is not full, and the number of layers in any of the four side images is not 1, resulting in an error message indicating that the quantity cannot be calculated.

[0047] Preferably, the method further comprises:

[0048] The object detection or instance segmentation results of each image are filtered according to a preset confidence threshold to remove false detection frames caused by texture interference.

[0049] According to a second aspect of an embodiment of the present application, a pallet and container stack inventory counting system based on a visual algorithm is provided, comprising:

[0050] processor and memory;

[0051] The processor and the memory are connected via a communication bus:

[0052] The processor is configured to call and execute the program stored in the memory;

[0053] The memory is used to store a program, and the program is used to at least execute a pallet and container stack inventory counting method based on a visual algorithm as described in any one of the above.

[0054] The technical solution provided by this application may have the following beneficial effects:

[0055] The visual algorithm-based pallet container stack inventory counting method in the present application includes: obtaining a top surface image, a top surface depth image and four side images of the pallet container stack; performing target detection or instance segmentation on the top surface image and the side surface image of the pallet container stack to obtain a container detection frame and a pallet detection frame contained in the image; calculating the number of containers on the top layer of the container stack based on the container detection frame and the top surface depth image of the top surface image; calculating the number of containers on non-top layers of the container stack based on the container detection frame of the side surface image; and calculating the sum of the number of containers on the top layer and non-top layers of the container stack as the total number of containers in the container stack.

[0056] In this technical solution, a visual algorithm is used to obtain the cargo box detection frame in the top surface image and the side image of the pallet cargo box stack, and then the total number of cargo boxes in the cargo box stack is determined based on the cargo box detection frame. The cargo box counting process is automated, saving labor costs while improving detection efficiency and accuracy.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 This is a flowchart of a method for counting pallets and container stacks based on a visual algorithm provided by an embodiment of the present application;

[0060] Figure 2a This is an incorrect way of stacking cargo boxes provided by one embodiment of the present application;

[0061] Figure 2b This is a correct way of stacking cargo boxes provided by one embodiment of the present application;

[0062] Figure 3a This is another incorrect cargo box stacking method provided by an embodiment of the present application;

[0063] Figure 3b This is another correct way of stacking cargo boxes provided by an embodiment of the present application;

[0064] Figure 4a This is another incorrect way of stacking cargo boxes provided by an embodiment of the present application;

[0065] Figure 4b This is another correct way of stacking cargo boxes provided by an embodiment of the present application;

[0066] Figure 5a This is a schematic diagram of side A of a cargo box provided by one embodiment of the present application;

[0067] FIG5 is a schematic diagram of side B of a cargo box provided by one embodiment of the present application;

[0068] Figure 6 This is a schematic diagram of a process for calculating the number of cargo boxes on the top layer of a cargo box stack provided by an embodiment of the present application;

[0069] Figure 7 This is a schematic diagram of a specific process of a pallet and container stack inventory counting method based on a visual algorithm provided by an embodiment of the present application;

[0070] Figure 8 This is a structural diagram of a pallet and container stack inventory counting system based on a visual algorithm provided by an embodiment of the present application.

[0071] Reference numerals: processor-21; memory-22. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0073] Example 1

[0074] Figure 1 This is a flowchart of a method for counting pallets and boxes based on a visual algorithm provided by an embodiment of the present application, with reference to Figure 1 , a pallet and container stack counting method based on visual algorithm, comprising:

[0075] S11: Acquire a top surface image, a top surface depth image, and four side surface images of the pallet container stack;

[0076] In specific practice, four side color images of the pallet container stack are captured by an RGB camera, and one top color image and one top depth image of the pallet container stack are captured by an RGBD camera.

[0077] S12: Performing object detection or instance segmentation on the top image and the side image of the pallet cargo box stack to obtain a cargo box detection frame and a pallet detection frame contained in the image;

[0078] Use a deep learning algorithm to perform object detection or instance segmentation on five color images and calculate the container and pallet detection boxes contained in each image.

[0079] Preferably, in this embodiment, the target detection or instance segmentation results of each image are filtered according to a preset confidence threshold to remove false detection frames caused by texture interference.

[0080] S13: Calculate the number of containers on the top layer of the container stack based on the container detection frame of the top surface image and the top surface depth image;

[0081] In this embodiment, the container detection frame of the top surface image and the top surface depth image are used to determine whether the top layer of the container stack is full and to calculate the number of containers on the top layer.

[0082] S14: Calculate the number of containers on the non-top layer of the container stack based on the container detection frame in the side image;

[0083] In this embodiment, the container detection frame of the side image is used to calculate the number of containers on non-top layers of the container stack. It should be noted that the side image is only used to calculate the number of containers on non-top layers.

[0084] S15: Calculate the sum of the number of cartons on the top layer and the number of cartons on the non-top layer of the cartons stack as the total number of cartons in the cartons stack.

[0085] It should be noted that in actual projects, fancy stacking (hereinafter referred to as "fancy stacking") is often used to ensure the stability of the box stack during transportation. To improve the stability and accuracy of the counting algorithm of this technical solution, this technical solution imposes restrictions on the stacking method of the cargo boxes during implementation.

[0086] Since industrial cameras can only capture the outermost boxes of a stack, the boxes cannot be stacked alternately vertically and horizontally on the same layer during stacking. An incorrect stacking method is shown in Figure 2(a). Stacking supports stacking boxes vertically on one side and horizontally on the other side. A correct stacking method is shown in Figure 2(b).

[0087] Because the number of cargo boxes is not enough for a full stack or a pressure box needs to be placed on the top layer to ensure the stability of the stack, for this type of stack with incomplete cargo boxes, it is stipulated that only the top layer can be unfull. An incorrect cargo box stacking method is shown in Figure 3(a), and a correct cargo box stacking method is shown in Figure 3(b).

[0088] For the stacking of different layers of a pallet, the stacking method of each layer must be consistent. The overall rotation angle can be used, but inconsistent stacking methods on different layers of the same pallet or one layer with no stacking method is not allowed. An incorrect palletizing method is shown in Figure 4(a), and a correct palletizing method is shown in Figure 4(b).

[0089] When implementing this technical solution, it is first necessary to train two deep neural networks, wherein deep neural network 1 is used to detect the surface of the cargo box in the five image planes of the front, back, left, right, and top of the cargo box stack, and deep neural network 2 is used to classify the detection frame according to shape or area, and distinguish the two sets of opposite sides of the cargo box, which are recorded as side A and side B. The data set used to train network model 1 should include stacks of all types of cargo boxes in the warehouse, and the labels only mark targets with an outermost surface of the visible cargo box with an occlusion area of less than 20%. The data set used by network model 2 should clearly distinguish the features of the two types of cargo box surfaces, A and B. One embodiment is: a group of surfaces with printed product names is A, and a group of surfaces with printed barcodes and detailed information is B, such as Figure 5a and Figure 5b Another embodiment is to print a uniform identification mark on a group of surfaces, such as an arrow, a stripe sequence, or a black and white grid, and the surfaces with the mark are classified as category A, and the surfaces without the mark are classified as category B.

[0090] Reference Figure 6 , based on the container detection frame and top depth image of the top surface image, calculate the number of containers on the top layer of the container stack, including:

[0091] Calculate the depth range of each container detection frame in the top surface image based on the top surface depth image;

[0092] Calculate the average depth value of the cargo box detection box in the top surface image;

[0093] Based on the average depth value, the outliers generated by the cargo box detection frames of non-top cargo boxes are removed, and the number of remaining cargo box detection frames is used as the number of cargo boxes on the top layer of the cargo stack.

[0094] Reference Figure 6 , the method further comprises:

[0095] Based on the number of cargo box detection frames on the top layer and the top surface depth image, calculate the area occupied by all cargo boxes on the top layer and the area occupied by a single cargo box;

[0096] Get the pallet area;

[0097] Based on the area occupied by all cargo boxes on the top floor, the area occupied by a single cargo box, and the area of the pallet, determine whether there is space on the top floor to accommodate a single cargo box;

[0098] If yes, make sure the top layer of the cargo stack is not full;

[0099] If not, make sure the top layer of the cargo box stack is full;

[0100] Record the status of the top layer of the stack.

[0101] In this embodiment, based on the cargo box detection frame of the top surface image and the top surface depth image, the number of cargo boxes on the top layer of the cargo box stack is calculated, and it is determined whether the top layer is full of cargo boxes. Figure 6 As shown in the figure, the depth map is first used to calculate the pixel set within the minimum depth range of the detection frame, that is, the pixel set corresponding to the top surface of the cargo box within the detection frame. The average depth value corresponding to the pixel set of each detection frame is then calculated, and outliers generated by non-top-level cargo boxes are removed. The number of top-level cargo boxes is obtained by the number of remaining detection frames. The area of the pixel set in the top-level detection frame filtered out in the previous step is then calculated. Combined with the average depth value, the actual spatial area of the cargo box top surface can be calculated. The area of the standard pallet is compared to determine whether another cargo box can be placed on the top layer. If it can, the top layer is not full.

[0102] Reference Figure 7 , based on the container detection frame in the side image, calculate the number of containers on the non-top layer of the container stack, including:

[0103] The detection frames of the four side images are layered on the y-axis to determine whether the layering results are consistent;

[0104] If the stratification results are consistent, determine whether the number of layers is greater than 1;

[0105] If the number of layers is greater than 1, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of non-top layers.

[0106] If the number of layers is not greater than 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

[0107] If the layering results are inconsistent, determine whether the difference between the left and right boundaries of the bottom layer detection frame of the four side images and the left and right boundaries of the pallet detection frame is within the preset difference threshold;

[0108] If the difference is within the preset threshold, the bottom layer of the stack is determined to be full, and the top layer of the stack is determined to be full.

[0109] If the top layer of the container stack is not full, determine whether the layering results of the four side images are consistent after removing the top layer of the container stack;

[0110] If the stratification results of the four side images are consistent, the number of single-layer cargo boxes is calculated based on the single-layer detection frames of the four side images, and the number of cargo boxes on the non-top layer of the cargo stack is calculated based on the number of single-layer cargo boxes and the number of non-top layers.

[0111] If the difference between the left and right boundaries of the bottom layer detection frame of the container stack in the four side images and the left and right boundaries of the pallet detection frame is not within the preset difference threshold, it is determined that the bottom layer of the container stack is not full, and whether there is a surface with a layer number of 1 in the four side images;

[0112] If the layer number of one of the four side images is 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

[0113] Specifically, the number of single-layer cargo boxes is calculated based on the single-layer detection frames of the four side images, including:

[0114] Compare the number of current single-layer detection frames of the two sets of relative side images to see if they are consistent;

[0115] If they are consistent, calculate the product of the current single-layer detection frame numbers of the two adjacent side images as the current single-layer cargo box number;

[0116] If only one set of side images has the same number of current single-layer detection frames, any one surface in the opposite side images with the same number of current single-layer detection frames is determined as the reference surface;

[0117] Classify the current single-layer detection frame of the container stack in the reference plane according to shape or area;

[0118] The sum of the number of first-category detection frames on the left side of the reference plane and the number of second-category detection frames on the right side of the reference plane is calculated as the current number of single-layer cargo boxes.

[0119] Reference Figure 7 , the method further comprises:

[0120] If the layering results of the detection frames of the four side images are inconsistent, and the bottom layer of the stack is full, and the top layer of the stack is also full, an error message indicating that the quantity cannot be calculated is displayed;

[0121] If the layering results of the detection frames of the four side images are inconsistent, the bottom layer of the stack is full, but the top layer is not, and after removing the top layer, the layering results of the four side images are inconsistent, an error message "Unable to calculate quantity" is displayed;

[0122] The layering results of the detection frames in the four side images are inconsistent, the bottom layer of the cargo box stack is not full, and the number of layers in any of the four side images is not 1, resulting in an error message indicating that the quantity cannot be calculated.

[0123] It should be noted that the detection frames of the four side image planes are layered in the y-axis direction of the pixel coordinates. If the layering results of the four side image planes are inconsistent, it proves that there is a front-to-back parallax caused by the insufficient placement of the cargo boxes. Compare whether the difference between the left and right boundaries of the bottom detection frame and the left and right boundaries of the pallet detection frame in each image plane is within the threshold. If not, it is judged that the bottom cargo box is not full. If the box stack has only one layer, it can be counted directly. If the box stack is more than one layer, the counting condition is not met. For the case where the bottom layer is full of goods, it is necessary to judge whether the top layer is full of goods. If the top layer is not full, it may be that the top cargo box is out of the field of view and the detection frame is missing, causing a layering error. Remove the top detection frame set to count the cargo boxes.

[0124] If the number of layers in the four side image planes is consistent, it means that the stack type meets the counting criteria, and then count according to different stacking methods. When the box stack has only one layer, the total number of cargo boxes is equal to the number of cargo box detection frames on the top surface. When the box stack is larger than one layer, compare the number of detection frames on the bottom layer in the two sets of relative image planes (front and back and left and right) to see if they are consistent. If they are consistent, it means that the stack type is a single layer, non-patterned or rounded shape, and the number of non-top cargo boxes is calculated layer by layer. The calculation formula can be expressed as:

[0125]

[0126] Where i is the current layer number.

[0127] The number of detection frames in each layer of the front and rear image planes is the same, and the number of detection frames in each layer of the left and right image planes is the same. They can be replaced with each other in the formula, that is, the product of the current single-layer detection frame numbers of the two adjacent side images is calculated as the current single-layer cargo box quantity.

[0128] If there is only one set of side images with the same number of current single-layer detection frames, it means that the stack type is a single-layer flower stack. Since the technical solution pre-sets the stack type rules, the detection frames of each layer of the four side surfaces in the flower stack type must have a set of relative image surfaces of the same number and a set of relative image surfaces of different numbers. Select one of the set of relative image surfaces of the same number as the reference plane F, and use the network model 2 to classify the current single-layer detection frame of the cargo box stack in the reference plane according to shape or area. The classification results A and B are bound to the information of each detection frame. It is stipulated that the reference plane F is the main view, and the detection frame images contained in the current single layer of the F surface are classified. The category of p detection frames is A, and the category of q detection frames is B. The positional relationship between the two detection frames is determined, such as p type A detection frames are to the left of q type B detection frames. Determine the left and right image surfaces adjacent to the F surface, and the relationship is shown in the following table:

[0129]

[0130] Here, F1 represents the left side of the reference plane F, and Fr represents the right side of the reference plane F.

[0131] Match the positional relationship between the two faces Fl and Fr and the two types of detection frames A and B. For example, if F is the front image face, then the left adjacent face Fl of F is the right image face, and the type A detection frame is matched with Fl; the right adjacent face Fr of F is the left image face, and the type B detection frame is matched with Fr.

[0132] Calculate the number of non-top-level containers layer by layer. The calculation formula can be expressed as:

[0133]

[0134] Where i is the current layer number.

[0135] It should be noted that the reason why the category of the detection frame needs to be determined by the reference plane in this embodiment is that Figure 2b As shown, Figure 2b The image surface is the reference surface F, then through Figure 2b The reference plane F in the image can be used to obtain the Class A detection frame on the left and the Class B detection frame on the right. However, when analyzing the detection frames on the left side of the reference plane F, not only the Class B detection frame is included, but also the interference of the detection frames at the edge. Therefore, it is necessary to eliminate the interference of these detection frames and only consider the Class B detection frame on the left side of the reference plane F.

[0136] It should be noted that after calculating the number of non-top-layer containers layer by layer, the number of containers on the non-top layer of the container stack can be summed up to obtain the number of containers on the non-top layer of the container stack.

[0137] In this technical solution, a visual algorithm is used to obtain the cargo box detection frame in the top surface image and the side image of the pallet cargo box stack, and then the total number of cargo boxes in the cargo box stack is determined based on the cargo box detection frame. The cargo box counting process is automated, saving labor costs while improving detection efficiency and accuracy.

[0138] Example 2

[0139] A pallet and container stack counting system based on visual algorithm, referring to Figure 8 ,include:

[0140] Processor 21 and memory 2;

[0141] The processor 21 and the memory 22 are connected via a communication bus:

[0142] The processor 21 is used to call and execute the program stored in the memory 22;

[0143] The memory 22 is used to store a program, and the program is used to at least execute a pallet and container stack inventory counting method based on a visual algorithm as described in the above embodiment.

[0144] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0145] It should be noted that, in the description of this application, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0147] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0148] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0149] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0150] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0151] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0152] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A pallet and container stack counting method based on visual algorithm, characterized in that: include: Obtain the top image, top depth image and four side images of the pallet container stack; Perform object detection or instance segmentation on the top and side images of the pallet cargo box stack to obtain the cargo box detection frame and pallet detection frame contained in the image; Calculate the number of containers on the top layer of the container stack based on the container detection frame and top depth image of the top surface image. Count the number of containers on the non-top layer of the stack based on the container detection frame in the side image. Calculate the sum of the number of cartons on the top and non-top layers of the stack as the total number of cartons in the stack; Calculate the number of containers on the top layer of the stack based on the container detection frame and top depth image of the top surface image, including: Calculate the depth range of each container detection frame in the top surface image based on the top surface depth image; Calculate the average depth value of the cargo box detection box in the top surface image; According to the average depth value, outliers generated by the container detection frames of non-top containers are removed, and the number of remaining container detection frames is used as the number of containers on the top layer of the container stack; The method further comprises: Based on the number of cargo box detection frames on the top layer and the top surface depth image, calculate the area occupied by all cargo boxes on the top layer and the area occupied by a single cargo box; Get the pallet area; Based on the area occupied by all cargo boxes on the top floor, the area occupied by a single cargo box, and the area of the pallet, determine whether there is space on the top floor to accommodate a single cargo box; If yes, make sure the top layer of the cargo stack is not full; If not, make sure the top layer of the cargo box stack is full; Record the status of the top layer of the container stack; Count the number of containers on the non-top layer of the stack based on the container detection frame in the side image, including: The detection frames of the four side images are layered on the y-axis to determine whether the layering results are consistent; If the stratification results are consistent, determine whether the number of layers is greater than 1; If the number of layers is greater than 1, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of non-top layers. If the number of layers is not greater than 1, the stack is determined to have only one layer, and the number of boxes on the top layer of the stack is taken as the total number of boxes in the stack; Calculate the number of single-layer cargo boxes based on the single-layer detection frames of the four side images, including: Compare the number of current single-layer detection frames of the two sets of relative side images to see if they are consistent; If they are consistent, calculate the product of the current single-layer detection frame numbers of the two adjacent side images as the current single-layer cargo box number; If only one set of side images has the same number of current single-layer detection frames, any one surface in the opposite side images with the same number of current single-layer detection frames is determined as the reference surface; Classify the current single-layer detection frame of the container stack in the reference plane according to shape or area; The sum of the number of first-category detection frames on the left side of the reference plane and the number of second-category detection frames on the right side of the reference plane is calculated as the number of single-layer cargo boxes.

2. The method according to claim 1, characterized in that Detecting the container frame based on the side image and counting the number of containers on the non-top layer of the stack also includes: If the layering results are inconsistent, determine whether the difference between the left and right boundaries of the bottom layer detection frame of the four side images and the left and right boundaries of the pallet detection frame is within the preset difference threshold; If the difference is within the preset threshold, the bottom layer of the stack is determined to be full, and the top layer of the stack is determined to be full. If the top layer of the container stack is not full, determine whether the layering results of the four side images are consistent after removing the top layer of the container stack; If the stratification results of the four side images are consistent, the number of single-layer cargo boxes is calculated based on the single-layer detection frames of the four side images, and the number of cargo boxes on the non-top layer of the cargo stack is calculated based on the number of single-layer cargo boxes and the number of non-top layers.

3. The method according to claim 2, characterized in that The method further comprises: If the difference between the left and right boundaries of the bottom layer detection frame of the container stack in the four side images and the left and right boundaries of the pallet detection frame is not within the preset difference threshold, it is determined that the bottom layer of the container stack is not full, and whether there is a surface with a layer number of 1 in the four side images; If the layer number of one of the four side images is 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

4. The method according to claim 3, characterized in that The method further comprises: If the layering results of the detection frames of the four side images are inconsistent, and the bottom layer of the stack is full, and the top layer of the stack is also full, an error message indicating that the quantity cannot be calculated is displayed; If the layering results of the detection frames of the four side images are inconsistent, the bottom layer of the stack is full, but the top layer is not, and after removing the top layer, the layering results of the four side images are inconsistent, an error message "Unable to calculate quantity" is displayed; The layering results of the detection frames in the four side images are inconsistent, the bottom layer of the cargo box stack is not full, and the number of layers in any of the four side images is not 1, resulting in an error message indicating that the quantity cannot be calculated.

5. The method according to claim 1, wherein The method further comprises: The object detection or instance segmentation results of each image are filtered according to a preset confidence threshold to remove false detection frames caused by texture interference.

6. A pallet and container stack counting system based on visual algorithms, characterized in that: include: processor and memory; The processor and the memory are connected via a communication bus: The processor is configured to call and execute the program stored in the memory; The memory is used to store a program, and the program is used to at least execute the pallet and container stack inventory counting method based on a visual algorithm as described in any one of claims 1 to 5.

Images