Pallet container stack checking and counting method and system based on visual algorithm

Abstract

The invention relates to a visual algorithm-based tray container stack checking and counting method and system. The method comprises the following steps of: obtaining a top surface image, a top surface depth image and four side surface images of a tray container stack; performing target detection or instance segmentation on the top surface image and the side surface image of the pallet container stack to obtain a container detection frame and a pallet detection frame included in the images; calculating the number of containers on the top layer of the container stack based on the container detection frame of the top surface image and the top surface depth image; calculating the number of containers on the non-top layer of the container stack based on the container detection frame of the side image; and calculating the sum of the number of containers on the top layer and the non-top layer of the container stack as the total number of containers of the container stack. According to the technical scheme, the container detection frames in the top surface image and the side surface image of the pallet container stack are obtained through the visual algorithm, then the total number of the containers of the container stack is determined according to the container detection frames, automation of the container counting process is achieved, the labor cost is saved, and meanwhile the detection efficiency and the detection accuracy are improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision algorithms, and in particular to a method and system for counting pallet and container stacks based on vision algorithms. Background Art

[0002] Inventory counting is a necessary process in warehousing and logistics. Inventory counting should be performed at both the inbound and outbound ends of the intermediate warehouses in the production process to ensure that the physical inventory is always consistent with the accounting records. For warehouses with large inbound and outbound volumes and frequent inbound and outbound operations, it is necessary to improve the efficiency of inventory counting while ensuring accuracy.

[0003] Traditional warehouse inventory counting requires manual counting. For warehouses that require a large number of warehousing and outbound operations, manual counting is inefficient, costly and prone to errors.

[0004] Another automatic counting solution using weighing method requires manual verification of the type of materials to be inspected and calculation of the total number based on the weight of each material. This method requires obtaining prior information on the weight of each inventory item in advance, and there will be errors in counting goods with large weight fluctuations in a single box. Summary of the invention

[0005] In order to at least to some extent overcome the problem that the inventory counting method in warehousing logistics in the related art is not only inefficient but also prone to errors, the present application provides a pallet cargo box stack inventory counting method and system based on a visual algorithm.

[0006] The scheme of this application is as follows:

[0007] According to a first aspect of an embodiment of the present application, a method for counting pallet cargo boxes based on a visual algorithm is provided, comprising:

[0008] Obtain the top image, top depth image and four side images of the pallet cargo box stack;

[0009] Perform object detection or instance segmentation on the top image and side image of the pallet cargo box stack to obtain the cargo box detection frame and pallet detection frame contained in the image;

[0010] Calculate the number of containers on the top layer of the container stack based on the container detection frame and the top depth image of the top surface image;

[0011] Count the number of containers on the non-top layer of the container stack based on the container detection frame in the side image;

[0012] Calculate the total number of cartons in the stack by adding the number of cartons on the top and non-top layers of the stack.

[0013] Preferably, calculating the number of cargo boxes on the top layer of the cargo box stack based on the cargo box detection frame of the top surface image and the top surface depth image includes:

[0014] Calculate the depth range of each cargo box detection frame in the top surface image according to the top surface depth image;

[0015] Calculate the average depth value of the cargo box detection frame in the top surface image;

[0016] According to the average depth value, outliers generated by the container detection frames of non-top containers are removed, and the number of remaining container detection frames is used as the number of containers on the top layer of the container stack.

[0017] Preferably, the method further comprises:

[0018] According to the number of cargo box detection frames on the top layer and the top surface depth image, calculate the area occupied by all cargo boxes on the top layer and the area occupied by a single cargo box;

[0019] Get the pallet area;

[0020] Based on the area occupied by all cargo boxes on the top floor, the area occupied by a single cargo box, and the area of ​​the pallet, determine whether there is space on the top floor to place a single cargo box;

[0021] If yes, make sure the top layer of the cargo stack is not full;

[0022] If not, make sure the top layer of the cargo box stack is full;

[0023] Record the status of the top layer of the stack.

[0024] Preferably, calculating the number of cargo boxes on non-top layers of a cargo box stack based on the cargo box detection frame of the side image includes:

[0025] The detection frames of the four side images are layered on the y-axis to determine whether the layering results are consistent;

[0026] If the stratification results are consistent, determine whether the number of layers is greater than 1;

[0027] If the number of layers is greater than 1, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of layers on the non-top layer.

[0028] If the number of layers is not greater than 1, it is determined that the stack of containers has only one layer, and the number of containers on the top layer of the stack of containers is taken as the total number of containers in the stack of containers.

[0029] Preferably, calculating the number of cargo boxes on non-top layers of a cargo box stack based on the cargo box detection frame of the side image further includes:

[0030] If the stratification results are inconsistent, determine whether the difference between the left and right boundaries of the bottom layer detection frame of the four side images and the left and right boundaries of the tray detection frame is within a preset difference threshold;

[0031] If the difference is within the preset threshold, it is determined that the bottom layer of the cargo stack is full, and whether the top layer of the cargo stack is full is determined;

[0032] If the top layer of the cargo box stack is not full, determine whether the layering results of the four side images are consistent after removing the top layer of the cargo box stack;

[0033] If the stratification results of the four side images are consistent, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of non-top layers.

[0034] Preferably, the method further comprises:

[0035] If the difference between the left and right boundaries of the bottom layer detection frame of the cargo stack in the four side images and the left and right boundaries of the pallet detection frame is not within the preset difference threshold, it is determined that the bottom layer of the cargo stack is not full, and it is determined whether the number of layers of one surface in the four side images is 1;

[0036] If the number of layers of one of the four side images is 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

[0037] Preferably, the number of single-layer cargo boxes is calculated according to the single-layer detection frames of the four side images, including:

[0038] Compare whether the number of current single-layer detection frames of two sets of relative side images is consistent;

[0039] If they are consistent, calculate the product of the current single-layer detection frame numbers of the two adjacent side images as the current single-layer cargo box number;

[0040] If only one set of side images has the same number of current single-layer detection frames, any one surface in the relative side images with the same number of current single-layer detection frames is determined as the reference surface;

[0041] Classify the current single-layer detection frame of the detection frame cargo box stack in the reference plane according to shape or area;

[0042] The sum of the number of first-category detection frames on the left side of the reference plane and the number of second-category detection frames on the right side of the reference plane is calculated as the current number of single-layer cargo boxes.

[0043] Preferably, the method further comprises:

[0044] When the layering results of the detection frames of the four side images are inconsistent, and the bottom layer of the cargo stack is full, and the top layer of the cargo stack is full, an error message is given that the quantity cannot be calculated;

[0045] When the layering results of the detection frames of the four side images are inconsistent, the bottom layer of the cargo stack is full, and the top layer of the cargo stack is not full. After removing the top layer of the cargo stack, if the layering results of the four side images are inconsistent, an error message is given that the quantity cannot be calculated.

[0046] The layering results of the detection frames in the four side images are inconsistent, the bottom layer of the cargo box stack is not full, and the number of layers of any of the four side images is not 1, so an error message is given that the quantity cannot be calculated.

[0047] Preferably, the method further comprises:

[0048] The object detection or instance segmentation results of each image are filtered according to the preset confidence threshold to remove the false detection frames caused by texture interference.

[0049] According to a second aspect of an embodiment of the present application, a pallet container stack inventory counting system based on a visual algorithm is provided, comprising:

[0050] Processor and memory;

[0051] The processor and the memory are connected via a communication bus:

[0052] Wherein, the processor is used to call and execute the program stored in the memory;

[0053] The memory is used to store a program, and the program is used to execute at least a pallet cargo box stack inventory counting method based on a visual algorithm as described in any one of the above items.

[0054] The technical solution provided by this application may have the following beneficial effects:

[0055] The visual algorithm-based pallet cargo box stack inventory counting method in the present application includes: obtaining a top surface image, a top surface depth image and four side images of the pallet cargo box stack; performing target detection or instance segmentation on the top surface image and the side image of the pallet cargo box stack to obtain a cargo box detection frame and a pallet detection frame contained in the image; calculating the number of cargo boxes on the top layer of the cargo box stack based on the cargo box detection frame and the top surface depth image of the top surface image; calculating the number of cargo boxes on non-top layers of the cargo box stack based on the cargo box detection frame of the side image; and calculating the sum of the number of cargo boxes on the top layer and non-top layers of the cargo box stack as the total number of cargo boxes in the cargo box stack.

[0056] In this technical solution, a visual algorithm is used to obtain the top surface image of the pallet cargo box stack and the cargo box detection frame in the side image, and then the total number of cargo boxes in the cargo box stack is determined based on the cargo box detection frame. The cargo box counting process is automated, saving labor costs while improving detection efficiency and accuracy.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 It is a flowchart of a pallet cargo box stack counting method based on a visual algorithm provided by an embodiment of the present application;

[0060] Figure 2a It is an incorrect cargo box stacking method provided by an embodiment of the present application;

[0061] Figure 2b This is a correct way of stacking cargo boxes provided by an embodiment of the present application;

[0062] Figure 3a It is another wrong cargo box stacking method provided by an embodiment of the present application;

[0063] Figure 3b This is another correct way of stacking cargo boxes provided by an embodiment of the present application;

[0064] Figure 4a This is another incorrect cargo box stacking method provided by an embodiment of the present application;

[0065] Figure 4b This is another correct way of stacking cargo boxes provided by an embodiment of the present application;

[0066] Figure 5a is a schematic diagram of side A of a cargo box provided by an embodiment of the present application;

[0067] FIG5 is a schematic diagram of side B of a cargo box provided by an embodiment of the present application;

[0068] Figure 6 It is a schematic diagram of a process for calculating the number of cargo boxes on the top layer of a cargo box stack provided by an embodiment of the present application;

[0069] Figure 7 This is a specific flow chart of a pallet cargo box stack inventory counting method based on a visual algorithm provided by an embodiment of the present application;

[0070] Figure 8 It is a structural schematic diagram of a pallet and cargo box stack inventory counting system based on a visual algorithm provided by an embodiment of the present application.

[0071] Reference numerals: processor-21; memory-22. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0073] Embodiment 1

[0074] Figure 1 This is a flowchart of a method for counting pallet cargo boxes based on a visual algorithm provided by an embodiment of the present application, referring to Figure 1 , a pallet and box stack counting method based on visual algorithm, comprising:

[0075] S11: Acquire a top surface image, a top surface depth image, and four side surface images of the pallet cargo box stack;

[0076] In specific practice, four side color images of the pallet cargo box stack are captured by an RGB camera, and one top color image and one top depth image of the pallet cargo box stack are captured by an RGBD camera.

[0077] S12: performing object detection or instance segmentation on the top image and the side image of the pallet cargo box stack to obtain a cargo box detection frame and a pallet detection frame contained in the image;

[0078] Use a deep learning algorithm to perform object detection or instance segmentation on the five color images and calculate the container detection box and pallet detection box contained in each image.

[0079] Preferably, in this embodiment, the target detection or instance segmentation result of each image is filtered according to a preset confidence threshold to remove false detection frames caused by texture interference.

[0080] S13: Calculate the number of cargo boxes on the top layer of the cargo box stack based on the cargo box detection frame of the top surface image and the top surface depth image;

[0081] In this embodiment, the cargo box detection frame of the top surface image and the top surface depth image are used to determine whether the top layer of the cargo box stack is full, and to calculate the number of cargo boxes on the top layer.

[0082] S14: Calculate the number of containers on the non-top layer of the container stack based on the container detection frame of the side image;

[0083] In this embodiment, the cargo box detection frame of the side image is used to calculate the number of cargo boxes on non-top layers of the cargo box stack. It should be noted that the side image is only used to calculate the number of cargo boxes on non-top layers.

[0084] S15: Calculate the sum of the number of cartons on the top layer and the number of cartons on the non-top layer of the cargo stack as the total number of cartons in the cargo stack.

[0085] It should be noted that in actual projects, fancy stacking (hereinafter referred to as "fancy stacking") is often used to ensure the stability of the box stack during transportation. In order to improve the stability and accuracy of the counting algorithm of this technical solution, this technical solution restricts the stacking method of the cargo box when it is implemented.

[0086] Since industrial cameras can only capture the outermost boxes of a stack, the boxes must not be placed alternately vertically and horizontally on the same layer during stacking. An incorrect stacking method is shown in Figure 2(a). Stacking supports stacking boxes vertically on one side and horizontally on the other side. A correct stacking method is shown in Figure 2(b).

[0087] When stacking, the number of cartons is not enough for a full stack or a pressure carton needs to be placed on the top layer to ensure the stability of the stack. For this type of stack with incomplete cartons, it is stipulated that only the top layer can be unfull. An incorrect way of stacking cartons is shown in Figure 3(a), and a correct way of stacking cartons is shown in Figure 3(b).

[0088] For the stacking of different layers of a pallet, it is stipulated that the stacking method of each layer must be consistent. The overall rotation angle can be allowed, but inconsistent stacking methods for different layers of the same pallet or one layer being stacked and another not being stacked are not allowed. An incorrect palletizing method is shown in Figure 4(a), and a correct palletizing method is shown in Figure 4(b).

[0089] When implementing this technical solution, it is first necessary to train two deep neural networks, wherein deep neural network 1 is used to detect the surface of the cargo box in the five image planes of the front, back, left, right, and top of the cargo box stack, and deep neural network 2 is used to classify the detection frame according to shape or area, and distinguish the two groups of opposite sides of the cargo box, which are recorded as side A and side B. The data set used to train network model 1 should include stacks of all types of cargo boxes in the warehouse, and the labels only mark targets with an outermost surface of the visible cargo box that blocks an area of ​​less than 20%. The data set used by network model 2 should clearly distinguish the features of the two types of cargo box surfaces, A and B. One embodiment is: a group of surfaces with printed product names is A, and a group of surfaces with printed barcodes and detailed information is B, such as Figure 5a and Figure 5b Another embodiment is: a uniform identification mark is printed on a group of faces, such as an arrow, a stripe sequence, or a black and white grid, and the faces with the mark are of category A, and the faces without the mark are of category B.

[0090] Reference Figure 6 , based on the container detection frame of the top surface image and the top surface depth image, calculate the number of containers on the top layer of the container stack, including:

[0091] Calculate the depth range of each cargo box detection frame in the top surface image according to the top surface depth image;

[0092] Calculate the average depth value of the cargo box detection frame in the top surface image;

[0093] According to the average depth value, the outliers generated by the container detection frames of non-top containers are removed, and the number of remaining container detection frames is taken as the number of containers on the top layer of the container stack.

[0094] Reference Figure 6 , the method further comprises:

[0095] According to the number of cargo box detection frames on the top layer and the top surface depth image, calculate the area occupied by all cargo boxes on the top layer and the area occupied by a single cargo box;

[0096] Get the pallet area;

[0097] Based on the area occupied by all cargo boxes on the top floor, the area occupied by a single cargo box, and the area of ​​the pallet, determine whether there is space on the top floor to place a single cargo box;

[0098] If yes, make sure the top layer of the cargo stack is not full;

[0099] If not, make sure the top layer of the cargo box stack is full;

[0100] Record the status of the top layer of the stack.

[0101] In this embodiment, based on the cargo box detection frame of the top surface image and the top surface depth image, the number of cargo boxes on the top layer of the cargo box stack is calculated, and it is determined whether the top layer is full. Figure 6 As shown in the figure, firstly, the pixel set with the minimum depth range in the detection frame is calculated according to the depth map, that is, the pixel set corresponding to the upper surface of the cargo box in the detection frame. Then, the average depth value corresponding to the pixel set of each detection frame is calculated, and the outliers generated by the non-top cargo boxes are removed. The number of top cargo boxes is obtained by the number of remaining detection frames. Then, the area of ​​the pixel set in the top detection frame filtered out in the previous step is calculated, and the actual space area of ​​the top surface of the cargo box can be calculated by combining the average depth value. Compare the area of ​​the standard pallet to determine whether another cargo box can be placed on the top layer. If it can be placed, it proves that the top layer is not full.

[0102] Reference Figure 7 , based on the container detection frame of the side image, calculate the number of containers on the non-top layer of the container stack, including:

[0103] The detection frames of the four side images are layered on the y-axis to determine whether the layering results are consistent;

[0104] If the stratification results are consistent, determine whether the number of layers is greater than 1;

[0105] If the number of layers is greater than 1, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of layers on the non-top layer.

[0106] If the number of layers is not greater than 1, it is determined that the stack of containers has only one layer, and the number of containers on the top layer of the stack of containers is taken as the total number of containers in the stack of containers.

[0107] If the stratification results are inconsistent, determine whether the difference between the left and right boundaries of the bottom layer detection frame of the four side images and the left and right boundaries of the tray detection frame is within a preset difference threshold;

[0108] If the difference is within the preset threshold, it is determined that the bottom layer of the cargo stack is full, and whether the top layer of the cargo stack is full is determined;

[0109] If the top layer of the cargo box stack is not full, determine whether the layering results of the four side images are consistent after removing the top layer of the cargo box stack;

[0110] If the stratification results of the four side images are consistent, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of non-top layers.

[0111] If the difference between the left and right boundaries of the bottom layer detection frame of the cargo stack in the four side images and the left and right boundaries of the pallet detection frame is not within the preset difference threshold, it is determined that the bottom layer of the cargo stack is not full, and it is determined whether the number of layers of one surface in the four side images is 1;

[0112] If the number of layers of one of the four side images is 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

[0113] Specifically, the number of single-layer cargo boxes is calculated based on the single-layer detection frames of the four side images, including:

[0114] Compare whether the number of current single-layer detection frames of two sets of relative side images is consistent;

[0115] If they are consistent, calculate the product of the current single-layer detection frame numbers of the two adjacent side images as the current single-layer cargo box number;

[0116] If only one set of side images has the same number of current single-layer detection frames, any one surface in the relative side images with the same number of current single-layer detection frames is determined as the reference surface;

[0117] Classify the current single-layer detection frame of the detection frame cargo box stack in the reference plane according to shape or area;

[0118] The sum of the number of first-category detection frames on the left side of the reference plane and the number of second-category detection frames on the right side of the reference plane is calculated as the current number of single-layer cargo boxes.

[0119] Reference Figure 7 , the method further comprises:

[0120] When the layering results of the detection frames of the four side images are inconsistent, and the bottom layer of the cargo stack is full, and the top layer of the cargo stack is full, an error message is given that the quantity cannot be calculated;

[0121] When the layering results of the detection frames of the four side images are inconsistent, the bottom layer of the cargo stack is full, and the top layer of the cargo stack is not full. After removing the top layer of the cargo stack, if the layering results of the four side images are inconsistent, an error message is given that the quantity cannot be calculated.

[0122] The layering results of the detection frames in the four side images are inconsistent, the bottom layer of the cargo box stack is not full, and the number of layers of any of the four side images is not 1, so an error message is given that the quantity cannot be calculated.

[0123] It should be noted that the detection frames of the four side image planes are layered in the y-axis direction of the pixel coordinates. If the layering results of the four side image planes are inconsistent, it proves that there is a front-to-back parallax caused by the insufficient placement of the cargo boxes. Compare whether the difference between the left and right boundaries of the bottom detection frame and the left and right boundaries of the pallet detection frame in each image plane is within the threshold. If not, it is judged that the bottom cargo box is not full. If the box stack has only one layer, it can be counted directly. If the box stack is more than one layer, the counting condition is not met. For the situation where the bottom layer is full, it is necessary to judge whether the top layer is full. If the top layer is not full, it may be that the top cargo box is out of the field of view and the detection frame is missing, causing a layering error. Remove the top detection frame set to count the cargo boxes.

[0124] If the number of layers on the four side image surfaces is consistent, it means that the stack type meets the counting criteria, and then count for different stacking methods. When the box stack has only one layer, the total number of cargo boxes is equal to the number of top surface cargo box detection frames. When the box stack is more than one layer, compare the number of detection frames on the bottom layer in the two sets of relative image surfaces (front and back and left and right) to see if they are consistent. If they are consistent, it means that the stack type is a single-layer non-flower code or a round shape, and the number of non-top-layer cargo boxes is calculated layer by layer. The calculation formula can be expressed as:

[0125]

[0126] Among them, i is the current layer number.

[0127] Among them, the number of detection frames in each layer of the front image plane and the rear image plane is the same, and the number of detection frames in each layer of the left image plane and the right image plane is the same, and they can be replaced with each other in the formula, that is, the product of the current single-layer detection frame numbers of two adjacent side images is calculated as the current single-layer cargo box quantity.

[0128] If only one group of side images has the same number of current single-layer detection frames, it means that the pallet type is a single-layer flower stack. Since the technical solution pre-sets the pallet type rules, the detection frames of each layer of the four side surfaces in the flower stack must have a group of relative image surfaces of the same number and a group of relative image surfaces of different numbers. Select one of the same group of relative image surfaces as the reference plane F, and use network model 2 to classify the current single-layer detection frame of the cargo box stack in the reference plane according to shape or area. The classification results A and B are bound to the information of each detection frame. It is stipulated that the reference plane F is the main view, and the detection frame image contained in the current single layer of the F surface is classified. The category of p detection frames is A, and the category of q detection frames is B. The positional relationship between the two detection frames is determined, such as p A-type detection frames are to the left of q B-type detection frames. Determine the left and right image surfaces adjacent to the F surface, and the relationship is shown in the following table:

[0129]

[0130] Here, F1 represents the left side surface of the reference plane F, and Fr represents the right side surface of the reference plane F.

[0131] Match the positional relationship between the two surfaces Fl and Fr and the two types of detection frames A and B. For example, if F is the front image surface, then the left adjacent surface Fl of F is the right image surface, and the type A detection frame is matched with Fl; the right adjacent surface Fr of F is the left image surface, and the type B detection frame is matched with Fr.

[0132] Calculate the number of non-top containers layer by layer. The calculation formula can be expressed as:

[0133]

[0134] Among them, i is the current layer number.

[0135] It should be noted that the reason why the category of the detection frame needs to be determined by the reference plane in this embodiment is that Figure 2b As shown, Figure 2b The image surface of is the reference surface F, then through Figure 2b The reference plane F in the figure can get the A-class detection frame on the left and the B-class detection frame on the right. However, when analyzing the detection frame on the left side of the reference plane F, there is not only the B-class detection frame, but also the interference of the edge detection frame, so it is necessary to eliminate the interference of this part of the detection frame and only consider the B-class detection frame on the left side of the reference plane F.

[0136] It should be noted that after calculating the number of non-top-layer containers layer by layer, the number of containers on the non-top layer of the container stack can be summed up to obtain the number of containers on the non-top layer.

[0137] In this technical solution, a visual algorithm is used to obtain the top surface image of the pallet cargo box stack and the cargo box detection frame in the side image, and then the total number of cargo boxes in the cargo box stack is determined based on the cargo box detection frame. The cargo box counting process is automated, saving labor costs while improving detection efficiency and accuracy.

[0138] Embodiment 2

[0139] A pallet and box stack counting system based on visual algorithm, referring to Figure 8 ,include:

[0140] Processor 21 and memory 2;

[0141] The processor 21 and the memory 22 are connected via a communication bus:

[0142] The processor 21 is used to call and execute the program stored in the memory 22;

[0143] The memory 22 is used to store programs, and the programs are used to execute at least a pallet cargo box stack inventory counting method based on a visual algorithm as described in the above embodiments.

[0144] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0145] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0147] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0148] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0150] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0152] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A pallet and container stack counting method based on visual algorithm, characterized in that: include: Obtain the top image, top depth image and four side images of the pallet cargo box stack; Perform object detection or instance segmentation on the top image and side image of the pallet cargo box stack to obtain the cargo box detection frame and pallet detection frame contained in the image; Calculate the number of containers on the top layer of the container stack based on the container detection frame and the top depth image of the top surface image; Count the number of containers on the non-top layer of the container stack based on the container detection frame in the side image; Calculate the total number of cartons in the stack by adding the number of cartons on the top and non-top layers of the stack.

2. The method according to claim 1, characterized in that Based on the container detection frame and top depth image of the top surface image, the number of containers on the top layer of the container stack is calculated, including: Calculate the depth range of each cargo box detection frame in the top surface image according to the top surface depth image; Calculate the average depth value of the cargo box detection frame in the top surface image; According to the average depth value, outliers generated by the container detection frames of non-top containers are removed, and the number of remaining container detection frames is used as the number of containers on the top layer of the container stack.

3. The method according to claim 2, characterized in that The method further comprises: According to the number of cargo box detection frames on the top layer and the top surface depth image, calculate the area occupied by all cargo boxes on the top layer and the area occupied by a single cargo box; Get the pallet area; Based on the area occupied by all cargo boxes on the top floor, the area occupied by a single cargo box, and the area of ​​the pallet, determine whether there is space on the top floor to place a single cargo box; If yes, make sure the top layer of the cargo stack is not full; If not, make sure the top layer of the cargo box stack is full; Record the status of the top layer of the stack.

4. The method according to claim 3, characterized in that Count the number of non-top boxes in the stack based on the box detection frame in the side image, including: The detection frames of the four side images are layered on the y-axis to determine whether the layering results are consistent; If the stratification results are consistent, determine whether the number of layers is greater than 1; If the number of layers is greater than 1, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of layers on the non-top layer. If the number of layers is not greater than 1, it is determined that the stack of containers has only one layer, and the number of containers on the top layer of the stack of containers is taken as the total number of containers in the stack of containers.

5. The method according to claim 4, characterized in that Count the number of containers on the non-top layer of the stack based on the container detection frame in the side image, and also include: If the stratification results are inconsistent, determine whether the difference between the left and right boundaries of the bottom layer detection frame of the four side images and the left and right boundaries of the tray detection frame is within a preset difference threshold; If the difference is within the preset threshold, it is determined that the bottom layer of the cargo stack is full, and whether the top layer of the cargo stack is full is determined; If the top layer of the cargo box stack is not full, determine whether the layering results of the four side images are consistent after removing the top layer of the cargo box stack; If the stratification results of the four side images are consistent, the number of single-layer containers is calculated based on the single-layer detection frames of the four side images, and the number of containers on the non-top layer of the container stack is calculated based on the number of single-layer containers and the number of non-top layers.

6. The method according to claim 5, characterized in that The method further comprises: If the difference between the left and right boundaries of the bottom layer detection frame of the cargo stack in the four side images and the left and right boundaries of the pallet detection frame is not within the preset difference threshold, it is determined that the bottom layer of the cargo stack is not full, and it is determined whether the number of layers of one surface in the four side images is 1; If the number of layers of one of the four side images is 1, it is determined that the container stack has only one layer, and the number of containers on the top layer of the container stack is taken as the total number of containers in the container stack.

7. The method according to any one of claims 4 to 5, characterized in that: The number of single-layer cargo boxes is calculated based on the single-layer detection frames of the four side images, including: Compare whether the number of current single-layer detection frames of two sets of relative side images is consistent; If they are consistent, calculate the product of the current single-layer detection frame numbers of the two adjacent side images as the current single-layer cargo box number; If only one set of side images has the same number of current single-layer detection frames, any one surface in the relative side images with the same number of current single-layer detection frames is determined as the reference surface; Classify the current single-layer detection frame of the detection frame cargo box stack in the reference plane according to shape or area; The sum of the number of first-category detection frames on the left side of the reference plane and the number of second-category detection frames on the right side of the reference plane is calculated as the number of single-layer cargo boxes.

8. The method according to claim 6, characterized in that The method further comprises: When the layering results of the detection frames of the four side images are inconsistent, and the bottom layer of the cargo stack is full, and the top layer of the cargo stack is full, an error message is given that the quantity cannot be calculated; When the layering results of the detection frames of the four side images are inconsistent, the bottom layer of the cargo stack is full, and the top layer of the cargo stack is not full. After removing the top layer of the cargo stack, if the layering results of the four side images are inconsistent, an error message is given that the quantity cannot be calculated. The layering results of the detection frames in the four side images are inconsistent, the bottom layer of the cargo box stack is not full, and the number of layers of any of the four side images is not 1, so an error message is given that the quantity cannot be calculated.

9. The method according to claim 1, characterized in that: The method further comprises: The object detection or instance segmentation results of each image are filtered according to the preset confidence threshold to remove the false detection frames caused by texture interference.

10. A pallet and container stack counting system based on visual algorithm, characterized in that: include: Processor and memory; The processor and the memory are connected via a communication bus: Wherein, the processor is used to call and execute the program stored in the memory; The memory is used to store a program, and the program is used to execute at least a pallet cargo box stack inventory counting method based on a visual algorithm as described in any one of claims 1-9.

Images