Cargo stacking quality detection method, device and system based on machine vision detection

Abstract

The application provides a kind of based on machine vision detection's cargo stacking quality detection method, equipment and system, it is related to cargo storage technical field.The method comprises: in the process that stacking mechanism executes cargo stacking operation, whenever stacking mechanism completes a layer of cargo stacking operation, the current cargo stacking picture is acquired;Based on cargo stacking picture, the position coordinates of each cargo are determined, and based on position coordinates, whether the cargo stacking quality in cargo stacking picture is qualified is detected;If the cargo stacking quality is qualified, then control stacking mechanism continues to execute the cargo stacking operation of next layer, until stacking mechanism completes cargo stacking operation.The application can improve the detection precision of stacking quality.

Description

Technical Field

[0001] This invention relates to the field of cargo storage technology, and in particular to a method, equipment and system for cargo palletizing quality inspection based on machine vision detection. Background Technology

[0002] In the field of automated warehousing and logistics, automated palletizing technology has been widely used. Palletizing systems typically rely on robotic arms and preset path programs, combined with vision sensors to identify and locate boxed goods in order to achieve automated stacking.

[0003] During the operation of a palletizing system, problems such as misalignment, tilting, or uneven spacing of goods after stacking may occur due to mechanical vibration, positioning errors, or external interference. If these quality problems are not detected and corrected in time, they will directly affect the stability of the stack, the utilization rate of storage space, and even lead to the risk of collapse.

[0004] In most related technologies, palletizing quality is inspected manually by visual inspection (e.g., neatness, misalignment, tilting, etc.). However, manual visual inspection is not only inefficient, but also subject to factors such as the experience and attention of the personnel, resulting in strong subjectivity, poor consistency, and a high risk of missed inspections or misjudgments. Summary of the Invention

[0005] This invention provides a method, equipment, and system for inspecting the quality of goods palletizing based on machine vision, in order to solve the problems of strong subjectivity, poor consistency, missed detection, or misjudgment that exist in manual inspection of palletizing quality during the goods palletizing process.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting the quality of goods palletizing based on machine vision inspection, comprising:

[0007] During the palletizing operation, the palletizing mechanism acquires a picture of the current goods after completing one layer of goods placement.

[0008] Based on the cargo stacking image, determine the position coordinates of each cargo, and based on the position coordinates, detect whether the cargo stacking quality in the cargo stacking image is qualified.

[0009] If the goods are stacked to the required quality, the palletizing mechanism is controlled to continue stacking the next layer of goods until the palletizing mechanism completes the goods stacking operation.

[0010] In one possible implementation, detecting whether the cargo stacking quality in the cargo stacking image is qualified based on the location coordinates includes:

[0011] Based on the location coordinates of each item, group the items by row or column to obtain at least one group of items;

[0012] For each group of goods, based on the position coordinates of the goods in that group, check whether the placement spacing and alignment of the goods in that group are up to standard;

[0013] If the spacing and alignment of the goods in this group are both up to standard, then the stacking quality of this group of goods is deemed to be up to standard.

[0014] If the stacking quality of all groups of goods in the stacking image is qualified, then the stacking quality of the goods in the stacking image is determined to be qualified.

[0015] In one possible implementation, the position coordinates of each cargo include: the x-coordinate and y-coordinate of each cargo;

[0016] Based on the location coordinates of each item, the items are grouped by row to obtain at least one group of items, including:

[0017] For each ungrouped item, calculate the difference between its ordinate and the ordinates of the remaining ungrouped items.

[0018] The remaining ungrouped goods whose coordinate difference is less than the set difference are grouped together with the ungrouped goods.

[0019] By iterating through all the goods in the image, at least one set of goods is obtained.

[0020] In one possible implementation, the step of detecting whether the placement spacing of each group of goods is acceptable based on the position coordinates of that group includes:

[0021] For each group of goods, arrange them in order according to the x-coordinate of each item in that group to obtain the goods sequence;

[0022] In the cargo sequence, the difference between the x-coordinates of each pair of adjacent cargoes is calculated to obtain multiple differences, and the variance corresponding to the multiple differences is calculated.

[0023] If the variance corresponding to the multiple differences is less than the first set variance threshold, then the placement spacing of the group of goods is determined to be qualified.

[0024] In one possible implementation, the step of detecting whether the alignment of each group of goods is acceptable based on the position coordinates of that group includes:

[0025] For each group of goods, the variance value corresponding to the ordinate is calculated based on the ordinate of each goods in the group.

[0026] If the variance value corresponding to the vertical axis is less than the second set variance threshold, then the alignment of the group of goods is determined to be qualified.

[0027] In one possible implementation, the method for determining the first set variance threshold includes:

[0028] Calculate the standard deviation and mean of the multiple differences;

[0029] The ratio of the standard deviation to the mean is determined as the first set variance threshold.

[0030] In one possible implementation, after determining whether the placement spacing and alignment of each group of goods are acceptable based on the position coordinates of that group, the method further includes:

[0031] If the spacing or alignment of the goods in this group is not up to standard, the target goods that are not up to standard will be determined based on the position coordinates of each goods in the group.

[0032] Based on the location coordinates of the target goods, an abnormal stacking information is sent to alert the user.

[0033] In one possible implementation, for each group of goods, if the spacing or alignment of the goods in that group is not up to standard, then based on the position coordinates of each item in that group, the target goods with substandard stacking quality are determined, including:

[0034] For each group of goods, if the spacing between the goods in the group is not up to standard, then based on the difference between the horizontal coordinates of every two adjacent goods in the sequence of goods in the group, at least one difference exceeding the set difference range is determined, and the goods corresponding to the at least one difference are determined as target goods with unqualified stacking quality.

[0035] If the alignment of the group of goods is not up to standard, then based on the ordinate of each item in the group, at least one ordinate that exceeds the set coordinate range is determined, and the item corresponding to the at least one ordinate is determined as the target item with unqualified stacking quality.

[0036] In a second aspect, embodiments of the present invention provide a control device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0037] Thirdly, embodiments of the present invention provide a cargo palletizing system, characterized in that it includes: a palletizing mechanism, an image acquisition device, and a control device as described in the second aspect.

[0038] Compared to existing technologies, the embodiments of the present invention determine the position coordinates of each item by using images of the goods being stacked, and then perform stacking quality inspection based on the position coordinates. This can eliminate the differences in subjective human judgment and avoid missed detections or misjudgments.

[0039] Furthermore, in each layer of stacking operation performed in this embodiment of the invention, a stacking quality inspection is performed. The quality inspection action can be embedded into the cyclical process of the stacking operation, so that the inspection node is strictly synchronized with the completion time of each layer of stacking, and the stacking quality is detected and fed back in real time. This not only avoids the problems of lag and missing layers that may exist in manual inspection, but also prevents unqualified layers from being covered or accumulated, thereby avoiding the risk of overall stacking stability decline or even collapse caused by bottom layer misalignment or tilting from the source. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the palletizing mechanism provided in an embodiment of the present invention;

[0041] Figure 2 This is a flowchart illustrating the implementation of a machine vision-based method for detecting the quality of palletizing goods, according to an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the structure of a palletizing platform provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of a goods stacking image provided in an embodiment of the present invention;

[0044] Figure 5 This is a flowchart illustrating the implementation of detecting whether the stacking quality is qualified according to an embodiment of the present invention;

[0045] Figure 6 This is a top view of the cargo stacking sequence of the first layer provided in an embodiment of the present invention;

[0046] Figure 7 This is a top view of the cargo stacking sequence of the second layer provided in an embodiment of the present invention;

[0047] Figure 8 This is a flowchart illustrating the implementation of a cargo palletizing quality inspection method based on machine vision detection, provided in another embodiment of the present invention.

[0048] Figure 9 This is a schematic diagram of the structure of a palletizing system provided in an embodiment of the present invention;

[0049] Figure 10 This is a schematic diagram of a cargo palletizing quality inspection device based on machine vision detection provided in an embodiment of the present invention. Detailed Implementation

[0050] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0051] First, a brief overview of the palletizing process will be provided using the palletizing mechanism as an example. (See also...) Figure 1 The palletizing mechanism mainly consists of a palletizer and a palletizing platform. During the palletizing operation, the palletizer first places the goods onto the palletizing platform according to a pre-set sequence. Then, the palletizer moves the stacked goods, along with the palletizing platform, to a storage compartment such as a truck bed or container. Subsequently, by controlling the retraction of the palletizing platform and using blocking devices to prevent the goods from following the platform, the goods on the top surface of the platform can fall into the storage compartment, completing one layer of stacking. This process is repeated, layer by layer, until all goods are stacked, completing the palletizing operation.

[0052] It should be noted that only one layer of goods needs to be stacked on the palletizing platform. The palletizer controls the retraction of the palletizing platform to smoothly drop the layer of goods into the storage bin, thus completing the stacking operation for that layer. Inside the storage bin, each layer of goods is stacked on top of the previous layer, and the palletizing operation is completed by stacking layers one by one.

[0053] The order of goods stacking for each layer can be determined based on a combination of factors, including the dimensions of the palletizing platform, the dimensions of the goods, palletizing stability, and the number of goods stacked. Specifically, after determining the dimensions of the palletizing platform and the goods, an enumeration method can be used to determine all possible stacking sequences. Next, sequences with excessively large spacing between adjacent goods are eliminated to ensure palletizing stability. Finally, the sequence with the largest number of goods stacked is determined as the final stacking sequence.

[0054] It is understandable that the greater the spacing between two adjacent goods, the worse the stacking stability. As for the number of stacking layers (i.e., the number of stacked layers of goods), it can be determined based on the quantity of goods to be stacked and the quantity of goods stacked in each layer. Here, it should be clarified that in this embodiment of the invention, the goods are limited to regular shapes such as cuboids or cubes.

[0055] During the palletizing process, the quality of palletizing (such as neatness, misalignment, tilting, etc.) is mostly checked manually in the storage warehouse. However, manual visual inspection is not only inefficient, but also subject to factors such as the experience and attention of the personnel, resulting in strong subjectivity, poor consistency, and easy omissions or misjudgments.

[0056] To address the issues of strong subjectivity, poor consistency, and susceptibility to missed detections or misjudgments in manual palletizing quality inspection, and to improve the accuracy of palletizing quality inspection, this invention, in the process of goods palletizing, after each layer of goods is completed, obtains the current goods stacking image to determine the position coordinates of each item, and based on the position coordinates of each item, detects whether the goods stacking quality is qualified, thereby eliminating the differences in subjective human judgment and avoiding missed detections or misjudgments.

[0057] See Figure 2 The document illustrates a flowchart of the implementation of a machine vision-based cargo palletizing quality inspection method provided in an embodiment of the present invention, detailed below:

[0058] Step 201: During the palletizing operation, the palletizing mechanism acquires a picture of the current goods after completing one layer of goods stacking.

[0059] As described above, the palletizing platform places only one layer of goods at a time. Once the palletizer drops one layer of goods from the palletizing platform into the goods storage bin, it is determined that the palletizing mechanism has completed the stacking operation of one layer of goods.

[0060] In this embodiment of the invention, after the palletizing mechanism has completed the stacking of one layer of goods, an image acquisition device can be used to acquire images of the goods stacked in the storage compartment for subsequent stacking quality inspection.

[0061] See Figure 3 The palletizing platform is connected to the palletizer via a connecting device on one side, and a lifting device is also installed on this side, on which an image acquisition device can be mounted. When the palletizer controls the retraction of the palletizing platform and the goods smoothly fall into the goods storage compartment, the lifting device can be controlled to descend vertically, allowing the image acquisition device on the lifting device to capture images of the goods stacked in the goods storage compartment. Considering the limited field of view of a single image acquisition device, for example, see [example description missing]. Figure 3 This embodiment of the invention employs two lifting devices, each equipped with a corresponding image acquisition device, to simultaneously capture images of the stacked goods from different angles. The fields of view of the two image acquisition devices can overlap to avoid omissions in the capture.

[0062] Here, the images of goods stacking acquired by each image acquisition device can be preprocessed by converting them to grayscale to eliminate the interference of color differences and lighting changes on subsequent goods recognition and improve the accuracy of the position coordinates of each item. Then, an image stitching algorithm is used to combine the preprocessed grayscale images into a complete image without blind spots, which is the final goods stacking image. For example, the final goods stacking image is as follows: Figure 4 As shown, in this embodiment of the invention, the position coordinates of each item are determined based on the final image of the goods being stacked.

[0063] Step 202: Based on the cargo stacking image, determine the position coordinates of each cargo, and based on the position coordinates, check whether the cargo stacking quality in the cargo stacking image is qualified.

[0064] This invention allows for pre-training of a YOLOv8 deep learning framework using different cargo stacking images to obtain a trained cargo recognition model. By inputting cargo stacking images into the cargo recognition model, the model can output the position coordinates of each item in the cargo stacking image.

[0065] In essence, the aforementioned cargo recognition model is a target detection model specifically trained for cargo palletizing detection scenarios based on the YOLOv8 deep learning framework. Its integration with the specific domain of cargo recognition and the technical details are as follows:

[0066] Specifically, embodiments of the present invention can collect complete cargo stacking images after image grayscale preprocessing and image stitching under different stacking layouts and lighting conditions, and annotate the bounding boxes and position coordinates of each cargo in each cargo stacking image to form a training dataset. Then, the cargo stacking images in the training dataset are input into the YOLOv8 deep learning framework for model training.

[0067] The YOLOv8 deep learning framework incorporates a backbone network, a neck network, and a detection head. The backbone network extracts features such as the shape and edges of the goods from the input image of stacked goods. These features are then enhanced by the neck network to improve feature fusion capabilities. Finally, the detection head outputs bounding box parameters and confidence scores. The confidence score is used to filter valid detection results, while the bounding box parameters (center coordinates, width, and height) are used to locate the coordinates of each target (goods) in the stacked image.

[0068] Based on the bounding box coordinates output by the YOLOv8 deep learning framework, and the standard bounding boxes and position coordinates in the training dataset, a loss function can be calculated. The model parameters are then continuously optimized with the goal of minimizing the loss function until the model's output reaches a preset accuracy, resulting in a trained model, i.e., a cargo recognition model. This cargo recognition model can output the confidence score and bounding box parameters (center point x-coordinate, y-coordinate, bounding box width, and bounding box height) for each detected target (i.e., cargo) based on the input image of cargo stacking.

[0069] In essence, the cargo recognition model is a target detection model. It can use bounding boxes to identify each target (i.e. cargo) in a cargo stacking image and output the confidence score of each target, as well as the center point coordinates, width, and height of each bounding box.

[0070] In this embodiment of the invention, targets with a confidence level lower than a set confidence level threshold can be eliminated, and the center point coordinates of the bounding boxes corresponding to the remaining targets can be determined as the location coordinates of each cargo.

[0071] Here, the position coordinates of each item can reflect the placement position of each item in the storage warehouse. Based on the position coordinates of each item, this embodiment of the invention can detect whether the placement spacing and alignment are qualified, and thus determine whether the current stacking quality of the goods is qualified.

[0072] It needs to be clarified that the cargo recognition model establishes a coordinate system based on images of cargo stacking to locate the center point coordinates of each bounding box, i.e., the position coordinates of each cargo. For example, using... Figure 4 Taking the cargo stacking image as an example, the upper left corner of the cargo storage compartment in the cargo stacking image is taken as the origin of the coordinate system (i.e., the positioning base point). A horizontal coordinate axis X is established along the horizontal direction of cargo stacking, where the positive direction of the X-axis is horizontally to the right of the positioning base point. A vertical coordinate axis Y is established along the vertical direction of cargo stacking, where the positive direction of the Y-axis is vertically downward of the positioning base point, thereby locating the position coordinates of each cargo.

[0073] Step 203: If the goods stacking quality is qualified, control the palletizing mechanism to continue to execute the goods stacking operation of the next layer until the palletizing mechanism completes the goods stacking operation.

[0074] In this embodiment of the invention, after determining that the goods stacking quality in the goods stacking image is qualified, the palletizing mechanism is controlled to continue to perform the goods stacking operation of the next layer.

[0075] If the goods stacking quality in the image is not up to standard, an alarm message can be sent to notify the user that the current goods stacking quality is not up to standard, so that the user can adjust the goods stacking position in time.

[0076] This invention uses the quality of goods palletizing as a feedback signal to control the subsequent actions of the palletizing mechanism, such as continuing to palletize or issuing an alarm, thereby forming a real-time feedback adjustment and control system with self-regulation and optimization capabilities.

[0077] Compared to existing technologies, the embodiments of the present invention determine the position coordinates of each item by using images of the goods being stacked, and then perform stacking quality inspection based on the position coordinates. This can eliminate the differences in subjective human judgment and avoid missed detections or misjudgments.

[0078] Furthermore, in each layer of stacking operation performed in this embodiment of the invention, a stacking quality inspection is performed. The quality inspection action can be embedded into the cyclical process of the stacking operation, so that the inspection node is strictly synchronized with the completion time of each layer of stacking, and the stacking quality is detected and fed back in real time. This not only avoids the problems of lag and missing layers that may exist in manual inspection, but also prevents unqualified layers from being covered or accumulated, thereby avoiding the risk of overall stacking stability decline or even collapse caused by bottom layer misalignment or tilting from the source.

[0079] The following section details the inspection process for the quality of goods stacking.

[0080] In some embodiments, see Figure 5 The specific steps for checking whether the stacking quality of goods in the images is up to standard are as follows:

[0081] Step 501: Based on the location coordinates of each cargo, group the cargo according to rows or columns to obtain at least one group of cargo.

[0082] Here, the position coordinates of each item include: the horizontal and vertical coordinates of each item. According to the above, in this embodiment of the invention, the horizontal direction of the item stacking is used as the horizontal axis, and the vertical direction of the item stacking is used as the vertical axis. The vertical coordinates of all items in each row (corresponding to each layer within the storage compartment) are the same, and the vertical coordinates of all items in each column are the same. The item stacking image contains at least one row and at least one column of items. This embodiment of the invention can group the items according to their horizontal and vertical coordinates, and then detect the spacing and alignment of each group of items.

[0083] This invention allows for grouping by row or column, enabling subsequent row-by-row or column-by-column inspection of the spacing and alignment of goods.

[0084] It should be noted here that, to ensure stacking stability, cross-stacking is generally used during the goods palletizing process. For example, if the stacking sequence of the first layer of goods during the palletizing operation is as follows... Figure 6 As shown, the stacking sequence of the goods in the second layer should be as follows: Figure 7 As shown, the stacking sequence of the first layer of goods is rotated 180°. The stacking sequence of goods in each odd-numbered layer remains consistent, and the stacking sequence of goods in each even-numbered layer remains consistent, resulting in the following... Figure 4 The cargo stacking structure shown is designed to ensure cargo stability.

[0085] In cross-stacking, if the length and width of the goods are different, the horizontal coordinates of the goods in the same column will be different, making it impossible to group the goods by column based solely on position coordinates. Only when the length and width of the goods are the same can the goods be grouped by column based on position coordinates.

[0086] In essence, grouping goods by row is more universally applicable. Therefore, this invention will describe specific grouping methods based on the principle of grouping by row:

[0087] For each ungrouped item in the goods stacking image, calculate the difference between its ordinate and the ordinates of the remaining ungrouped items. Then, group the remaining ungrouped items whose ordinate difference is less than the set difference with the ungrouped item. Finally, iterate through all items in the goods stacking image to obtain at least one group of items.

[0088] In other words, goods with the same or similar vertical coordinates are grouped into the same group. "Similar" can be simply understood as the difference in vertical coordinates being less than a set value. The specific value of this set value can be determined based on actual circumstances, and this invention does not impose any specific limitations on it.

[0089] It should be noted that, given a predetermined goods stacking sequence, the number of goods in each row is fixed when grouping by row. In this embodiment of the invention, when grouping goods by row, the quantity of goods in each group can be detected. If this quantity does not correspond to the quantity in the goods stacking sequence, it indicates a possible stacking problem in that layer or significant damage causing a drastic change in appearance, preventing the goods recognition model from accurately identifying the target. In this case, the palletizer can be paused, and an alarm message can be sent to prompt the user to inspect or adjust the operation.

[0090] Still with Figure 6 and Figure 7 For example, according to Figure 6 and Figure 7 When stacking goods in the shown sequence, the quantity of goods in odd-numbered layers (e.g., layers 1, 3, 5, etc.) is fixed at 5. The quantity of goods in even-numbered layers (e.g., layers 2, 4, 8, etc.) is fixed at 9. If, after grouping by row, it is found that the quantity of goods in odd-numbered layers is not 5, or the quantity of goods in even-numbered layers is not 9, the palletizer can be controlled to pause operation and send an alarm message to prompt the user to check or adjust.

[0091] Based on the row-based grouping, each group of goods can be sorted sequentially according to the vertical axis, thereby restoring the actual number of layers of goods in the storage warehouse.

[0092] Step 502: For each group of goods, based on the position coordinates of the goods in that group, check whether the placement spacing and alignment of the goods in that group are up to standard.

[0093] For each group of goods (corresponding to each layer of goods in the storage warehouse), this embodiment of the invention mainly detects whether the placement quality of the group of goods is qualified by measuring the horizontal spacing and the vertical alignment.

[0094] In some embodiments, for each group of goods, when checking whether the lateral placement spacing is qualified, the goods can first be arranged in order according to the horizontal coordinates of each goods in the group to obtain a goods sequence; then, in the goods sequence, the difference between the horizontal coordinates of every two adjacent goods is calculated to obtain multiple differences, and then the variance corresponding to the above multiple differences is calculated. If the variance corresponding to the above multiple differences is less than a first set variance threshold, then the placement spacing of the group of goods is determined to be qualified.

[0095] For each layer of goods, this embodiment of the invention arranges the corresponding horizontal coordinates of each goods in sequence, thus reconstructing the actual placement order of the goods in the storage warehouse and obtaining a goods sequence. In the goods sequence, the difference in the horizontal coordinates of every two adjacent goods can proportionally reflect the actual placement distance between every two adjacent goods.

[0096] This invention, through calculating the variance corresponding to the aforementioned multiple differences, can use this variance to reflect the uniformity of the lateral spacing of the goods in that layer. The smaller the variance value, the more uniform the lateral spacing of the goods in that layer.

[0097] Here, considering that the pixel size of the goods stacking images acquired by different image acquisition devices may differ, it is possible that the actual placement spacing between two adjacent goods is the same, but the difference in the horizontal coordinates in the goods stacking images may be different. To overcome the influence of the deviation caused by the image acquisition device, for each goods sequence, this embodiment of the invention can determine a first preset variance threshold based on the difference in the horizontal coordinates of every two adjacent goods in the goods sequence.

[0098] In some embodiments, for each group of goods corresponding to a goods sequence, the difference in the abscissa of every two adjacent goods can be calculated to obtain multiple differences. Then, the standard deviation and mean of the above multiple differences are calculated. Finally, the ratio of the standard deviation and the mean is determined as the first predetermined variance threshold for that group of goods. In essence, the first predetermined variance threshold for each group of goods is the coefficient of variation calculated by the standard deviation and the mean.

[0099] This invention uses the position coordinates of each item in each group of goods to set a corresponding first set variance threshold for that group of goods, thereby avoiding the influence of deviations caused by image acquisition equipment and improving the accuracy of stacking quality detection.

[0100] In some embodiments, for each group of goods, when detecting whether the alignment of the group of goods is qualified, the variance value corresponding to the vertical coordinate of each goods in the group can be calculated; if the variance value corresponding to the vertical coordinate is less than a second set variance threshold, then the alignment of the group of goods is determined to be qualified.

[0101] Here, for each group of goods (corresponding to each layer of goods in the storage warehouse), the variance value corresponding to the above-mentioned vertical axis can reflect the degree of alignment of the goods in that layer in the vertical direction. The smaller the variance value corresponding to the vertical axis, the more neatly the goods in that layer are arranged in the vertical direction.

[0102] Similarly, to overcome the influence of image acquisition equipment, this embodiment of the invention can calculate the standard deviation and mean value of the ordinate of each item in each group of goods. Then, the ratio of the standard deviation to the mean value is determined as the second predetermined variance threshold for that group of goods. Essentially, the second predetermined variance threshold for each group of goods is the coefficient of variation calculated from the standard deviation and mean value.

[0103] Step 503: If the spacing and alignment of the goods in this group are both up to standard, then the stacking quality of this group of goods is deemed to be up to standard.

[0104] Step 504: If the stacking quality of all groups of goods in the goods stacking image is qualified, then the stacking quality of the goods in the goods stacking image is determined to be qualified.

[0105] This invention, through layer-by-layer detection of the spacing and alignment of goods in each layer of the storage compartment, enables real-time feedback control during the palletizing process by detecting the palletizing quality after each layer of goods is placed.

[0106] In some embodiments, after step 502 above, the following may also be included:

[0107] Step 505: If the spacing or alignment of the goods in this group is not up to standard, then based on the position coordinates of each goods, determine the target goods that are not up to standard in terms of stacking quality.

[0108] For each group of goods, if the spacing between the goods in the group is not up to standard, then based on the difference between the horizontal coordinates of every two adjacent goods in the corresponding goods sequence of the group, at least one difference exceeding the set difference range is determined, and the goods corresponding to at least one difference are identified as target goods with unqualified stacking quality.

[0109] As described above, the difference in the horizontal coordinates of any two adjacent goods reflects the lateral spacing between them. For each group of goods, this embodiment of the invention can, based on the determination that the spacing of the goods in that group is substandard, identify goods whose horizontal coordinate differences exceed a set range as target goods with substandard stacking quality.

[0110] Considering that too small a spacing has little impact on the stacking quality of goods, while too large a spacing has a significant impact, this embodiment of the invention can identify goods with the largest difference in horizontal coordinate as goods with substandard stacking quality. Alternatively, goods with a difference in horizontal coordinate greater than a set difference threshold can be identified as target goods with substandard stacking quality.

[0111] Here, the range of differences and the threshold of differences can be determined according to the actual situation, and the embodiments of the present invention do not impose specific limitations on them.

[0112] Based on the above, the longitudinal coordinate of each item in each group of goods reflects the longitudinal alignment degree of each item. For each group of goods, this embodiment of the invention can, based on determining that the alignment degree of the group of goods is unqualified, identify goods whose longitudinal coordinates exceed the set coordinate range as target goods with unqualified stacking quality.

[0113] Alternatively, in this embodiment of the invention, the goods with the largest vertical coordinate can be identified as the target goods with unqualified stacking quality. Alternatively, goods with a horizontal coordinate greater than a set coordinate threshold can be identified as the target goods with unqualified stacking quality.

[0114] Here, the set coordinate range and set coordinate threshold can be determined according to the actual situation, and the embodiments of the present invention do not impose specific limitations on this.

[0115] Step 506: Based on the location coordinates of the target goods, send a stacking anomaly information to alert the user.

[0116] Based on the identification of target goods with substandard stacking quality, this embodiment of the invention sends stacking anomaly information to alert the user. Simultaneously, the palletizing mechanism can be controlled to pause operation. Here, the stacking anomaly information includes the location coordinates of the target goods to assist the user in positioning and adjusting them. After the user adjusts the goods, the palletizing mechanism can be controlled to resume the palletizing operation.

[0117] The following is combined with Figure 8 This paper provides an overview of methods for inspecting the quality of cargo palletizing.

[0118] In this embodiment of the invention, when performing palletizing quality inspection, a YOLOv8-based cargo recognition model is pre-trained, and the optimal palletizing layout is calculated based on the dimensions of the palletizing platform and the cargo dimensions. This optimal palletizing layout may include the stacking sequence for each layer and the number of palletizing layers. The stacking sequence for each layer may include the number of horizontal stacks, the number of vertical stacks, and the number of rows.

[0119] Next, the control palletizing mechanism palletizes the goods layer by layer according to the optimal palletizing layout. After each layer is completed, images are captured and stitched together using dual cameras, and the captured images are then processed into grayscale.

[0120] Subsequently, using a pre-trained YOLOv8-based cargo recognition model, image recognition was performed on the grayscale image, filtering out low-confidence bounding boxes and retaining high-confidence bounding boxes.

[0121] Next, for high-confidence bounding boxes, the center coordinates of the bounding boxes are extracted, and a coordinate list is generated. Based on the coordinate list, the boxes are grouped by the ordinate and sorted by the abscissa to form a cargo sequence. Subsequently, the placement spacing and alignment are detected by calculating the variance of the difference in the abscissas of adjacent boxes and the variance of their ordinates.

[0122] If all the above variances are less than the corresponding set variance thresholds, then the palletizing quality is determined to be qualified, and the next layer can be continued.

[0123] If the above variance is greater than or equal to the corresponding set variance threshold, the palletizing quality is determined to be unqualified, and an alarm message is triggered to remind the user.

[0124] Based on the above-described method for detecting the quality of cargo palletizing, this invention also provides a cargo palletizing system. For example... Figure 9 As shown, the cargo palletizing system includes a palletizing mechanism, image acquisition equipment, and control equipment.

[0125] The palletizing mechanism is used to perform goods palletizing operations. During the palletizing operation, the control equipment executes the aforementioned goods palletizing quality inspection method to detect the palletizing quality. When the palletizing quality is determined to be acceptable, the control equipment controls the palletizing mechanism to continue the goods palletizing operation. When the palletizing quality is determined to be unacceptable, the control equipment can control the palletizing mechanism to pause the goods palletizing operation and send an alarm message to notify the user.

[0126] Here, the image acquisition device is used to acquire images of the goods being stacked and send them to the control device so that the control device can perform the above-mentioned goods stacking quality inspection method based on the images of the goods being stacked.

[0127] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0128] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0129] Figure 10 The diagram shows a schematic representation of a goods palletizing quality inspection device based on machine vision inspection according to an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment of the present invention are shown, and are described in detail below:

[0130] like Figure 10 As shown, the cargo palletizing quality inspection device 10 based on machine vision inspection includes: an acquisition module 101 and an inspection module 102.

[0131] The acquisition module 101 is used to acquire the current cargo stacking image after the palletizing mechanism completes one layer of cargo stacking operation during the cargo stacking operation.

[0132] Detection module 102 is used for:

[0133] Based on the images of goods being stacked, the location coordinates of each item are determined, and based on these location coordinates, the stacking quality of the goods in the images is checked to see if it is up to standard.

[0134] If the goods are stacked to the required quality, the palletizing mechanism will continue to perform the next layer of goods stacking operation until the palletizing mechanism completes the goods stacking operation.

[0135] In one possible implementation, the detection module 102 is specifically used for:

[0136] Based on the location coordinates of each item, group the items by row or column to obtain at least one group of items;

[0137] For each group of goods, based on the position coordinates of the goods in that group, check whether the placement spacing and alignment of the goods in that group are up to standard;

[0138] If the spacing and alignment of the goods in this group are both up to standard, then the stacking quality of this group of goods is deemed to be up to standard.

[0139] If the stacking quality of all groups of goods in the goods stacking image is qualified, then the goods stacking quality in the goods stacking image is qualified.

[0140] In one possible implementation, the position coordinates of each cargo include: the x-coordinate and y-coordinate of each cargo;

[0141] Detection module 102 is specifically used for:

[0142] For each ungrouped item, calculate the difference between its ordinate and the ordinates of the remaining ungrouped items.

[0143] The remaining ungrouped goods whose coordinate difference is less than the set difference are grouped together with the ungrouped goods.

[0144] Iterate through all the goods in the goods stacking image to obtain at least one set of goods.

[0145] In one possible implementation, the detection module 102 is specifically used for:

[0146] For each group of goods, arrange them in order according to the x-coordinate of each item in that group to obtain the goods sequence;

[0147] In the cargo sequence, calculate the difference in the x-coordinates of each pair of adjacent cargoes to obtain multiple differences, and then calculate the variance of each difference.

[0148] If the variance corresponding to multiple differences is less than the first set variance threshold, then the placement spacing of the group of goods is determined to be qualified.

[0149] In one possible implementation, the detection module 102 is specifically used for:

[0150] For each group of goods, calculate the variance value corresponding to the ordinate based on the ordinate of each goods in that group;

[0151] If the variance value corresponding to the vertical axis is less than the second set variance threshold, then the alignment of the group of goods is determined to be qualified.

[0152] In one possible implementation, the method for determining the first set variance threshold includes:

[0153] Calculate the standard deviation and mean for multiple differences;

[0154] The ratio of the standard deviation to the mean is determined as the first set variance threshold.

[0155] In one possible implementation, the detection module 102 is further configured to:

[0156] If the spacing or alignment of the goods in this group is not up to standard, the target goods that are not up to standard will be determined based on the position coordinates of each goods in the group.

[0157] Based on the location coordinates of the target goods, send a stacking anomaly message to alert the user.

[0158] In one possible implementation, the detection module 102 is specifically used for:

[0159] For each group of goods, if the spacing between the goods in the group is not up to standard, then based on the difference between the horizontal coordinates of every two adjacent goods in the corresponding goods sequence of the group, at least one difference exceeding the set difference range is determined, and the goods corresponding to at least one difference are determined as target goods with unqualified stacking quality.

[0160] If the alignment of the group of goods is not up to standard, then based on the ordinate of each item in the group, at least one ordinate that exceeds the set coordinate range will be determined, and the item corresponding to at least one ordinate will be identified as the target item with unqualified stacking quality.

[0161] This device embodiment is used to implement the above method embodiment, and its technical principle and implementation effect are the same as those of the above method embodiment, so they will not be repeated here.

[0162] This invention also provides a control device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the above method embodiments.

[0163] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0164] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting the quality of cargo stacking based on machine vision detection, characterized in that, The method comprises the following steps: During the execution of the cargo stacking operation by the stacking mechanism, a current cargo stacking picture is obtained each time the stacking mechanism completes a layer of cargo stacking operation; Based on the cargo stacking picture, the position coordinates of each cargo are determined, and based on the position coordinates, it is detected whether the cargo stacking quality in the cargo stacking picture is qualified; If the cargo stacking quality is qualified, the stacking mechanism continues to execute the next layer of cargo stacking operation until the stacking mechanism completes the cargo stacking operation; The detection of whether the cargo stacking quality in the cargo stacking picture is qualified based on the position coordinates comprises the following steps: Based on the position coordinates of each cargo, the cargos are grouped by rows or columns to obtain at least one group of cargos; For each group of cargos, based on the position coordinates of the group of cargos, it is detected whether the placement interval and the alignment degree of the group of cargos are qualified; If the placement interval and the alignment degree of the group of cargos are both qualified, it is determined that the cargo stacking quality of the group of cargos is qualified; If the cargo stacking quality of all groups of cargos in the cargo stacking picture is qualified, it is determined that the cargo stacking quality in the cargo stacking picture is qualified; The position coordinates of each cargo comprise the horizontal coordinates and the vertical coordinates corresponding to each cargo; and the detection of whether the placement interval of each group of cargos is qualified based on the position coordinates of the group of cargos comprises the following steps: For each group of cargos, the cargos in the group are arranged in sequence according to the horizontal coordinates corresponding to each cargo to obtain a cargo sequence; In the cargo sequence, the difference values of the horizontal coordinates corresponding to each two adjacent cargos are calculated to obtain a plurality of difference values, and the variance corresponding to the plurality of difference values is calculated; If the variance corresponding to the plurality of difference values is less than a first set variance threshold, it is determined that the placement interval of the group of cargos is qualified. 2.The method of claim 1, wherein, The grouping of each cargo by rows based on the position coordinates of each cargo to obtain at least one group of cargos comprises the following steps: For each ungrouped cargo, the coordinate difference values between the vertical coordinates corresponding to the cargo and the vertical coordinates corresponding to the remaining ungrouped cargos are calculated respectively; The remaining ungrouped cargos with a coordinate difference value less than a set difference value are divided into a group with the ungrouped cargo; All cargos in the cargo stacking picture are traversed to obtain at least one group of cargos. 3.The method of claim 2, wherein, The detection of whether the alignment degree of each group of cargos is qualified based on the position coordinates of the group of cargos comprises the following steps: For each group of cargos, the variance value corresponding to the vertical coordinates of the cargos in the group is calculated based on the vertical coordinates corresponding to each cargo in the group; If the variance value corresponding to the vertical coordinates is less than a second set variance threshold, it is determined that the alignment degree of the group of cargos is qualified. 4.The method of claim 1, wherein, The determination method of the first set variance threshold comprises the following steps: The standard deviation and the average value corresponding to the plurality of difference values are calculated; The ratio of the standard deviation to the average value is determined as the first set variance threshold. 5.The method of claim 1 or 2, wherein, After the detection of whether the placement interval and the alignment degree of each group of cargos are qualified based on the position coordinates of the group of cargos, the following steps are further included: If the placement interval or the alignment degree of the group of cargos is not qualified, the target cargo with unqualified stacking quality is determined based on the position coordinates of each cargo in the group of cargos; Based on the position coordinates of the target goods, code abnormal information is sent to remind the user. 6.The method of claim 5, wherein, For each group of goods, if the placement interval or the alignment degree of the group of goods is unqualified, based on the position coordinates of each good in the group of goods, a target good with unqualified code quality is determined, including: For each group of goods, if the placement interval of the group of goods is unqualified, according to the difference between the horizontal coordinates corresponding to each two adjacent goods in the goods sequence corresponding to the group of goods, at least one difference value exceeding the set difference value range is determined, and the goods corresponding to the at least one difference value are determined as the target goods with unqualified code quality. If the alignment degree of the group of goods is unqualified, according to the longitudinal coordinates corresponding to each good in the group of goods, at least one longitudinal coordinate exceeding the set coordinate range is determined, and the goods corresponding to the at least one longitudinal coordinate are determined as the target goods with unqualified code quality.

7. A control device characterized by comprising: A device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method of any one of claims 1 to 6 when executing the computer program.

8. A cargo palletizing system characterized by, The device comprises: A control device according to claim 7, a stacking mechanism, and an image acquisition device.

Images