Illegal stacking detection method

Abstract

The invention relates to the technical field of image processing, in particular to an illegal stacking detection method, which comprises an illegal stacking detection method for acquiring three-dimensional image data based on an RGBD depth camera and combining depth information analysis. And making image data into a segmentation data set, and extracting road line mask, ground, tray and AGV road marker area information. And optimizing the road line mask through a straight line detection and line segment similarity algorithm, constructing an AGV road detection area, and obtaining a depth foreground area. By means of triple verification of tray line pressing, AGV road foreign matter and packing material positioning and in combination with candidate foreign matter rectangle and YOLO figure detection, accurate recognition of tray border crossing, road foreign matter invasion and illegal packing material stacking is achieved, the detection robustness in a complex scene is remarkably improved, and an efficient solution is provided for industrial automatic management.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a detection method for illegal stacking. Background Art

[0002] The detection methods for illegal stacking in workshops mainly include three types: manual inspection, video monitoring, and sensor monitoring. Although each method has specific applicable scenarios, there are also limitations to varying degrees.

[0003] Traditional manual inspection discovers problems of illegal stacking through regular personnel inspections. Its advantages lie in the intuitiveness of on-site observation and the timeliness of disposal. However, it has defects such as high labor costs, low detection efficiency, and being easily affected by subjective factors. Especially in large or complex-structured workshops, monitoring blind spots are likely to occur.

[0004] With the development of security technology, video monitoring systems have been widely used in enterprise security management. The system based on video monitoring uses cameras to achieve real-time monitoring of the workshop environment, improving management efficiency while reducing labor requirements. However, this method is limited by technical bottlenecks such as large storage space occupation and poor imaging quality in low-light environments, and there is also a problem of low recognition efficiency when relying on manual viewing of videos.

[0005] In addition, using sensor devices such as weight sensors and infrared sensors to monitor the status of items is also an effective method. The above technologies can provide accurate data support, which helps to establish an automated management and early warning mechanism. However, the cost of sensor deployment is relatively high, and it needs to be configured specifically according to the characteristics of the monitoring object, and there are adaptation challenges in actual applications.

[0006] In view of this, the present invention is specifically proposed. Summary of the Invention

[0007] In order to solve the above technical problems existing in the prior art, the present invention provides a detection method for illegal stacking. This method combines the three-dimensional image data collected by an RGBD depth camera with depth information analysis, and can solve the automated monitoring of the illegal stacking situation of items in the workshop.

[0008] To achieve the above object, the technical solution of the present invention is as follows:

[0009] A detection method for illegal stacking, comprising:

[0010] Making the collected three-dimensional image data into an image segmentation dataset, and respectively obtaining a road line mask, ground area information, tray area information, and AGV road marker area information based on the image segmentation dataset;

[0011] Optimize the road line mask through a line detection algorithm and a line segment similarity algorithm to generate optimized road line region information; obtain the AGV road detection region through the ground region information, the road line region information, and the AGV road marker region information; obtain the foreground region of the depth channel from the depth channel image through distance parameters and the tray region information.

[0012] Perform illegal stacking detection based on the road line region information, the AGV road detection region, the tray region information, the depth foreground region, and the packaging material region information. The illegal stacking detection specifically includes:

[0013] Obtain the intersection region between the tray region information and the road line region information. If it exists and meets the predetermined range, it is determined that there is a tray pressing line region.

[0014] Generate a foreign object rectangle parameter array through the AGV road detection region, the tray pressing line region, and the foreground region of the depth channel, and then obtain the foreign object region in the AGV road according to the intersection rate between the foreign object rectangle parameter arrays.

[0015] Obtain the packaging material region that is not placed on the tray as required through the packaging material region information, the tray region information, and the ground region information.

[0016] The tray pressing line region, the foreign object region in the AGV road, and the packaging material region that is not placed on the tray as required are illegal stackings.

[0017] Further, the specific steps for generating the road line region information include:

[0018] Obtain the endpoint information of the line segments based on the road line mask through a line detection algorithm.

[0019] Eliminate redundant line segments based on the endpoint information of the line segments through line segment similarity analysis to obtain a list of line segments after duplicate removal.

[0020] Divide the line segments in the line segment list into horizontal line segments and vertical line segments by calculating the line segment angles.

[0021] Calculate the combination of adjacent horizontal and vertical line segments, obtain the extended intersection points, obtain the connection lines and extension lines from the intersection points to the endpoints of the horizontal and vertical line segments respectively, and make them into a second road line mask.

[0022] Calculate the remaining vertical line segments, then perform line fitting on them and make them into a third road line mask.

[0023] Add the second road line mask and the third road line mask to obtain the optimized road line region information.

[0024] Furthermore, the line segment similarity analysis uses the cosine similarity algorithm, and the specific formula is as follows:

[0025]

[0026] A i ,B i represent the components of line segment A and line segment B respectively; among them, the endpoint coordinates of line segment A are (x1, y1), (x2, y2), and its vector form is A i The calculation method is as follows:

[0027] A i =(x2 - x1, y2 - y1).

[0028] Furthermore, to obtain the AGV road detection area, the specific steps include:

[0029] Subtract the road line area information from the ground area information to obtain the second ground area information;

[0030] Convert the AGV road marker area information into marker information;

[0031] Calculate the inclusion relationship between the second ground area information and the outer contour in the marker information, extract the contour of the marker information contained in the second ground area information, and fill and draw to obtain the third ground area information;

[0032] When the third ground area information is a completely black image, extract the largest connected area in the second ground area information and perform convex hull fitting, fill and draw in the third ground area information, subtract the road line area information from the third ground area information, and extract the largest connected area and fill and draw to obtain the fourth ground area information. The fourth ground area information B4 is the AGV road detection area;

[0033] When the third ground area information is not a completely black image, perform convex hull fitting on the third ground area information, subtract the road line area information, and extract the largest connected area and fill and draw to obtain the fourth ground area information. The fourth ground area information is the AGV road detection area.

[0034] Furthermore, the specific steps to obtain the tray pressing line area include:

[0035] Extract the intersection area of the tray area information and the road line area information as the second tray area information, extract the contour of the second tray area information, and filter out the areas that do not meet the predetermined size;

[0036] Calculate the inclusion relationship between the unfiltered area and the outer contour of the tray area information, extract the contour in the tray area information that contains the contour of the unfiltered area and perform filling and drawing to obtain the third tray area information, and the third tray area information is the tray wire pressing area.

[0037] Further, the specific steps for obtaining the foreground area of the depth channel include:

[0038] Obtain the first depth channel image information by excluding the set area in the distance parameter of the depth channel image;

[0039] Exclude the tray area information in the first depth channel image information and extract the largest connected area. After performing erosion and dilation processing on this area and then adding the tray area information, extract the largest connected area to obtain the second depth channel image information, and the second depth channel image information E2 is the foreground area of the depth channel.

[0040] Further, the foreign object rectangle parameter array includes: a candidate foreign object rectangle parameter array, a person area rectangle parameter array, and a tray wire pressing area parameter array.

[0041] Furthermore, the specific steps for generating the candidate foreign object rectangle parameter array include:

[0042] Add the AGV road detection area to the tray area information, extract the largest connected area and exclude the fifth ground area information obtained from the ground area information. Perform an image logical AND operation on the fifth ground area information and the foreground area of the depth channel for the binary image and the grayscale or color image to obtain the third depth channel image information;

[0043] Extract the intersection area of the third depth channel image information and the tray area information. If this area is not a completely black image, verify the inclusion relationship between its contour and the outer contour of the tray area information, extract the conforming contour for filling to generate the fourth depth channel image information, and perform rectangular fitting on the outer contour of the fourth depth channel image information and store the result in the candidate foreign object rectangle parameter array;

[0044] Extract the non-intersection area of the third depth channel image information and the tray area information, traverse its outer contour and extract the center point: if the center point is inside the contour, generate a mask through the SAM model and fit a rectangle and store it in the candidate foreign object rectangle parameter array; if the center point is not inside the contour, directly fit the contour rectangle and store it in the candidate foreign object rectangle parameter array.

[0045] The specific steps for generating the person area rectangle parameter array include:

[0046] The character region information of the image data is extracted through the YOLO universal character model, and the rectangular parameters of the character region information are obtained and stored in the character region rectangular parameter array.

[0047] Generate the pallet pressing area parameter array, the specific steps include:

[0048] The outer contour of the pallet pressing line area information D3 is extracted, and a rectangle fitting is performed on each outer contour and rectangle parameters are extracted and stored in the pallet pressing line area parameter array.

[0049] Furthermore, the specific steps of obtaining the foreign object area in the AGV road include:

[0050] Traverse the foreign object rectangular parameter array, calculate the intersection rate of the rectangles in the candidate foreign object rectangular parameter array with the human area rectangular parameter array and the pallet pressure line area parameter array; if the intersection rate is greater than 0.75, delete the corresponding rectangle information in the candidate foreign object rectangular parameter array and update the data; the updated rectangular area of the foreign object area parameter array is the foreign object area detected in the AGV road.

[0051] Furthermore, the specific steps of obtaining the packaging material area that is not placed on the pallet in accordance with regulations include:

[0052] The packaging material area is extracted from the RGB channel image of the RGBD camera through the HSV color threshold, and the area whose contour does not meet the preset range standard is removed, thereby obtaining the packaging material area information;

[0053] Extracting a rectangular area below the circumscribed rectangle of the packaging material area information outline, and extending the rectangular area downward as second packaging material area information, wherein each packaging material in the second packaging material area information corresponds to a pallet detection area;

[0054] Calculating the contour intersection relationship between the second packaging material area information and the pallet area information;

[0055] If they do not intersect, the packaging material area information in the second packaging material area information is eliminated, and the overlap rate between the remaining area and the outline of the ground area information is calculated. If the overlap rate is greater than 0.5, the remaining area is drawn to obtain the third packaging material area information;

[0056] The packaging material area information and the area corresponding to the third packaging material area information are extracted, and the area is the packaging material area that is not placed on the pallet according to regulations.

[0057] Compared with the prior art, the above-mentioned method for detecting illegal stacking provided by the present invention realizes accurate identification through multi-modal image processing and logical verification mechanisms. Through a triple verification mechanism of pallet line pressing detection, AGV road foreign object detection, and packaging material positioning detection, problems such as pallet overstepping, road foreign object intrusion, and illegal placement of packaging materials are respectively determined. The fusion technology of foreign object contour fitting assisted by the SAM model and YOLO person detection is introduced, combined with the intersection rate calculation of the rectangular parameter array and the contour spatial relationship verification, which significantly improves the detection robustness in complex scenarios, realizes the full-dimensional monitoring of warehouse illegal stacking behaviors, and provides an efficient solution for automated management in industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the method for detecting illegal stacking provided by the embodiment of the present invention;

[0059] Figure 2 It is a schematic diagram of pallet line pressing detection provided by the second embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of obtaining the AGV road detection area provided by the third embodiment of the present invention;

[0061] Figure 4 It is a schematic diagram of AGV road foreign object detection provided by the fourth embodiment of the present invention;

[0062] Figure 5 It is a schematic diagram of detecting whether the packaging material is placed on the pallet provided by the fifth embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] The technical solutions of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are not all the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0064] It should be noted that unless otherwise specifically stated, the relative arrangements of components and steps described in these embodiments and numerical expressions should not be construed as limiting the scope of the present invention.

[0065] The following description of the exemplary embodiments is merely illustrative and in no way restricts the present invention and its application or use. Technologies, methods, and devices known to those of ordinary skill in the relevant fields may not be discussed in detail here, but when applicable, these technologies, methods, and devices should be regarded as part of this specification.

[0066] Embodiment 1

[0067] Refer to Figure 1, the present invention provides a detection method for illegal stacking, specifically related to a detection method for illegal stacking based on an RGBD depth camera, image processing technology, and image segmentation algorithm. A depth camera is mounted on a robot to obtain real-time three-dimensional image data and analyze it in combination with depth information to achieve automatic monitoring of the illegal stacking situation of items in the workshop. The specific implementation process may include the following steps:

[0068] A1. Image data processing

[0069] A11. Multi-modal data acquisition

[0070] Collect RGB channel image data through an RGBD camera in the workshop and create an image segmentation data set;

[0071] A12. Instance segmentation model training

[0072] Train a model based on the YOLO instance segmentation model to achieve image segmentation detection of road lines, the ground, pallets, and AGV road markers. The output mask information includes: road line mask, ground area information, pallet area information, and AGV road marker area information.

[0073] A2. Obtain road line area information

[0074] A21. Basic road line extraction

[0075] Use the segmentation model to extract the road line area information of the RGB channel image of the RGBD camera and make it into a road line mask; use Canny edge detection to detect the edges of the road line area information, and obtain the endpoint information of the line segments through a line detection algorithm; among them, the line detection algorithm adopts the Hough transform;

[0076] A22. Redundant line segment removal

[0077] Based on the endpoint information of the line segments, remove redundant line segments through line segment similarity analysis to obtain a list of line segments after duplicate removal; among them, the similarity analysis adopts the cosine similarity algorithm. Given two attribute vectors A and B, their cosine similarity θ is given by the dot product and vector length. The specific formula is:

[0078]

[0079] Among them, the endpoint coordinates of line segment A are (x1, y1), (x2, y2), and its vector form is A i The calculation method is as follows:

[0080] A i =(x2 - x1, y2 - y1)

[0081] A i , B iRespectively representing the components of vectors A and B, the given similarity ranges from -1 to 1: -1 means that the directions in which the two vectors point are exactly opposite, 1 indicates that they point in exactly the same direction, 0 usually means they are independent of each other, and values in between represent intermediate similarity or dissimilarity.

[0082] Use a double loop to traverse the endpoint information of the line segments obtained in step A21, and calculate the cosine similarity between the direction vectors of pairwise line segments. If the cosine similarity of two line segments is greater than 0.95, it is considered that they are too similar, and one of them is removed. Obtain the list of line segments after duplicate removal.

[0083] A23. Classification of line segment directions

[0084] Classify the line segments in the said list of line segments into horizontal direction line segments and vertical direction line segments by calculating the line segment angles; the horizontal or vertical direction classification of the line segments is achieved through the following steps:

[0085] A231. Remove the lines with a similarity greater than 95 through cosine similarity calculation based on the endpoint information of the line segments; use a double loop to traverse the endpoint information of the line segments obtained in B12, and calculate the cosine similarity between the direction vectors of pairwise line segments. If the cosine similarity of two line segments is greater than 0.95, it is considered that they are too similar and one of them is removed to obtain the list of line segments after duplicate removal.

[0086] A232. Let the coordinates of the starting endpoint of the line segment be (x1, y1), and the coordinates of the ending endpoint be (x2, y2), then the horizontal difference dx = x2 - x1, and the vertical difference dy = y2 - y1. Use the arctan2 function to calculate the radian value from the origin to the point (dx, dy), and accurately determine the quadrant where the angle is located according to the positive and negative of dx and dy.

[0087] A233. Convert the obtained radian value to an angle value, and the conversion formula is: angle value = radian value × (180 / π), and ensure that the angle value is positive. If the angle value is negative, add 360°;

[0088] A234. Extract the line segments with angles between 6° and 174° and classify them as vertical direction line segments; classify the remaining line segments with angles close to 0° or 180° as horizontal direction line segments.

[0089] A24. Generate road line area information

[0090] A241. Traverse each vertical line segment and check if it intersects any horizontal line segment. When an intersection occurs, calculate the distances from the intersection coordinates to the horizontal and vertical line segments respectively. When both distances are less than 95 - 105 pixel points, determine that the horizontal and vertical line segments are adjacent line segments. Draw the connecting lines from the intersection points to the horizontal and vertical line segments and extend them by 25 - 35 pixel points, and make it into the second road line mask.

[0091] A242. Calculate the remaining vertical line segments, then extend them by 25 - 35 pixel points at both ends and make them into the third road line mask.

[0092] A243. Add the second road line mask and the third road line mask to obtain the optimized road line area information. A3. Obtain the AGV road detection area

[0093] A31. Ground area processing

[0094] A311. Use the segmentation model to extract the ground area of the RGB channel image of the RGBD camera and make it into the ground area information.

[0095] A312. Subtract the road line area information from the ground area information to obtain the second ground area information.

[0096] A313. Use the image segmentation model to extract the AGV road marker area information and convert it into marker information.

[0097] A314. Calculate the inclusion relationship between the outer contours in the second ground area information and the marker information, extract the contour in the second ground area information that includes the marker information, and fill and draw to obtain the third ground area information.

[0098] A32. Dynamic area correction

[0099] A321. All - black image processing

[0100] When the third ground area information is an all - black image, extract the largest area in the second ground area information and perform convex hull fitting, fill and draw in the third ground area information, subtract the road line area information from the third ground area information, and extract the largest connected area and fill and draw to obtain the fourth ground area information, which is the AGV road detection area.

[0101] A322. Non - all - black image processing

[0102] When the third ground area information is not a completely black image, the third ground area information is subjected to convex hull fitting, and the road line area information is subtracted, and the maximum connected area is extracted and filled and drawn to obtain the fourth ground area information, which is the AGV road detection area.

[0103] A4. Pallet pressure line detection

[0104] A41. Regional intersection analysis

[0105] Based on the image segmentation dataset, the image segmentation model is used to extract the pallet area information and convert it into the pallet area information, the intersection area of the pallet area information and the road line area information is extracted as the second pallet area information, the contour of the second pallet area information is extracted and the area with a smaller area is filtered;

[0106] A42, Determination of the pressing area

[0107] The inclusion relationship between the unfiltered area and the outer contour of the pallet area information is calculated, and the contour including the unfiltered area contour in the pallet area information is extracted and filled and drawn to obtain the third pallet area information, where the third pallet area information is the pallet pressing line area.

[0108] A5. AGV road foreign body detection

[0109] A51. Deep foreground segmentation

[0110] A511. Use the depth channel image of the RGBD camera through the distance parameter, and eliminate the set area through the distance parameter to obtain the first depth channel image information; wherein the set area includes: an area above 5 meters and below 0.3 meters.

[0111] A512. Remove the pallet area information from the first depth channel image information and extract the maximum connected area. Perform corrosion and expansion processing on the area and then add the pallet area information to extract the maximum connected area to obtain the second depth channel image information. The second depth channel image information is the depth channel foreground area.

[0112] A52, Foreign body area positioning

[0113] A521, adding the AGV road detection area to the pallet area information, extracting the maximum connected area and eliminating the ground area information to obtain fifth ground area information;

[0114] A522, performing an image logic AND operation on the fifth ground area information and the depth channel foreground area to obtain third depth channel image information;

[0115] A53. Create a foreign body rectangle parameter array

[0116] A531, generating a candidate foreign body region parameter array, the specific steps include:

[0117] Extracting the intersection area of the third depth channel image information and the pallet area information, when the image in the area is not a completely black image, calculating the inclusion relationship between the image and the outer contour of the pallet area information, extracting the contour containing the obtained image contour in the pallet area information, filling and drawing to obtain the fourth depth channel image information, extracting the outer contour information of the four depth channel image information for rectangle fitting, and storing the rectangle parameters in the candidate foreign body area parameter array;

[0118] Extract the non-intersection area of the third depth channel image information and the tray area information, obtain the outer contour information of the area, extract the center point of each contour, when the point is in the contour to which it belongs, input the point in the SAM model to obtain mask information, obtain the contour information for rectangular fitting and store the rectangular parameters in the candidate foreign body area parameter array; if the point is not in the contour to which it belongs, perform rectangular fitting on the contour, extract the rectangular parameters and store them in the candidate foreign body area parameter array.

[0119] A532, generating a character area rectangular parameter array, the specific steps include:

[0120] Extracting character region information of the image data through the YOLO universal character model, obtaining rectangular parameters of the character region information and storing them in a character region rectangular parameter array;

[0121] A533. Generate the pallet pressing area parameter array. The specific steps include:

[0122] The outer contour of the pallet pressing line area information is extracted, and a rectangle fitting is performed on each outer contour and rectangle parameters are extracted and stored in a pallet pressing line area parameter array.

[0123] A54, Generate foreign body area in AGV road

[0124] Traverse the candidate foreign body area parameter array, and calculate the intersection rate of the candidate foreign body area parameter array with the rectangles in the human area rectangle parameter array and the pallet pressure line area parameter array;

[0125] If the intersection rate is greater than 0.75, the corresponding rectangular information is deleted from the candidate foreign object area parameter array and the data is updated; the updated rectangular area of the candidate foreign object area parameter array is the foreign object area detected in the AGV road.

[0126] A6. Check whether the packaging materials are placed on the pallet

[0127] A61, packaging material area screening

[0128] The packaging material area in the RGB channel image of the RGBD camera is extracted according to the HSV color threshold, and the area whose contour does not meet the preset range standard is removed. The preset standard can be an area with a too small contour or an area with a too large aspect ratio, which is made into packaging material area information; wherein, the preset range standard can be set independently according to demand.

[0129] A62. Pallet correlation analysis

[0130] A621, extracting the lower 1 / 4 rectangular area of the circumscribed rectangle of the packaging material area information outline, and extending the rectangular area downward by one half as the second packaging material area information, wherein each packaging material in the second packaging material area information corresponds to a pallet detection area;

[0131] A622, calculating the contour intersection relationship between the second packaging material area information and the pallet area information;

[0132] A623, when they do not intersect, the packaging material area information in the second packaging material area information is eliminated, and the overlap rate between the remaining area and the outline of the ground area information is calculated. If the overlap rate is greater than 0.5, the remaining area is drawn to obtain the third packaging material area information;

[0133] If they intersect, calculate the overlap ratio of the second packaging material area information and the pallet area information, and the percentage of the area of the intersection of the two rectangles to the area of the previous rectangle. If the overlap ratio is greater than 0.5, it is determined that the packaging material is placed on the pallet and there is no need to record the information.

[0134] A63. Generate packaging material areas that are not placed on the pallet in accordance with regulations

[0135] The packaging material area information and the area corresponding to the third packaging material area information are extracted, and the area is the packaging material area that is not placed on the pallet according to regulations.

[0136] A7. Information integration and output of test results

[0137] The three violations, namely the pallet pressure line area, the foreign object area in the AGV road, and the packaging material area that is not placed on the pallet in accordance with regulations, are marked on the image.

[0138] A test set test was performed based on the violation detection method provided by the present invention, using 235 test set images, and the test results are shown in Table 1: in the table, Box represents the prediction box, Mask represents the prediction mask, P represents the accuracy, R represents the recall rate, mAP represents the area of the closed area formed after the precision rate and the recall rate are used as two-axis plots, mAP50 represents the average mAP with a threshold greater than 0.5, mAP50-95 represents the threshold value of 0.5 to the mAP threshold of 0.95, with an interval of 0.05, and 10 mAP values are obtained, and then the ten values are averaged.

[0139] Table 1 Model training results

[0140]

[0141]

[0142] Example 2

[0143] Refer to Figure 2 , in this example, taking the case of extracting the illegal line pressing situation of an image as an example, Figure 2 Figure a in [reference] is the tray area extracted by segmentation, Figure 2 Figure b in [reference] is the road line area extracted by segmentation, Figure 2 Figure c in [reference] is the road line area after algorithm optimization, Figure 2 Figure d in [reference] is the intersection area between the optimized road line and the tray area, Figure 2 Figure e in [reference] is the detected tray line pressing area.

[0144] In this embodiment, all algorithms are compiled using Python (v3.9.12; Python Software Foundation, 2022); specifically, it includes the following steps:

[0145] B1. Obtain road line area information

[0146] B11. Load the image to be detected, use the segmentation model to extract the road line area information of the image to be detected, and make it into a road line area;

[0147] B12. Use Canny edge detection to detect the edges of the road line area, and use Hough transform to detect straight lines to obtain the endpoint information of the straight line segments;

[0148] B13. Calculate the line segment angles and divide the line segments in the line segment list into horizontal direction line segments and vertical direction line segments, specifically including:

[0149] B131. Based on the straight line segment endpoint information, calculate and eliminate the straight lines with a similarity greater than 95 through cosine similarity; use a double loop to traverse the straight line segment endpoint information obtained in B12, and calculate the cosine similarity between the direction vectors of two line segments. If the cosine similarity of two line segments is greater than 0.95, it is considered that they are too similar, and one of them is eliminated. Obtain a deduplicated line segment list.

[0150] B132. Let the coordinates of the starting endpoint of the line segment be (x1, y1), and the coordinates of the ending endpoint be (x2, y2), then the horizontal difference dx = x2 - x1, and the vertical difference dy = y2 - y1. Use the arctan2 function to calculate the radian value from the origin to the point (dx, dy), and accurately determine the quadrant where the angle is located according to the positive and negative of dx and dy.

[0151] B133. Convert the obtained radian value into an angle value. The conversion formula is: Angle value = Radian value × (180 / π), and ensure that the angle value is positive. If the angle value is negative, add 360°;

[0152] B134. Classify the line segments with angles between 6° and 174° as vertical direction line segments; classify the remaining line segments with angles close to 0° or 180° as horizontal direction line segments.

[0153] B14. Generate the determination area information of the road line for detecting the pressed line

[0154] B141. Traverse each vertical direction line segment and check whether it intersects with any horizontal direction line segment; when intersecting, calculate the distances from the intersection point coordinates to the horizontal direction line segment and the vertical direction line segment respectively. When both distances are less than 100 pixel points, determine that the horizontal direction line segment and the vertical direction line segment are adjacent line segments, draw the connection lines from the intersection point to the horizontal and vertical direction line segments and extend them by 30 pixel points, and make it into the second road line mask;

[0155] B142. Calculate the remaining vertical direction line segments, then extend them by 30 pixel points at both ends respectively and make them into the third road line mask;

[0156] B143. Add the second road line mask and the third road line mask to obtain the optimized determination area of the road line for detecting the pressed line.

[0157] B2. Pallet pressed line detection

[0158] B21. Load the image to be detected, use the segmentation model to extract the pallet area information of the image to be detected, and make it into the pallet area; extract the intersection area of the optimized determination area of the road line for detecting the pressed line and the pallet area, extract the contour of this area and filter out the contour areas with the number of points less than 6 in the contour to obtain the intersection area of the road line and the pallet area.

[0159] B22. Calculate the inclusion relationship between the intersection area and the outer contour of the pallet area, and extract the contour in the pallet area that contains the contour of the intersection area; perform rectangular fitting on the contour, and draw this contour on the image to be detected to obtain the pressed line detection result. The rectangular box selection area in the pressed line detection result is the detected pallet pressed line area.

[0160] Example 3

[0161] Refer to Figure 3 , taking the extraction of the AGV road line area of an image in this example as an example, Figure 3 In Figure a, it is the AGV road marker area extracted by segmentation, Figure 3 In Figure b, it is the road line area extracted by segmentation, Figure 3 In Figure c, it is the ground area mask extracted by segmentation,Figure 3 In Figure d, the horizontal and vertical lines are adjacent and intersecting fitted road lines. Figure 3 In Figure e, there are no horizontally adjacent and intersecting fitted road lines in the vertical line. Figure 3 In Figure f, it is a fitted road line mask. Figure 3 In Figure g, it is the extracted AGV road detection area. Figure 3 In Figure h, it is the optimized AGV road detection area.

[0162] In this embodiment, all algorithms are compiled using Python (v3.9.12; Python Software Foundation, 2022); specifically, it includes the following steps:

[0163] C1. Obtain road line area information

[0164] C11. Load the image to be detected, use the segmentation model to extract the road line area information of the image to be detected, and make it into a road line area.

[0165] C12. Use Canny edge detection to detect the edges of the road line area, and use Hough transform to detect straight lines to obtain the endpoint information of the straight line segments.

[0166] C13. Calculate the line segment angles and divide the line segments in the line segment list into horizontal direction line segments and vertical direction line segments, specifically including:

[0167] C131. Based on the straight line segment endpoint information, calculate and eliminate the straight lines with a similarity greater than 95 through cosine similarity; use a double loop to traverse the straight line segment endpoint information obtained in C12, and calculate the cosine similarity between the direction vectors of two line segments. If the cosine similarity of two line segments is greater than 0.95, it is considered that they are too similar, and one of them is eliminated to obtain a deduplicated line segment list.

[0168] C132. Let the starting endpoint coordinates of the line segment be (x1, y1), and the ending endpoint coordinates be (x2, y2), then the horizontal difference dx = x2 - x1, and the vertical difference dy = y2 - y1. Use the arctan2 function to calculate the radian value from the origin to the point (dx, dy), and accurately determine the quadrant where the angle is located according to the positive and negative of dx and dy.

[0169] C133. Convert the obtained radian value to an angle value, and the conversion formula is: angle value = radian value × (180 / π), and ensure that the angle value is positive. If the angle value is negative, add 360°.

[0170] C134. Extract the line segments with angles between 6° and 174° and classify them as vertical direction line segments; classify the remaining line segments with angles close to 0° or 180° as horizontal direction line segments.

[0171] C14. Generate road line area information

[0172] C141. Traverse each vertical line segment and check whether it intersects with any horizontal line segment; when intersecting, calculate the distances from the intersection coordinates to the horizontal line segment and the vertical line segment respectively. When both distances are less than 100 pixel points, determine that the horizontal line segment and the vertical line segment are adjacent line segments, draw the connecting lines from the intersection points to the horizontal line and the vertical line respectively and extend them to the image boundary, and obtain the fitting road lines that are adjacent and intersecting in the horizontal line and the vertical line;

[0173] C142. Calculate the remaining vertical line segments, then perform linear fitting on them and make them into the fitting road lines in the vertical lines that have no horizontal adjacent and intersecting ones;

[0174] C143. Add the fitting road lines that are adjacent and intersecting in the horizontal line and the vertical line to the fitting road lines in the vertical lines that have no horizontal adjacent and intersecting ones to obtain an optimized road line mask.

[0175] C2. Obtain the AGV road detection area

[0176] C21. Ground area processing

[0177] C211. Use a segmentation model to extract the ground area information of the image to be detected and make it into a ground area mask;

[0178] C212. Subtract the road line mask from the ground area mask to obtain the area outer contour information;

[0179] C213. Use an image segmentation model to extract the AGV road marker area information and convert it into an AGV road marker area;

[0180] C314. Calculate the inclusion relationship between the area outer contour information and the outer contour of the AGV road marker area, extract the contour including the marker information in the second ground area information, and fill and draw to obtain the AGV road detection area;

[0181] C3. AGV road detection area correction

[0182] C321. All - black image processing

[0183] If the AGV road detection area is an all - black image, extract the largest connected area in the area outer contour information for convex hull fitting, perform filling and drawing in the AGV road detection area. After removing the optimized road line mask from the AGV road detection area, extract the largest connected area and fill and draw to obtain the optimized AGV road detection area.

[0184] C322. Non - all - black image processing

[0185] When the third ground area information is not a completely black image, the third ground area information h is subjected to convex hull fitting, then the road line area information is subtracted, and the largest connected area is extracted and filled to obtain the optimized AGV road detection area.

[0186] Example 4

[0187] Refer to Figure 4 , in this example, the illegal line pressing situation of an image is detected. Among them, Figure a is the tray area information extracted by segmentation, Figure 4 Figure b is the ground area information extracted by segmentation, Figure 4 Figure c is the depth channel image captured by the RGBD camera, Figure 4 Figure d is the first depth channel image information after removing the tray area, Figure 4 Figure e is the optimized depth channel foreground area, Figure 4 Figure f is the AGV road detection area mask extracted, Figure 4 Figure g is the foreign object area in the detected AGV road.

[0188] In this embodiment, all algorithms are compiled using Python (v3.9.12; Python Software Foundation, 2022); specifically, it includes the following steps:

[0189] D1. AGV road foreign object detection

[0190] D11. Depth foreground segmentation

[0191] D111. Use the segmentation model to extract the tray area information of the RGB channel image, and make it into a tray area mask; load the RGB channel image captured by the RGBD camera and the depth channel image captured by the RGBD camera, perform grayscale processing, and set the color threshold of the background area more than 5 meters and less than 0.3 meters to 255 through the distance parameter to obtain the first depth channel image information;

[0192] D112. After performing an inversion operation on the first depth channel image information, remove the tray area information in the first depth channel image information and extract the largest connected area. After performing erosion and dilation processing on this area and then adding the tray area information, extract the largest connected area to obtain the optimized depth channel foreground area.

[0193] D2. Foreign object area positioning

[0194] D21. Use the segmentation model to extract the ground area information of the RGB channel image, and make it into the ground area information; extract the AGV road detection area to obtain the AGV road detection area mask;

[0195] Add the AGV road detection area mask and the pallet area mask, extract the maximum connected area and remove the ground area information to obtain the fifth ground area information;

[0196] D22, performing an image logic AND operation on the fifth ground area information and the depth channel foreground area to obtain the AGV road foreign object area;

[0197] D3, create a foreign body rectangle parameter array

[0198] D31. Generate a candidate foreign body region parameter array. The specific steps include:

[0199] Extract the intersection area of the AGV road foreign body area and the pallet area information. If the image in the obtained area is not a completely black image, calculate the inclusion relationship between the image and the outer contour of the pallet area information, extract the contour of the obtained image in the pallet area information, fill and draw to obtain the fourth depth channel image information, extract the outer contour information of the four depth channel image information for rectangle fitting, and store the rectangle parameters in the candidate foreign body area parameter array;

[0200] Extract the non-intersecting area of the AGV road foreign object area and the pallet area information, obtain the outer contour information of the area, extract the center point of each contour, and when the point is in the contour to which it belongs, input the point in the SAM model to obtain the mask information, obtain the contour information for rectangular fitting and store the rectangular parameters in the candidate foreign object area parameter array; if the point is not in the contour to which it belongs, perform rectangular fitting on the contour, extract the rectangular parameters and store them in the candidate foreign object area parameter array.

[0201] D32, using the YOLO general character model to extract character region information of the image data from the RGB channel image, obtaining rectangular parameters of the character region information and storing them in a character region rectangular parameter array;

[0202] D33. Execute the pallet pressing line detection and store the rectangular parameters of the rectangular selected area in the pressing line detection result into the pallet pressing line area parameter array.

[0203] D4. Generate foreign body area in AGV road

[0204] Traverse the candidate foreign body area parameter array, and calculate the intersection rate of the candidate foreign body area parameter array with the rectangles in the human area rectangle parameter array and the pallet pressure line area parameter array;

[0205] If the intersection rate is greater than 0.75, the corresponding rectangular information is deleted from the candidate foreign object area parameter array and the data is updated; the updated candidate foreign object area parameter array is drawn on image a, and the drawn rectangular area is the foreign object area detected in the AGV road.

[0206] Embodiment 5

[0207] See also Figure 5 This example takes the detection of whether there is illegal placement of packaging materials in an image as an example. Figure 5 Figure a in the middle shows the segmented and extracted pallet area information. Figure 5 Figure b in the middle shows the ground area information extracted by segmentation. Figure 5 Figure c in the middle shows the extracted packaging material area. Figure 5 Figure d in the middle shows the pad determination area under the extracted packaging material. Figure 5 Figure e in the middle shows the area in the packaging area that is determined to have no pallets. Figure 5 Figure f in the middle shows the area of packaging materials that are not placed on the pallet in accordance with regulations.

[0208] All algorithms in this example are compiled using Python (v3.9.12; Python Software Foundation, 2022); specifically, the following steps are included:

[0209] E1. Packaging material area screening

[0210] The image to be detected is converted into an HSV color image, and the packaging material color area is screened out in the image according to the HSV color threshold to obtain the packaging material area. The area with an area less than 100 pixels in the packaging material area is eliminated, and the minimum circumscribed rectangle of each packaging material color area is calculated. The packaging material area with an aspect ratio greater than 4.5 is eliminated and made into packaging material area information;

[0211] E2. Pallet correlation analysis

[0212] E21, extract the lower 1 / 4 rectangular area of the circumscribed rectangle of the packaging material area information outline, extend the rectangular area downward by one time and fill it as the pad determination area under the packaging material;

[0213] E22, calculating the contour intersection relationship between the tray determination area under the packaging material and the pallet area information;

[0214] E23. If they do not intersect, the packaging material area information in the pallet determination area information below the packaging material is removed, and the overlap ratio between the remaining area and the contour of the ground area information is calculated. If the overlap ratio is greater than 0.5, the remaining area is drawn to obtain an area without a pallet;

[0215] E24. Extract the packaging material area information and the area corresponding to the area without a pallet, perform rectangular fitting on it and draw it on image a to generate a new image. The rectangular framed area on the new image is the packaging material area that is not placed on the pallet in accordance with regulations.

[0216] In summary, the present invention has the following advantages:

[0217] (1) Replace manual inspection with automated monitoring by depth cameras to reduce labor costs and time consumption, overcome the problems of low efficiency, difficult coverage in traditional manual inspections, high storage costs in video monitoring, and dependence on manual analysis in video monitoring, and achieve cost reduction and efficiency improvement.

[0218] (2) Based on three-dimensional image data and combined with depth information analysis, depth data is not affected by light and can accurately detect violations such as pallet wire pressing, foreign objects on the AGV road, and packaging material placement in complex environments such as poor lighting, ensuring the reliability of detection.

[0219] (3) Relying solely on depth cameras can complete various violation detections, with the characteristics of simple hardware requirements, flexible deployment, and strong adaptability.

[0220] The above specific implementation manners are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the examples, those of ordinary skill in the art should understand that the technical solutions of the invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A detection method for illegal stacking, characterized in that, Including: Making the collected three-dimensional image data into an image segmentation dataset, and respectively obtaining a road line mask, ground area information, tray area information, and AGV road marker area information based on the image segmentation dataset; Optimizing the road line mask through a line detection algorithm and a line segment similarity algorithm to generate optimized road line area information; obtaining an AGV road detection area through the ground area information, road line area information, and AGV road marker area information; obtaining a depth channel foreground area from the depth channel image through a distance parameter and tray area information; Performing illegal stacking detection based on the road line area information, AGV road detection area, tray area information, depth foreground area, and packaging material area information. The illegal stacking detection specifically includes: Obtaining the intersection area of the tray area information and the road line area information. If it exists and meets the predetermined range, it is determined that there is a tray pressing line area; Generating a foreign object rectangle parameter array through the AGV road detection area, tray pressing line area, and depth channel foreground area, and then obtaining the foreign object area in the AGV road according to the intersection rate between the foreign object rectangle parameter arrays; Obtaining the packaging material area that is not placed on the tray as required through the packaging material area information, tray area information, and ground area information; The tray pressing line area, the foreign object area in the AGV road, and the packaging material area that is not placed on the tray as required are illegal stackings.

2. The detection method for illegal stacking according to claim 1, wherein The specific steps for generating the road line area information include: Obtaining the endpoint information of line segments through a line detection algorithm based on the road line mask; Removing redundant line segments through line segment similarity analysis based on the line segment endpoint information to obtain a list of line segments after duplicate removal; Dividing the line segments in the line segment list into horizontal line segments and vertical line segments by calculating the line segment angles; Calculating the combination of adjacent horizontal and vertical line segments, obtaining the extended intersection points, obtaining the connecting lines and extended lines from the intersection points to the endpoints of the horizontal and vertical line segments respectively, and making them into a second road line mask; Calculating the remaining vertical line segments, then performing linear fitting on them and making them into a third road line mask; Adding the second road line mask and the third road line mask to obtain the optimized road line area information.

3. The detection method for illegal stacking according to claim 2, wherein The line segment similarity analysis uses a cosine similarity algorithm, and the specific formula is: A i , B i represent the components of line segment A and line segment B respectively; among them, the endpoint coordinates of line segment A are (x1, y1) and (x2, y2), and its vector form is A i The calculation method is as follows: A i = (x2 - x1, y2 - y1).

4. The detection method for illegal stacking according to claim 1, characterized in that, Obtaining the AGV road detection area, and the specific steps include: Subtracting the road line area information from the ground area information to obtain second ground area information; Converting the AGV road marker area information into marker information; Calculating the inclusion relationship between the second ground area information and the outer contour in the marker information, extracting the contour of the marker information included in the second ground area information, and filling and drawing to obtain third ground area information; When the third ground area information is a completely black image, extracting the largest connected area in the second ground area information and performing convex hull fitting, filling and drawing in the third ground area information, subtracting the road line area information from the third ground area information, and extracting the largest connected area and filling and drawing to obtain fourth ground area information. The fourth ground area information B4 is the AGV road detection area; When the third ground area information is not a completely black image, the third ground area information is subjected to convex hull fitting, then the road line area information is subtracted, and the largest connected area is extracted and filled to obtain the fourth ground area information, and the fourth ground area information is the AGV road detection area.

5. The detection method for illegal stacking according to claim 1, characterized in that, The specific steps for obtaining the pallet pressing line area include: Extract the intersection area of the pallet area information and the road line area information as the second pallet area information, extract the contour of the second pallet area information, and filter out the areas that do not meet the predetermined size; Calculate the inclusion relationship between the unfiltered area and the outer contour of the pallet area information, extract the contour of the pallet area information that contains the contour of the unfiltered area and perform filling to obtain the third pallet area information, and the third pallet area information is the pallet pressing line area.

6. The detection method for illegal stacking according to claim 1, wherein The specific steps for obtaining the foreground area of the depth channel include: Obtain the first depth channel image information by excluding the set area in the distance parameter of the depth channel image; Exclude the pallet area information in the first depth channel image information and extract the largest connected area. After performing erosion and dilation processing on this area and then adding the pallet area information, extract the largest connected area to obtain the second depth channel image information, and the second depth channel image information E2 is the foreground area of the depth channel.

7. The detection method for illegal stacking according to claim 1, characterized in that, The foreign object rectangle parameter array includes: a candidate foreign object rectangle parameter array, a human area rectangle parameter array, and a pallet pressing line area parameter array.

8. The detection method for illegal stacking according to claim 7, characterized in that, The specific steps for generating the candidate foreign object rectangle parameter array include: Add the AGV road detection area and the pallet area information, extract the largest connected area and exclude the ground area information to obtain the fifth ground area information. Perform an image logical AND operation on the fifth ground area information and the foreground area of the depth channel for the binary image and the grayscale or color image to obtain the third depth channel image information; Extract the intersection area of the third depth channel image information and the pallet area information. If this area is not a completely black image, verify the inclusion relationship between its contour and the outer contour of the pallet area information, extract the conforming contours for filling to generate the fourth depth channel image information, and perform rectangle fitting on the outer contour of the fourth depth channel image information and store the result in the candidate foreign object rectangle parameter array; Extract the non-intersection area of the third depth channel image information and the pallet area information, traverse its outer contour and extract the center point: if the center point is inside the contour, generate a mask through the SAM model and fit a rectangle and store it in the candidate foreign object rectangle parameter array; if the center point is not inside the contour, directly fit the contour rectangle and store it in the candidate foreign object rectangle parameter array; The specific steps for generating the human area rectangle parameter array include: Extract the human area information of the image data through the YOLO general human model, and obtain the rectangle parameters of the human area information and store them in the human area rectangle parameter array; The specific steps for generating the pallet pressing line area parameter array include: Extract the outer contour of the pallet pressing line area information D3, perform rectangle fitting on each outer contour, and extract the rectangle parameters and store them in the pallet pressing line area parameter array.

9. The detection method for illegal stacking according to claim 1, characterized in that, The specific steps for obtaining the foreign object area in the AGV road include: Traverse the foreign object rectangle parameter array, and calculate the intersection rate of the rectangles in the candidate foreign object rectangle parameter array with the rectangles in the person area rectangle parameter array and the tray pressing line area parameter array; if the intersection rate is greater than 0.75, delete the corresponding rectangle information in the candidate foreign object rectangle parameter array and update the data; the rectangle area of the updated foreign object area parameter array is the foreign object area in the detected AGV road.

10. The detection method for illegal stacking according to claim 1, characterized in that The specific steps for obtaining the packaging material area that is not placed on the tray as required include: Extract the packaging material area from the RGB channel image of the RGBD camera through HSV color thresholds, and at the same time remove the areas whose contours do not meet the preset range standard, so as to obtain the packaging material area information; Extract the lower rectangle area of the circumscribed rectangle of the contour of the packaging material area information, and extend this rectangle area downward as the second packaging material area information, and the second packaging material area information corresponds to the tray detection area for each packaging material; Calculate the contour intersection relationship between the second packaging material area information and the tray area information; If there is no intersection, eliminate the packaging material area information in the second packaging material area information, calculate the overlap rate of the remaining area with the contour of the ground area information, and if the overlap rate is greater than 0.5, draw the remaining area to obtain the third packaging material area information; Extract the areas corresponding to the packaging material area information and the third packaging material area information, and the areas are the packaging material areas that are not placed on the tray as required.

Images