Empty basket detection method based on 3D vision
Through the combination of depth camera and infrared detection, combined with deep learning and point cloud conversion technology, the existing empty basket detection efficiency and high error detection rate are solved, and efficient empty basket recognition in a completely black environment is achieved, reducing the number of new baskets The labor and usage cost are improved, and the identification accuracy and efficiency are improved.
Patent Information
- Application Number
- CN202411008452.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The existing empty basket detection methods are inefficient, are susceptible to ambient light and background interference, and cannot be effectively identified in a completely black environment. The scratch error rate of the basket installation is high, resulting in an increase in labor costs and usage costs.
Using a combination of depth cameras and infrared detection, a binary classification model is established through the key point detection and object detection of deep learning to determine whether the basket is pasted with advertisements, and through point cloud conversion and clustering, it is judged whether the basket is empty, reducing Mis-checking improves identification accuracy and efficiency.
Achieve efficient empty basket identification in a black environment, reduce labor costs and usage costs, improve detection efficiency, reduce false detection caused by scratches, and enhance identification accuracy.
Smart Images

Figure CN118537851B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to an empty basket detection method based on 3D vision. Background Art
[0002] In the security inspection places of the prior art, luggage is usually placed in baskets, which are transported by means of a transport device so as to be sequentially inspected through various security inspection equipment. After the luggage inspection is completed, the passengers take away the luggage, and finally the baskets are recycled. During the recycling process of the baskets, it is necessary to check whether the passengers have taken away all the items placed on the baskets, that is, whether the baskets are empty. If there are any leftovers, an alarm is required to remind the user. Otherwise, the baskets will have to be traced back after they flow into the recycling queue. This has poor timeliness and complicated procedures. Therefore, it is necessary to check whether the baskets are empty before entering the recycling queue.
[0003] Existing empty basket detection is mainly divided into two methods: manual judgment and conventional visual algorithm judgment. Manual judgment is to manually check whether there are any items left in the basket. This method has the advantages of simple and accurate recognition, but it has the disadvantages of time-consuming, labor-intensive, low efficiency, and high labor costs. Visual analysis technology is used to classify the background features of the basket or the target features that may exist in the basket, so as to achieve the purpose of determining whether there is a target on the basket. It has the advantages of accurate recognition and high recognition efficiency, but traditional image analysis is easily affected by environmental interference, such as the brightness of the ambient light, the type of target, and the feature distinction between the target and the background. Especially when advertisements with different patterns are posted in the background frame, the above defects are particularly prominent. When the above method is used for empty basket detection, it is required that no advertisements can be posted and the basket cannot have scratches. However, the advertisements posted on the basket are part of the airport's revenue. Giving up advertisements will cause the airport to lose revenue, and the basket is prone to scratches. Frequent replacement of the basket will lead to excessively high use costs and poor feasibility. Summary of the invention
[0004] One object of the present invention is to provide an empty basket detection method based on 3D vision, which adopts a combination of depth and infrared detection. It not only overcomes the problem that the current visual detection cannot detect the empty basket after advertisements are posted in the basket, but also overcomes the problem of requiring external fill light and ambient light interference. It can identify the empty basket in a completely dark environment, and at the same time introduces depth recognition to increase the accuracy of recognition and reduce false detections caused by scratches on the basket. It saves time and has low delay, improves detection efficiency, and reduces labor costs.
[0005] Other advantages and features of the present invention will become more apparent from the following detailed description and may be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims.
[0006] According to one aspect of the present invention, a 3D vision-based empty basket detection method of the present invention that can achieve the aforementioned objectives and other objectives and advantages includes the following steps:
[0007] Step A: calibrate the installation position of the depth camera to obtain the bottom plane equation of the basket based on the camera coordinate system;
[0008] Step B: Using a deep learning key point detection method to determine the detection area, the detection area is the entire bottom area of the basket;
[0009] Step C: A binary classification model is established for the baskets with and without advertisements through a deep learning target detection method to determine whether advertisements are posted before detection, and then different detection methods are used to perform detection based on different judgment results;
[0010] Step D: If there are advertisements in the placing basket, the detection area is converted into a point cloud. If there are no advertisements in the placing basket, infrared image noise reduction, edge extraction, and target area point cloud conversion are performed, and the distance from the point cloud to the bottom plane of the placing basket is calculated. If it exceeds the preset height threshold, the point cloud is retained, and then the retained point cloud is clustered to generate clusters. It is determined whether the number of point clouds in each cluster exceeds the number threshold. If the number of point clouds exceeds the number threshold, the cluster is retained. If the number of point clouds does not exceed the number threshold, after the overall cluster is filtered, check whether there are still clusters. If there are remaining clusters, the placing basket is determined to be non-empty. If there are no remaining clusters, the placing basket is determined to be empty.
[0011] According to one embodiment of the present invention, step D includes sub-step D1: if there are advertisements in the placing basket, the depth data in the detection area is converted into point cloud data through a conversion formula in combination with the depth map, and the distance from the point cloud to the bottom plane of the placing basket is calculated through target detection.
[0012] According to one embodiment of the present invention, step D includes sub-step D2: if there is no advertisement in the placing basket, then for the collected infrared image, Gaussian filtering is used to reduce the noise data of the image, an edge detection algorithm is used to perform edge detection, the amplitude and direction of the gradient are obtained, a non-maximum suppression method is used to obtain a more detailed edge, the final edge information is determined by a double threshold algorithm, the obtained edge is converted into a target area, and the target area is converted into point cloud information accordingly, and the distance from the point cloud to the bottom plane of the placing basket is calculated.
[0013] According to one embodiment of the present invention, step A includes sub-step A1: collecting depth data of the empty basket, manually calibrating the bottom area of the basket, marking the coordinates of the four vertex key points of the basket, and determining them as the bottom fitting area.
[0014] According to one embodiment of the present invention, step B includes sub-step B1: collecting infrared images of the placing basket at different angles through a depth camera, establishing a depth detection model of the infrared image and the four vertex key points of the bottom surface of the placing basket, applying the above-mentioned trained model, and using a test set to verify the effect.
[0015] According to one embodiment of the present invention, step C includes sub-step C1: using a depth camera arranged at the top, continuously collecting infrared images of baskets without advertisements and baskets with advertisements of different specifications, unifying the sizes of all images, and using a labeling tool to label the collected data, wherein the labels are divided into those with advertisements and those without advertisements. The labeled data are divided into three categories according to a ratio of 8:1:1, including a training set, a verification set, and a test set, and are further input into a YOLOV5 model for training. By adjusting the training parameters, the better trained model is retained, the trained model is applied, and the effect is verified using a test set.
[0016] According to one embodiment of the present invention, step D includes sub-step D1: obtaining a depth map and an infrared map of the placing basket through a depth camera, applying a depth detection model to perform key point detection on the infrared map, obtaining a detection area, and applying a YOLOV5 model to determine whether an advertisement is pasted in the detection area.
[0017] According to one embodiment of the present invention, step A includes sub-step A2: converting the bottom surface fitting area into a point cloud, and the conversion formula is: , where X, Y, and Z correspond to the spatial coordinates in the camera coordinate system, and depth represents the depth value of the pixel. and Represents the coordinates of the pixel point, ppx and ppy represent the coordinates of the optical center of the depth camera, and fx and fy represent the focal lengths of the depth camera in the horizontal and vertical directions.
[0018] According to one embodiment of the present invention, step A includes sub-step A3: calling a depth camera to take images, obtaining the area within the four vertex key points and converting them into a spatial point cloud, using a random sampling consistency algorithm to perform plane fitting, and obtaining the bottom plane equation of the basket.
[0019] According to one embodiment of the present invention, step D includes sub-step D3: preprocessing the infrared image, the preprocessing step includes noise reduction and brightness normalization, and then edge detection is performed on the detection area. The edge detection generates a gradient area, and the detected gradient area is converted into a point cloud in combination with the depth data, and the distance between the point cloud and the bottom plane equation of the placing basket is calculated. The point cloud exceeding the height threshold is retained, and clustering is also performed and compared with the quantity threshold. Clusters greater than the quantity threshold are retained. If a cluster exists, it indicates that the placing basket is not empty. If no cluster exists, it indicates that the placing basket is empty.
[0020] The beneficial effects of the present invention are as follows: in addition to providing the RGB images output by traditional cameras, the depth camera used in the present invention can also provide depth information of the target by actively transmitting and receiving lasers and analyzing the laser reflection characteristics, thereby avoiding the need for an additional lighting system and eliminating the influence of external light. Moreover, by analyzing the depth of the target, the foreground and background can be effectively distinguished. By combining depth and infrared detection, empty baskets can be identified in a completely dark environment, depth recognition is introduced, the accuracy of recognition is increased, and false detections caused by scratches on the loading baskets are reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 The present invention is a flowchart of a method for detecting an empty basket based on 3D vision according to an embodiment of the present invention.
[0022] Figure 2 These are the four vertex key points and detection areas of infrared image detection of the empty basket detection method based on 3D vision in the above-mentioned embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.
[0024] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.
[0025] It is understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number. In addition, in the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more than two.
[0026] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0027] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0028] like Figure 1 to Figure 2 As shown, an empty basket detection method based on 3D vision of an embodiment of the present invention is disclosed. The empty basket detection method based on 3D vision can identify empty baskets in a completely dark environment, introduces depth recognition, increases recognition accuracy, and reduces false detections caused by scratches on the baskets. It consumes less time and has lower delays, thereby improving detection efficiency and reducing labor costs.
[0029] In short, the detection process of the empty basket detection method based on 3D vision includes the following steps: first, fix the depth camera at an obliquely upper position of the empty basket detection system, place the empty basket for imaging, use the pre-trained key point detection model to perform key point detection, and detect the area surrounded by the four vertex key points of the bottom area of the basket, which is the detection area; then, call the depth camera to take images, obtain the area within the four vertex key points and convert it into a spatial point cloud, use the random sampling consistency algorithm to perform plane fitting, and obtain the bottom plane equation of the basket; then, use the infrared image collected by the depth camera to establish a binary classification model to identify whether there are advertisements pasted on the basket. If there are advertisements, all depth data in the detection area are converted into point clouds, and calculate whether the distance from each point cloud to the fitting plane exceeds the height threshold. If it exceeds the height threshold, the point cloud exceeding the height threshold is retained, and then clustering is performed to obtain different clusters. The number of point clouds in a single cluster is traversed and counted to see if it exceeds the number threshold. If the number of point clouds exceeds the number threshold, it indicates that the basket is not empty. If the number of point clouds does not exceed the number threshold, it indicates that the basket is empty. If there is no advertisement, the infrared image is first preprocessed. The preprocessing steps include noise reduction and brightness normalization. Then, edge detection is performed on the detection area. The edge detection generates a gradient area, and the detected gradient area is converted into a point cloud in combination with the depth data. The distance between the point cloud and the bottom plane equation of the basket is calculated. The point cloud exceeding the height threshold is retained. Clustering is also performed and compared with the number threshold. Clusters greater than the number threshold are retained. If a cluster exists, it indicates that the basket is not empty. If no cluster exists, it indicates that the basket is empty.
[0030] In detail, the empty basket detection method based on 3D vision includes the following steps:
[0031] Step 1: Calibrate the installation position of the depth camera to obtain the bottom plane equation of the mounting basket based on the camera coordinate system. Specifically, after being installed in the application scenario, the relative position of the depth camera and the shooting area is rigidly connected and will not change. Therefore, after the installation is completed, only one calibration is required. In detail, it includes the following sub-steps:
[0032] Step 1: Fix the depth camera. In order to achieve a better viewing angle, the depth camera is mounted directly above the basket in this embodiment.
[0033] Step 2: Get the basket data. First, collect the basket depth data when the basket is empty, and manually calibrate the bottom area of the basket. Mark the coordinates of the four vertex key points of the basket and determine them as the bottom surface fitting area. It is worth noting that the four vertex key points selected must be points on the bottom surface.
[0034] Step 3: Depth to point cloud: Convert the bottom surface fitting area selected in the above step into a point cloud. The conversion formula is: Among them, X, Y, and Z correspond to the spatial coordinates in the camera coordinate system, and depth represents the depth value of the pixel point. and Represents the coordinates of the pixel point, ppx and ppy represent the optical center coordinates of the depth camera, and fx and fy represent the horizontal and vertical focal lengths of the depth camera;
[0035] Step 4: Fitting the plane, using the random sampling consistency algorithm for plane fitting;
[0036] Step 2: Determine the detection area. Since there are both the basket area and the non-basket area in the captured basket image, and the conveying device will slightly change the position it reaches each time when conveying the basket, it is necessary to first eliminate the non-detection area before actual detection and retain the detection area. The detection area in the present invention is the entire bottom area of the basket. Since the shape and size of the basket are consistent, it can be determined by using both the traditional image template matching method and the key point detection method in deep learning. The present invention uses the Kapao key point detection method in deep learning to determine the detection area. In detail, it includes the following sub-steps:
[0037] Step 1: Collect data, using a depth camera to collect infrared images of the basket at different angles;
[0038] Step 2: Model making, build the depth detection model of infrared image and four vertex key points of the bottom surface of the basket;
[0039] Step 3: Model verification. Specifically, the trained model is applied and the test set is used to verify the effect.
[0040] Step 3: Detect whether there are advertisements. Specifically, when there are advertisement stickers at the bottom of the basket, if traditional image detection is used alone, a large number of false detections will occur. Determining whether there are advertisements is critical for adopting different detection strategies later. In the present invention, a deep learning target detection method is first used to establish a binary classification model for baskets with and without advertisements. Before detection, it is determined whether there are advertisements. Then, different detection methods are used to perform detection for different judgment results. Specifically, the model recognizes whether there are advertisements, including the following sub-steps:
[0041] Step 1: Data collection. Specifically, the depth camera installed at the top continuously collects infrared images of baskets without advertisements and baskets with advertisements of different specifications. The advertisements of different specifications may be, for example, different colors, different patterns, and different pasting areas and areas. In a specific application scenario, the present invention uses 10,000 images of empty baskets and 10,000 images of baskets with advertisements of different specifications.
[0042] Step 2: Data preprocessing and labeling. Specifically, the sizes of all images are unified, and the collected data are labeled using labeling tools, where the labels are divided into those with ads and those without ads.
[0043] Step 3: Data training. Specifically, the labeled data is divided into three categories according to the ratio of 8:1:1, including training set, validation set and test set, and then input into the YOLOV5 model for training. The training parameters are adjusted to retain the well-trained model.
[0044] Step 4: Model verification. Specifically, the trained model is applied and the test set is used to verify the effect.
[0045] Step 4: Detect whether the basket is empty. Through steps 1, 2, and 3, the bottom equation of the basket, the detection area of the basket, and whether there are advertisements pasted in the basket are determined. Specifically, the following sub-steps are included:
[0046] Step 1: Data collection, obtain depth map and infrared image through the depth camera;
[0047] Step 2: Key point detection: Use the model in step 2 to detect key points on the infrared image and obtain the detection area.
[0048] Step 3: Determine whether there is an advertisement. Apply the model in step 3 to determine whether there is an advertisement in the detection area.
[0049] Furthermore, if there is an advertisement, the following steps are performed:
[0050] Step 4 (1): Perform point cloud conversion on the detection area. Extract the detection area through the vertex key points of the detection area in step 2. Combined with the depth map, convert the depth data in the detection area into point cloud data through the conversion formula.
[0051] Step 5 (1): Target detection. Calculate the distance from the point cloud to the bottom of the basket. If it exceeds a certain height threshold, keep the point cloud. Then cluster the retained point clouds and determine whether the number of point clouds in each cluster exceeds the threshold. If so, keep the cluster. If not, judge it as noise. After filtering the entire cluster, check whether there are still clusters. If there are any remaining clusters, judge the basket as not empty. If not, judge the basket as empty.
[0052] Furthermore, if it is detected in step 3 that no advertisement is pasted, the following steps are performed:
[0053] Step 4 (2): Image preprocessing: For the acquired infrared image, Gaussian filtering is first used to reduce the image noise data;
[0054] Step 5 (2): Edge detection: Use edge detection algorithm to detect edges, obtain the magnitude and direction of gradients, then use non-maximum suppression method to obtain more detailed edges, and finally use double threshold algorithm to determine the final edge information.
[0055] Step 6: Region conversion, converting the acquired edge into the target region, and correspondingly converting the target region into point cloud information;
[0056] Step 7: Target filtering, calculate the distance from the point cloud to the bottom plane of the basket, retain the point cloud whose distance exceeds the height threshold for clustering, traverse each cluster, if the number of clusters is less than the number threshold, remove the cluster, and define the detected target as scratches or other stains, otherwise retain it, and determine whether there are any clusters left. If there are clusters, the basket is not empty, if there are no clusters, the basket is empty.
[0057] The 3D vision-based empty basket detection method of the present invention first uses a depth camera to collect data on the basket, applies a key point detection model to detect the collected infrared image to determine the target detection area, and then uses a binary classification model to determine whether there is an advertisement pasted in the basket. If there is an advertisement, all points in the entire detection area are converted into a point cloud. For all points in the area surrounded by the key points, the distance between the point cloud in the detection area and the bottom surface is calculated, and the point clouds exceeding the height threshold are retained and clustered. The number of each category is calculated. If the number is less than the number threshold, the cluster is excluded. If there are still clusters at the end, the basket is determined to be non-empty. If there are no clusters at the end, the basket is determined to be empty. If there is no billboard, the infrared image is denoised, edge information is extracted, and the edge area is obtained. Each area is converted into a point cloud, and the distance from the point cloud to the bottom plane of the basket is calculated. If it exceeds the height threshold, it is retained, and then clustering is performed to obtain clusters. Each cluster is traversed, and the clusters whose number is less than the number threshold are removed. Check whether there are any remaining clusters. If there are clusters, it indicates that the basket is not empty. If there are no clusters, it indicates that the basket is empty.
Claims
1. A 3D vision-based empty basket detection method, characterized in that: The following steps are involved: Step A: calibrate the installation position of the depth camera, collect the depth data of the basket when the basket is empty, manually calibrate the bottom area of the basket, mark the coordinates of the four vertex key points of the basket, determine them as the bottom surface fitting area, and obtain the bottom plane equation of the basket based on the camera coordinate system; Step B: Use a depth camera to collect infrared images of the basket at different angles, establish a depth detection model for the infrared image and the key points of the four vertices of the bottom surface of the basket, apply the established model, and use a test set to verify the effect; use a deep learning key point detection method to determine the detection area, which is the entire bottom surface area of the basket; Step C: A binary classification model is established for the baskets with and without advertisements through a deep learning target detection method to determine whether advertisements are posted before detection, and then different detection methods are used to perform detection based on different judgment results; Step D: If there are advertisements in the basket, the depth data in the detection area is converted into point cloud data through the conversion formula in combination with the depth map, and the distance from the point cloud to the bottom plane of the basket is calculated through target detection; if there are no advertisements in the basket, the collected infrared image is subjected to Gaussian filtering to reduce the noise data of the image, and the edge detection algorithm is used for edge detection to obtain the amplitude and direction of the gradient, and the non-maximum suppression method is used to obtain more detailed edges, and the final edge information is determined by the double threshold algorithm, and the obtained edges are converted into target areas, and the target areas are correspondingly The area is converted into point cloud information, and the distance from the point cloud to the bottom plane of the basket is calculated. If it exceeds the preset height threshold, the point cloud is retained, and then the retained point clouds are clustered to generate clusters. It is determined whether the number of point clouds in each cluster exceeds the number threshold. If the number of point clouds exceeds the number threshold, the cluster is retained. If the number of point clouds does not exceed the number threshold, the cluster is removed. The detection target is defined as scratches or stains. After the overall cluster is filtered, check whether there are still clusters. If there are remaining clusters, the basket is determined to be non-empty. If there are no remaining clusters, the basket is determined to be empty.
2. The empty basket detection method based on 3D vision according to claim 1, characterized in that: The step C includes sub-step C1: using a depth camera arranged at the top, continuously collecting infrared images of baskets without advertisements and baskets with advertisements of different specifications, unifying the sizes of all images, and using a labeling tool to label the collected data, wherein the labels are divided into those with advertisements and those without advertisements. The labeled data are divided into three categories according to a ratio of 8:1:1, including a training set, a verification set, and a test set, and are further input into a YOLOV5 model for training. The trained model is retained by adjusting the training parameters, and the trained model is applied to verify the effect using a test set.
3. The empty basket detection method based on 3D vision according to claim 2 is characterized in that: The step D includes sub-step D1: obtaining a depth map and an infrared map of the basket through a depth camera, applying a depth detection model to perform key point detection on the infrared map, obtaining a detection area, and applying a YOLOV5 model to determine whether an advertisement is pasted in the detection area.
4. The empty basket detection method based on 3D vision according to claim 1, characterized in that: The step A includes sub-step A2: converting the bottom surface fitting area into a point cloud, and the conversion formula is: , , , Among them, X, Y, and Z correspond to the spatial coordinates in the camera coordinate system, and depth represents the depth value of the pixel point. and Represents the coordinates of the pixel point, ppx and ppy represent the coordinates of the optical center of the depth camera, and fx and fy represent the focal lengths of the depth camera in the horizontal and vertical directions.
5. The 3D vision-based empty basket detection method according to claim 4, characterized in that: The step A includes sub-step A3: calling a depth camera to take an image, obtaining the area within the four vertex key points and converting it into a spatial point cloud, and using a random sampling consistency algorithm to perform plane fitting to obtain a bottom plane equation of the basket.
6. The empty basket detection method based on 3D vision according to claim 1, characterized in that: The step D includes sub-step D3: pre-processing the infrared image, the pre-processing steps include noise reduction and brightness normalization, and then edge detection is performed on the detection area. The edge detection generates a gradient area, and the detected gradient area is converted into a point cloud in combination with the depth data, and the distance between the point cloud and the bottom plane equation of the basket is calculated. The point cloud exceeding the height threshold is retained, and clustering is also performed and compared with the quantity threshold. Clusters greater than the quantity threshold are retained. If a cluster exists, it indicates that the basket is not empty. If no cluster exists, it indicates that the basket is empty.
