A fast multi-scale weakly supervised box edge perception method and device

Abstract

The present invention provides a fast, multi-scale, weakly supervised box edge perception method and device, comprising the following steps: Step S1, training a deep learning model; Step S2, obtaining a ROI of an image to be identified and segmenting it; Step S3, if the number of segments without pixels after edge conversion is less than a first threshold, proceeding to Step S4; otherwise, performing edge conversion after cluster analysis and proceeding to Step S4; Step S4, filtering out untrustworthy data from candidate data to obtain final candidate data; Step S5, fitting a candidate line, and determining whether to proceed to Step S6 based on the distance variance between the final candidate data and the candidate line; Step S6, obtaining an ideal line, dividing the data points around the candidate line into two categories, fitting two candidate lines for each of the two categories, and determining the final edge line. The present invention can perceive the edge line of a box in a complex environment, making detection faster and more accurate, thereby reducing enterprise costs.

Description

Technical Field

[0001] The present invention relates to a fast multi-scale weakly supervised box edge perception method and device. Background Art

[0002] In recent years, the mechanization and automation of logistics operations have become crucial tools for companies to reduce costs and improve efficiency within the logistics industry. With the rapid development of artificial intelligence, the use of machine vision to simulate human vision is enabling forklifts to perform more intelligent tasks. The application of machine vision in industrial forklifts is not limited to guidance and positioning; it can also be used to detect the appearance and contours of target products and identify high-precision edges.

[0003] Currently, popular unmanned forklift systems in China use radio frequency identification technology to locate cargo, requiring a large number of electronic tags to be labeled. Furthermore, due to a lack of sufficient sample data, deep learning alone is unable to meet logistics requirements. These efforts undoubtedly incur significant costs and overhead. Addressing this pain point, how to quickly and accurately detect the edges of boxes while keeping costs low is a major challenge. Summary of the Invention

[0004] The present invention proposes a fast, multi-scale, weakly supervised box edge perception method and device, which can perceive the edge straight lines of the box in complex environments without a large amount of sample data. The detection is faster and more accurate, which is conducive to reducing enterprise costs.

[0005] The present invention is achieved through the following technical solutions:

[0006] A fast multi-scale weakly supervised box edge perception method includes the following steps:

[0007] Step S1: Obtain a box image as training data to train a deep learning model for identifying box contours;

[0008] Step S2: Obtain an image to be identified, identify the image using a trained deep learning model, select a ROI based on the identification result, and segment the ROI;

[0009] Step S3: Perform edge conversion on the ROI. If the number of segments without pixels after edge conversion is less than a set first threshold, proceed to step S4. Otherwise, perform cluster analysis on the RGB of the ROI and then perform edge conversion on the ROI, and proceed to step S4.

[0010] Step S4: extract candidate data from each segment according to the nearest neighbor strategy, filter out untrustworthy data in the candidate data according to the confidence of the pixels in the defined ROI to obtain credible data, find the outlier boundary in the credible data, and filter the credible data again according to the outlier boundary to obtain the final candidate data;

[0011] Step S5: Fit the final candidate data to a candidate line using the least squares method, and determine whether the distance variance between the final candidate data and the candidate line is less than the set variance threshold. If so, the candidate line is the final identified edge line; otherwise, proceed to step S6.

[0012] Step S6, according to the recognition result of step S3, obtain the ideal straight line, divide the data points around the candidate straight line into two categories of data according to the second principal component direction of PCA, and use quadratic PCA to fit the two categories of data to obtain two candidate straight lines, and determine the final edge straight line according to the similarity between the two candidate straight lines and the ideal straight line. By selecting a small amount of candidate data to perceive the edge straight line of the box, the amount of calculation is greatly reduced. If there are too many pixel-free areas after the edge conversion, it means that the data is seriously missing. In this case, cluster analysis is first performed on this part of the image, and then edge conversion is performed to improve the edge conversion results of the locally blurred and unclear image, which is conducive to improving the final recognition result. Therefore, the present invention does not require a large amount of sample data to perceive the edge straight line of the box in a complex environment. It is less affected by the complex internal and external environment or pattern texture. The detection is faster and more accurate, which is conducive to reducing enterprise costs. The present invention is not only suitable for the recognition of boxes in intelligent forklifts, but also can provide assistance for future labeling work.

[0013] Furthermore, in step S1, after obtaining the box image, the image is annotated and saved as an XML or JSON format file to serve as training data for the deep learning model, and the deep learning model is Rotated Faster R-CNN.

[0014] Furthermore, in step S2, the recognition result is a plurality of box contour lines, and the ROI is determined based on these contour lines.

[0015] Furthermore, in step S2, the ROI is segmented by geometric division.

[0016] Furthermore, in step S3, the Canny operator is used to perform edge conversion, and the KMeans algorithm is used to perform cluster analysis.

[0017] Furthermore, in step S4, the outlier boundary is found from the credible data using a box plot method.

[0018] Furthermore, in step S5, the least square method is used to fit the straight line, and the candidate data is continuously updated during the fitting process. The fitting process is terminated when the candidate data before and after the update are consistent, thereby obtaining the candidate straight line.

[0019] Furthermore, in step S6, the recognition result includes the box width, height and the contour line, and the ideal line is calculated by combining the recognition result with the rectangular feature of the box.

[0020] The present invention is also achieved through the following technical solutions:

[0021] A fast multi-scale weakly supervised box edge perception device, including the following modules:

[0022] Training module: used to obtain box images as training data and train the deep learning model for identifying box contours;

[0023] ROI segmentation module: used to obtain the image to be identified, identify the image to be identified using the trained deep learning model, select the ROI based on the recognition result and segment the ROI;

[0024] Edge conversion module: used to perform edge conversion on the ROI. If the number of segments without pixels after edge conversion is less than the set first threshold, candidate data is extracted from each segment according to the nearest neighbor strategy. Otherwise, the RGB of the image to be identified corresponding to the segment without data is clustered to make its edge differentiation more obvious. Then, edge conversion is performed on each segment and candidate data is extracted from each segment according to the nearest neighbor strategy.

[0025] Candidate data determination module: used to filter out untrustworthy data from candidate data according to the confidence of the pixels in the defined ROI to obtain credible data, find the outlier boundary from the credible data, and filter the credible data again according to the outlier boundary to obtain the final candidate data;

[0026] Edge line determination module: The least squares method is used to fit the final candidate data to obtain a candidate line. When the distance variance between the final candidate data and the candidate line is less than the set variance threshold, the candidate line is the final identified edge line. Otherwise, the ideal line is obtained according to the recognition result of the deep learning model. The data points around the candidate line are divided into two categories of data according to the second principal component direction of PCA, and the quadratic PCA is used to fit these two categories of data to obtain two candidate lines. The final edge line is determined based on the similarity between the two candidate lines and the ideal line. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described in detail below with reference to the accompanying drawings.

[0028] Figure 1 Flowchart of the present invention.

[0029] Figure 2 This is a schematic diagram of the edge conversion results combined with cluster analysis in the present invention. DETAILED DESCRIPTION

[0030] like Figure 1 As shown in Figure 2, the fast multi-scale weakly supervised box edge perception method includes the following steps:

[0031] Step S1: Obtain a small sample of box images as training data to train a deep learning model for identifying box contours; wherein, the box images are obtained by photographing, and the photographed images are annotated and saved as XML or JSON format files to serve as training data for the deep learning model. The deep learning model in this embodiment is Rotated Faster R-CNN;

[0032] Step S2: Capture and acquire an image to be recognized, use the trained deep learning model to recognize the image to be recognized, select a ROI (region of interest) based on the recognition result, and segment the ROI; the recognition result is a plurality of box contour lines, and the ROI is determined based on these contour lines, and the ROI is divided into N segments by geometric segmentation;

[0033] Step S3, edge conversion of the ROI using the Canny operator can obtain an edge image. However, since the captured image to be identified cannot guarantee that all parts are clear enough, in order to obtain a more accurate edge image, a judgment needs to be made after edge conversion of the ROI using the Canny operator. That is, if the number of segments without pixels after edge conversion is less than a set first threshold, it means that an edge image with a clear outline is obtained, and the process proceeds to step S4. Otherwise, cluster analysis is performed on the RGB of the ROI using the KMeans algorithm to make the edge differentiation of this part of the image to be identified more obvious. Then, edge conversion is performed on the ROI after clustering processing to obtain an edge image with a clear outline, and the process proceeds to step S4.

[0034] exist Figure 2 middle, Figure 2-1 is the image to be recognized, Figure 2-2 This is the edge image after edge conversion using only the Canny operator. The right edge of the image is almost invisible. Figure 2 -3 to Figure 2 -5 are the images after cluster analysis with K=4, K=5, and K=6 respectively. Figure 2 -6 is the edge image after cluster analysis and edge conversion. It can be seen that the entire contour is very clear. This step of processing makes the final edge line more accurate. In this embodiment, the first threshold is set to 50%;

[0035] Step S4: Filter out untrustworthy data from candidate data according to the confidence of the pixels in the defined ROI, and obtain N m Credible data (where N≤N m ), and then find the outlier boundaries from the trusted data and filter the trusted data again according to the outlier boundaries to obtain the final candidate data; the outlier boundaries are found from the trusted data using the box plot method, where outliers refer to straight lines that are obviously not box outlines, and outlier boundaries refer to the boundaries of the range where these straight lines are located;

[0036] Step S5: Fit the final candidate data to a candidate line using the least squares method, and determine whether the distance variance between the final candidate data and the candidate line is less than a set variance threshold. If so, it indicates that the image to be identified is a single box, and the candidate line is the final identified edge line. Otherwise, it indicates that the image to be identified is likely to be two or more boxes overlapping, and then proceed to step S6; the set variance threshold is set according to the actual situation; the candidate data is continuously updated during the process of fitting the candidate line, and the fitting process is terminated when the candidate data before and after the update are consistent, and the candidate line is obtained. Otherwise, the fitting process is continued;

[0037] Step S6: Obtain an ideal straight line based on the recognition result of step S3, divide the data points around the candidate straight line into two categories of data according to the second principal component direction of PCA, and use quadratic PCA to fit these two categories of data to obtain two candidate straight lines, and determine the final edge straight line based on the similarity between the two candidate straight lines and the ideal straight line. The recognition result specifically refers to the width, height and contour straight line of the box. The ideal straight line is calculated by combining the recognition result with the rectangular feature of the box. The ideal straight line is the true edge straight line equation of the box.

[0038] Correspondingly, the fast multi-scale weakly supervised box edge perception device includes the following modules:

[0039] Training module: used to obtain box images as training data and train the deep learning model for identifying box contours;

[0040] ROI segmentation module: used to obtain the image to be identified, identify the image to be identified using the trained deep learning model, select the ROI based on the recognition result and segment the ROI;

[0041] Edge conversion module: used to perform edge conversion on the ROI. If the number of segments without pixels after edge conversion is less than the set first threshold, candidate data is extracted from each segment according to the nearest neighbor strategy. Otherwise, the RGB of the image to be identified corresponding to the segment without data is clustered to make its edge differentiation more obvious. Then, edge conversion is performed on each segment and candidate data is extracted from each segment according to the nearest neighbor strategy.

[0042] Candidate data determination module: used to filter out untrustworthy data from candidate data according to the confidence of the pixels in the defined ROI to obtain credible data, find the outlier boundary from the credible data, and filter the credible data again according to the outlier boundary to obtain the final candidate data;

[0043] Edge line determination module: The least squares method is used to fit the final candidate data to obtain a candidate line. When the distance variance between the final candidate data and the candidate line is less than the set variance threshold, the candidate line is the final identified edge line. Otherwise, the ideal line is obtained according to the recognition result of the deep learning model. The data points around the candidate line are divided into two categories of data according to the second principal component direction of PCA, and the quadratic PCA is used to fit these two categories of data to obtain two candidate lines. The final edge line is determined based on the similarity between the two candidate lines and the ideal line.

[0044] The above description is merely a preferred embodiment of the present invention and therefore cannot be used to limit the scope of the present invention. In other words, equivalent changes and modifications made according to the scope of the patent application and the contents of the specification should still fall within the scope of the patent of the present invention.

Claims

1. A fast multi-scale weakly supervised box edge perception method, characterized by: The steps include: Step S1: Obtain a box image as training data to train a deep learning model for identifying box contours; Step S2: Obtain an image to be identified, identify the image using a trained deep learning model, select a ROI based on the identification result, and segment the ROI; Step S3: Perform edge conversion on the ROI. If the number of segments without pixels after edge conversion is less than a set first threshold, proceed to step S4. Otherwise, perform cluster analysis on the RGB of the ROI and then perform edge conversion on the ROI, and proceed to step S4. Step S4: extract candidate data from each segment according to the nearest neighbor strategy, filter out untrustworthy data in the candidate data according to the confidence of the pixels in the defined ROI to obtain credible data, find the outlier boundary in the credible data, and filter the credible data again according to the outlier boundary to obtain the final candidate data; Step S5: Fit the final candidate data to a candidate line using the least squares method, and determine whether the distance variance between the final candidate data and the candidate line is less than the set variance threshold. If so, the candidate line is the final identified edge line; otherwise, proceed to step S6. Step S6: Obtain the ideal straight line based on the recognition result of step S3, divide the data points around the candidate straight line into two categories according to the second principal component direction of PCA, and use quadratic PCA to fit these two categories of data to obtain two candidate straight lines respectively. Determine the final edge straight line based on the similarity between the two candidate straight lines and the ideal straight line.

2. The fast multi-scale weakly supervised box edge perception method according to claim 1, characterized in that: In step S1, after obtaining the box image, the image is annotated and saved as an XML or JSON format file to serve as training data for the deep learning model, which is Rotated Faster R-CNN.

3. The fast multi-scale weakly supervised box edge perception method according to claim 1, characterized in that: In step S2, the recognition result is a plurality of box contour lines, and the ROI is determined based on these contour lines.

4. The fast multi-scale weakly supervised box edge perception method according to claim 3, characterized in that: In step S2, the ROI is segmented by geometric division.

5. The fast multi-scale weakly supervised box edge perception method according to claim 1, 2 or 3, characterized in that: In step S3, the Canny operator is used to perform edge conversion, and the KMeans algorithm is used to perform cluster analysis.

6. The fast multi-scale weakly supervised box edge perception method according to claim 1, 2 or 3, characterized in that: In step S4, the outlier boundary is found from the credible data using the box plot method.

7. The fast multi-scale weakly supervised box edge perception method according to claim 1, 2 or 3, characterized in that: In step S5, the least square method is used to fit the straight line, and the candidate data is continuously updated during the fitting process. The fitting process is terminated when the candidate data before and after the update are consistent, thereby obtaining the candidate straight line.

8. The fast multi-scale weakly supervised box edge perception method according to claim 4, characterized in that: In step S6, the recognition result includes the box width, height and the outline straight line, and the ideal straight line is calculated by combining the recognition result with the rectangular feature of the box.

9. A fast multi-scale weakly supervised box edge perception device, characterized by: Includes the following modules: Training module: used to obtain box images as training data and train the deep learning model for identifying box contours; ROI segmentation module: used to obtain the image to be identified, identify the image to be identified using the trained deep learning model, select the ROI based on the recognition result and segment the ROI; Edge conversion module: used to perform edge conversion on the ROI. If the number of segments without pixels after edge conversion is less than the set first threshold, candidate data is extracted from each segment according to the nearest neighbor strategy. Otherwise, the RGB of the image to be identified corresponding to the segment without data is clustered to make its edge differentiation more obvious. Then, edge conversion is performed on each segment and candidate data is extracted from each segment according to the nearest neighbor strategy. Candidate data determination module: used to filter out untrustworthy data from candidate data according to the confidence of the pixels in the defined ROI to obtain credible data, find the outlier boundary from the credible data, and filter the credible data again according to the outlier boundary to obtain the final candidate data; Edge line determination module: The least squares method is used to fit the final candidate data to obtain a candidate line. When the distance variance between the final candidate data and the candidate line is less than the set variance threshold, the candidate line is the final identified edge line. Otherwise, the ideal line is obtained according to the recognition result of the deep learning model. The data points around the candidate line are divided into two categories of data according to the second principal component direction of PCA, and the quadratic PCA is used to fit these two categories of data to obtain two candidate lines. The final edge line is determined based on the similarity between the two candidate lines and the ideal line.

Images