Intelligent glass dirt detection method and system based on deep learning and edge detection
Through intelligent detection methods based on deep learning and edge detection, the problems of low glass dirt detection accuracy and insufficient data set in the prior art are solved, and high-precision glass dirt positioning and contour segmentation are realized, which is suitable for intelligent robot cleaning systems.
Patent Information
- Application Number
- CN202411962513.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-29
AI Technical Summary
The prior art relies on image data in glass dirt detection and lacks large-scale glass dirt data sets, limiting the accurate positioning and cleaning of glass dirt by intelligent robots.
Using intelligent detection methods based on deep learning and edge detection, the improved YOLOv8 model is trained by constructing the glass dirt object detection image dataset, and combining the Sobel edge detection method, the positioning and contour segmentation of the glass dirt area is achieved.
It improves the accuracy and efficiency of glass dirt detection, can achieve high-precision detection with few samples, reduces data acquisition costs, and provides effective positioning information for robot cleaning systems.
Smart Images
Figure CN119904422A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of glass dirt detection, and specifically relates to an intelligent glass dirt detection method and system based on deep learning and edge detection. Background Art
[0002] Curtain wall glass often produces a certain amount of dirt during use, which is usually cleaned by workers. However, cleaning in some scenes is not only time-consuming and laborious, but also dangerous. With the development of robotics and artificial intelligence, using robots to clean curtain wall glass can save labor costs, improve efficiency, and avoid accidents. For intelligent robots, the premise of cleaning curtain wall glass is to be able to accurately locate the dirt on the glass, which requires the assistance of computer vision technology.
[0003] Current computer vision technology is highly dependent on image data, but the image acquisition of glass dirt not only requires the assistance of professionals, but also the types of glass dirt are numerous and complex. There is still no large-scale glass dirt dataset, which limits the development and application of intelligent robots to clean glass dirt. Summary of the invention
[0004] In order to solve the shortcomings of the prior art and achieve the purpose of saving costs and improving detection accuracy, the present invention adopts the following technical solutions:
[0005] The intelligent detection method of glass dirt based on deep learning and edge detection includes the following steps:
[0006] Step 1: Build a glass dirt target detection image dataset;
[0007] Step 2: construct and train a glass dirt image detection model; use the image dataset to train the constructed image detection model;
[0008] Step 3: Use the trained glass dirt image detection model to detect the glass dirt area of the image to be detected and obtain the detection frame;
[0009] Step 4: Perform edge detection on the image to be detected by edge detection method to obtain an edge area map;
[0010] Step 5: Combine the detection frame with the edge region map to locate and segment the glass dirt region in the image to be detected.
[0011] Furthermore, in step 1, a glass dirt image is collected and a target bounding box is annotated for the dirt area on the collected image. The image dataset and its annotation file are divided into a training set and a validation set as a glass dirt target detection image dataset. The annotation content is the center point coordinates (X, Y) of the rectangular box surrounding the target dirt area and the width and height (W, H) of the rectangular box.
[0012] Furthermore, in step 2, the glass dirt image detection model includes a feature extraction module, a feature fusion module and a multi-scale detection head. The feature extraction module performs feature extraction on the input image to be detected using a convolution kernel, batch normalization, and an activation function to obtain a multi-scale feature map. The feature fusion module performs fusion feature extraction on the obtained multi-scale feature map, and inputs the fused features into the multi-scale detection head. The multi-scale detection head outputs predicted detection frames at each scale, and obtains the final glass dirt predicted detection frame through non-maximum suppression.
[0013] Furthermore, the image detection model introduces a linear dynamic convolution module to replace the convolution kernel, obtains image features, performs further feature extraction through a predefined linear convolution kernel, and learns the offset of the predefined linear convolution kernel through an additional convolution layer to adapt to the shape of glass dirt, so as to enhance the feature perception capability of complex and changeable glass dirt and improve the recognition and positioning accuracy of dirt areas on the glass.
[0014] Furthermore, in step 4, the grayscale value of the horizontal and vertical edge detection of the image to be detected by the Sobel operator is solved to obtain the global image edge grayscale value and binarize it. After binarization, a preliminary edge area map E1 is obtained, in which the background area pixel value is 0 and the edge area pixel value is 1; the edge area map E1 is eroded and expanded to achieve edge area enhancement, and a smooth and coherent edge area map E2 is obtained.
[0015] Furthermore, the edge contour is extracted by the edge detection method based on the Sobel operator, and the edge point is found by finding the point with the largest first-order derivative of the pixel value, and then all the edge points are connected to obtain the edge area of the glass dirt. For the segmentation of the glass dirt area, for the image to be processed I, the edge detection formula is as follows:
[0016]
[0017] Among them, G x and G y They represent the gradient changes in the horizontal and vertical directions of the image I to be detected, respectively. G represents the global gradient change, which is used to extract the edge area in the image. This type of contour edge extraction is mainly used to further detect flaky dirt on curtain wall glass, and can reflect a clearer glass dirt area based on the improved YOLOv8 positioning detection.
[0018] Furthermore, the edge area enhancement is to select a core of a certain size to further perform multiple iterative processes of corrosion and / or expansion on the preliminary glass dirt edge result obtained by Sobel edge detection.
[0019] An intelligent glass dirt detection system based on deep learning and edge detection, including an image data set building module, an image detection module, an edge detection module, and a detection result combination module;
[0020] The image data set construction module is used to construct a glass dirt target detection image data set;
[0021] The image detection module is used to construct and train a glass dirt image detection model; use the image data set to train the constructed image detection model, and use the trained glass dirt image detection model to detect the glass dirt area of the image to be detected to obtain a detection frame;
[0022] The edge detection module performs edge detection on the image to be detected by using an edge detection method to obtain an edge area map;
[0023] The detection result combining module combines the detection frame with the edge region map to locate and segment the contour of the glass dirt region in the image to be detected.
[0024] Furthermore, the image detection model includes a feature extraction module, a feature fusion module and a multi-scale detection head;
[0025] The feature extraction module performs feature extraction on the input image to be detected to obtain a multi-scale feature map, wherein a linear dynamic convolution module is introduced to replace the convolution kernel for feature extraction to obtain image features, further extract physical signs through a predefined linear convolution kernel, and learn the offset of the predefined linear convolution kernel through an additional convolution layer to adapt to the shape of glass dirt, so as to enhance the feature perception capability of complex and changeable glass dirt and improve the recognition and positioning accuracy of the dirt area on the glass;
[0026] The feature fusion module extracts fusion features from the obtained multi-scale feature map;
[0027] The multi-scale detection head obtains the fused features, outputs the predicted detection frames at each scale, and obtains the final glass dirt predicted detection frame through non-maximum suppression.
[0028] Furthermore, the edge detection module solves the grayscale value of the horizontal and vertical edge detection of the image to be detected by the Sobel operator, obtains the global image edge grayscale value and binarizes it, and obtains a preliminary edge area map E1 after binarization, in which the background area pixel value is 0 and the edge area pixel value is 1; the edge area map E1 is corroded and expanded to achieve edge area enhancement, and a smooth and coherent edge area map E2 is obtained; the edge contour is extracted based on the Sobel operator, the edge point is found by obtaining the point with the largest first-order derivative of the pixel value, and then all the edge points are connected to obtain the glass dirt edge area, and the glass dirt area is segmented for the image to be processed.
[0029] The advantages and beneficial effects of the present invention are:
[0030] The present invention first collects the glass dirt images to be detected on site, annotates and produces a glass dirt target detection image data set; then uses the glass dirt target detection image data set to train an improved YOLOv8 model, the improved model introduces a linear dynamic convolution module, enhances the learning ability of irregular dirt areas, and obtains a trained model parameter data file; then solves the grayscale value of the image after horizontal and vertical edge detection, and then mixes, corrodes, and expands the image to extract the edge of the curtain wall glass dirt; finally, the detection frame predicted based on the improved YOLOv8 and the edge area obtained based on the Sobel edge detection are combined to realize the positioning detection and contour segmentation of the glass dirt area in the image to be detected, and complete the intelligent detection of glass dirt, which can detect white dust, white water stains, yellow mud marks and other glass curtain wall dirt. The present invention has low cost and does not need to collect and annotate a large number of glass dirt images; has good precision performance, and improves the model performance by combining deep learning with edge detection methods; the positioning information is effective, and the bounding box information can be transmitted to the robot control system to assist in completing industrial garbage cleaning. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 is a flow chart of a method in an embodiment of the present invention.
[0032] Figure 2 It is a model framework diagram of the improved YOLOv8 in an embodiment of the present invention.
[0033] Figure 3 It is an execution flow chart of linear dynamic convolution in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.
[0035] like Figure 1As shown, the glass dirt intelligent detection method based on deep learning and edge detection includes the following steps:
[0036] Step 1: Construct a glass dirt target detection image dataset; collect some glass dirt images that need to be detected in the field, and annotate the dirt area on the collected image with a target enclosing rectangular box. The annotation content is the center point coordinates (X, Y) of the rectangular box enclosing the target dirt area and the width and height (W, H) of the rectangular box. The image dataset and annotation files are divided into a training set and a validation set at a ratio of 4:1 to create a glass dirt target detection image dataset A;
[0037] Step 2: Build and train a glass dirt image detection model;
[0038] The image detection model uses an improved YOLOv8 model. The self-built glass dirt target detection dataset A is used to train the improved YOLOv8 model. The improved model introduces a linear dynamic convolution module to enhance the learning ability of irregular dirt areas. The training iteration batch is 150 times, the optimizer uses the AdamW optimizer, the learning rate adjustment class is StepLR, and the learning rate is 0.01. The training model parameter data file M is obtained;
[0039] like Figure 2 As shown in FIG, the improved YOLOv8 model consists of three modules: a feature extraction module, a feature fusion module and a multi-scale detection head. The feature extraction module extracts features from the input image I to be detected using convolution kernels, batch normalization and activation function operations to obtain multi-scale feature maps C1, C2, C3 and C4. The feature fusion module obtains the multi-scale feature maps and performs fusion feature extraction, and inputs the fused features into the multi-scale detection head. The multi-scale detection head outputs the predicted detection frames at each scale, and obtains the final glass dirt predicted detection frame through non-maximum suppression.
[0040] like Figure 3 As shown in the figure, a linear dynamic convolution module is introduced in the improved YOLOv8 model to replace the traditional conventional convolution kernel. The module is predefined as a linear convolution kernel, and the offset of the linear convolution kernel is learned by setting an additional convolution layer to adapt to the shape of glass dirt, enhance the feature perception ability of complex and changeable glass dirt, and improve the recognition and positioning accuracy of dirt areas on the glass.
[0041] Step 3: Use the trained glass dirt image detection model to detect the glass dirt area of the image to be detected and obtain the detection frame;
[0042] The model parameter data file M is loaded to predict the detection frame of the image I to be detected, so as to identify and locate the glass dirt area, and finally obtain the predicted detection frame P, whose center point coordinates are (Xp, Yp), width and height are (Wp, Hp);
[0043] Step 4: Perform edge detection on the image to be detected by edge detection method to obtain an edge area map;
[0044] Solve the grayscale value of the horizontal and vertical edge detection of the image I to be detected by the Sobel operator, obtain the grayscale value of the global image edge and perform binarization, and obtain a preliminary edge region map E1 after binarization, in which the pixel value of the background region is 0 and the pixel value of the edge region is 1; perform erosion and expansion processing on the edge region map E1 to achieve edge region enhancement, and obtain a smooth and coherent edge region map E2;
[0045] Among them, the preliminary glass dirt edge result obtained by Sobel edge detection is further corroded and expanded to achieve edge area enhancement. The kernel size of the corrosion and expansion operations is 3×3, the number of corrosion iterations is 2, and the number of expansion iterations is 5.
[0046] Edge contour extraction is achieved by the edge detection method based on the Sobel operator. The edge point is found by finding the point with the largest first-order derivative of the pixel value, and then all edge points are connected to obtain the edge area of the glass dirt to achieve the segmentation of the glass dirt area. For the image I to be processed, the edge detection formula is as follows:
[0047]
[0048] Among them, G x and G y Respectively represent the gradient changes in the horizontal and vertical directions of the image to be detected I, G represents the global gradient change, and is used to extract the edge area in the image. This type of contour edge extraction is mainly used to further detect flaky dirt on curtain wall glass, and can reflect a clearer glass dirt area based on the improved YOLOv8 positioning detection.
[0049] Step 5: Combine the detection frame obtained in step 3 with the edge area map obtained in step 4 to locate and segment the glass dirt area in the image to be detected.
[0050] The detection box P obtained based on the improved YOLOv8 prediction and the edge area map E2 obtained based on the Sobel edge detection are combined to realize the positioning detection and contour segmentation of the glass dirt area in the image I to be detected, and complete the intelligent detection of glass dirt.
[0051] Example:
[0052] According to the above-mentioned glass dirt intelligent detection method based on deep learning and edge detection, the following operations are performed:
[0053] First, a glass dirt target detection image dataset is constructed; glass dirt images that need to be detected are collected in industrial scenarios, and target boxes are annotated for them. The dataset format is the YOLO dataset format; then, a glass dirt image detection module is trained;
[0054] Secondly, train the glass dirt image detection model; first load the self-built glass dirt dataset, and then fine-tune the training on YOLOv8. The model training is completed on a computer with 1 Intel(R) Xeon(R) CPU E5-2680 v3@2.50GHz CPU and 128G memory and 2 RTX 2080Ti GPUs. The training iteration batch is 100 times, the optimizer uses AdamW optimizer, the learning rate adjustment class is StepLR, and the learning rate is 0.01.
[0055] Then, the glass dirt image detection model was tested; after completing the model training in step 2, the test set and the trained model parameter data file were input into the curtain wall glass dirt particle recognition program and the program was run to output the deep learning recognition detection frame and edge detection recognition detection frame of the glass curtain wall dirt particles. The deep learning recognition frame is used to detect block dirt, and the edge detection recognition frame is used to detect flake dirt. The test indicator uses the dirt particle recognition accuracy of the overall input result > 90%. The model test was completed on a computer with 1 Intel(R) Xeon(R) CPU E5-2680 v3@2.50GHz CPU and 128G memory and 2 RTX 2080Ti GPUs.
[0056] Finally, the application of intelligent detection of glass dirt is carried out; the trained model can detect glass dirt and generate detection frame positioning. The detection frame position and size information can be transmitted to the intelligent robot control system to assist in glass dirt cleaning.
[0057] The present invention can realize high-precision detection of glass dirt with few samples through the intelligent glass dirt detection method based on deep learning and edge detection, effectively reducing the cost of data collection and improving the model accuracy of glass dirt detection with few samples.
[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. Intelligent glass dirt detection method based on deep learning and edge detection, characterized by The steps include: Step 1: Build a glass dirt target detection image dataset; Step 2: construct and train a glass dirt image detection model; use the image dataset to train the constructed image detection model; Step 3: Use the trained glass dirt image detection model to detect the glass dirt area of the image to be detected and obtain the detection frame; Step 4: Perform edge detection on the image to be detected by edge detection method to obtain an edge area map; Step 5: Combine the detection frame with the edge region map to locate and segment the glass dirt region in the image to be detected.
2. The glass dirt intelligent detection method based on deep learning and edge detection according to claim 1, characterized in that: In the step 1, a glass dirt image is collected and a target bounding box is annotated for the dirt area on the collected image. The image dataset and its annotation file are divided into a training set and a validation set as a glass dirt target detection image dataset. The annotation content is the center point coordinates (X, Y) of the rectangular box surrounding the target dirt area and the width and height (W, H) of the rectangular box.
3. The glass dirt intelligent detection method based on deep learning and edge detection according to claim 1, characterized in that: In the step 2, the glass dirt image detection model includes a feature extraction module, a feature fusion module and a multi-scale detection head. The feature extraction module extracts features from the input image to be detected by using a convolution kernel, batch normalization, and an activation function to obtain a multi-scale feature map. The feature fusion module extracts fused features from the obtained multi-scale feature map, inputs the fused features into the multi-scale detection head, and the multi-scale detection head outputs predicted detection frames at each scale, and obtains the final glass dirt predicted detection frame through non-maximum suppression.
4. The glass dirt intelligent detection method based on deep learning and edge detection according to claim 3 is characterized in that: The image detection model introduces a linear dynamic convolution module to replace the convolution kernel, obtains image features, performs further sign extraction through a predefined linear convolution kernel, and learns the offset of the predefined linear convolution kernel through an additional convolution layer to adapt to the shape of glass dirt.
5. The glass dirt intelligent detection method based on deep learning and edge detection according to claim 1, characterized in that: In step 4, the grayscale values of the horizontal and vertical edge detection of the image to be detected by the Sobel operator are solved to obtain the global image edge grayscale value and binarize it, and a preliminary edge area map is obtained after binarization; the edge area map is enhanced to obtain a smooth and coherent edge area map.
6. The glass dirt intelligent detection method based on deep learning and edge detection according to claim 5, characterized in that: The edge contour is extracted by the edge detection method based on the Sobel operator. The edge point is found by finding the point with the largest first-order derivative of the pixel value. Then all the edge points are connected to obtain the edge area of the glass dirt. For the segmentation of the glass dirt area, for the image to be processed I, the edge detection formula is as follows: Among them, G x and G y They represent the gradient changes in the horizontal and vertical directions of the image I to be detected, and G represents the global gradient change, which is used to extract the edge area in the image.
7. The glass dirt intelligent detection method based on deep learning and edge detection according to claim 1, characterized in that: The edge region enhancement is to select a kernel of a certain size for further iterative processing of multiple corrosion and / or expansion of the preliminary glass dirt edge result obtained by Sobel edge detection.
8. The glass dirt intelligent detection system based on deep learning and edge detection includes an image data set construction module, an image detection module, an edge detection module, and a detection result combination module, and is characterized by: The image data set construction module is used to construct a glass dirt target detection image data set; The image detection module is used to construct and train a glass dirt image detection model; use the image data set to train the constructed image detection model, and use the trained glass dirt image detection model to detect the glass dirt area of the image to be detected to obtain a detection frame; The edge detection module performs edge detection on the image to be detected by using an edge detection method to obtain an edge area map; The detection result combining module combines the detection frame with the edge region map to locate and segment the contour of the glass dirt region in the image to be detected.
9. The glass dirt intelligent detection system based on deep learning and edge detection according to claim 8, characterized in that: The image detection model includes a feature extraction module, a feature fusion module and a multi-scale detection head; The feature extraction module extracts features from the input image to be detected to obtain a multi-scale feature map, wherein a linear dynamic convolution module is introduced to replace the convolution kernel for feature extraction to obtain image features, further extracts physical signs through a predefined linear convolution kernel, and learns the offset of the predefined linear convolution kernel through an additional convolution layer to adapt to the shape of glass dirt; The feature fusion module extracts fusion features from the obtained multi-scale feature map; The multi-scale detection head obtains the fused features, outputs the predicted detection frames at each scale, and obtains the final glass dirt predicted detection frame through non-maximum suppression.
10. The glass dirt intelligent detection method based on deep learning and edge detection according to claim 3, characterized in that: The edge detection module solves the grayscale value of the horizontal and vertical edge detection of the image to be detected by the Sobel operator, obtains the global image edge grayscale value and binarizes it, and obtains a preliminary edge area map after binarization; performs edge area enhancement on the edge area map to obtain a smooth and coherent edge area map; extracts the edge contour based on the Sobel operator, finds the edge point by finding the point with the largest first-order derivative of the pixel value, and then connects all the edge points to obtain the glass dirt edge area, and segments the glass dirt area for the image to be processed.