A data fusion-based automatic garbage recognition method and system

Abstract

A garbage automatic identification method and system based on data fusion. It relates to the field of garbage identification technology, specifically to the field of deep learning garbage automatic identification technology of multi-source data. It can accurately identify the types and locations of garbage around rural areas under different environmental and weather conditions, improving garbage identification efficiency. The method comprises the following steps: obtaining garbage image data captured by a camera and meteorological data of the area at that time, and preprocessing the garbage image; removing, classifying and data enhancing the garbage image; motion compensation processing of the garbage image combined with meteorological data; classifying and clustering the garbage image; calculating the color, texture and edge features of the garbage image and performing normalization processing; optimizing the deep learning model yolov11 to obtain the optimized yolov11 model; adjusting the hyperparameters of the optimized yolov11 model; and identifying garbage using the optimized yolov11 model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of garbage recognition, specifically to the technical field of deep learning garbage automatic recognition of multi-source data. BACKGROUND

[0002] With the continuous advancement of urbanization, the garbage problem in rural areas is becoming increasingly prominent, especially the garbage disposal and recognition around rural areas still have many challenges. In practical applications, the garbage around rural areas is scattered, and the shape and background environment of the garbage are complex, so the traditional garbage recognition method is difficult to adapt to these complex scenes. In addition, the image quality in rural areas is often affected by weather, lighting and other factors, which further reduces the accuracy of garbage recognition. Therefore, how to improve the accuracy of garbage recognition in complex environmental conditions has become a problem to be solved in the current garbage recognition field. The traditional garbage recognition method relies on manual patrol, which is low in efficiency and easy to cause misrecognition and missed recognition. With the development of computer vision and deep learning technology, automatic garbage recognition technology has gradually become an effective means to solve this problem. SUMMARY

[0003] The present application proposes a garbage automatic recognition method and system based on data fusion, which can accurately recognize the types and locations of garbage around rural areas under different environmental and weather conditions, improve the efficiency of garbage recognition, and provide effective technical support for garbage cleaning and environmental protection.

[0004] The method comprises the following steps:

[0005] S1, acquiring garbage image data captured by a camera and meteorological data of the area at the moment, and preprocessing the garbage image;

[0006] S2, performing de-duplication, classification and data enhancement processing on the garbage image;

[0007] S3, performing motion compensation processing on the garbage image combined with the meteorological data;

[0008] S4, performing category labeling and clustering on the garbage image;

[0009] S5, calculating the color, texture and edge features of the garbage image and performing normalization processing;

[0010] S6, optimizing the deep learning model yolov11 to obtain an optimized yolov11 model;

[0011] S7, adjusting the hyperparameters of the optimized yolov11 model;

[0012] S8, using the optimized yolov11 model for garbage recognition.

[0013] Further, when the garbage image data is photographed by the camera, data of different seasons needs to be photographed, and the garbage image preprocessing includes: evaluating the image quality, and the evaluation result is divided into good, general and poor, and the image with the poor evaluation result is deleted.

[0014] Further, when the garbage image is de-duplicated, the color texture similarity feature and the edge similarity feature of each pair of comparison images are calculated, the two features are normalized and then spliced, the similarity between the images is measured by the cosine similarity, and if the similarity is greater than 0.5, one of the garbage images is deleted; when the garbage image is classified, the garbage is classified according to the type and shape of the image, and the classification includes plastic garbage, straw garbage, construction garbage and pile-shaped garbage; when the garbage image is data enhanced, the method includes rotation, cutting and splicing and high-frequency enhancement.

[0015] Further, when the garbage image is combined with the meteorological data for motion compensation processing, the object of the motion compensation processing is the garbage image with the general evaluation result.

[0016] Further, the flow of the motion compensation processing is:

[0017] S51, determine the meteorological data corresponding to the garbage image;

[0018] S52, estimate the garbage image blur kernel direction according to the pitch angle of the garbage image and the wind direction of the meteorological data;

[0019] S53, estimate the garbage image blur kernel size according to the wind speed of the meteorological data;

[0020] S54, determine the rotation matrix according to the blur kernel size, the rotation matrix is used to rotate the blur kernel to ensure that the spatial distribution of the blur kernel is consistent with the component of the wind direction of the meteorological data on the garbage image;

[0021] S55, use the convolution kernel combined with the Wiener filter to deblur the garbage image.

[0022] Further, when the garbage image is classified and clustered, Labelimg software is used, and plastic garbage, straw garbage and construction garbage are respectively clustered; plastic garbage, straw garbage and construction garbage use DBSCAN clustering, and pile-shaped garbage uses IOU clustering.

[0023] Further, the color feature includes color histogram and HSV color space feature, the texture feature includes LBP and Tamura feature, and the edge feature includes the feature extracted by the LOG feature extraction algorithm.

[0024] Further, in the normalization process, first, the color feature, edge feature and texture feature are fused, the fused features are subjected to Z-score standardization, the mean of the fused features is changed to 0 and the standard deviation is changed to 1, the fused features are subjected to PCA operation, and the principal components are extracted.

[0025] Further, the deep learning model yolov11 is optimized, and the optimization is specifically as follows:

[0026] S91, an MLCA attention mechanism is added to the shallow feature layer and the intermediate feature layer of the backbone part of the deep learning model yolov11.

[0027] S92, a global attention mechanism is added to the deep feature layer of the backbone part of the deep learning model yolov11.

[0028] S93, color feature loss, texture feature loss and edge feature loss are added to the loss function of the deep learning model yolov11.

[0029] Further, the system comprises the following modules:

[0030] A module for acquiring garbage image data photographed by a camera and meteorological data of the region at the moment and pre-processing the garbage image;

[0031] A module for performing de-duplication, classification and data enhancement processing on the garbage image;

[0032] A module for performing motion compensation processing on the garbage image in combination with the meteorological data;

[0033] A module for performing category labeling and clustering processing on the garbage image;

[0034] A module for calculating color, texture and edge features of the garbage image and performing normalization processing;

[0035] A module for optimizing the deep learning model yolov11 to obtain an optimized yolov11 model;

[0036] A module for adjusting the hyperparameters of the optimized yolov11 model;

[0037] A module for performing garbage recognition by using the optimized yolov11 model.

[0038] The method has the beneficial effects that the multi-element data processing flow is adopted to perform data enhancement, data de-duplication and data de-blurring processing on the original rural garbage data, the clarity and classification accuracy of the garbage picture are effectively improved, more high-quality data is provided for subsequent garbage recognition based on the model, and the model precision is improved.

[0039] And, the present application does not directly use the existing recognition model as the existing garbage recognition method, but according to the characteristics of the data to be processed, the existing model is improved, thereby improving the adaptability and recognition ability of the model, and improving the accuracy of garbage recognition in special places, i.e. rural areas. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A rural garbage detection flowchart is provided for the embodiments of the present application;

[0041] Figure 2 A wind speed, wind direction polynomial interpolation graph is provided for the embodiments of the present application;

[0042] Figure 3 A rural wind speed, wind direction time graph is provided for the embodiments of the present application;

[0043] Figure 4 A graph of the angle relationship between the wind direction and the image is provided for the embodiments of the present application;

[0044] Figure 5 A local attention mechanism MLCA is provided for the embodiments of the present application;

[0045] Figure 6 A global attention mechanism Global Context is provided for the embodiments of the present application;

[0046] Figure 7 A raw yolov11 network structure graph is provided for the embodiments of the present application;

[0047] Figure 8 An optimized yolov11 network structure graph is provided for the embodiments of the present application;

[0048] Figure 9 A model garbage recognition effect graph is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0049] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0050] Embodiment 1,

[0051] The present embodiment provides a garbage automatic recognition method based on data fusion, which improves the garbage recognition accuracy and efficiency through data fusion, such as Figure 1 As shown in the figure, the method comprises the following steps:

[0052] S1, acquire garbage image data photographed by a camera and meteorological data of the region at the moment, and pre-process the garbage image;

[0053] S2, perform de-duplication, classification and data enhancement processing on the garbage image;

[0054] S3, perform motion compensation processing on the garbage image in combination with the meteorological data;

[0055] S4, perform category labeling and clustering on the garbage image;

[0056] S5, calculate color, texture and edge features of the garbage image, and perform normalization processing;

[0057] S6, optimize the deep learning model yolov11 to obtain an optimized yolov11 model;

[0058] S7, adjust the hyperparameters of the optimized yolov11 model;

[0059] S8, use the optimized yolov11 model to perform garbage recognition.

[0060] Embodiment 2,

[0061] This embodiment is a further limitation of embodiment 1, and further describes step S1.

[0062] S11: collect image data and corresponding meteorological data in different seasons;

[0063] S12: select appropriate camera multiple images, use the Laplacian operator in opencv to calculate the gradient of the image to evaluate the image quality, the threshold is 500-1000 for general, higher than 1000 for good, and lower than 500 for poor.

[0064] S13: delete poor data, do not operate on good data, and perform motion compensation on data with general image quality.

[0065] Embodiment 3,

[0066] This embodiment is a further limitation of embodiment 1, and further describes step S2.

[0067] S21: there may be repeated shooting of regions in the collected data, and image similarity algorithm is used for de-duplication operation.

[0068] S22: calculate the texture similarity and edge similarity of each pair of comparison pictures.

[0069] The texture similarity is extracted using a Gabor filter, and the calculation is as follows:

[0070]

[0071] x' = x cos θ + y sin θ;

[0072] y' = y sin θ + y cos θ;

[0073] where σ represents the standard deviation of the Gaussian function, θ represents the angle of the filter in the image, λ represents the filter wavelength, γ represents the aspect ratio parameter of the Gabor filter, which is used to control the elliptical shape of the filter, by adjusting this parameter, the shape and direction sensitivity of the filter in the image can be changed, and φ represents the phase offset, x and y represent the pixel coordinates.

[0074] The edge similarity uses an eight-neighborhood Laplacian operator, and the Laplacian operator is as follows,

[0075]

[0076] S23: The two features are normalized respectively, and feature splicing is performed, and the two features are texture similarity and edge similarity.

[0077] S24: The similarity between each pair of comparison pictures is measured by cosine similarity, and the image is removed if the similarity exceeds a certain threshold.

[0078] S25: The garbage image is classified into plastic garbage, straw garbage, construction garbage and pile garbage.

[0079] S26: The garbage image is enhanced in multiple ways, including rotation, cropping and splicing, flipping and high-frequency enhancement. In order to easily distinguish between snow in northern regions and plastic garbage, the image is enhanced in high frequency. The convolution kernel of the high-frequency enhancement operation is:

[0080] Embodiment 4,

[0081] This embodiment is a further limitation of embodiment 1, and step S3 is further described.

[0082] S31: The closest wind speed and wind direction within 0 hours within 15 minutes of shooting garbage image data of general quality are selected, as shown in Figure 3 If there is no close data, polynomial interpolation is used for calculation, and a one-variable cubic function is used for interpolation, as shown in Figure 2 i-2 , x i-1 , x i , x i+1 are the corresponding times, F(x i-2 ), F(x​i-1 ), F(x i ), F(x i+1 ) are the corresponding wind speed or wind direction, the four coordinates (x1, y1), (x2, y2), (x3, y3), (x4, y4) are calculated as follows:

[0083] a(x1) 3 +b(x1) 2 +c(x1)+d=y1;

[0084] a(x2) 3 +b(x2) 2 +c(x2)+d=y2;

[0085] a(x3) 3 +b(x3) 2 +c(x3)+d=y3;

[0086] a(x4) 3 +b(x4) 2 +c(x4)+d=y4;

[0087] By solving the linear equation system, as follows:

[0088]

[0089] a cubic function of one variable is obtained.

[0090] S32: Estimate the blur kernel direction according to the pitch angle in the image and the wind direction of the meteorological data, as shown in Figure 4 . Specifically, the direction of the blur kernel is consistent with the component direction of the wind direction in the imaging plane, and the blur kernel direction needs to be aligned with the wind speed direction (0-360°). The rotation matrix is used to rotate the blur kernel to the direction consistent with the wind direction. The size of the blur kernel is determined according to the wind speed. The direction is consistent with the component of the wind direction in the image.

[0091] S33: Estimate the blur kernel size by the wind speed of the meteorological data, and the blur kernel size calculation formula is as follows

[0092] kernel_size=int(base+(wind_speed**alpha));

[0093] Where kernel_size is the size of the blur kernel, base is the basic blur kernel size, wind_speed is the wind speed, alpha is the adjustment parameter, where the wind direction is in the range of 0-360, and the operation represented by "**" is the power operation.

[0094] S34: Determine the rotation matrix using kernel_size, and the rotation matrix is used to rotate the blur kernel toFigure 3 θ1 direction in the image, ensuring that the spatial distribution of the blur kernel aligns with the wind direction component on the image. Given the angle and a kernel_size, to rotate the blur kernel, a rotation matrix is needed. The blur kernel is originally not aligned with the wind direction component, so a rotation matrix is needed. Using the kernel_size and the angle of the wind direction on the image with the horizontal direction, the rotation matrix can be determined using cv2.getRotationMatrix2D in opencv. Using the angle and the kernel_size to get the rotation matrix, the blur kernel is rotated using the rotation matrix, so that the blur kernel is aligned with the wind direction component on the image.

[0095] S35: Deblurring the image using the convolution kernel combined with Wiener Filtering, where the Wiener Filtering formula is as follows:

[0096]

[0097] F(u,v) is the frequency domain representation of the input image, G(u,v) is the frequency domain representation of the blur kernel, S(u,v) is the power spectral density of the original image, σ 2 is the noise power spectral density, |G(u,v)| 2 is the power spectral density of the blur kernel, and H(u,v) is the frequency domain representation of the Wiener filter.

[0098] Embodiment 5,

[0099] This embodiment is a further limitation of embodiment 1, and further describes step S4.

[0100] When classifying and clustering the garbage images, Labelimg software is used to cluster plastic garbage, straw garbage and construction garbage respectively; plastic garbage, straw garbage and construction garbage are clustered using DBSCAN, and heap garbage is clustered using IOU.

[0101] Through the clustering method, garbage objects with similar characteristics (such as color, texture, shape, etc.) can be classified into one category. For example, the different types of garbage mentioned (plastic garbage, straw garbage, construction garbage and heap garbage) each have their unique characteristics. Clustering can help more accurately identify these different types of garbage. After clustering, features can be extracted according to the characteristics of each category of garbage (such as color histogram, LBP feature, Tamura feature, etc.), which helps to further improve the relevance and effectiveness of feature representation, thereby enhancing the learning ability and generalization ability of the subsequent deep learning model.

[0102] Embodiment 6,

[0103] This embodiment is a further limitation of embodiment 1, further illustrating step S5.

[0104] S51: Calculate the color, texture and edge features of the garbage image and perform normalization processing;

[0105] S511: Among them, the color feature calculates the color histogram, and the HSV color space feature.

[0106] S512: Texture feature calculation LBP and Tamura feature

[0107] S513: LBP feature calculation is as follows:

[0108]

[0109] P represents the total number of pixels, i represents the ith pixel, where I(i) is the gray value of the neighborhood pixel, I(x, y) is the gray value of the center pixel, 1 is an indicator function, indicating whether the pixel gray is greater than or equal to the gray of the center pixel.

[0110] S514: Tamura feature, select roughness, edge density, detail three dimensions for feature extraction, the roughness formula is as follows:

[0111]

[0112] I(x k ) is the gray value at position x k , which represents the gray value of a pixel point in the image; is the gray value at position , which represents the gray value of a pixel point in the image; M is the sample number, usually refers to the total number of pixel points used in the calculation process; x k and are the coordinates of two adjacent pixel points in the image.

[0113] The edge density calculation formula is as follows:

[0114]

[0115] M is the sample number, usually refers to the total number of pixel points used in the calculation process.

[0116] Edgeatx k is an indicator function, indicating whether there is an edge at position x k , if there is, it is 1, otherwise it is 0.

[0117] The detail calculation formula is as follows:

[0118]

[0119] U(x k ) and respectively represent the pixel gray value at x k and ; is the gray difference of two pixels in the image, indicating the degree of fine texture change; N is the number of samples, indicating the number of pixel pairs in the calculation process.

[0120] S515: The edge features of the garbage are mostly irregular shapes, so LOG is used for extraction, and the calculation is as follows:

[0121]

[0122] is the Laplacian operator, representing the second-order spatial derivative; the Laplacian operator performs second-order derivative operation on the image, emphasizing the most dramatic change area (i.e. edge)

[0123]

[0124] x, y represent the spatial coordinates in the image; σ is the standard deviation of the Gaussian function, which controls the smoothing degree of the Gaussian filter. The larger σ is, the wider the range of the filter, the smoother the image, and vice versa.

[0125] S52: Fuse color features, edge features, and texture features.

[0126] S53: Perform Z-score standardization on the fused features, so that the mean of the features is 0 and the standard deviation is 1.

[0127] S54: Perform PCA on the fused features to extract principal components, reduce data dimensionality, remove redundancy, retain important information in the features, and reduce computational complexity.

[0128] Example 7,

[0129] This embodiment is a further limitation of Example 1, which further describes steps S6-S8.

[0130] S6 is specifically:

[0131] S61: Optimize the garbage recognition model, add local attention, global attention, color feature loss, edge feature loss, and texture feature loss.

[0132] S62: Add 2 layers of local attention mechanism MLCA-mixed local channel attention to the shallow layer and intermediate feature layer of the backbone, respectively. The module can integrate channel information and spatial information at the same time, and can better detect small frame garbage, as shown in Figure 5 .

[0133] S63: Add a layer of global attention mechanism Global Context in the deep layer of backbone, as shown in the figure Figure 6 When there is a large area of snow, the model can reallocate attention from a global perspective, focus on important areas, and improve the global consistency of feature representation to better detect large frame garbage.

[0134] S64: Add color feature loss based on the original loss function, use KL divergence to measure the difference between the real image and the predicted image, as follows:

[0135] Where i represents the i position, H true is the real color feature histogram, H pred is the predicted color feature histogram.

[0136] S65: Add texture feature loss, calculated using mean square error (MSE): L texture = MSE (F pred , F true );

[0137] Where F pred represents the texture feature of the predicted image, F true represents the texture feature of the real image.

[0138] S66: Add edge feature loss, use structural similarity index (SSIM) to measure the similarity of edge feature maps, as follows: L edge = 1-SSIM (E pred , E true );

[0139] S67: The final comprehensive function is:

[0140] L total = λ1L box + λ2L obj + λ3L cls + λ4L color + λ5L texture + λ6L edge ;

[0141] Where λ1……λ6 are the hyperparameters of each loss, L box , L obj , L cls are the original loss functions, where L color , L texture , L edge are collectively referred to as L featureGlobal Context is a global attention mechanism, and MLCA is a local attention mechanism. The original yolov11 is shown in Fig. 7, and the optimized yolov11 is shown in Fig. Figure 8 .

[0142] S7: Adjust the model hyperparameters, including λ1… λ6, optimizer, batch_size, learning_rate, etc.

[0143] S71: Manually adjust the basic fuzzy kernel size: first, set an initial value for the fuzzy kernel size (kernel_size), which is set to 3. After multiple experiments, the fuzzy image is compensated for motion, and whether the Laplacian gradient variance is improved is evaluated. The quality of the improvement is measured by calculating the cumulative improvement of the gradient of multiple images.

[0144] S8: Identify and detect garbage, as shown in Fig. Figure 9 , an effect diagram of garbage identification using the optimized yolov11 model described in the embodiments of the present application is provided.

[0145] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0146] The specific embodiments of the present application do not constitute a limitation on the scope of protection of the present application. Any various other corresponding changes and modifications made according to the technical concept of the present application shall be included in the scope of protection of the claims of the present application.

Claims

1. A method for automatic waste identification based on data fusion, characterized in that, The method includes the following steps: S1. Acquire garbage image data captured by the camera and meteorological data of the area at that time, and preprocess the garbage images; S2. Perform deduplication, classification, and data augmentation on garbage images; S3. Perform motion compensation processing on garbage images combined with meteorological data; S4. Perform category labeling and clustering on garbage images; S5. Calculate the color, texture, and edge features of the garbage image and perform normalization processing; S6. Optimize the deep learning model YOLOv11 to obtain the optimized YOLOv11 model, specifically: S61. Add MLCA attention mechanism to the shallow feature layer and intermediate feature layer of the backbone part of the deep learning model yolov11 respectively; S62. Add a global attention mechanism to the deep feature layers of the backbone part of the YOLOv11 deep learning model. S63. Add color feature loss, texture feature loss and edge feature loss to the loss function of the deep learning model yolov11; S7. Adjust the hyperparameters of the optimized YOLOv11 model; S8. Use the optimized yolov11 model for garbage identification.

2. The automatic waste identification method based on data fusion according to claim 1, characterized in that, When capturing junk image data by a camera, it is necessary to capture data from different seasons. Preprocessing of junk images includes: evaluating image quality, with evaluation results categorized as good, average, and poor, and deleting images with poor evaluation results.

3. The automatic waste identification method based on data fusion according to claim 2, characterized in that, When deduplicating junk images, the color texture similarity features and edge similarity features of the pairwise comparison images are calculated. The two features are normalized and then concatenated. The similarity between images is measured by cosine similarity. If the similarity is greater than 0.5, one of the junk images is deleted. When classifying garbage images, the garbage is classified according to its type and shape. The classification includes plastic garbage, straw garbage, construction garbage, and piled garbage. Data augmentation techniques used for garbage images include rotation, cropping and stitching, and high-frequency enhancement.

4. The automatic waste identification method based on data fusion according to claim 3, characterized in that, When performing motion compensation processing on garbage images combined with meteorological data, the object of motion compensation processing is the garbage image with a general evaluation result.

5. The automatic waste identification method based on data fusion according to claim 4, characterized in that, The process of motion compensation is as follows: S51. Determine the meteorological data corresponding to the garbage images; S52. Estimate the direction of the blur kernel in the garbage image based on the elevation angle of the garbage image and the wind direction from meteorological data; S53. Estimate the size of the blur kernel in the garbage image based on the wind speed data from meteorological data; S54. Determine the rotation matrix based on the size of the blur kernel. The rotation matrix is ​​used to rotate the blur kernel to ensure that the spatial distribution of the blur kernel is consistent with the wind direction component of the meteorological data on the garbage image. S55. Use convolution kernels combined with Wiener filtering to deblur the garbage image.

6. The automatic waste identification method based on data fusion according to claim 5, characterized in that, When performing category labeling and clustering on waste images, the Labelimg software was used to cluster plastic waste, straw waste, and construction waste separately; DBSCAN clustering was used for plastic waste, straw waste, and construction waste, while IOU clustering was used for piled waste.

7. The automatic waste identification method based on data fusion according to claim 6, characterized in that, Color features include color histograms and HSV color space features; texture features include LBP and Tamura features; and edge features include features extracted using the LOG feature extraction algorithm.

8. The automatic waste identification method based on data fusion according to claim 7, characterized in that, During normalization, color features, edge features, and texture features are first fused. The fused features are then standardized using Z-score, with the mean of the fused features set to 0 and the standard deviation set to 1. Finally, PCA is performed on the fused features to extract principal components.

9. A waste automatic identification system based on data fusion, characterized in that, The system includes the following modules: A module that acquires garbage image data captured by a camera and meteorological data of the area at that time, and preprocesses the garbage images; A module for deduplication, classification, and data augmentation of garbage images; A module for motion compensation processing of garbage images combined with meteorological data; A module for classifying and clustering garbage images; A module that calculates the color, texture, and edge features of garbage images and performs normalization processing; The deep learning model yolov11 is optimized to obtain the modules of the optimized yolov11 model; The optimizations are as follows: S91. Add MLCA attention mechanism to the shallow feature layer and intermediate feature layer of the backbone part of the deep learning model yolov11 respectively; S92. Add a global attention mechanism to the deep feature layers of the backbone part of the YOLOv11 deep learning model. S93. Add color feature loss, texture feature loss, and edge feature loss to the loss function of the deep learning model yolov11; A module for adjusting and optimizing the hyperparameters of the YOLOv11 model; A module for garbage identification using an optimized YOLOv11 model.

Images