Garbage classification method based on intelligent vision, related device thereof and storage medium
Through the multi-scale feature extraction and multi-task learning module of intelligent vision technology, the problems of low efficiency and low accuracy in the existing garbage classification methods are solved, efficient and accurate garbage classification and processing are achieved, and the degree of automation of the system is improved. Interactivity with users.
Patent Information
- Application Number
- CN202410952984.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-26
AI Technical Summary
The existing garbage classification methods rely on manual operation efficiency and low accuracy, making it difficult for automated systems to deal with complex and diverse garbage, and intelligent vision technology has shortcomings in image data acquisition and preprocessing, classification model design, etc., resulting in limited classification effects.
Using a garbage classification method based on intelligent vision, the image data is preprocessed by obtaining garbage image data, and the complex network structure of multi-scale feature extraction module, feature fusion module, recursive neural network module, attention module and multi-task learning module are used to prepare the image data. Carry out multi-level and multi-angle feature extraction and classification to achieve efficient and accurate classification of garbage, and transmit classification results through wireless communication modules.
Improve the accuracy and automation of garbage classification, enhance the reliability and adaptability of the system, solve the problems of unstable data collection, high processing complexity and low classification accuracy in traditional methods, and realize high-precision garbage identification and handle.
Smart Images

Figure CN119027712A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of garbage classification, and in particular to a garbage classification method based on intelligent vision and its related devices and storage media. Background Art
[0002] Garbage sorting is an important part of modern urban management and environmental protection, aiming to classify different types of garbage for effective resource recovery and environmental protection. However, traditional garbage sorting methods usually rely on manual operations, which have technical problems such as low efficiency, low accuracy, and high labor intensity. Manual sorting not only consumes a lot of manpower and time, but is also prone to classification errors due to human factors, affecting subsequent resource recovery and processing. In addition, manual sorting is difficult to ensure the comprehensiveness and accuracy of classification when faced with complex and diverse garbage.
[0003] Although some existing automated garbage sorting systems have introduced automated equipment such as robotic arms and conveyor belts, they still rely mainly on simple sensors and rule-based algorithms, making it difficult to cope with complex and changing garbage sorting scenarios. The systems can usually only handle a relatively single type of garbage, and are not very effective in sorting mixed garbage. At the same time, the image processing and recognition technologies in existing systems are relatively backward, making it difficult to achieve high-precision garbage recognition and classification.
[0004] In order to improve the efficiency and accuracy of garbage classification, intelligent vision technology has gradually been applied to the field of garbage classification in recent years. Intelligent vision technology combines computer vision and machine learning algorithms to automatically identify and classify different types of garbage. However, the current intelligent vision garbage classification methods still have some technical problems in the acquisition and preprocessing of image data, the design and training of classification models, etc. For example, most of the existing classification models use a single-scale convolutional neural network (CNN), which has limited classification effects when faced with garbage of different sizes and shapes. At the same time, the existing system also has shortcomings in the transmission and feedback of classification results, making it difficult to achieve efficient and accurate garbage disposal and user interaction. Summary of the invention
[0005] The technical problem to be solved by the present application is to provide a garbage classification method based on intelligent vision and its related devices and storage media in view of the deficiencies of the existing technology, so as to solve the existing problem of how to improve the accuracy and automation of garbage classification.
[0006] In order to solve the above technical problems, the first aspect of the embodiment of the present application provides a garbage classification method based on intelligent vision, including:
[0007] Acquire junk image data, and preprocess the junk image data to obtain preprocessed data;
[0008] The preprocessed image data is classified based on the pre-trained garbage classification model to obtain the classification result.
[0009] The process of obtaining the junk image data and preprocessing the junk image data to obtain the preprocessed junk image data specifically includes:
[0010] Get real-time garbage image data;
[0011] Performing grayscale processing and noise filtering processing on the real-time junk image data to obtain first junk image data;
[0012] Performing edge detection and enhancement processing on the first garbage image data to obtain second garbage image data;
[0013] The second junk image data is normalized to obtain pre-processed junk image data.
[0014] Among them, the pre-training-based garbage classification model specifically includes a multi-scale feature extraction module, a feature fusion module, a recursive neural network module, an attention module, and a multi-task learning module.
[0015] The process of classifying the preprocessed image data based on the pre-trained garbage classification model to obtain the classification result specifically includes:
[0016] Performing feature extraction on the preprocessed image data based on the multi-scale feature extraction module to obtain multi-scale features;
[0017] Inputting the multi-scale features into the feature fusion module to obtain fused features;
[0018] Inputting the fused features into a recursive neural network module to obtain global features;
[0019] Inputting the global feature into an attention module to obtain an attention feature;
[0020] The attention features are input into a multi-task learning module to obtain a classification result corresponding to the preprocessed image data.
[0021] The feature fusion module includes multiple parallel convolutional neural network branches, and the process of extracting features from the preprocessed image data based on the multi-scale feature extraction module to obtain multi-scale features specifically includes:
[0022] Based on multiple parallel convolutional neural network branches, convolution operations of different scales are performed on the preprocessed image data to obtain a multi-scale feature map, wherein each convolution branch uses a convolution kernel and a step size of different size;
[0023] The multi-scale feature maps are respectively subjected to pooling operations to obtain multi-scale features.
[0024] The multi-task learning module uses a shared underlying feature representation, including task branches of garbage type classification and garbage form detection, and the process of inputting the attention feature into the multi-task learning module to obtain the classification result corresponding to the preprocessed image data specifically includes:
[0025] Mapping the attention feature to a garbage category label based on a fully connected layer and a Softmax function;
[0026] Mapping the attention features into garbage morphology parameters based on a fully connected layer and a regression function;
[0027] A classification result corresponding to the processed image data is determined based on the garbage category label and the garbage morphology parameter.
[0028] Wherein, after classifying the preprocessed image data based on the pre-trained garbage classification model to obtain the classification result, the method further includes:
[0029] Transmitting the classification result to a garbage disposal system via a wireless communication module;
[0030] After receiving the classification result, the garbage disposal system controls the corresponding garbage bin to operate and feeds back the classification result to the user through a display screen or a voice prompt device.
[0031] A second aspect of an embodiment of the present application provides a garbage classification device based on intelligent vision, comprising:
[0032] An acquisition module, used for acquiring junk image data, and preprocessing the junk image data to obtain preprocessed data;
[0033] The classification module classifies the preprocessed image data based on the pre-trained garbage classification model.
[0034] The third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the intelligent vision-based garbage classification method.
[0035] A fourth aspect of an embodiment of the present application provides a terminal device, which includes: a processor and a memory;
[0036] The memory stores a computer-readable program executable by the processor;
[0037] When the processor executes the computer-readable program, the steps in the garbage classification method based on intelligent vision are implemented.
[0038] Beneficial effects:
[0039] The present application discloses a garbage classification method based on intelligent vision. The method obtains garbage image data and pre-processes the garbage image data to obtain pre-processed data. The method solves the problems of unstable data collection and high processing complexity in the traditional manual garbage classification process, thereby improving the quality and consistency of garbage classification data and providing reliable basic data for subsequent classification.
[0040] The method is a technical means of classifying preprocessed image data based on a pre-trained garbage classification model, solving the problems of low classification accuracy and poor model generalization ability in existing garbage classification methods, thereby achieving high-precision garbage classification and improving the reliability and adaptability of the system;
[0041] The method adopts a multi-scale feature extraction module and uses multiple parallel convolutional neural network branches to perform convolution operations on image data at different scales, thereby solving the problem that traditional single-scale convolutional neural networks are not effective in processing garbage of different sizes and shapes, thereby improving the recognition ability and accuracy of the classification model;
[0042] The method adopts a feature fusion module to perform splicing and dimensionality reduction on multi-scale feature maps, thereby solving the problem that multi-scale feature information is dispersed and difficult to process uniformly, and then forms a high-dimensional and comprehensive feature vector, providing a rich and compact feature representation for subsequent processing;
[0043] The method adopts a recursive neural network module and uses a bidirectional long short-term memory network (Bi-LSTM) to process high-dimensional feature vectors to solve the problem that traditional convolutional neural networks are difficult to capture dependencies and contextual information between image regions, thereby enhancing the model's ability to understand complex image content.
[0044] The method adopts an attention mechanism module and uses a self-attention mechanism to weight the feature vector to solve the problem that it is difficult to distinguish important information from irrelevant information in the feature vector, thereby highlighting key features and improving the accuracy and efficiency of classification;
[0045] The method adopts a multi-task learning module and uses shared underlying feature representation to simultaneously perform garbage type classification and garbage morphology detection, thereby solving the problems of poor robustness and generalization ability of single-task learning models, and thus improving the comprehensive performance and practicality of the classification model. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without creative work.
[0047] Figure 1 A flowchart of a garbage classification method based on intelligent vision provided in this application.
[0048] Figure 2 A structural schematic diagram of a garbage sorting device based on intelligent vision provided in this application.
[0049] Figure 3 This is a schematic diagram of the structure of the terminal device provided in this application.
[0050] Description of Figure Numbers:
[0051] 100. Get module;
[0052] 200, classification module;
[0053] 20. Processor;
[0054] 21. Display screen;
[0055] 22. Memory;
[0056] 23. Communication interface;
[0057] 24. Bus. DETAILED DESCRIPTION
[0058] The present application provides a garbage classification method based on intelligent vision and its related devices and storage media. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.
[0059] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0061] Intelligent vision refers to a technical means of identifying, detecting and classifying objects in images or videos through computer vision technology and machine learning algorithms. Intelligent vision technology is widely used in autonomous driving, security monitoring, medical diagnosis and other fields.
[0062] In the field of image processing, preprocessing refers to a series of preliminary processing operations on raw image data in order to improve the quality and processability of the image. Common preprocessing operations include grayscale, denoising, edge detection, and image enhancement.
[0063] Convolutional Neural Network (CNN) is a deep learning model that is widely used in image and video processing tasks. CNN can effectively extract image features and classify them through structures such as convolutional layers, pooling layers, and fully connected layers.
[0064] LSTM is a special recurrent neural network (RNN) suitable for processing and predicting time series data. LSTM can capture long-term dependencies and is widely used in natural language processing, speech recognition and other fields.
[0065] The attention mechanism is a technology that simulates human attention by assigning different weights to different parts of the input data, thereby highlighting key information and suppressing irrelevant information.
[0066] Multi-task learning is a machine learning framework that simultaneously learns and optimizes multiple related tasks by sharing the underlying feature representation, thereby improving the generalization ability and robustness of the model.
[0067] A wireless sensor network (WSN) is a network consisting of multiple sensor nodes that work together through wireless communication to monitor and collect various data in the environment. LPWAN is a wireless communication technology with wide coverage and low power consumption, suitable for long-distance communication of Internet of Things (IoT) devices. Common LPWAN technologies include LoRa and NB-IoT.
[0068] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
[0069] In modern urban management and environmental protection, garbage classification is an important link. Accurate garbage classification can improve resource recovery efficiency and reduce environmental pollution. However, traditional garbage classification methods mainly rely on manual operations, which have technical problems such as low efficiency, poor accuracy, and high labor intensity. Manual classification not only consumes a lot of manpower and time, but is also prone to classification errors due to human factors, affecting subsequent resource recovery and processing processes.
[0070] In response to the above problems, existing technologies have gradually introduced automated equipment and intelligent technologies. For example, the use of robotic arms and conveyor belts for automated garbage disposal has reduced the manual burden to a certain extent, but because it mainly relies on simple sensors and rule algorithms, the classification effect of the system is still not ideal when dealing with complex and diverse garbage.
[0071] Intelligent vision technology combines computer vision and machine learning algorithms and is gradually being applied to the field of garbage classification. By acquiring garbage image data through cameras and using deep learning models for automatic identification and classification, the accuracy and efficiency of classification can be effectively improved. However, the existing intelligent visual garbage classification system still has some technical problems in the acquisition and preprocessing of image data, the design and training of classification models, etc. For example, the existing system usually adopts a single-scale convolutional neural network (CNN), which has limited classification effect when facing garbage of different sizes and shapes. In addition, the existing system also has shortcomings in the transmission and feedback of classification results, making it difficult to achieve efficient and accurate garbage disposal and user interaction.
[0072] In order to solve the above problems, a garbage classification method based on intelligent vision and its related devices and storage media are disclosed in the embodiment of the present application. The method includes obtaining garbage image data, preprocessing the garbage image data to obtain preprocessed data; and classifying the preprocessed image data based on a pre-trained garbage classification model to obtain a classification result. The method of the present application first obtains garbage image data in real time through a camera installed at the garbage delivery port, and performs grayscale processing, noise filtering, edge detection, image enhancement and normalization processing on the image. Secondly, a complex network structure including a multi-scale feature extraction module, a feature fusion module, a recursive neural network module, an attention module and a multi-task learning module is used to perform multi-level and multi-angle feature extraction and classification on the preprocessed image data to achieve efficient and accurate classification of garbage. Thirdly, the classification results are transmitted wirelessly to the garbage disposal system and fed back to the user through a display screen or a voice prompt device, ultimately improving the automation and accuracy of garbage classification, reducing manual intervention, and improving the efficiency and quality of garbage disposal.
[0073] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.
[0074] This embodiment provides a garbage classification method based on intelligent vision and its related devices and storage media, such as Figure 1 As shown, the method includes:
[0075] S10, acquiring junk image data, and preprocessing the junk image data to obtain preprocessed data.
[0076] The process of obtaining the junk image data and preprocessing the junk image data to obtain the preprocessed junk image data specifically includes:
[0077] S110, acquiring real-time garbage image data.
[0078] A camera is installed at the garbage delivery port. The camera can be a high-resolution camera or other alternative camera equipment. The camera is connected to a data processing unit. The camera captures the image data of the garbage delivered in real time and transmits the image data to the data processing unit. In order to ensure the continuity and real-time nature of the data, the camera has autofocus and high-speed continuous shooting functions. The resolution and frame rate of the camera must meet the actual application requirements. It is generally recommended that the resolution be no less than 1080p and the frame rate be no less than 30 frames per second.
[0079] S111 . Perform grayscale processing and noise filtering processing on the real-time junk image data to obtain first junk image data.
[0080] Firstly, grayscale processing is performed on the real-time garbage image data to convert the color image into a grayscale image to simplify the subsequent processing steps and reduce the computational complexity. The grayscale processing can adopt a weighted average method to calculate the values of the RGB three channels as grayscale values according to certain weights.
[0081] After grayscale processing is completed, noise filtering is performed to remove random noise in the image and improve image quality. Noise filtering usually uses Gaussian filtering to reduce high-frequency noise in the image while maintaining the edge information of the image. The processing process of Gaussian filtering is as follows:
[0082] Define the Gaussian kernel function and select an appropriate kernel size (such as 3x3, 5x5);
[0083] The grayscale image is convolved using a Gaussian kernel function to obtain a smoothed image.
[0084] Through the above steps, the first garbage image data is obtained.
[0085] S112: Perform edge detection and enhancement processing on the first garbage image data to obtain second garbage image data.
[0086] Perform edge detection on the first junk image data after noise filtering to highlight the contour and boundary information of the object in the image. The edge detection can adopt the Canny edge detection algorithm, and the specific steps are as follows:
[0087] Use Gaussian filter to smooth the image and reduce the impact of noise on edge detection;
[0088] Calculate the gradient magnitude and direction of the image and use the Sobel operator for gradient calculation;
[0089] Apply non-maximum suppression to refine edges and retain local maxima;
[0090] Use the double threshold method to connect edges and determine the final edge map;
[0091] After edge detection, the image is enhanced to improve the contrast and detail information of the image.
[0092] Among them, the image enhancement processing can adopt the histogram equalization method, and its steps are as follows:
[0093] Calculate the grayscale histogram of the image and count the number of pixels at each grayscale level;
[0094] Calculate the cumulative distribution function (CDF) to map the grayscale value of the original image to the new grayscale value;
[0095] Generate an enhanced image according to the mapping relationship;
[0096] The second garbage image data is obtained through edge detection and image enhancement processing.
[0097] S113: Normalize the second junk image data to obtain pre-processed junk image data.
[0098] In order to ensure that different images have consistent sizes and scales when input into the classification model, the second junk image data is normalized. The normalization process includes two steps: size normalization and pixel value normalization:
[0099] Size normalization: Resize the image to a fixed size (such as 224x224 pixels) to meet the input requirements of the subsequent classification model. Size normalization can use bilinear interpolation to scale the image.
[0100] Pixel value normalization: Normalize the pixel values of the image to the range of [0,1] to facilitate the training and reasoning of the neural network model. The pixel value normalization formula can be expressed as:
[0101] Normalized pixel = pixel value / 255;
[0102] After the above normalization process, the preprocessed garbage image data is obtained.
[0103] S20. Classify the preprocessed image data based on the pre-trained garbage classification model to obtain a classification result.
[0104] Specifically, the pre-training-based garbage classification model includes a multi-scale feature extraction module, a feature fusion module, a recursive neural network module, an attention module, and a multi-task learning module.
[0105] The multi-scale feature extraction module extracts features from the pre-processed garbage image data through multiple parallel convolutional neural network (CNN) branches to capture the multi-scale information of the image. The specific process is as follows:
[0106] Input layer: receives the preprocessed junk image data.
[0107] Parallel convolution branches:
[0108] Each branch uses convolution kernels of different sizes (such as 3x3, 5x5, 7x7) and different strides to perform convolution operations on the input image.
[0109] Each convolution branch generates a feature map of the corresponding scale.
[0110] Pooling operation:
[0111] A pooling operation (such as maximum pooling or average pooling) is performed on the feature map generated by each convolution branch to reduce the size and computational complexity of the feature map and retain important features.
[0112] Output layer: Outputs multiple feature maps of different scales as input to the feature fusion module.
[0113] The feature fusion module fuses the feature maps from the multi-scale feature extraction module to form a high-dimensional comprehensive feature vector. The specific process is as follows:
[0114] Concatenation operation: Multiple feature maps of different scales are concatenated in the channel dimension to form a high-dimensional feature vector.
[0115] Dimensionality reduction: Use a 1x1 convolutional layer to reduce the dimensionality of the concatenated feature vector to reduce the computational complexity while retaining important feature information.
[0116] Output layer: Outputs the fused feature vector as the input of the recursive neural network module.
[0117] The recursive neural network module processes the fused feature vectors through a bidirectional long short-term memory network (Bi-LSTM) to capture the dependencies and contextual information between different regions in the image. The specific process is as follows:
[0118] Input layer: receives the fused feature vector.
[0119] The bidirectional long short-term memory network processes the input feature vector from both the forward and backward directions to obtain global context information.
[0120] Bi-LSTM can capture long-term dependencies and enhance the understanding of complex image content.
[0121] Time step aggregation: The feature vector output by Bi-LSTM is aggregated in time steps to form a global feature representation.
[0122] Output layer: Outputs global feature representation as input to the attention mechanism module.
[0123] The attention mechanism module weights the global feature representation by calculating the importance weight of each position in the feature vector, highlighting key features and suppressing irrelevant information. The specific process is as follows:
[0124] Input layer: receives the global feature representation.
[0125] Attention weight calculation:
[0126] The importance weight of each position in the feature vector is calculated using the self-attention mechanism.
[0127] The attention weights are calculated by taking the dot product with the global feature representation.
[0128] Weighted operation: The global feature representation is weighted according to the calculated attention weights to generate a weighted feature vector.
[0129] Output layer: Outputs the weighted feature vector as the input of the multi-task learning module.
[0130] The multi-task learning module uses the shared underlying feature representation to simultaneously perform garbage type classification and garbage morphology detection, thereby improving the robustness and generalization ability of the classification model. The specific process is as follows:
[0131] Input layer: receives weighted feature vectors.
[0132] Shared underlying feature representation: Use shared underlying feature representation to learn multiple tasks simultaneously.
[0133] Garbage type classification task:
[0134] A fully connected layer and a Softmax function are used to map the weighted feature vector to a garbage class label.
[0135] The Softmax function calculates the probability distribution of each category and selects the category with the highest probability as the classification result.
[0136] Garbage form detection task:
[0137] The weighted feature vector is mapped to garbage morphological parameters (such as size and shape) using a fully connected layer and a regression function.
[0138] The regression function calculates the specific values of the garbage shape parameters.
[0139] Output layer: Outputs garbage category labels and garbage morphological parameters to determine the classification results corresponding to the preprocessed image data.
[0140] The process of classifying the preprocessed image data based on the pre-trained garbage classification model to obtain the classification result specifically includes:
[0141] S210: Perform feature extraction on the preprocessed image data based on the multi-scale feature extraction module to obtain multi-scale features.
[0142] The feature fusion module includes a plurality of parallel convolutional neural network branches, and the process of extracting features from the preprocessed image data based on the multi-scale feature extraction module to obtain multi-scale features specifically includes:
[0143] S2101, based on multiple parallel convolutional neural network branches, performing convolution operations of different scales on the preprocessed image data to obtain a multi-scale feature map, wherein each convolution branch uses a convolution kernel and step size of a different size;
[0144] S2102: Perform pooling operations on the multi-scale feature maps to obtain multi-scale features.
[0145] First, the multi-scale feature extraction module receives the pre-processed junk image data, which has been processed through grayscale processing, noise filtering, edge detection and normalization to ensure data quality and consistency.
[0146] The multi-scale feature extraction module includes multiple parallel convolutional neural network branches, each of which uses convolution kernels of different sizes (such as 3x3, 5x5, 7x7) and different step sizes (such as 1, 2) to perform convolution operations on the input image. Each convolution kernel can capture features of different scales, for example, a small convolution kernel (such as 3x3) can capture detail features, and a large convolution kernel (such as 7x7) can capture global features.
[0147] For example:
[0148] Branch 1: Use a 3x3 convolution kernel with a stride of 1 to perform a convolution operation. This branch mainly captures the detailed features of the image, such as edges and textures.
[0149] Branch 2: Use a 5x5 convolution kernel with a stride of 2 to perform convolution operations. This branch can capture medium-scale features, including both details and some global information.
[0150] Branch 3: Use a 7x7 convolution kernel with a step size of 2 to perform convolution operations. This branch mainly captures the global features of the image and is suitable for extracting large-scale features.
[0151] Each convolution branch generates a feature map of the corresponding scale through convolution operation. The feature map retains the characteristic information of the input image at different scales.
[0152] A pooling operation is performed on the feature map generated by each convolution branch to reduce the size and computational complexity of the feature map while retaining important features. The pooling operation includes maximum pooling and average pooling.
[0153] For example:
[0154] Branch-1 pooling: Perform a 2x2 maximum pooling operation on the feature map generated by the 3x3 convolution kernel with a step size of 2 to retain the significance of the detail features.
[0155] Branch two pooling: Perform a 2x2 average pooling operation on the feature map generated by the 5x5 convolution kernel with a step size of 2 to smooth medium-scale features.
[0156] Branch triple pooling: Perform a 3x3 maximum pooling operation on the feature map generated by the 7x7 convolution kernel with a step size of 3 to reduce the size of the global feature map.
[0157] The feature maps of each convolution branch after the pooling operation are spliced in the channel dimension to form a high-dimensional feature vector. The splicing operation combines features of different scales to provide richer feature representation.
[0158] The pooled feature maps of branches 1, 2, and 3 are concatenated in the channel dimension to form a high-dimensional feature vector containing multi-scale features. When the size of each feature map is NxN, the size of the concatenated feature vector is NxNx3.
[0159] Finally, the concatenated high-dimensional feature vector is output as the input of subsequent modules (such as feature fusion module and recursive neural network module). This high-dimensional feature vector contains the multi-scale information of the preprocessed image, providing a rich feature basis for subsequent processing.
[0160] S211, inputting the multi-scale features into the feature fusion module to obtain fused features.
[0161] Specifically, in step (such as S210), the pre-processed image data generates a plurality of feature maps of different scales through a multi-scale feature extraction module. The feature maps retain information of the image at different scales, such as detail features and global features.
[0162] In the feature fusion module, first, the multi-scale feature maps generated from different convolution kernels (such as 3x3, 5x5, 7x7) are spliced in the channel dimension. Assuming that the size of each feature map is NxNxC1, NxNxC2, NxNxC3, the size of the spliced feature map is NxNx(C1+C2+C3).
[0163] In order to reduce computational complexity and improve computational efficiency, a 1x1 convolution layer is used to reduce the dimensionality of the concatenated high-dimensional feature vector. The 1x1 convolution kernel can effectively integrate the information of each channel, reduce the dimension of the feature vector, and maintain important features. For example, a 1x1 convolution layer is applied with an output channel number of C (e.g., 128), thereby converting the concatenated high-dimensional feature vector (NxNx(C1+C2+C3)) into a NxNxC feature vector.
[0164] Subsequently, the feature vector after dimensionality reduction is subjected to nonlinear activation processing (such as ReLU activation function) to introduce nonlinear features.
[0165] The final output fused feature vector (NxNxC) contains the comprehensive features of the preprocessed image, including both detail features and global features, providing a rich information basis for the subsequent recursive neural network module.
[0166] S212: Input the fused features into a recursive neural network module to obtain global features.
[0167] In the specific solution of the present application, the recursive neural network model can select a bidirectional long short-term memory network. The bidirectional long short-term memory network Bi-LSTM is an improved recursive neural network that can process input data from both forward and backward directions to capture global context information.
[0168] Forward LSTM: processes feature vectors from left to right, capturing forward dependencies.
[0169] Backward LSTM: Processes feature vectors from right to left, capturing backward dependencies.
[0170] The output of Bi-LSTM is a bidirectional feature vector (NxNx2H), where H is the dimension of the LSTM hidden layer.
[0171] First, the recursive neural network module receives the fused feature vector (NxNxC) output by the feature fusion module.
[0172] Aggregate the bidirectional feature vectors in the time dimension to generate a global feature representation. The aggregation operation can be performed by summing or concatenating. For example, concatenate the outputs of the forward LSTM and the backward LSTM to obtain a global feature vector (NxNx2H).
[0173] The global feature vector (NxNx2H) used for output contains the global information and context dependency of the image.
[0174] S213: Input the global feature into the attention module to obtain the attention feature.
[0175] The attention module receives the global feature vector (NxNx2H) output by the recursive neural network module.
[0176] The importance weight of each position in the global feature vector is calculated using the self-attention mechanism. The attention weight is calculated by the dot product with the global feature representation.
[0177] The global feature vector is weighted according to the attention weight to generate a weighted feature vector. The weighting operation can highlight key features and suppress irrelevant information.
[0178] S214: Input the attention feature into a multi-task learning module to obtain a classification result corresponding to the preprocessed image data.
[0179] The multi-task learning module uses a shared underlying feature representation, including task branches of garbage type classification and garbage form detection, and the process of inputting the attention feature into the multi-task learning module to obtain the classification result corresponding to the pre-processed image data specifically includes:
[0180] S2140, mapping the attention feature to a garbage category label based on a fully connected layer and a Softmax function;
[0181] S2141, mapping the attention feature into garbage morphology parameters based on a fully connected layer and a regression function;
[0182] S2142. Determine a classification result corresponding to the processed image data based on the garbage category label and the garbage morphology parameter.
[0183] Receive the attention feature vector (NxNx2H) processed by the attention mechanism module;
[0184] The attention feature vector is used as a shared underlying feature representation for both the garbage type classification task and the garbage form detection task. The sharing mechanism can make full use of feature information and improve the learning efficiency and generalization ability of the model.
[0185] In step S2140, the attention feature vector is first input into the fully connected layer for feature conversion. The fully connected layer maps the high-dimensional feature vector to the low-dimensional space required for the classification task through linear transformation.
[0186] For example, assume that the dimension of the attention feature vector is 2H and the output dimension of the fully connected layer is C (the number of garbage categories).
[0187] Linear transformation formula: Z = WX + b,
[0188] Among them, X is the attention feature vector, W is the weight matrix, b is the bias vector, and Z is the output of the fully connected layer.
[0189] The output of the fully connected layer is passed through the Softmax function to calculate the probability distribution of each category. The Softmax function converts the original output value into a probability value, and the sum of the probabilities is 1.
[0190] The category with the highest probability is selected as the junk category label, indicating that the image data belongs to this category.
[0191] In step S2141, the attention feature vector is input into another fully connected layer for feature conversion. Similar to the classification task, the fully connected layer maps the high-dimensional feature vector to the low-dimensional space required for the regression task through linear transformation.
[0192] Specific implementation: Assume that the dimension of the attention feature vector is 2H, and the output dimension of the fully connected layer is P (the number of garbage morphology parameters, such as size, shape, etc.).
[0193] The linear transformation formula is the same as above: Y = W'X + b',
[0194] Where W' is the weight matrix, b' is the bias vector, and Y is the output of the fully connected layer.
[0195] Subsequently, a regression function is used to perform regression analysis on the output of the fully connected layer to predict the specific values of the garbage morphological parameters. The regression function may be linear regression, LASSO regression, etc.
[0196] Output predicted garbage shape parameters, such as garbage size, shape, and weight.
[0197] In step S2142, first, the garbage category label and garbage morphology parameters are integrated to form a complete classification result. The comprehensive result includes the category of garbage and the corresponding morphology information, which can provide richer classification information. The complete classification result is output as the decision basis of the garbage classification system.
[0198] S30. After classifying the preprocessed image data based on the pretrained garbage classification model to obtain a classification result, the method further includes:
[0199] S310, transmitting the classification result to the garbage disposal system via a wireless communication module;
[0200] S311, after receiving the classification result, the garbage disposal system controls the corresponding garbage bin to operate and feeds back the classification result to the user through a display screen or a voice prompt device.
[0201] Specifically, the multi-task learning module obtains the category labels and morphological parameters of the garbage to form a complete classification result. The classification result includes the garbage category (such as recyclable garbage, kitchen waste, hazardous waste and other garbage) and the corresponding morphological information (such as size, weight, etc.).
[0202] The classification results are transmitted to the garbage disposal system using a wireless sensor network (WSN). The wireless sensor network uses low-power wide area network (LPWAN) technology, such as LoRa or NB-IoT, to ensure the reliability and stability of data transmission. During the data transmission process, the classification results are packaged and sent by the data processing unit through the wireless communication module. The wireless communication module transmits the packaged data to the control center of the garbage disposal system through LPWAN. Among them, a reliable transmission protocol, such as MQTT or CoAP, is used to ensure the integrity and security of data transmission. The data packet contains detailed information on the classification results, such as garbage category, morphological parameters, timestamp, etc.
[0203] In step S311, the control center of the garbage disposal system receives the classification result data packet transmitted by the wireless communication module, and analyzes the received data to extract detailed information of the classification result.
[0204] According to the classification result, the control center sends a control signal to the corresponding garbage bin through the central control unit (CCU). The opening and closing of the corresponding garbage bin is controlled by the control signal. The garbage bin operating device can use an electromagnetic lock or an electric push rod to achieve automatic control. For example, when the classification result is recyclable garbage, the control signal instructs the recyclable garbage bin to open and automatically close after the garbage is put in. The control signal is transmitted to the control device of the garbage bin by wire or wireless means. The wireless method can use short-range wireless communication technology, such as Zigbee or Bluetooth.
[0205] In addition, the garbage disposal system integrates an embedded display screen to display garbage classification results and operation prompts in real time. The display screen can be a liquid crystal display (LCD) or an organic light-emitting diode display (OLED) to ensure the clarity and readability of the information displayed. The specific display content includes garbage category, morphological parameters, current operation status, etc.
[0206] In addition, the garbage disposal system integrates an intelligent voice prompt device, which uses speech synthesis technology to broadcast classification results and operation prompts to users. The voice prompt device includes a speech synthesis module and a speaker, which can generate natural speech to enhance the user experience. The content of the voice prompt includes garbage category, operation steps (such as placement location), etc. The voice prompt device also integrates a speech recognition module, which can adjust the prompt content and volume according to user instructions.
[0207] In summary, this embodiment provides a garbage classification method based on intelligent vision and its related devices, computer-readable storage media and terminal devices. The method described in the present application obtains garbage image data, preprocesses the garbage image data to obtain preprocessed data, and classifies the preprocessed image data based on a pre-trained garbage classification model to obtain a classification result. Furthermore, by adopting modules such as multi-scale feature extraction, feature fusion, recursive neural network, attention mechanism and multi-task learning, the accuracy and efficiency of garbage classification can be significantly improved.
[0208] Specifically, by acquiring real-time garbage image data and performing grayscale processing, noise filtering, edge detection, image enhancement and normalization processing on it, the pre-processed garbage image data is obtained, ensuring the quality and consistency of the input data. Based on the pre-trained garbage classification model, including multi-scale feature extraction module, feature fusion module, recursive neural network module, attention module and multi-task learning module, the classification process is further optimized. Among them, the multi-scale feature extraction module uses multiple parallel convolutional neural network branches to perform convolution operations of different scales on the pre-processed image data to obtain multi-scale features, and fuses them through the feature fusion module to form a high-dimensional feature representation. The recursive neural network module captures the dependencies and contextual information between different regions in the image through a bidirectional long short-term memory network (Bi-LSTM), and the attention mechanism module weights the global features, highlights the key features, and suppresses irrelevant information. The multi-task learning module uses the shared underlying feature representation to simultaneously perform garbage type classification and garbage morphology detection, improving the robustness and generalization ability of the classification model.
[0209] In addition, the classification results are transmitted to the garbage disposal system through the wireless communication module, realizing real-time transmission and efficient processing of the classification results. After receiving the classification results, the garbage disposal system sends a control signal through the central control unit (CCU) to control the automatic opening and closing of the corresponding garbage bins, realizing automated garbage disposal. The system also displays the garbage classification results and operation prompts in real time through the embedded display screen, and broadcasts the classification results and operation prompts to the user through the intelligent voice prompt device, improving the user experience and interactivity.
[0210] Based on the above-mentioned garbage classification method based on intelligent vision, this embodiment provides a garbage classification method based on intelligent vision, such as Figure 2 As shown, the device comprises:
[0211] An acquisition module 100 is used to acquire junk image data and preprocess the junk image data to obtain preprocessed data;
[0212] The classification module 200 classifies the preprocessed image data based on the pretrained garbage classification model.
[0213] Based on the above-mentioned intelligent vision-based garbage classification method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the intelligent vision-based garbage classification method as described in the above embodiment.
[0214] Based on the above-mentioned garbage classification method based on intelligent vision, the present application also provides a terminal device, such as Figure 3As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communication interface 23 and a bus 24. The processor 20, the display screen 21, the memory 22 and the communication interface 23 may communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call the logic instructions in the memory 22 to execute the method in the above embodiment.
[0215] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.
[0216] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.
[0217] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.
[0218] In addition, the specific process of loading and executing the multiple instruction processors in the above storage medium and the terminal device has been described in detail in the above method and will not be described here one by one.
[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, the modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A garbage classification method based on intelligent vision, characterized in that: include: Acquire junk image data, and preprocess the junk image data to obtain preprocessed data; The preprocessed image data is classified based on the pre-trained garbage classification model to obtain the classification result.
2. The garbage classification method based on intelligent vision according to claim 1 is characterized in that: The process of obtaining the junk image data and preprocessing the junk image data to obtain the preprocessed junk image data specifically includes: Get real-time garbage image data; Performing grayscale processing and noise filtering processing on the real-time junk image data to obtain first junk image data; Performing edge detection and enhancement processing on the first garbage image data to obtain second garbage image data; The second junk image data is normalized to obtain pre-processed junk image data.
3. The garbage classification method based on intelligent vision according to claim 1 is characterized in that: The pre-training-based garbage classification model specifically includes a multi-scale feature extraction module, a feature fusion module, a recursive neural network module, an attention module, and a multi-task learning module.
4. The garbage classification method based on intelligent vision according to claim 3 is characterized in that: The process of classifying the preprocessed image data based on the pre-trained garbage classification model to obtain the classification result specifically includes: Performing feature extraction on the preprocessed image data based on the multi-scale feature extraction module to obtain multi-scale features; Inputting the multi-scale features into the feature fusion module to obtain fused features; Inputting the fused features into a recursive neural network module to obtain global features; Inputting the global feature into an attention module to obtain an attention feature; The attention features are input into a multi-task learning module to obtain a classification result corresponding to the preprocessed image data.
5. The garbage classification method based on intelligent vision according to claim 4 is characterized in that: The feature fusion module includes a plurality of parallel convolutional neural network branches, and the process of extracting features from the preprocessed image data based on the multi-scale feature extraction module to obtain multi-scale features specifically includes: Based on multiple parallel convolutional neural network branches, convolution operations of different scales are performed on the preprocessed image data to obtain a multi-scale feature map, wherein each convolution branch uses a convolution kernel and a step size of different size; The multi-scale feature maps are respectively subjected to pooling operations to obtain multi-scale features.
6. The garbage classification method based on intelligent vision according to claim 4 is characterized in that: The multi-task learning module uses a shared underlying feature representation, including task branches of garbage type classification and garbage form detection, and the process of inputting the attention feature into the multi-task learning module to obtain the classification result corresponding to the pre-processed image data specifically includes: Mapping the attention feature to a garbage category label based on a fully connected layer and a Softmax function; Mapping the attention features into garbage morphology parameters based on a fully connected layer and a regression function; A classification result corresponding to the processed image data is determined based on the garbage category label and the garbage morphology parameter.
7. The garbage classification method based on intelligent vision according to claim 1 is characterized in that: After classifying the preprocessed image data based on the pretrained garbage classification model to obtain a classification result, the method further includes: Transmitting the classification result to a garbage disposal system via a wireless communication module; After receiving the classification results, the garbage disposal system controls the corresponding garbage bins to operate and feeds back the classification results to the user through a display screen or a voice prompt device.
8. A garbage sorting device based on intelligent vision, characterized in that: include: An acquisition module, used for acquiring junk image data, and preprocessing the junk image data to obtain preprocessed data; The classification module classifies the preprocessed image data based on the pre-trained garbage classification model.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the intelligent vision-based garbage classification method as described in any one of claims 1-7.
10. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps of the intelligent vision-based garbage classification method as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Intelligent garbage classification method and device, storage medium and robot
CN111414926A
Household garbage classification method based on deep learning
CN112364944A
Garbage classification method based on classification and detection joint judgment
CN113657143A
Garbage classification and identification method and system based on deep learning
CN116416476A
Deep learning-based junk image classification and identification method
CN116434061A
