A garbage classification and identification system for smart cities

Abstract

The present invention relates to the field of waste disposal technology, and more specifically to a smart city-oriented waste sorting and identification system, comprising an image acquisition and restoration module, a multimodal feature fusion module, a dynamic classification decision module, and a classification result output module. Specifically, the image acquisition and restoration module acquires and restores images of waste disposal points, outputting restored three-dimensional images; the multimodal feature fusion module fuses extracted features into multimodal fusion features using a spatiotemporal alignment algorithm; the dynamic classification decision module uses a parallel dual-branch convolutional neural network with an integrated attention mechanism for classification; and the classification result output module generates control signals for triggering waste recycling equipment. The present invention, through the collaborative design of multimodal feature fusion and parallel classification structures, achieves accurate identification of waste materials and uses in complex environments, and can automatically link recycling equipment, thereby improving the overall intelligence level of smart city waste sorting systems.

Description

Technical Field

[0001] The present invention relates to the field of garbage disposal technology, and in particular to a garbage classification and identification system for smart cities. Background Art

[0002] With the continuous growth of the total amount of urban domestic waste, traditional manual sorting methods are no longer able to meet the needs of modern urban management due to their slow speed, high error rate and high labor intensity. Automatic classification systems based on single two-dimensional images are prone to missed detections and misjudgments when there is insufficient lighting or occlusion, and lack the comprehensive use of garbage weight and volume information, and cannot accurately reflect the physical characteristics of the objects placed.

[0003] Most current smart city waste sorting solutions rely solely on visual recognition or weight detection, making it difficult to achieve truly high-precision, real-time sorting and coordinated control of recycling equipment. Therefore, a smart city-specific waste sorting and identification system is needed to overcome the limitations of existing technologies in classification accuracy and system robustness in complex environments. Summary of the Invention

[0004] Based on the above objectives, the present invention provides a garbage classification and identification system for smart cities.

[0005] A garbage classification and identification system for smart cities includes an image acquisition and restoration module, a multimodal feature fusion module, a dynamic classification decision module, and a classification result output module; wherein:

[0006] Image acquisition and restoration module: used to obtain RGB-D images of garbage disposal points, perform illumination compensation on low-light areas, perform pixel-level restoration on occluded areas, and output the restored 3D images;

[0007] Multimodal feature fusion module: This module receives the restored 3D image, extracts its surface texture features and 3D geometric features, simultaneously accesses real-time data from the weight sensor and volume sensor, and fuses the extracted features into multimodal fusion features using a spatiotemporal alignment algorithm.

[0008] Dynamic classification decision module: Based on multimodal fusion features, a parallel two-branch convolutional neural network with an integrated attention mechanism is used for classification. The first branch identifies the type of garbage material, and the second branch identifies the type of garbage use. The output of the two branches is combined to generate a garbage classification label and its confidence score.

[0009] Classification result output module: used to encapsulate garbage classification labels and confidence scores into a structured data format, output it to the smart city management platform, and trigger the control signal of the corresponding garbage recycling equipment.

[0010] Optionally, the image acquisition and restoration module includes an image acquisition unit, an illumination compensation unit, a pixel-level restoration unit, and an image output unit; wherein:

[0011] Image acquisition unit: used to synchronously use a structured light depth camera and a wide-angle RGB camera to acquire RGB-D data of the garbage disposal point, wherein the structured light depth camera collects depth maps at a rate of 30 frames per second with a field of view of 70°, and the RGB camera has a resolution of 1920×1080 pixels;

[0012] The illumination compensation unit is used to perform multi-scale Retinex processing on the RGB image output by the image acquisition unit. The multi-scale Retinex processing is to perform logarithmic domain transformation, color restoration and gamma correction in sequence to compensate the brightness of pixels in low-light areas to a grayscale range of 50 to 200.

[0013] Pixel-level restoration unit: This unit is used to perform texture matching restoration on the RGB pixel blocks corresponding to the missing areas in the depth map caused by occlusion based on the PatchMatch algorithm. The pixel blocks with the highest texture similarity to the surrounding missing areas are selected for filling, and the bilateral filter is used to smooth the restoration boundaries.

[0014] Image output unit: used to generate an RGB-D three-dimensional image by mapping the compensated RGB image and the repaired depth map at the pixel level, and output the three-dimensional image to the multimodal feature fusion module.

[0015] Optionally, the pixel-level restoration unit includes:

[0016] Occlusion detection subunit: used to identify pixels in the input depth map whose pixel value is zero or whose depth difference with adjacent pixels exceeds a set threshold as the boundary of the occlusion area;

[0017] Candidate initialization subunit: It is used to randomly select multiple candidate blocks with the same size as the pixel block to be repaired in the non-occluded area. Let the target repair block be , the candidate blocks are ;

[0018] Similarity calculation subunit, used for each candidate block With the target block Perform texture similarity calculation;

[0019] Iterative update subunit: used to traverse the matching results of adjacent areas in the target image to guide the iteration of candidate blocks and retain the matching position corresponding to the minimum texture similarity;

[0020] Repair filling subunit: used to copy the pixel information of the candidate block with the smallest similarity to cover the target occluded area, and use the bilateral filtering algorithm to smooth the repair boundary.

[0021] Optionally, the multimodal feature fusion module includes a surface texture feature extraction unit, a three-dimensional geometric feature extraction unit, a physical sensor data acquisition unit, and a spatiotemporal alignment fusion unit; wherein:

[0022] Surface texture feature extraction unit: used to perform local binary pattern encoding on the input repaired RGB image, extract detailed texture information of the garbage object surface, and form a surface texture feature vector;

[0023] 3D geometric feature extraction unit: used to construct a spatial point cloud based on the input repaired depth map, and perform surface normal vector estimation, curvature calculation and shape histogram analysis on the point cloud to extract the 3D shape features of the garbage objects and form a 3D geometric feature vector;

[0024] Physical sensor data acquisition unit: used to collect data generated by weight sensors and volume sensors during garbage disposal in real time, and to form standardized physical feature vectors after eliminating outliers and interference information through data preprocessing;

[0025] Spatiotemporal alignment fusion unit: It is used to establish a spatiotemporal alignment mapping relationship based on the timestamp information of the received 3D image features and physical features, and use interpolation and data synchronization methods to ensure that the image features and physical features are strictly aligned under the same spatiotemporal reference, and then splice the aligned feature vectors into a single multimodal fusion feature vector.

[0026] Optionally, the three-dimensional geometric feature extraction unit includes:

[0027] Point cloud generation subunit: converts two-dimensional pixel coordinates and depth information into three-dimensional spatial coordinate point cloud based on the intrinsic parameter matrix of the depth image, and outputs spatial coordinate point cloud data;

[0028] Normal vector calculation subunit: used to estimate the normal vector of each spatial point in the point cloud data using the neighborhood covariance analysis method to obtain the normal vector of each point;

[0029] Curvature calculation subunit: used to calculate the curvature of each spatial point in the point cloud based on the normal vector obtained by the normal vector calculation subunit, and thus construct a local curvature feature description of the point cloud;

[0030] Shape histogram construction subunit: used to construct a shape histogram based on the direction distribution of the normal vector and the curvature value statistics of the point cloud data, and normalize the statistical results and output them as a three-dimensional geometric feature vector.

[0031] Optionally, the spatiotemporal alignment fusion unit includes:

[0032] Timestamp matching subunit: used to extract timestamps of 3D image feature data and timestamps of physical sensor data , and calculate the time difference between the two ;

[0033] Time alignment interpolation subunit: used to set the time alignment threshold , when the time difference When , it is determined that the current image feature and the physical feature are in the same time and space reference, otherwise the physical feature data is interpolated using linear interpolation;

[0034] The feature splicing subunit is used to concatenate the time-aligned image feature vector and the interpolated physical feature vector in a fixed order to generate a multimodal fusion feature vector.

[0035] Optionally, the dynamic classification decision module includes a feature input unit, an attention enhancement unit, a material identification branch unit, a usage identification branch unit, and a label fusion unit; wherein:

[0036] Feature input unit: used to receive the fused feature vector output by the multimodal feature fusion module, perform dimension verification and normalization on it, and use it as the unified input of the two-branch convolutional neural network;

[0037] Attention Enhancement Unit: An integrated mechanism for applying channel attention and spatial attention to the fused feature vector at the network input stage;

[0038] Material recognition branch unit: This is constructed as a first-branch convolutional neural network structure, which receives the feature vector after attention enhancement. Through successive convolutional layers, batch normalization layers, and activation functions, it extracts discriminative features reflecting the properties of garbage materials and outputs the corresponding material classification result vector.

[0039] Usage identification branch unit: This is constructed as a second-branch convolutional neural network structure, which processes the same input feature vector in parallel, focusing on extracting the functional usage characteristics of garbage items and outputting a usage classification result vector;

[0040] Label fusion unit: It is used to integrate the output results of the material identification branch and the purpose identification branch, map the output indexes of the two branches into a unique garbage classification label based on a predefined classification mapping table, and calculate the joint confidence score using a probabilistic combination rule.

[0041] Optionally, the attention enhancement unit includes:

[0042] Channel attention subunit: It is used to perform global average pooling and global maximum pooling operations on the input fusion feature vector to obtain the corresponding channel-level statistical information. It also learns the weight factors of each feature channel through a shared multi-layer perceptron network to obtain the attention weight vector of the channel dimension.

[0043] Channel enhancement subunit: used to apply the channel attention weight vector output by the channel attention subunit to the original fusion feature vector in a channel-by-channel multiplication manner to obtain channel enhancement features;

[0044] Spatial attention subunit: Based on the channel enhancement features output by the channel enhancement subunit, maximum pooling and average pooling are performed along the channel dimension to generate spatial attention map features, and a two-dimensional spatial attention mask is generated through convolution operation;

[0045] Spatial enhancement subunit: It is used to apply the spatial attention mask output by the spatial attention subunit to the channel enhancement feature element by element, enhance the local area features with significant recognition ability for the garbage material and use classification task in the spatial dimension, form a fused feature vector that is doubly enhanced in channel and space, and output it to the material recognition branch unit and the use recognition branch unit.

[0046] Optionally, the label fusion unit includes:

[0047] Index mapping subunit: used to receive the material category index output by the material identification branch unit and the usage category index output by the usage identification branch unit , based on a predefined two-dimensional classification mapping table , mapping the two indexes to unique garbage classification labels , the mapping relationship is: ;

[0048] Probability combination subunit: used to receive the material classification probability value output by the material identification branch unit and the use classification probability value output by the use identification branch unit, and calculate the joint confidence score based on the multiplication fusion rule ;

[0049] Label output subunit: used to map the garbage classification labels and joint confidence score Output to the classification result output module.

[0050] Optionally, the classification result output module includes a structured encapsulation unit, a data communication unit, a device matching unit, and a control signal triggering unit; wherein:

[0051] The structured encapsulation unit is used to receive the garbage classification label and the joint confidence score, and uses the predefined JSON data structure to encapsulate the label and score in the form of key-value pairs to form a standard structured data message;

[0052] Data communication unit: used to send the structured data message generated by the structured encapsulation unit to the data interface of the smart city management platform through the network interface using the MQTT protocol, and perform data packet integrity verification and security encryption processing before sending;

[0053] Device matching unit: used to call the predefined association table of garbage classification labels and garbage recycling equipment according to the received garbage classification labels, determine the device identification code of the corresponding garbage recycling equipment, and form the target device control index;

[0054] Control signal trigger unit: Generates control instruction messages that comply with the communication protocol of the garbage recycling equipment based on the target device control index, and sends control signals to the corresponding garbage recycling equipment through the wireless communication module to execute the actual recycling operations of starting and stopping the equipment and opening or closing the garbage sorting door.

[0055] Beneficial effects of the present invention:

[0056] The present invention, by constructing an image acquisition and restoration module, a multimodal feature fusion module, a dynamic classification decision module, and a classification result output module, can realize the acquisition and restoration of RGB-D images of garbage disposal points in low-light and occluded scenes, and combine physical sensor data such as weight and volume to fuse multi-source feature information under a unified spatiotemporal benchmark, effectively improving the integrity and accuracy of garbage feature extraction.

[0057] The present invention realizes parallel recognition of garbage materials and uses through a dual-branch convolutional neural network with an integrated attention mechanism, and outputs unique classification labels and scores based on label fusion and confidence calculation mechanisms. Combined with the device mapping table, it automatically triggers corresponding recycling equipment control instructions, realizing seamless linkage between garbage identification results and urban recycling hardware systems, and enhancing the intelligence and automation capabilities of the overall garbage classification system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 This is a schematic diagram of a garbage classification and identification system according to an embodiment of the present invention;

[0060] Figure 2 Schematic diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It is also noted that, to provide a more detailed description, the following embodiments are best and preferred embodiments, and those skilled in the art may employ alternative methods for implementing certain known technologies. Furthermore, the accompanying drawings are intended only to provide a more detailed description of the embodiments and are not intended to limit the present invention.

[0062] It should be noted that references in the specification to "one embodiment," "an embodiment," "exemplary embodiments," "some embodiments," etc. indicate that the described embodiments may include specific features, structures, or characteristics, but not necessarily every embodiment will include such specific features, structures, or characteristics. Furthermore, when specific features, structures, or characteristics are described in conjunction with an embodiment, it is within the knowledge of persons skilled in the relevant art to implement such features, structures, or characteristics in conjunction with other embodiments (whether or not explicitly described).

[0063] In general, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending at least in part on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0064] like Figure 1-Figure 2 As shown, a garbage classification and recognition system for smart cities includes an image acquisition and restoration module, a multimodal feature fusion module, a dynamic classification decision module, and a classification result output module; wherein:

[0065] Image acquisition and restoration module: This module is used to obtain RGB-D images (Red-Green-Blue-Depth images) of garbage disposal sites, perform illumination compensation on low-light areas, perform pixel-level restoration on occluded areas, and output the restored 3D images.

[0066] Multimodal feature fusion module: This module receives the restored 3D image, extracts its surface texture features and 3D geometric features, simultaneously accesses real-time data from the weight sensor and volume sensor, and fuses the extracted features into multimodal fusion features using a spatiotemporal alignment algorithm.

[0067] Dynamic classification decision module: Based on multimodal fusion features, a parallel two-branch convolutional neural network with an integrated attention mechanism is used for classification. The first branch identifies the type of garbage material, and the second branch identifies the type of garbage use. The output of the two branches is combined to generate a garbage classification label and its confidence score.

[0068] Classification result output module: used to encapsulate garbage classification labels and confidence scores into a structured data format, output it to the smart city management platform, and trigger the control signal of the corresponding garbage recycling equipment.

[0069] The image acquisition and restoration module includes an image acquisition unit, an illumination compensation unit, a pixel-level restoration unit, and an image output unit; wherein:

[0070] Image acquisition unit: used to synchronously acquire RGB-D data of garbage disposal points using a structured light depth camera and a wide-angle RGB camera. The structured light depth camera acquires depth maps at a rate of 30 frames per second with a field of view of 70°. The RGB camera has a resolution of 1920×1080 pixels.

[0071] Illumination compensation unit: used to perform multi-scale Retinex processing on the RGB image output by the image acquisition unit. The multi-scale Retinex processing is to perform logarithmic domain transformation, color restoration and gamma correction in sequence to compensate the brightness of pixels in low-light areas to the grayscale range of 50 to 200.

[0072] Pixel-level restoration unit: This unit is used to perform texture matching restoration on the RGB pixel blocks corresponding to the missing areas in the depth map caused by occlusion based on the PatchMatch algorithm. The pixel blocks with the highest texture similarity to the surrounding missing areas are selected for filling, and the bilateral filter is used to smooth the restoration boundaries.

[0073] Image output unit: used to generate an RGB-D three-dimensional image by mapping the compensated RGB image and the repaired depth map at the pixel level, and output the three-dimensional image to the multimodal feature fusion module; based on the coordinated cooperation of the above units, the image acquisition and repair module can generate high-precision and complete RGB-D three-dimensional images under low light and occlusion conditions, providing accurate and reliable input for the multimodal feature fusion module, thereby improving the classification accuracy of the entire garbage classification and recognition system.

[0074] The pixel-level restoration unit includes:

[0075] Occlusion detection subunit: used to identify pixels in the input depth map whose pixel value is zero or whose depth difference with adjacent pixels exceeds a set threshold as the boundary of the occlusion area;

[0076] Candidate initialization subunit: It is used to randomly select multiple candidate blocks with the same size as the pixel block to be repaired in the non-occluded area. Let the target repair block be , the candidate blocks are ,in is the number of candidate blocks;

[0077] Similarity calculation subunit, used for each candidate block With the target block To calculate texture similarity, the formula is: , where Represents the target block With candidate blocks The pixel L2 distance between them; Indicates relative coordinates within the block; Represents the target block In position RGB three-channel pixel vector; Represents candidate blocks RGB pixel vector at the same position; represents the Euclidean square norm;

[0078] Iterative update subunit: used to traverse the matching results of adjacent areas in the target image to guide the iteration of candidate blocks and retain the matching position corresponding to the minimum texture similarity;

[0079] Inpainting subunit: used to copy the pixel information of the candidate block with the least similarity to cover the target occluded area, and use the bilateral filtering algorithm to smooth the inpainting boundary to ensure the continuity of color and texture between the inpainting area and the surrounding environment; the above-mentioned pixel-level inpainting unit introduces the PatchMatch texture matching mechanism with clear parameter definitions, which can quickly identify and copy the most similar candidate image blocks in the occluded area, achieve high-fidelity image reconstruction, effectively improve the image inpainting quality, and provide complete and stable input data for subsequent image feature extraction.

[0080] The multimodal feature fusion module includes a surface texture feature extraction unit, a three-dimensional geometric feature extraction unit, a physical sensor data acquisition unit, and a spatiotemporal alignment fusion unit; wherein:

[0081] Surface texture feature extraction unit: used to perform local binary pattern encoding on the input repaired RGB image, extract detailed texture information of the garbage object surface, and form a surface texture feature vector;

[0082] 3D geometric feature extraction unit: used to construct a spatial point cloud based on the input repaired depth map, and perform surface normal vector estimation, curvature calculation and shape histogram analysis on the point cloud to extract the 3D shape features of the garbage objects and form a 3D geometric feature vector;

[0083] Physical sensor data acquisition unit: used to collect data generated by weight sensors and volume sensors during garbage disposal in real time, and to form standardized physical feature vectors after eliminating outliers and interference information through data preprocessing;

[0084] Spatiotemporal alignment fusion unit: It is used to establish a spatiotemporal alignment mapping relationship based on the received three-dimensional image features (including surface texture feature vectors and three-dimensional geometric feature vectors) and the timestamp information of the physical features. It uses interpolation and data synchronization methods to ensure that the image features and physical features are strictly aligned under the same spatiotemporal reference, and splices the aligned feature vectors into a single multimodal fusion feature vector, which is output to the dynamic classification decision module.

[0085] The 3D geometric feature extraction unit includes:

[0086] Point cloud generation subunit: converts two-dimensional pixel coordinates and depth information into three-dimensional spatial coordinate point cloud based on the intrinsic parameter matrix of the depth image, and outputs spatial coordinate point cloud data;

[0087] Normal vector calculation subunit: used to estimate the normal vector of each spatial point in the point cloud data using the neighborhood covariance analysis method to obtain the normal vector of each point;

[0088] The normal vector calculation formula is: ,in, is the covariance matrix of the neighborhood points, with dimension ; The first The three-dimensional coordinate vector of a point; is the average vector of the three-dimensional coordinates of all points in the neighborhood; is the number of points in the neighborhood; the eigenvector corresponding to the minimum eigenvalue obtained by eigenvalue decomposition is the normal vector of the current point;

[0089] Curvature calculation subunit: used to calculate the curvature of each spatial point in the point cloud based on the normal vector obtained by the normal vector calculation subunit, and thus construct a local curvature feature description of the point cloud;

[0090] The curvature calculation formula is: ,in, is the curvature eigenvalue of the current spatial point; is the minimum eigenvalue of the covariance matrix; is the sum of all eigenvalues of the covariance matrix;

[0091] Shape histogram construction subunit: used to construct a shape histogram based on the direction distribution of the normal vector and the curvature value statistics of the point cloud data, and normalize the statistical results and output them as a three-dimensional geometric feature vector.

[0092] The specific steps to construct a shape histogram are as follows:

[0093] Step 1: Convert the normal vector of each point in the point cloud into a spherical coordinate system and express the normal vector as polar angle and azimuth angle. The formulas are:

[0094] ;

[0095] Where, is the polar angle of the normal vector; is the azimuth of the normal vector; is the normal vector in three-dimensional space Directional component;

[0096] Step 2: Normalize the curvature value calculated for each point in the point cloud so that the curvature eigenvalue falls within the interval [0, 1]. The formula is: , where is the normalized curvature value; is the original curvature value of the current point; They represent the maximum and minimum curvature values of all points in the point cloud respectively;

[0097] Step 3: Evenly divide the feature space composed of the normal vector direction and the normalized curvature value into intervals, where the polar angle The scope is divided into interval; azimuth The scope is divided into intervals; curvature values The scope is divided into intervals;

[0098] Step 4: For each point in the point cloud data, determine its interval index in the above feature space based on the polar angle, azimuth and normalized curvature value of its normal vector, where the polar angle index is , the azimuth index is , the curvature index is ; The interval index is defined as: ; ; ,in, is the polar angle quantization step size; is the azimuth quantization step size; Quantize the step size for curvature; This is a rounding down operation; then the corresponding positions of each interval index are counted, and the formula is: ;

[0099] Step 5: Normalize the shape histogram obtained by statistics to form the final three-dimensional geometric feature vector, which is expressed as: , where are the dimensional components of the final three-dimensional geometric feature vector; the denominator is the total count of all intervals of the shape histogram, which plays a normalization role.

[0100] The spatiotemporal alignment fusion unit includes:

[0101] Timestamp matching subunit: used to extract timestamps of 3D image feature data and timestamps of physical sensor data , and calculate the time difference between the two , the formula is: ;

[0102] Time alignment interpolation subunit: used to set the time alignment threshold , when the time difference When , it is determined that the current image feature and the physical feature are in the same time and space reference, otherwise the physical feature data is interpolated using linear interpolation, the formula is: ,in: Represents the interpolated physical characteristic data; Respectively represent the two physical feature data points closest to the image feature timestamp; Respectively represent the timestamps of the two physical feature data closest to the image feature timestamp;

[0103] Feature splicing subunit, used to combine the time-aligned image feature vectors and the interpolated physical eigenvectors Perform serial concatenation in a fixed order to generate a multimodal fusion feature vector. The calculation formula is: ,in, is the final output fusion feature vector; the symbol [;] represents the concatenation operation of feature vectors; through the precise matching and interpolation operations of the above-mentioned spatiotemporal alignment fusion unit, the precise time synchronization of image features and physical sensor data is achieved, which effectively reduces the spatiotemporal error in the multimodal data fusion process and significantly improves the accuracy and robustness of garbage classification decisions.

[0104] The dynamic classification decision module includes a feature input unit, an attention enhancement unit, a material recognition branch unit, a usage recognition branch unit, and a label fusion unit; wherein:

[0105] Feature input unit: used to receive the fused feature vector output by the multimodal feature fusion module, perform dimension verification and normalization on it, and use it as the unified input of the two-branch convolutional neural network;

[0106] Attention Enhancement Unit: This unit integrates channel attention and spatial attention on the fused feature vector at the network input stage to enhance the responsiveness of salient features of material and usage, suppress irrelevant feature dimensions, and improve classification accuracy.

[0107] Material recognition branch unit: This is constructed as a first-branch convolutional neural network structure, which receives the feature vector after attention enhancement. Through successive convolutional layers, batch normalization layers, and activation functions, it extracts discriminant features reflecting the material properties of the garbage (such as metal, plastic, glass, etc.) and outputs the corresponding material classification result vector.

[0108] Usage identification branch unit: This is constructed as a second-branch convolutional neural network structure, which processes the same input feature vector in parallel, focusing on extracting the functional use characteristics of garbage items (such as kitchen waste, hazardous, recyclable, etc.) and outputting a use classification result vector;

[0109] Label fusion unit: It is used to integrate the output results of the material identification branch and the purpose identification branch, map the output indexes of the two branches into a unique garbage classification label based on a predefined classification mapping table, and use probabilistic combination rules to calculate the joint confidence score to measure the credibility of the final classification result.

[0110] The attention enhancement unit includes:

[0111] Channel attention subunit: It is used to perform global average pooling and global maximum pooling operations on the input fusion feature vector to obtain the corresponding channel-level statistical information. It also learns the weight factors of each feature channel through a shared multi-layer perceptron network to obtain the attention weight vector of the channel dimension.

[0112] Channel enhancement subunit: This subunit is used to apply the channel attention weight vector output by the channel attention subunit to the original fusion feature vector in a channel-by-channel multiplication manner, thereby enhancing the response strength of the garbage classification salient feature channel and obtaining channel enhancement features.

[0113] Spatial attention subunit: Based on the channel enhancement features output by the channel enhancement subunit, maximum pooling and average pooling are performed along the channel dimension to generate spatial attention map features, and a two-dimensional spatial attention mask is generated through convolution operation;

[0114] Spatial enhancement subunit: It is used to apply the spatial attention mask output by the spatial attention subunit to the channel enhancement feature element by element, enhance the local area features with significant recognition for the garbage material and use classification task in the spatial dimension, form a fusion feature vector that is doubly enhanced by the channel and space, and output it to the material recognition branch unit and the use recognition branch unit; through the above-mentioned clear integration of the channel and spatial attention mechanism, the attention enhancement unit effectively improves the network's attention to important information in the fusion feature, suppresses the interference of irrelevant information, and significantly improves the classification performance of the material and use recognition branches and the classification accuracy of the overall system.

[0115] The label fusion unit includes:

[0116] Index mapping subunit: used to receive the material category index output by the material identification branch unit and the usage category index output by the usage identification branch unit , based on a predefined two-dimensional classification mapping table , mapping the two indexes to unique garbage classification labels , the mapping relationship is: ,in, It is a preset two-dimensional label mapping table used to represent the garbage classification label corresponding to any pair of material category and use category combination; The material category index output by the material identification branch; The use category index output by the use identification branch; is the garbage classification label obtained after mapping;

[0117] Table 1 Preset two-dimensional label mapping example

[0118]

[0119] In Table 1 above, the rows represent the category index output by the material recognition branch , columns represent the category index output by the usage identification branch The value of each intersection is the unique garbage classification label for the final output; some combinations are logically inapplicable (such as "glass + kitchen waste") and can be set to the invalid item "L-" or the default output is "other garbage".

[0120] The specific application examples of Table 1 are as follows:

[0121] Assumption: The output of the material recognition branch is metal, that is, the index is ; The output of the usage identification branch is the recyclable class, that is, the index is ;but: .

[0122] Probability combination subunit: used to receive the material classification probability value output by the material identification branch unit and the use classification probability value output by the use identification branch unit, and calculate the joint confidence score based on the multiplication fusion rule , the formula is as follows: ,in, The predicted probability value of the corresponding index of the material identification branch; The predicted probability value of the corresponding index of the usage identification branch; The joint confidence score indicates the credibility of the final garbage classification label;

[0123] Label output subunit: used to map the garbage classification labels and joint confidence score Output to the classification result output module; through the classification label index mapping and probability fusion rules of the above-mentioned label fusion unit, accurate comprehensive decision-making of material and usage information is effectively achieved, the confidence and stability of the garbage classification results are improved, and the overall decision-making quality of the smart city garbage classification system is further guaranteed.

[0124] The classification result output module includes a structured encapsulation unit, a data communication unit, a device matching unit, and a control signal triggering unit; wherein:

[0125] The structured encapsulation unit is used to receive the garbage classification label and the joint confidence score, and uses the predefined JSON data structure to encapsulate the label and score in the form of key-value pairs to form a standard structured data message;

[0126] Data communication unit: used to send the structured data message generated by the structured encapsulation unit to the data interface of the smart city management platform through the network interface using the MQTT protocol, and perform data packet integrity verification and security encryption processing before sending;

[0127] Device matching unit: used to call the predefined association table of garbage classification labels and garbage recycling equipment according to the received garbage classification labels, determine the device identification code of the corresponding garbage recycling equipment, and form the target device control index;

[0128] Table 2 Example of association between garbage classification labels and garbage recycling equipment identification codes

[0129]

[0130] In Table 2 above, the garbage classification label number (L) is the final label output by the dynamic classification decision module, such as "L1", "L5", etc.; the label name corresponds to the Chinese label name for easy management and display; the device number (Device_ID) represents the corresponding garbage recycling device number in the system, which serves as the target device for the control instruction; the control instruction type represents the specific control signal generated by the system through this field (such as "open the recyclables door", "close other doors", etc.).

[0131] Control signal trigger unit: Generates control instruction messages that comply with the communication protocol of the garbage recycling equipment based on the control index of the target device, and sends control signals to the corresponding garbage recycling equipment through the wireless communication module to execute the actual recycling operations of starting and stopping the equipment and opening or closing the garbage sorting door; the above unit uses the classification result output module to structure the encapsulation and secure transmission of classification labels and confidence scores, as well as the generation of automatic control instructions that accurately match the garbage recycling equipment, thereby achieving seamless collaboration between the system and the smart city platform and recycling equipment, effectively improving the automation level and management efficiency of urban garbage sorting and recycling.

[0132] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are described in detail below in connection with the preferred embodiments of the present invention, but those skilled in the art will be able to fully understand the present invention without these detailed descriptions. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits have not been described in detail.

[0133] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A garbage classification and identification system for smart cities, characterized by: It includes image acquisition and restoration module, multimodal feature fusion module, dynamic classification decision module and classification result output module; among them: Image acquisition and restoration module: used to obtain RGB-D images of garbage disposal points, perform illumination compensation on low-light areas, perform pixel-level restoration on occluded areas, and output the restored 3D images; Multimodal feature fusion module: This module receives the restored 3D image, extracts its surface texture features and 3D geometric features, simultaneously accesses real-time data from the weight sensor and volume sensor, and fuses the extracted features into multimodal fusion features using a spatiotemporal alignment algorithm. The multimodal feature fusion module includes a surface texture feature extraction unit, a three-dimensional geometric feature extraction unit, a physical sensor data acquisition unit, and a spatiotemporal alignment fusion unit; wherein: Surface texture feature extraction unit: used to perform local binary pattern encoding on the input repaired RGB image, extract detailed texture information of the garbage object surface, and form a surface texture feature vector; 3D geometric feature extraction unit: used to construct a spatial point cloud based on the input repaired depth map, and perform surface normal vector estimation, curvature calculation and shape histogram analysis on the point cloud to extract the 3D shape features of the garbage objects and form a 3D geometric feature vector; Physical sensor data acquisition unit: used to collect data generated by weight sensors and volume sensors during garbage disposal in real time, and to form standardized physical feature vectors after eliminating outliers and interference information through data preprocessing; The spatiotemporal alignment fusion unit is used to establish a spatiotemporal alignment mapping relationship based on the timestamp information of the received 3D image features and physical features. It uses interpolation and data synchronization methods to ensure that the image features and physical features are strictly aligned under the same spatiotemporal reference, and then splices the aligned feature vectors into a single multimodal fusion feature vector. Dynamic classification decision module: Based on multimodal fusion features, a parallel two-branch convolutional neural network with an integrated attention mechanism is used for classification. The first branch identifies the type of garbage material, and the second branch identifies the type of garbage use. The output of the two branches is combined to generate a garbage classification label and its confidence score. The dynamic classification decision module includes a feature input unit, an attention enhancement unit, a material recognition branch unit, a usage recognition branch unit, and a label fusion unit; wherein: Feature input unit: used to receive the fused feature vector output by the multimodal feature fusion module, perform dimension verification and normalization on it, and use it as the unified input of the two-branch convolutional neural network; Attention Enhancement Unit: An integrated mechanism for applying channel attention and spatial attention to the fused feature vector at the network input stage; Material recognition branch unit: This is constructed as a first-branch convolutional neural network structure, which receives the feature vector after attention enhancement. Through successive convolutional layers, batch normalization layers, and activation functions, it extracts discriminative features reflecting the properties of garbage materials and outputs the corresponding material classification result vector. Usage identification branch unit: This is constructed as a second-branch convolutional neural network structure, which processes the same input feature vector in parallel, focusing on extracting the functional usage characteristics of garbage items and outputting a usage classification result vector; Label fusion unit: This unit is used to integrate the output results of the material identification branch and the purpose identification branch, map the output indexes of the two branches into a unique garbage classification label based on a predefined classification mapping table, and calculate the joint confidence score using a probabilistic combination rule; Classification result output module: used to encapsulate garbage classification labels and confidence scores into a structured data format, output it to the smart city management platform, and trigger the control signal of the corresponding garbage recycling equipment.

2. The garbage classification and identification system for smart cities according to claim 1 is characterized in that: The image acquisition and restoration module includes an image acquisition unit, an illumination compensation unit, a pixel-level restoration unit, and an image output unit; wherein: Image acquisition unit: used to synchronously use a structured light depth camera and a wide-angle RGB camera to acquire RGB-D data of the garbage disposal point, wherein the structured light depth camera collects depth maps at a rate of 30 frames per second with a field of view of 70°, and the RGB camera has a resolution of 1920×1080 pixels; The illumination compensation unit is used to perform multi-scale Retinex processing on the RGB image output by the image acquisition unit. The multi-scale Retinex processing is to perform logarithmic domain transformation, color restoration and gamma correction in sequence to compensate the brightness of pixels in low-light areas to a grayscale range of 50 to 200. Pixel-level restoration unit: This unit is used to perform texture matching restoration on the RGB pixel blocks corresponding to the missing areas in the depth map caused by occlusion based on the PatchMatch algorithm. The pixel blocks with the highest texture similarity to the surrounding missing areas are selected for filling, and the bilateral filter is used to smooth the restoration boundaries. Image output unit: used to generate an RGB-D three-dimensional image by mapping the compensated RGB image and the repaired depth map at the pixel level, and output the three-dimensional image to the multimodal feature fusion module.

3. The garbage classification and identification system for smart cities according to claim 2 is characterized in that: The pixel-level repair unit includes: Occlusion detection subunit: used to identify pixels in the input depth map whose pixel value is zero or whose depth difference with adjacent pixels exceeds a set threshold as the boundary of the occlusion area; Candidate initialization subunit: It is used to randomly select multiple candidate blocks with the same size as the pixel block to be repaired in the non-occluded area. Let the target repair block be , the candidate blocks are ; Similarity calculation subunit, used for each candidate block With the target block Perform texture similarity calculation; Iterative update subunit: used to traverse the matching results of adjacent areas in the target image to guide the iteration of candidate blocks and retain the matching position corresponding to the minimum texture similarity; Repair filling subunit: used to copy the pixel information of the candidate block with the smallest similarity to cover the target occluded area, and use the bilateral filtering algorithm to smooth the repair boundary.

4. The garbage classification and identification system for smart cities according to claim 1 is characterized in that: The three-dimensional geometric feature extraction unit includes: Point cloud generation subunit: converts two-dimensional pixel coordinates and depth information into three-dimensional spatial coordinate point cloud based on the intrinsic parameter matrix of the depth image, and outputs spatial coordinate point cloud data; Normal vector calculation subunit: used to estimate the normal vector of each spatial point in the point cloud data using the neighborhood covariance analysis method to obtain the normal vector of each point; Curvature calculation subunit: used to calculate the curvature of each spatial point in the point cloud based on the normal vector obtained by the normal vector calculation subunit, and thus construct a local curvature feature description of the point cloud; Shape histogram construction subunit: used to construct a shape histogram based on the direction distribution of the normal vector and the curvature value statistics of the point cloud data, and normalize the statistical results and output them as a three-dimensional geometric feature vector.

5. The garbage classification and identification system for smart cities according to claim 1 is characterized in that: The spatiotemporal alignment fusion unit includes: Timestamp matching subunit: used to extract timestamps of 3D image feature data and timestamps of physical sensor data , and calculate the time difference between the two ; Time alignment interpolation subunit: used to set the time alignment threshold , when the time difference When , it is determined that the current image feature and the physical feature are in the same time and space reference, otherwise the physical feature data is interpolated using linear interpolation; The feature splicing subunit is used to concatenate the time-aligned image feature vector and the interpolated physical feature vector in a fixed order to generate a multimodal fusion feature vector.

6. The garbage classification and identification system for smart cities according to claim 1 is characterized in that: The attention enhancement unit comprises: Channel attention subunit: It is used to perform global average pooling and global maximum pooling operations on the input fusion feature vector to obtain the corresponding channel-level statistical information. It also learns the weight factors of each feature channel through a shared multi-layer perceptron network to obtain the attention weight vector of the channel dimension. Channel enhancement subunit: used to apply the channel attention weight vector output by the channel attention subunit to the original fusion feature vector in a channel-by-channel multiplication manner to obtain channel enhancement features; Spatial attention subunit: Based on the channel enhancement features output by the channel enhancement subunit, maximum pooling and average pooling are performed along the channel dimension to generate spatial attention map features, and a two-dimensional spatial attention mask is generated through convolution operation; Spatial enhancement subunit: It is used to apply the spatial attention mask output by the spatial attention subunit to the channel enhancement feature element by element, enhance the local area features with significant recognition ability for the garbage material and use classification task in the spatial dimension, form a fused feature vector that is doubly enhanced in channel and space, and output it to the material recognition branch unit and the use recognition branch unit.

7. The garbage classification and identification system for smart cities according to claim 1 is characterized in that: The label fusion unit includes: Index mapping subunit: used to receive the material category index output by the material identification branch unit and the usage category index output by the usage identification branch unit , based on a predefined two-dimensional classification mapping table , mapping the two indexes to unique garbage classification labels , the mapping relationship is: ; Probability combination subunit: used to receive the material classification probability value output by the material identification branch unit and the use classification probability value output by the use identification branch unit, and calculate the joint confidence score based on the multiplication fusion rule ; Label output subunit: used to map the garbage classification labels and joint confidence score Output to the classification result output module.

8. The garbage classification and identification system for smart cities according to claim 1 is characterized in that: The classification result output module includes a structured encapsulation unit, a data communication unit, a device matching unit, and a control signal triggering unit; wherein: The structured encapsulation unit is used to receive the garbage classification label and the joint confidence score, and uses the predefined JSON data structure to encapsulate the label and score in the form of key-value pairs to form a standard structured data message; Data communication unit: used to send the structured data message generated by the structured encapsulation unit to the data interface of the smart city management platform through the network interface using the MQTT protocol, and perform data packet integrity verification and security encryption processing before sending; Device matching unit: used to call the predefined association table of garbage classification labels and garbage recycling equipment according to the received garbage classification labels, determine the device identification code of the corresponding garbage recycling equipment, and form the target device control index; Control signal trigger unit: Generates control instruction messages that comply with the communication protocol of the garbage recycling equipment based on the target device control index, and sends control signals to the corresponding garbage recycling equipment through the wireless communication module to execute the actual recycling operations of starting and stopping the equipment and opening or closing the garbage sorting door.

Images